For most of the internet’s history, looking someone up meant reading a list. You typed a name, you got ten blue links, and you decided for yourself which ones mattered. The search engine ranked the sources. The reader did the synthesis.
That arrangement is quietly ending. Google now places an AI-generated summary above the results for many queries. Millions of people ask AI assistants direct questions instead of searching at all. In both cases the machine does the synthesis and hands over a conclusion. The links, when they appear, are footnotes to an answer that has already been written.
This is a genuine improvement for a great many questions. It is a strange and underexamined problem when the subject of the question is a person.
How these systems actually build an answer
The failure modes follow directly from the mechanism, so it is worth being precise about it.
Modern AI search generally works by retrieval augmented generation. When a user asks about a person, the system does not consult a curated dossier. It searches an index, pulls back a set of documents it judges relevant, and passes those documents to a language model with an instruction along the lines of: using these sources, answer the question. The model then writes fluent prose summarizing what the retrieved documents say.
Notice what the architecture assumes. It assumes the retrieved documents are accurate. It assumes they are current. It assumes a document’s prominence in the index correlates with its truthfulness. And it assumes that summarizing a set of sources produces something meaningfully like the truth about their subject.
For a question about the boiling point of water, these are safe assumptions. For a question about a human being, every one of them can fail at once.
The compounding problem
Consider what the open web actually contains about a typical professional.
It contains what they published themselves, usually accurate and usually old. It contains what institutions published about them, such as a university bio from a job they left in 2019. It contains whatever the local press wrote, which may be a single story from the worst week of their life. It contains aggregator sites that scraped court dockets or arrest records, republished them for traffic, and never checked whether the case was later dismissed. And it contains dozens of near-identical copies of all of the above, because syndication and scraping are cheap.
Now ask a retrieval system to summarize this person.
The system has no reliable way to know that the 2019 bio is stale, that the news story described charges that were dropped, or that eleven of the fourteen sources it retrieved are copies of one original. It sees a corpus, and it produces a fluent summary of that corpus.
Kevin Curran, founder of NewReputation, a firm that handles content removal and reputation repair, argues that this is where the deepest misunderstanding lies. “Repetition looks like corroboration,” he says. “When one arrest record gets scraped and republished across forty sites, a retrieval system encountering those copies is not seeing forty independent confirmations. It is seeing one document forty times. But the shape of that evidence is exactly the shape both ranking algorithms and language models have learned to treat as reliable.”
The internet’s copy-paste economy, on this account, manufactures the appearance of consensus. AI summarization then reports that manufactured consensus in a calm, authoritative voice with no visible seams.
Fluency is not accuracy, and readers cannot tell the difference
A ranked list of links wore its uncertainty on its face. Ten results, in some order, of varying quality, obviously assembled by a machine. The reader knew they were being handed raw material.
A paragraph of clean prose reads as a conclusion. It has no hedging, no visible sourcing hierarchy, no signal that source three contradicted source seven and the model quietly picked one. Research on human interaction with automated systems has long documented automation bias, the tendency to over-trust a confident machine output and under-verify it. Replacing a list with synthesized text is, among other things, an interface change that suppresses the reader’s instinct to check.
The stakes are asymmetric in a way that is easy to miss. If an AI summary of a restaurant is slightly wrong, someone has a mediocre dinner. If an AI summary of a job candidate leads with a dismissed charge from 2014, because that is what the retrievable corpus emphasized, the candidate never learns why the call did not come.
The subject is the one party with no seat at the table
Every other participant has some form of recourse. The publisher can correct an article. The platform can adjust its ranking. The model developer can tune the system.
The person being described has, in most cases, nothing. They typically do not know an AI system is summarizing them. They cannot see what it says without asking it themselves, and the answer may vary between users, sessions, and models. There is no appeals process, no correction mechanism, and often no way to identify which retrieved source produced the sentence that is costing them work.
“The people we work with almost never find out from the system,” Curran says. “They find out from a friend, or from a job that goes quiet. By then the summary has been repeating itself for months.”
This is a structural gap rather than anyone’s malice. The systems were built to answer questions about the world, and a person is a subject in the world. But people are the one category of subject that can be harmed by an inaccurate summary and that has a legitimate interest in accuracy about themselves.
What can actually be done
Approached as a systems problem, the leverage points are not where most people look.
No one can edit the model, and Curran is blunt about the vendors who imply otherwise. “Be deeply skeptical of anyone who tells you they can control what an AI system says about you. Model behavior is not a dial anyone exposes, and outputs vary in ways no third party can guarantee. What is tractable is the corpus.”
Retrieval systems can only summarize what they retrieve. If the accessible, well-structured, authoritative material about a person is thin, outdated, or dominated by a single bad episode, the summary will reflect that. If the accessible material is substantial, current, and accurate, the summary has better raw material to work with.
That reframes the work as something closer to data quality than public relations, and the interventions that matter are unglamorous. Getting genuinely false or unlawfully published material removed at the source, so it stops being retrievable at all. Attacking duplication, because forty copies of a scraped record are the thing manufacturing false consensus, and removing the original does nothing if the copies persist. Publishing accurate, current, well-structured information a retrieval system can find and parse. Monitoring what the systems say over time, since the corpus is not static.
There is a cynical reading of this, in which the answer to machine summarization is simply to feed the machine better propaganda. Curran rejects the framing. “When a system summarizes a person by retrieving documents about them, the accuracy of the summary is bounded by the accuracy of the retrievable record. Improving that record is not gaming anything. It is the only honest way to be described accurately by it.”
The design question nobody has answered
The engineering community has spent considerable effort on hallucination, the failure mode where a model invents facts that appear in no source. That work matters.
The failure mode described here is different, and in some ways harder. The model invents nothing. Every sentence is faithfully grounded in a retrieved document. The summary is a correct summary of an incorrect corpus, and no amount of grounding, citation, or retrieval fidelity fixes it. You cannot cite your way out of the problem that the sources were wrong.
Which leaves an open question for anyone building these systems. When the subject of a query is a living person, should the system treat the retrievable web as a reliable description of them? It plainly is not one. It is a partial, stale, duplicated, and adversarially seeded record that was never assembled with accuracy about individuals as a design goal.
The answer will probably not be a better retriever. It may be that queries about people warrant different treatment altogether, with more visible uncertainty, more explicit recency signals, or a lower threshold for declining to summarize at all. Those are product decisions rather than model decisions, and they are being made right now by a small number of teams.
Everyone else, meanwhile, is being described.
