How Search Actually Decides What to Show You

Most explanations of how search works are either marketing material or folklore. This one tries to be neither: a plain-English walkthrough of the actual pipeline, with the parts you can influence marked clearly and the parts you can't marked just as clearly.

I've written it for someone who publishes things and wants to understand the machine, not for someone looking for tactics.

The pipeline in one paragraph

A crawler fetches pages. An indexer processes them into a searchable structure. When someone searches, a retrieval step pulls a few thousand plausible candidates from that index very quickly. A ranking step then orders those candidates using a much more expensive set of signals. Increasingly, a final step generates a written answer on top of the ranked results. Five stages, each with different rules, and confusing them is where most bad SEO advice comes from.

1. Crawling

A crawler follows links and fetches pages. It has a finite budget per site, and it decides how often to come back based on how often you've changed things before.

What matters here is narrow and mechanical: the page must be reachable by a link, it must return a success status, it must not be blocked by robots.txt, and it should load without requiring JavaScript to produce its main content. That last one is worth stressing — JavaScript-rendered content is crawled, but on a delayed, lower-priority path.

You control: internal linking, site speed, whether content requires JS, and your sitemap. You don't control: how much crawl budget you get, which is largely a function of how important the site already appears to be.

2. Indexing

The indexer turns a fetched page into structured data: the text, its structure, the links out, the language, the canonical version, and a growing amount of semantic representation — embeddings that capture roughly what the page is about rather than which words it contains.

Crucially, being crawled does not mean being indexed. Pages get dropped for being near-duplicates of something already held, for being thin, or for simply not seeming worth the storage. This is the stage where most large auto-generated sites quietly fail.

You control: whether each page says something the index doesn't already have. That's genuinely most of it.

3. Retrieval

Someone searches. In a few milliseconds, the system pulls a candidate set — typically a few thousand documents — from an index of many billions. This happens two ways at once: lexical matching (documents containing these terms) and semantic matching (documents whose meaning is close to this query's meaning).

The practical consequence: you no longer need to contain the exact phrase to be retrieved, but you do need to be clearly, unambiguously about the thing. Pages that hedge across four topics match none of them strongly.

4. Ranking

Now the expensive part runs on those few thousand candidates. Hundreds of signals, weighted differently by query type, combined by systems that are themselves partly learned rather than hand-written.

The broad families are well established even though the details aren't public:

• Relevance — how well the document answers this specific query, including semantic closeness rather than word overlap.
• Quality — signals that correlate with the content being trustworthy and original.
• Links — still real, still weaker than it was in 2012, and far more sensitive to whether links are editorially given.
• Interaction data — aggregated, noisy, and used in ways the search companies are deliberately vague about. It exists; it is not a dial you can turn by clicking your own result.
• Context — location, language, device, and what the query pattern suggests the person wants.

The single most useful thing to understand: ranking is per-query, not per-site. There is no site-wide score being applied. The same domain can rank first for one query and nowhere for a closely related one, and that isn't a contradiction.

5. The generated answer on top

Increasingly a written answer appears above the results, synthesised from several retrieved sources. This changes the economics rather than the mechanics: you can be a source for an answer that the reader never clicks through from.

Two observable consequences, both well documented by now: informational queries with simple answers lose clicks, and pages that are clearly structured with direct claims are more likely to be drawn from. Being the source of an answer is worth something in credibility, but it is not worth the traffic that ranking used to bring, and pretending otherwise leads to bad planning.

Ranking is per-query, not per-site. Most confusion about search dissolves once that clicks.

What you can actually influence

Ordered by how much control you genuinely have:

Everything in the "total control" rows is unglamorous and it's where essentially all the durable results come from. That's the honest summary of a decade of watching this: the boring rows are the whole job.

Five things that are widely repeated and wrong

Understanding the pipeline makes it easier to see why certain advice persists despite being wrong. These are the five I'm asked about most.

"There's a site-wide quality score." There isn't a single number applied to your domain. Ranking is computed per query against a candidate set, which is why a domain can rank first for one phrasing and be absent for a near-identical one. Most of the folklore about "the site being penalised" describes ordinary per-query variance.

"Keyword density matters." It hasn't meaningfully for over a decade, and semantic retrieval has made it less relevant still. What matters is whether the page is unambiguously about one thing — which correlates with using the relevant words, but the density isn't the mechanism.

"Publishing more often improves rankings." Crawl frequency adapts to how often you change things, so publishing often gets you crawled more often. That's not the same as ranking better, and thin pages published frequently actively hurt at the indexing stage.

"Meta keywords do something." No major engine has used them for many years. They're harmless and pointless.

"You can improve rankings by clicking your own result." Interaction data is aggregated across enormous populations and heavily denoised. Individual behaviour is not a lever, and the services selling it are selling nothing.

What changes when an answer is generated on top

The generated-answer layer is the biggest shift in a decade, and most of the advice about it is speculation. Here's what can be said with reasonable confidence, separated from what can't.

Reasonably well established: queries with short factual answers lose clicks, because the answer is delivered without a visit. Pages that state claims directly and are structured with clear headings are easier to draw from. Being cited is worth something in credibility and referral, but substantially less than the ranking position used to be worth in traffic.

Not established, despite confident claims: that there's a separate "optimise for AI answers" discipline with its own techniques. Everything I've seen that reliably helps — be clearly about one thing, state claims plainly, structure with real headings, be the primary source rather than a summary — is the same list as before.

The strategic consequence is about content selection rather than technique. If a topic can be fully answered in two sentences, that topic is now a poor investment regardless of how well you write it. The work that survives is the work that can't be compressed: first-hand experience, original measurement, and reasoning a reader wants to follow rather than skip.

That's an uncomfortable conclusion for a lot of content strategies, and I think it's the right one. It's also why this site publishes twelve posts a year rather than two hundred.

Questions I get about this

Found an error in this piece? Tell me — corrections get made in the open with a dated note, never a silent edit. The log is public.