Everything you need to know about how a search engine works

Chapter 3 of the SEO from zero guide. Published in November 2019 and revised in August 2026. This revision corrects an error in the original text: we said the engine sends a spider out to crawl pages the moment you type your question. It does not, and the difference completely changes what you have to do to rank.

We already know every search engine has one main function: to resolve a person’s search intent by listing results that are relevant to them. The technical question is still open: how do they do it?

A search engine is a computer system that locates the documents published on the internet, organises them, and displays them ordered by relevance, so that whoever is searching can open the pages that best answer their question.

The real order: crawling comes first, the question second

Here is the important correction. When you type something into a search engine, nobody goes out and walks the internet at that moment. It would be impossible: it would take weeks.

What actually happens is that the engine has spent months or years walking the web on its own, storing an organised copy of what it finds. Your query is not resolved against the internet: it is resolved against that index, which was already built before you asked.

The practical consequence is enormous. If your page was not crawled and indexed beforehand, it cannot appear, however well written it is and however exact the match. That is the answer to the question we get most: “the page is published, so why doesn’t it show up?”

The three stages

1 · Crawling

Programs called spiders or bots walk the web hopping from link to link. That is their main path: links are the roads. A page with no link pointing at it, internal or external, may never be discovered at all.

Crawling is not simple: documents come in different formats — HTML, PDF, images, video — and the web is vast. So an engine gives each site a limited amount of attention; if it spends that attention on duplicate or pointless pages, it stops spending it on the ones that matter.

You can help at this stage with three concrete things: a sitemap.xml listing your pages, a robots.txt that does not block what you actually want seen, and an internal link structure that reaches the whole site in a few hops.

2 · Indexing

When the engine finds a page, it processes it: it interprets the code, renders it — these days it executes JavaScript too, though on a second pass and with fewer guarantees — and stores what matters in enormous databases spread across data centres worldwide.

Being crawled does not mean being indexed. The engine decides whether the page is worth storing, and discards whatever it judges duplicate, thin or irrelevant. This is the stage where most “missing” pages actually fall.

And there is a direct consequence for how a site is built: if your content only exists after JavaScript runs, you are betting that the second pass arrives and goes well. Serving it in the HTML removes the bet.

3 · Ranking

With the index built, your query arrives. The engine identifies documents matching what you asked and orders them, weighing hundreds of signals: content relevance, site reputation, language, location, device, page speed and stability, and the observed behaviour of people who ran that same search before you.

All of it happens in fractions of a second, because searchers demand instant results: even a one or two second delay produces dissatisfaction.

What changed since 2019?

The three-stage model still holds, but two layers have been added on top of it that are worth knowing about.

The first is that the result is no longer always a list of links. There are featured snippets, question boxes, local results with a map and, since 2024, AI-generated answers above the results. All of those are built from the same index, so the rule does not change: if you are not indexed, you are in none of them.

The second is that page experience is an explicit signal since 2021, through Core Web Vitals. One of them, the metric for responsiveness to interaction, became INP in March 2024.

What this means for you

Ranking is, at bottom, about helping the engine do its job well at each of the three stages:

• Let it reach your page (links, sitemap, nothing blocked by mistake). And if you change address too — a redesign, a change of platform — the redirect matrix is what stops the index holding on to the old version.
• Let it understand the page (content in the HTML, clear titles, orderly structure).
• Make it worth showing (genuinely answer the question, better than the alternatives).

When any of the three fails, the rest of the effort is wasted. And in that order: the best article in the world is worth nothing on a page nobody can crawl.

---

Previous: Access to information · Next: Experience tasters · Guide index