How a search engine finds the right page
A plain-language look at crawling, inverted indexes, ranking, and the path from a query to a useful result.
Search feels like a text box, but the useful work happens long before anyone types. Pages must be found, cleaned, indexed, and ranked. Building SearchX made that hidden pipeline much easier to see.
Fast search begins by doing the expensive work before the user asks the question.
Ahmad Ali, One Percent
flowchart LR
A[Seed pages] --> B[Crawler]
B --> C[Clean text]
C --> D[Inverted index]
Q[User query] --> E[Query parser]
E --> D
D --> F[Rank matches]
F --> G[Search results] Crawling is careful exploration
A crawler starts with a small list of pages, reads their links, and follows the useful ones. It also needs boundaries. Without limits it can revisit the same pages, drift into unrelated sites, or place too much load on a server.
A good crawler keeps a queue, records visited URLs, respects site rules, and normalizes links so two versions of the same address do not become duplicate documents.
The inverted index is the key idea
Scanning every document for every query is slow. An inverted index flips the problem. Instead of storing only the words inside each document, it stores the documents that contain each word.
A search for “cloud pipeline” can jump straight to the lists for cloud and pipeline, combine them, and rank the overlap. This is the same basic idea behind much larger search systems.
- Normalize case and punctuation
- Remove words that add little meaning when appropriate
- Store term frequency and document frequency
- Keep enough source text to build a helpful snippet
Ranking turns matches into answers
Matching a word is not the same as finding a useful page. Ranking considers how often a term appears, how rare it is across the collection, where it appears, and sometimes how pages link to each other.
The important product lesson is that ranking is a set of choices. A documentation search, a shop, and a public web search should not value the same signals.
Measure what people actually find
A fast response with poor results is still a poor search experience. Useful checks include whether the expected page appears in the first few results, how often users reformulate a query, and where they leave the results page.
Search improves when query logs become product feedback. Missed queries show gaps in content, synonyms, spelling support, and ranking rules.
Ahmad Ali
Co-founder and CTO at One Percent. Writes about the practical choices behind products that need to work in the real world.