By Mohamed Mohamoud

The Reranker: Why the First Vector Match Isn't Always Right

What reranking does in search systems, and why Retrievals currently ranks results with exact cosine similarity and no reranker.

The Reranker: Why the First Vector Match Isn't Always Right

Architecture update, 24 October 2026: Retrievals' current search implementation embeds text queries on Modal CPU and ranks 40,368 normalized 768-dimensional image vectors by exact cosine similarity. It returns 64 results, with pagination through the top 256. It does not use a reranker.

Reranking is a second ranking step used by some search systems. A first step retrieves candidate items. A second model then scores each query and candidate together and reorders the candidates.

How reranking works

An embedding search maps the query and items into a shared space, then ranks items by vector similarity. The system can limit its next step to the first set of candidates.

A cross-encoder takes the query and one candidate together and returns a score for that pair. A reranker applies that step to the candidate set, then sorts the candidates by the new scores.

flowchart LR
    A["Query"] --> B["First-stage retrieval"]
    B --> C["Candidate set"]
    A --> D["Pairwise scoring"]
    C --> D
    D --> E["Reordered results"]

This design can help when an evaluation shows that the first-stage ranking misses relevant items. It also adds another model call, compute use, and a component to operate. Whether it improves results depends on tests against representative queries.

Retrievals' current ranking

Retrievals does not use a reranker. Its Modal CPU service embeds the text query with quantized LiteRT EmbeddingGemma 2, then computes exact cosine scores against the normalized image vectors. The query vector and first 64 results come back in the first response. Later requests use that vector to page through the top 256 ranked results.

Artwork images are processed ahead of time. During a search, the service compares your text with those prepared image descriptions. It does not run a second model over the candidate images.

If a future search evaluation finds ranking errors that exact cosine cannot resolve, reranking would be one approach to assess. The decision should follow measured results on a representative query set.

Try a search

#architecture#reranking#vector search#cross-encoder#search quality