Nebius (acq. Tavily) · Oct 2025 – May 2026 · Abu Dhabi
Making AI search return the right answer more often
Two workstreams inside a sub-millisecond budget: turning raw BM25 scores into a reranker-style confidence score, and a relevance layer with per-document summaries that decides which chunks are worth an AI agent's context. Roughly 29% precision improvement.
Role
Lead DS, sole owner
Constraint
<1 ms per query
Outcome
~29% precision
The stakes
When a human searches, a mediocre result costs them a glance. When an agent searches, a mediocre result gets read in full, occupies context, and shapes an answer someone will act on. Precision stops being a nice-to-have and becomes the product.
The retrieval layer was BM25-based and fast, and fast was non-negotiable. The search API sat inside other people's agent loops, so any budget I spent was a budget they could not spend on reasoning. The brief was uncomfortable on purpose: raise precision measurably, in under a millisecond, without a cross-encoder, a second network hop, or touching the index.
The system, in three stages
1 · Retrieve
BM25 top-k candidates for the agent's query
2 · Calibrate
BM25 scores converted into a confidence score that behaves like a reranker's
3 · Select chunks
a relevance layer over TF-IDF and score-distribution signals - plus per-document summaries - picks what is worth reading
Everything after retrieval is CPU-local and feature-cached. No model call and no extra hop. That is the entire reason it fits the budget.
Three decisions, and what I turned down
01
Turn BM25 scores into a confidence score
Raw BM25 scores are not comparable across queries - a rare-term query produces a completely different score distribution from a common-term one, so no global threshold works. I converted the scores into a calibrated confidence score that behaves the way a neural reranker's confidence does: a number you can set one policy against, for any query. Downstream consumers get reranker semantics at lexical-search cost.
Rejected: a cross-encoder reranker. Best quality on paper and I would have enjoyed building it. It costs tens of milliseconds - ten thousand percent over budget is not a tradeoff, it is a different product.
02
Judge chunks by their score distributions
Lexical match tells you a chunk contains the words; it does not tell you the chunk is worth reading. On top of Elasticsearch keyword search I built a relevance layer that identifies the chunks actually worth returning - reading TF-IDF signals and the shape of the BM25 score distribution rather than trusting any single score, alongside standard retrieval-quality techniques from the RAG playbook.
Rejected: trusting the top-k as ranked. The k-th result of a strong query and the first result of a weak one carry very different amounts of evidence, and treating them the same is where bad context comes from.
03
Give every document a summary, and search that
Each document in the index was an extract of a full webpage - long, noisy, and expensive to match against. I added a short summary to every document describing what the page actually contains, so search can decide relevance from the summary instead of wading through the whole extract. Results got faster and noticeably more precise at the same time, which is not a trade you get offered often.
Rejected: matching on full page content. More text to search is not more signal - a page mentions a hundred things, its summary says what it is about.
What came of it
~29%
precision improvement at the depth agents read, on a held-out slice, with the recall cost stated openly in the same report.
<1 ms
held. The layer shipped into the production search path used by external agent developers, with no added network hop.
The part I would argue is most valuable is not the number. It is that the relevance layer became a place where quality signals could be added later. Normalisation made new signals composable, so each new one did not need a threshold of its own.