Nebius (acq. Tavily) · Oct 2025 – May 2026 · Abu Dhabi
Making AI search return the right answer more often
A content-relevance and ML filtering layer in front of BM25 retrieval, inside a sub-millisecond latency budget. Roughly 29% precision improvement from a hybrid score-normalisation method.
Role
Lead DS, sole owner
Constraint
<1 ms per query
Outcome
~29% precision
The stakes
When a human searches, a mediocre result costs them a glance. When an agent searches, a mediocre result gets read in full, occupies context, and shapes an answer someone will act on. Precision stops being a nice-to-have and becomes the product.
The retrieval layer was BM25-based and fast, and fast was non-negotiable. The search API sat inside other people's agent loops, so any budget I spent was a budget they could not spend on reasoning. The brief was uncomfortable on purpose: raise precision measurably, in under a millisecond, without a cross-encoder, a second network hop, or touching the index.
The system, in three stages
1 · Retrieve
BM25 top-k candidates for the agent's query
2 · Normalise
per-query, distribution-aware score normalisation
3 · Filter & rank
lightweight ML on cached features, cutting the list down to what is worth reading
Everything after retrieval is CPU-local and feature-cached. No model call and no extra hop. That is the entire reason it fits the budget.
Three decisions, and what I turned down
01
Normalise scores inside each query
Raw BM25 scores are not comparable across queries. A rare-term query produces a completely different score distribution from a common-term one, so any global threshold is simultaneously too strict and too loose. I normalised within each query's candidate set, so “relevant” became a statement about a document's position in this distribution.
Rejected: a single tuned global cutoff. Cheap, and it was what existed. It hid its failures: it silently over-pruned narrow queries, which are exactly the queries an agent asks when it is close to an answer.
02
Blend cheap signals into one score
Lexical match tells you a document contains the words. It does not tell you the document is worth reading. I combined the normalised lexical score with cheap content-quality and semantic-proximity signals into one blended score, with the weights fit against a labelled relevance set.
Rejected: a cross-encoder reranker. Best quality on paper and I would have enjoyed building it. It costs tens of milliseconds. Ten thousand percent over budget is not a tradeoff, it is a different product.
03
Measure precision at the depth agents actually read
The improvement is quoted at the top of the list because that is the only part that reaches the model's context. I built the eval set from real query patterns, held out a slice the weight-fitting never saw, and reported precision on that slice, alongside the recall I was giving up. A filter that only reports precision is a filter that is hiding something.
Rejected: optimising average precision over the full result list. It rewards improvements at rank 40 that no agent will ever see.
What came of it
~29%
precision improvement at the depth agents read, on a held-out slice, with the recall cost stated openly in the same report.
<1 ms
held. The layer shipped into the production search path used by external agent developers, with no added network hop.
The part I would argue is most valuable is not the number. It is that the relevance layer became a place where quality signals could be added later. Normalisation made new signals composable, so each new one did not need a threshold of its own.