Component benchmarks lie. Tune the stack.

Every team that builds retrieval starts in the same place: a leaderboard. You compare candidate retrievers, sort by the headline metric, and pick whatever sits at the top that you can afford to run. It feels rigorous: the number is right there, measured on a public benchmark, reproducible. It is also, for the decision you're actually making, close to meaningless.

The benchmark measures one component, alone, on someone else's corpus. Your product runs that component inside a stack: a tokenizer, a sparse signal, a quantization scheme, an index with its own recall/latency knobs, a reranker, and an LLM that consumes whatever survives. The number that matters is the one at the bottom of that stack, on your data, under your latency budget. Nothing on the leaderboard measures that.

The leaderboard is the wrong starting point

A public retrieval benchmark is built to isolate a variable. That's the whole point of it: hold everything else fixed so you can compare retrievers on equal footing. It is a good instrument for the question "which sparse retriever scores highest alone, all else equal." It is a bad instrument for the question you have, which is "given my data and my one-second latency budget, what is the most accurate retrieval system I can ship."

Those are not the same question, and the gap between them is where most of the engineering value lives. In production, changing one component changes the behavior and operating constraints of the stack around it. Your documents are heterogeneous, your queries are domain-specific, the sparse retriever is fused with a dense one, and there's a reranker downstream that the benchmark didn't run at all. The relevant question is not whether the component lead is real. It is how much of that lead survives the stack.

Ten points in isolation. One tenth end to end.

In a production legal-search engagement, the goal was to match the accuracy of an all-premium retrieval stack without inheriting its latency, cost, and operational burden. The reference stack used the leaderboard winner at each layer; for sparse retrieval, that component was a learned sparse model from OpenSearch.

BM25 is a CPU-native lexical retriever built on fixed term statistics. The premium sparse candidate was OpenSearch's OpenSearch Neural Sparse Encoding Doc v3 GTE, a learned sparse model that uses a transformer to encode documents and a lightweight token-weight lookup for queries. It improved standalone recall, but increased indexing time and infrastructure complexity relative to BM25.

In isolation, the OpenSearch model reached 60.3 Recall@40, while classical BM25 scored roughly 50, a gap of about ten points. But the comparison was asymmetric: the learned sparse model had learned which terms mattered from labeled examples; BM25 had not.

We first made that fixed component trainable. We assigned each token a static weight learned from labeled relevance pairs, producing a variant we call L-BM25.¹ That recovered roughly seven points and reduced the standalone gap to 3.1, without changing BM25's serving profile.

We then compared both sparse candidates in the hybrid search setup they would actually serve in: each was fused with the same fine-tuned dense retriever, then followed by the same reranker. Hybrid search reduced the remaining gap to 0.5 points; reranking reduced it again to 0.1 at the product boundary.² The benchmark's ten-point preference had effectively disappeared, while the BM25 stack retained its simpler CPU-native sparse pipeline and avoided transformer-based document encoding.

Why the gap collapses

The result is not that BM25 secretly beats learned sparse retrieval. It is that a ten-point component advantage can become immaterial after adaptation, fusion, and reranking, while its indexing and infrastructure costs remain.

The gap collapsed through two different forms of optimization. The first happened inside the component. Classical BM25 derives term importance from corpus statistics; L-BM25 added a static learned weight for each token, trained offline from the engagement's relevance pairs. The online query path remained CPU-native lexical retrieval, but the component now knew that a statute number or defined term could matter more than its raw frequency implied.

The second happened between components. Sparse retrieval was one source of candidates, fused with a fine-tuned dense retriever and followed by a reranker. Some matches the weaker sparse component missed were supplied by the dense branch; some ordering errors were corrected downstream. Only errors that survived the complete stack counted against the product. A standalone benchmark measures none of that recovery because it holds those layers fixed or leaves them out.

The advantage was real in isolation. Isolation is not what you ship.

The narrower claim is the useful one: an isolated component advantage does not tell you how much of that advantage survives the complete system, or whether what survives is worth a different serving architecture.

What "tuning the stack" actually means

Tuning the stack is not just grid-searching a few coefficients around frozen components. It means treating apparently fixed components as trainable where possible, evaluating every candidate inside the pipeline it will actually inhabit, and optimizing the complete system against the product metric and serving constraint.

Concretely: make the sparse candidate, its learned token weights, the dense partner, the fusion weight, and the reranker one search space. Evaluate complete configurations, not naked components. When configuration search stops paying off, train the components that can still move the product metric, then retain an update only if it survives held-out and regression sets.

# Compare complete retrieval systems under the serving constraint.
for cfg in search_space(
        sparse      = ["bm25", "l-bm25", "splade"],
        dense       = ["e5-base", "bge-small"],
        fusion_weight = [0.2, 0.4, 0.6, 0.8],
        reranker    = ["none", "distilled-ce"]):

    stack = build(cfg)
    stack = finetune(stack, on=corpus)        # token weights, dense adapters
    recall, latency = evaluate(stack, queries)

    if latency <= budget:
        frontier.add(cfg, recall, latency)  # keep every feasible point

What comes out the other side isn't a single "best" sparse retriever: it's a frontier of complete systems. You pick the one that fits the product, and the learned-sparse-or-BM25 question that started all of this turns out to have been a knob, not the decision.

What this means for how you buy

If you're technical, the takeaway is operational: stop selecting components in isolation, and start tuning against the metric your product is judged by. The leaderboard is a fine place to assemble a shortlist of parts. It is a terrible place to make the decision.

If you're a leader, the takeaway is about procurement. The instinct is to assemble the all-premium build, the leaderboard winner at every layer, and trust that the sum is good. But every vendor in that stack optimizes its own layer and benchmarks against others in the same layer: none of them is moving the product-level number you actually care about.

Every vendor is optimizing a layer. Nobody is optimizing your number.

So the thing worth buying isn't a row on a leaderboard. It's the operating point on your frontier, and the tuned, fine-tuned, distilled assets that put it there. That's a different purchase, and it's the one that actually ships.

The market-structure half of this argument, why nobody selling you a layer is paid to find your operating point, is the anchor essay: The stack is the product. And if you'd rather have your frontier measured than argued: talk to us.