Why your retrieval scores cannot be averaged
Our unified search compared three different quantities in one column. The exact answer ranked below a weaker one, purely because of scale. What we measured, and what we changed.
1 Nov 2026 · 3 min readAI-generated audio overview (NotebookLM) of the sources behind this article. A discussion, not a measurement: the figures in the text are the checked ones.
Our system has one read tool that searches six kinds of data at once: memories, library docs, code, the knowledge graph, plans, and past sessions. For months it returned a single list, sorted by a field called relevance.
That field held three different quantities.
What was in the column
- Memory reported its biological score: similarity shaped by decay and importance.
- Docs, sessions and plans reported raw cosine similarity.
- Knowledge and code passed through a rescaling step that stretched
[0.42, 0.78]onto[0, 1].
The rescaling constants came from an earlier embedding model. We had moved to bge-m3 without recomputing them, so one model’s distribution was being rescaled with another model’s bounds.
Measured on 31 August 2026, for the query “where are the hive’s values and axioms stored”:
| result | cosine | shown as |
|---|---|---|
| the anchor record for the values and axioms | 0.5529 | 0.369 |
| a docs page about a browser API | 0.4353 | 0.435 |
The exact answer was semantically closer and displayed lower. Nothing was wrong with either retriever. The comparison was never between like and like.
A second way a score lies
Keyword search returns a position, not a measure. One layer turned that position into a score, which handed the top keyword hit a flat 1.0 however poorly it matched. The query “Landauer principle” returned Landauer’s Principle at similarity 0.335 and score 1.0.
Inside that one layer, harmless: the order was already decided. In a list shared with five other sources, a flat 1.0 outranks every honest cosine.
The bands move when the model moves
After the bge-m3 cutover we measured where each source’s scores sit. Memory and knowledge had swapped ends compared with the note written under the previous model. Any rule like “memory scores are usually higher” silently inverts the day you change embedders.
Normalising does not rescue it. Score fusion earns its place only once you have labelled data to tune weights against 2; until then rank fusion is hard to beat out of the box 3. Min-max scaling is destroyed by one outlier, and z-scores assume a normal distribution that relevance scores do not have.
What we do instead
1. Fuse by position, not by score. Reciprocal rank fusion scores a document as Σ 1/(k + rank) across retrievers, with k = 60, the value from Cormack et al. (2009) 1. Scale never enters, so it cannot leak.
2. Rerank inside one kind of data only. A cross-encoder reads the query and each candidate together and is far more precise than a vector distance. We run it per source, never across a mixed list.
3. Return one group per source. Since 31 October 2026 the unified tool returns groups[source], each reranked inside its own source and capped at five, with no fused list across sources. A reader compares within a group, never between groups.
The numbers, and what they do not say yet
On our fixed set of 52 labelled queries, with the reranker on:
| measure | reranker off | reranker on |
|---|---|---|
| right item first (recall@1) | 0.5192 | 0.5385 |
| right item in the top 5 | 0.6731 | 0.7308 |
| nDCG@10 | 0.6227 | 0.6537 |
Two honest limits:
- The per-source shape did not move these numbers, and that was expected. Every one of the 52 queries names its source, so none exercises the mixed path the change fixed. We added a separate query set for that, rather than claim a gain we had not measured.
- Slovak queries are weak: the right item came first for 1 of 9. Our keyword index runs with English stemming. That is the next thing to fix, not something to hide.
One more measurement shaped the design: when the biological rescore ran before any reranking, exact-name retrieval fell from 93% to 43% on 30 records queried by their own names. A memory that fades is a good idea. Letting it fade an exact match out of the results is not. In the next revision it moves after the reranker, and only for exploratory questions.
A score is only meaningful next to scores made the same way. Compare positions across sources, and scores only within one.
- G. V. Cormack, C. L. A. Clarke, S. Büttcher. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. SIGIR 2009not in corpus
- TopK. RRF vs Score Fusion vs True Hybrid: Which Should You Use?. topk.io (2026)saved pagecorpus: ~/hybrid-retrieval
- Guillaume Laforge. Advanced RAG — Understanding Reciprocal Rank Fusion in Hybrid Search. glaforge.dev (2026)saved pagecorpus: ~/hybrid-retrieval
The measurements in this article come from these records in the running system.
factorium/src/server/core/hive/hive-db.ts— rescaling defect, measured 2026-08-31factorium/src/server/core/hive/rrf.ts— score bands after the bge-m3 cutover, 2026-09-29~/.factorium/logs/retrieval-eval.jsonl— v2 probe set, 52 queries, 2026-10-31hive chat #356, #390, #391— exact-name measurement; per-source shape, verified
Our embedder and both rerankers we tested are sold as multilingual. Measured with paired Slovak and English queries, Slovak failed 55% of the time against 20%, and a cross-encoder gave three different documents the identical score.
Retrieval finds the nearest item. Search composes an answer from parts, the way a chemist plans a molecule backwards from what is on the shelf. The map we took from the chemistry, and what it changes in an agent.
Ask our memory for a record by its own name: 93% rank-1 on plain semantic search, 43% after the biological rescore. Why a smarter score lost the easiest query, and the fix we chose.