Benchmarks
The numbers we quote, what each one measures, and where it comes from. All four were measured in our lab on the public LoCoMo long-conversation benchmark. They say how the system behaved on that test, not what your data or workload will do. Results vary with dataset, configuration and workload.
Tokens: the harness
87.9% fewer tokens than resending the conversation, by token accounting on the LoCoMo benchmark: the tokens sent when only the recalled context is included, against the tokens needed to resend the whole conversation each time. It measures context size, not answer quality; we do not claim the answers are identical. Your savings depend on how long your conversations are and what you ask.
Speed: the retrieval step
About 3.5 ms per lookup, measured in our lab on LoCoMo: the time for the retrieval engine to
return its results for one query. It does not include the network, authentication or any model call. A full
memory_query request to the hosted service takes longer, and times vary with store size, query length and load.
| measure | result | scope |
|---|---|---|
| retrieval step time | 3.5 ms | per lookup, LoCoMo, lab; excludes network and model calls |
| LLM calls in retrieval | 0 | no large language model is called to find memories; a small embedding model is used to match by meaning |
Quality: recall on LoCoMo
98.4% any-gold recall at k=512 on the LoCoMo benchmark: for each question, does at least one correct (gold) memory appear among the top 512 retrieved? LoCoMo is a public long-conversation memory benchmark. This is a lab benchmark figure, not a live figure, and not a promise about your data. k=512 is a wide net by design; it measures whether the right memory is findable at all. The hosted memory tools return at most 20 results per query, so this figure does not describe what a single query returns.
How this is measured
We list only numbers we can reproduce, and we say which benchmark each one comes from. If we find a flaw in a number, we withdraw it.
Why the retrieval step is fast
Retrieval is lexical plus semantic, fused at query time and scored by our own engine, with no large language model in the retrieval path. The embedding model used for memory runs on our own servers, so the memory step does not send your text to a third-party AI service. Inference requests are different: they go to the Model Provider that serves them, as the Privacy Policy explains.
Run it yourself
Export your store any time (/wiki/export.ndjson; today it holds each node's id, namespace, title and a
content snippet) and evaluate retrieval against your own questions.