// Evaluation
SPM-Core is evaluated against standard_rag, hybrid_rag, and mem0_platform on locked public test sets. All results use fixed data versions and are reproducible.
LHCSB primary score
85.57%
T5 abstention
100%
Test samples
291
LHCSB primary score
85.57%
BABILong full-250
99.60%
T5 abstention accuracy
100%
Formal test samples
1,461
// Systems Compared
All systems receive the same context, the same questions, and are scored by the same metrics. No system has access to test labels during evaluation.
The primary memory system: precision long-context recall, explicit abstention on unanswerable questions, and bounded prompt budgets.
The default external-memory baseline. Retrieves top-k chunks by TF-IDF cosine similarity. No topic scoping or abstention mechanism.
Combined semantic and lexical retrieval (0.65 × cosine + 0.35 × lexical overlap). Stronger on keyword-heavy queries than standard RAG.
Hosted memory platform using semantic compression on ingestion. Retrieves structured memory entries rather than raw text chunks.
// Cross-Benchmark Results
| Benchmark | SPM-Core | Best baseline | System | Lead |
|---|---|---|---|---|
| LHCSB | 85.57% | 41.58% | standard_rag | +43.99pp |
| LoCoMo 100q | 86.47% | 73.33% | standard_rag | +13.14pp |
| BABILong full-250 | 99.60% | 46.99% | hybrid_rag | +52.61pp |
| LongMemEval 100q | 73.00% | 52.00% | hybrid_rag | +21.00pp |
| RULER 720-case | 86.47% | 81.08% | standard_rag | +5.39pp |
LHCSB
85.57%
+43.99pp
LoCoMo 100q
86.47%
+13.14pp
BABILong full-250
99.60%
+52.61pp
LongMemEval 100q
73.00%
+21.00pp
RULER 720-case
86.47%
+5.39pp
// Key Deltas
Wrong rate (LHCSB)
25.35% → 0.75%
Standard RAG to SPM-Core.
T5 abstention
100%
All baselines below 2.5%.
BABILong lead
+52.61pp
vs hybrid_rag (46.99%).
Evidence recall
94.58%
LongMemEval 100q.
// Pass Rate by Benchmark
// Wrong-Answer Rate Comparison
SPM-Core
Standard RAG
// Standout Notes
SPM-Core leads all comparison systems on every formal test set.
LHCSB T5 abstention accuracy: 100%. All comparison systems score below 2.5%.
BABILong full-250: 99.60% — QA1 through QA4 all pass. Hybrid RAG scores 46.99%.
LongMemEval evidence recall: 94.58%. Primary score 73.00% vs 52.00% for the best baseline.
// Ultra-Long Validation
The ultra-long set asks a narrow but important question: when baseline context becomes operationally extreme (0.8M+ tokens), can SPM still keep the final prompt bounded while preserving the facts required by the task?