// Datasets
These datasets define the public test surface used across the SPM evidence pages. Together they cover long-horizon conversational state, multi-session recall, synthetic state tracking, and extreme long-context stress conditions.
// Why This Suite Exists
Public claims on this site are grounded in explicit test surfaces rather than loosely sampled conversations. Each case follows a recognizable structure: target facts or constraints are introduced early, long or distracting history is added, and a fact-sensitive probe is asked at the end.
// Public Metrics
// Suite Inventory
LHCSB
Long-horizon conversational state
Internal benchmark covering fact tracking (T4), absence recognition (T5), and temporal reasoning (T6). Derived from LoCoMo and MSC Personas corpora. SHA-256 locked, seed=42 reproducible.
Representative scenario types
T4: cross-session fact retrieval
T5: explicit abstention on unanswerable questions
T6: relative-time expression inference
Test split: T4:120, T5:120, T6:51
LoCoMo 100q
Multi-session long conversation recall
Real multi-session conversations with questions spanning multiple sessions. Tests whether the system can retrieve facts from earlier sessions without confusion.
Representative scenario types
cross-session fact recall
temporal ordering
speaker attribution
long-range reference resolution
BABILong full-250
Synthetic state tracking at 16k context
Synthetic QA tasks (QA1–QA5) embedded in long distractor context. Tests whether the system can track state through noise at fixed context length.
Representative scenario types
QA1: single supporting fact
QA2: two supporting facts
QA3: three supporting facts
QA4: two argument relations
QA5: three argument relations
LongMemEval 100q
Long-term memory question answering
Questions requiring retrieval from long personal conversation histories. Tests evidence recall and answer accuracy over extended interaction logs.
Representative scenario types
single-session recall
cross-session recall
temporal reasoning
preference tracking
RULER 720-case
Long-context stress across full category coverage
Balanced sample across all RULER categories. Tests retrieval, multi-hop reasoning, and aggregation under long-context pressure.
Representative scenario types
NIAH single
NIAH multi-key
NIAH multi-value
variable tracking
common words extraction
frequent words extraction
question answering
Ultra-long validation
Extreme-length factual retention
Focused validation at operationally extreme history sizes above 0.8M baseline tokens. Tests whether the system preserves factual usefulness without prompt collapse.
Representative scenario types
host and port recall
config value recall
environment separation
planning target recall