Verbatim Chunks Beat Extracted Artifacts: A Controlled Ablation of Memory Representations for Long LLM Conversations
Tao An Hawaii Pacific University
Abstract
A growing class of conversational-memory systems compresses dialogue history into structured artifacts—extracted facts, decisions, or events—on the premise that distilled structure retrieves better than raw text. We test this premise with a controlled ablation: within one fixed retrieval–rerank–reasoning pipeline, we swap only the stored representation—LLM-extracted typed artifacts versus verbatim conversation chunks—holding the model, retriever, reranker, and judge constant. Verbatim chunks win by 15.9 points on LoCoMo (43.9% vs. 28.0%) and 22.0 points on LongMemEval-S (67.4% vs. 45.4%); a 1-hop semantic graph does not recover the gap, and five confound controls reproduce the effect. The mechanism is lossy distillation: extraction discards verbatim detail that chunks retain for free, and the extracted-artifact pipeline never beats naive RAG in overall accuracy. Concurrent positive results with near-verbatim, provenance-preserving units fit the same account: retrieval accuracy tracks how far the representation departs from the source. For the extraction designs we test, structured memory should augment verbatim text rather than replace it: a chunks $\cup$ artifacts union store matc
原文 arXiv:2601.00821;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2601.00821v3