Fidelity Before Structure: Verbatim Chunks Beat Lossy Artifact Extraction in Long-Conversation LLM Memory
Tao An Affiliation: Hawaii Pacific University Email:
Abstract
A growing class of conversational-memory systems compresses dialogue history into structured artifacts (extracted facts, decisions, or events) on the premise that distilled structure retrieves better than raw text. We test this premise with a controlled ablation: within one fixed retrieval–rerank–reasoning pipeline, we swap only the stored representation (LLM-extracted typed artifacts versus verbatim conversation chunks), holding the model, retriever, reranker, and judge constant. Verbatim chunks win by 15.9 points on LoCoMo (43.9% vs. 28.0%) and 22.0 points on LongMemEval-S (67.4% vs. 45.4%); a 1-hop semantic graph does not recover the gap, and six confound controls reproduce the effect. The mechanism is lossy distillation, not structure per se: accuracy tracks how much source text survives in the store, and the extracted-artifact pipeline does not beat naive RAG in overall accuracy (though chunks abstain worse, §6). For the extraction designs we test, structured memory should augment verbatim text rather than replace it: adding artifacts alongside chunks preserves accuracy; substituting them forfeits the gap. Code and data: https://github.com/tao-hpu/cog-canvas.
中文速览
长对话记忆系统常把原文压缩成事实、决定等结构化条目，但关键问题是这种“提炼”会不会丢掉回答问题所需的细节。研究者在同一套检索、重排和推理流程中，只替换存储形式，严格比较原始对话片段与大模型抽取的结构化记忆，并用语义图和多项控制实验排除其他因素。结果显示，原文片段在 LoCoMo 上领先 15.9 个百分点、在 LongMemEval-S 上领先 22 个百分点，语义图也无法弥补差距，说明主要损失来自有损压缩而非缺少结构。这个结论提醒我们，结构化记忆更适合与原文并存，而不应直接取代原文；在当前这类无训练、无状态的系统里，保真度比“看起来更聪明”的抽象结构更重要。
原文 arXiv:2601.00821;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2601.00821v4