Verbatim Chunks Beat Extracted Artifacts: A Controlled Ablation of Memory Representations for Long LLM Conversations
Tao An Hawaii Pacific University
Abstract
A growing class of conversational-memory systems compresses dialogue history into structured artifacts—extracted facts, decisions, or events—on the premise that distilled structure retrieves better than raw text. We test this premise with a controlled ablation: within one fixed retrieval–rerank–reasoning pipeline, we swap only the stored representation—LLM-extracted typed artifacts versus verbatim conversation chunks—holding the model, retriever, reranker, and judge constant. Verbatim chunks win by 15.9 points on LoCoMo (43.9% vs. 28.0%) and 22.0 points on LongMemEval-S (67.4% vs. 45.4%); a 1-hop semantic graph does not recover the gap, and five confound controls reproduce the effect. The mechanism is lossy distillation: extraction discards verbatim detail that chunks retain for free, and the extracted-artifact pipeline never beats naive RAG in overall accuracy. Concurrent positive results with near-verbatim, provenance-preserving units fit the same account: retrieval accuracy tracks how far the representation departs from the source. For the extraction designs we test, structured memory should augment verbatim text rather than replace it: a chunks $\cup$ artifacts union store matc
中文速览
现有的对话记忆系统流行一种做法：用大语言模型把历史对话提炼成「结构化片段」（如关键事实、决策、事件），再从这些提炼物中检索，理由是结构化内容比原始文本更容易检索。这篇论文用控制变量实验直接检验这个假设——在完全相同的检索-重排-推理流水线中，只换掉存储格式：一组用LLM提取的结构化片段，另一组直接保留原始对话文本块，其余所有组件不变。结果是原始文本块大幅胜出：在LoCoMo和LongMemEval-S两个基准上，原始文本块分别领先结构化提取15.9分和22.0分，加上语义图扩展也无法弥合差距。失败根源在于「有损蒸馏」——提取过程本身就丢弃了原文中细节，例如「到处都用类型注解」被压缩成「用户偏好类型注解」，精确约束就此消失，而保留原文的方案天然免疫这一损失。这一发现对记忆系统的工程设计有直接指导意义：结构化提取应作为原始文本的补充而非替代，两者合并使用可兼顾两个基准的表现，而单独依赖结构化提取在准确率和每次正确回答的成本上均不占优势。
原文 arXiv:2601.00821;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2601.00821v3