(Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliable

摘要

Large language models (LLMs) are increasingly delegated tasks once reserved for trained researchers: generating hypotheses, choosing specifications, drafting conclusions. Whether this delegation produces trustworthy science is not solely a technical question about model capability; it also depends on how cognitive labour is structured between humans and machines. We study this problem by organising an AI-assisted research workflow around behavioural-science principles, pre-commitment, decision sequencing, accountability, and attention allocation, rather than by directly measuring human psychology at the gates. We propose that reliability in AI-assisted research is a property of decision architecture: the placement, sequencing, and binding force of the choices that humans and AI components are each permitted to make. In a pre-specified $2\times 4$ factorial experiment ( $N=280$ complete research runs across four datasets), an unconstrained multi-agent baseline produced critical failures in 72% of runs; the same underlying model and the same agent decomposition, with identical prompts on the reasoning agents shared by both arms, failed in 16% once organised by three architectural commitments (LLMs restricted to reasoning, data and estimation executed deterministically, three human decision gates; Fisher’s exact $p<0.001$ ). Reliability gains were largest on the least publicly represented dataset, a Qing-dynasty population register, in a pattern consistent with a task-based production model with Fréchet-distributed output quality. An 80-run ablation suggests that deterministic computation and human gates contribute independently, with exploratory evidence consistent with complementarity. We interpret HLER as a research harness rather than an autonomous AI scientist. Its contribution is to reduce failures sharply, make residual weaknesses more visible, and prevent unreliable claims from being advanced as publication-ready outputs.