When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL

摘要

For sparse, structured reinforcement-learning tasks with semantic reward-function interfaces, LLM-generated reward shaping is better framed as a debugging problem than as pure one-shot generation. We study PPO-trained agents, using MiniGrid as the core sparse-structured evaluation and MuJoCo reaching/locomotion as boundary stress tests. Our qualitative audit finds two recurring one-shot failure modes—reward flooding and semantic/API misunderstanding—plus a rarer, less reliably distinguished weak-shaping case. We propose diagnostic-driven iterative refinement, where training diagnostics and a failure-mode taxonomy guide targeted reward-function revision. On sparse structured tasks with diagnosable reward failures, refinement improves DoorKey-8 $\times$ 8 from $2.3\%$ success without shaping to $97.6\%$ , and KeyCorridor from $31.2\%$ one-shot to $86.7\%$ with high seed-to-seed variance. Controls indicate that these gains are not attributable to generic retrying or extra training alone: metrics-only re-prompting produces large drops, while a static-vocabulary control recovers much of the gap, showing that the taxonomy prompt itself is a major mechanism and that dynamic trigger labels provide only partially isolated incremental evidence. In a new prompt-matched comparison across Haiku 4.5, Opus 5, and DeepSeek V4 Pro, the refinement effect is task- and model-dependent and none of the six contrasts survives family-wise correction; a four-call, episode-budget-matched population control is also competitive. Component-removal stress tests and sensitivity analyses reveal calibration limits. Continuous-control results expose the boundary: success-based diagnostics can misfire in dense-reward locomotion, and return-trend feedback removes one false-positive mechanism without producing robust locomotion gains. The method’s scope is deliberately bounded to sparse structured tasks with reliable structured interfaces under PPO training.