Optimizing Long-Horizon Agent Memory with Ranked Recall

The Memory Harness Architecture

Stefania Dug proposes treating memory as a control loop—a write-manage-read cycle—rather than a static database. The harness is designed for local agents (tested on Qwen 2.5 7B and DeepSeek V4 Flash) that lack durable memory. The system consists of three primary blocks: a core trace that is always visible to the agent, a recall block for retrieving information, and an archival block for cross-session tracking.

Ranked Recall vs. Standard RAG

Experiments on the Xbench benchmark demonstrate that standard vector RAG often fails to improve performance when the relevant information is buried deep in the task history. Instead, a "ranked decision ledger"—which explicitly tracks and prioritizes decisions made at each turn—consistently yields higher accuracy.

• Baseline: No memory recall.
• Vector RAG: Standard similarity-based retrieval.
• Ranked Ledger: A structured policy that ranks historical decisions based on relevance to the current step.
• Oracle: A ground-truth control that provides the correct memory to the model, revealing that even with perfect retrieval, models may still ignore or misinterpret the data.

Performance and Cost Implications

When tasks fit entirely within the context window, memory harnesses provide no performance gain and only increase token costs. However, for long-horizon tasks where the answer lies outside the immediate context, a ranked recall policy reduces token expenditure compared to naive retrieval. Bad memory is expensive because it consumes tokens while potentially steering the agent toward incorrect conclusions. The author emphasizes that the recall policy should be treated as a first-class metric in system design, as it directly impacts both agent reliability and compute budget.