AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

Abstract: Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see. The simplest contract appends past observations, tool calls, and reflections to every prompt, which makes prior context easy to access but also turns it into a jumbled mixture in which the effect of any single memory component is hard to isolate. We introduce and instrument an alternative bounded contract: every decision is made from a fresh user message assembled by typed retrieval, with no raw cross-decision transcript appended. The prompt thus stays bounded across runs of any length, and any single layer can be ablated in isolation. We instantiate the contract in Slay the Spire 2, a closed-rule stochastic deck-building game whose runs require hundreds of tactical and strategic decisions. A public online benchmark of frontier LLMs on the same game reports zero wins at the lowest difficulty across five configurations, and the developer-reported human win rate at the same difficulty is 16%; the task is hard but not saturated. Within our harness, a fixed-A0 ablation shows the largest observed difference when triggered strategic skills are enabled: the no-store baseline wins 3/10 games and adding the skill layer 6/10. At this sample size the comparison is directional rather than statistically decisive (Fisher exact p\approx0.37); a cross-backbone probe and public accumulating-context baselines are reported as operational comparisons rather than controlled tests of the contract variable itself. We release a reproducible testbed: 298 completed trajectories with condition tags, frozen memory/skill snapshots, prompt records, and analysis scripts -- an agent design and a validated, reusable methodology for studying how explicit memory layers shape long-horizon LLM-agent decisions.

• The paper introduces a bounded, typed-memory contract that replaces accumulating transcripts with a compositional five-layer retrieval design.
• It systematically ablates memory components, showing that skill augmentation (L5) yields the largest performance gain in win rates at A0 and beyond.
• The study demonstrates reduced token costs and enhanced auditability, setting a new benchmark for long-horizon LLM agent evaluations in complex, closed environments.

AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

Motivation and Context

Long-horizon autonomous agents driven by LLMs are fundamentally constrained by memory architectures that dictate which prior experiences and knowledge are accessible at each decision step. Historically, the dominant paradigm has been transcript accumulation—appending the entire prior action/observation history to the prompt, thus creating prohibitively large contexts, occluding causal attribution, and hindering precise ablation of memory components. The "AgenticSTS" paper (2607.02255) addresses these issues by introducing a formally bounded, typed-memory contract for LLM agent memory in a difficult sequential decision testbed, enabling layer-wise ablation and systematic evaluation of memory contributions.

Testbed: Slay the Spire 2 as a Long-Horizon Evaluation Platform

Slay the Spire 2 (STS2), a stochastic deck-building roguelike, serves as the testbed, selected for its closed, symbolically enumerable rule space, substantial long-horizon character (hundreds of decisions per run), and rich multi-axis stochasticity. The environment enables text-based representation of all relevant state/action variables, making it a favorable substrate for LLM-agent studies of non-trivial planning, state-conditioned reasoning, and the demands on persistent agent memory.

The practical evaluation focuses on the "Silent" character at the lowest difficulty (Ascension 0, or A₀). Public benchmarks report $0$ victories at A₀ across five high-profile LLM-agent configurations, with a developer-quoted human win rate at $16\%$, underscoring the challenge and suitability for studying architectural interventions in agent memory.

Figure 1: The paper's core proposal: bounding memory through typed retrieval layers creates an ablatable evaluation surface for long-horizon memory in LLM agents.

Architecture: The Bounded Typed-Memory Contract

AgenticSTS replaces the monolithic accumulating transcript with a five-layer, type-annotated retrieval system:

1. $L_1$ (Protocol Prompts): Immutable, state-generic role and protocol instructions.
2. $L_2$ (State-Specific Schemas): Fixed per-decision templates and legal action formats.
3. $L_3$ (Game Rules): Dynamically retrieved symbolic rule database (cards, relics, events, etc).
4. $L_4$ (Episodic Memory): Postrun summaries indexed by character/ascension/act/enemy class, supporting case-based retrieval.
5. $L_5$ (Skills): Triggerable, state-indexed tactical/strategic guides distillable from logs with explicit gating, capable of author-driven or automatically discovered population.

This design ensures that only decision-relevant, type-labelled context is included at each step, bounding both prompt size (tokens scale as $O(1)$ wrt number of decisions) and facilitating clean ablation and statistical attribution. Unbounded transcripts are strictly excluded.

Figure 2: Typed retrieval decomposes per-decision context into five ablatable memory layers, enabling explicit audit and manipulation of what experience or knowledge feeds each agent action.

Experimental Methodology

The evaluation comprises three primary empirical streams:

1. Fixed-$A_0$ Five-Condition Ablation: Systematically toggles $L_4$ and $16\%$0 to isolate their contributions, using a balanced subset of runs (10 per condition).
2. Cross-Backbone Transfer: Applies frozen $16\%$1/$16\%$2 stacks generated by Gemini 3.1 Pro to other LLMs (Qwen, DeepSeek) to probe backbone-sensitivity.
3. Auto-Mode Ascension Ladder: Activates post-run memory/skill updates, measuring the highest difficulty level reached under active memory evolution.

Each run records complete game states, condition tags, prompt records, skill/memory snapshots, and employs rigorous statistical protocol (Wilson/Clopper-Pearson intervals for win rates, bootstrap intervals for scores).

Numerical Results and Ablation Analysis

The central ablation reveals that the addition of the $16\%$3 skill layer results in the largest observed point-difference in $16\%$4 win rate: moving from a 'no-scaffold' baseline (3/10 wins) to a skill-scaffolded condition (6/10 wins). The difference is directional at the available sample size ($16\%$5; Wilson 95% CIs for win rates overlap), but is robust across both hand-authored ($16\%$6 mode-a) and template-generated ($16\%$7 mode-b) skills.

Figure 3: Five-condition ablation at fixed $16\%$8 quantifies the lift provided by each memory or skill layer, revealing $16\%$9 as the dominant differentiator.

Importantly, $L_1$0 episodic recall does not further increase win rates in the $L_1$1 regime (mode-a vs full-frozen: both 6/10), but is critical at higher difficulties; runs with active postrun memory reach A$L_1$2–A$L_1$3 on the auto-mode ladder, while those with no updates stall at A$L_1$4–A$L_1$5.

Figure 4: Ascension ladder endpoints for different memory update regimes, demonstrating the practical necessity of evolvable memory/skills in more challenging scenarios.

Cross-backbone tests show that frozen skill/episodic stacks are not universally portable: Gemini-trained stacks improve Qwen's score ($L_1$6), but harm DeepSeek ($L_1$7), and do not lift either above zero wins in $L_1$8 trials, confirming transfer is an empirical, nontrivial property.

Comparison to Accumulating-Context Agents

AgenticSTS was benchmarked against prominent, transcript-accumulating open-source baselines (STS2MCP, CharTyr), both of which failed to win any games (0/5 wins), despite consuming $L_1$9 orders of magnitude more LLM tokens per score point than the bounded contract agent. Mechanistically, their token usage balloons (>500k tokens per action in late runs), while AgenticSTS remains flat ($L_2$05k tokens per strategic call).

Figure 5: Prompt growth profiles: transcript-accumulating agents suffer catastrophic context bloat, while bounded context remains tightly capped.

Figure 6: Cost-effectiveness frontier: bounded-memory agents achieve superior performance with exponentially fewer tokens.

Implications and Prospects

The AgenticSTS framework substantiates several critical claims:

• Explicit memory contracts, disambiguated by type and mutability, enable clean empirical attribution of which slice of experience is functionally relevant in long-horizon policy execution.
• The bounded-memory contract—implemented via explicit per-decision typed reassembly rather than transcript accumulation—decouples agent performance from history size, dramatically reducing computation, bandwidth, and inference cost without sacrificing effectiveness at non-trivial task difficulty.
• The release—298 archived runs with condition tags, memory/skill snapshots, prompt logs, and scripts—provides a platform for reproducible, ablation-driven studies of agent memory design, supporting granular audits and future controlled comparisons (e.g., direct same-codebase matched comparisons to transcript-accumulating conditions).

This methodology shifts the analytic focus from aggregate "more context is better" heuristics toward precise layer-wise intervention, and the testbed architecture is readily generalizable to other closed-state agentic tasks. Additionally, the template/case-based skill insertion protocol supports a hybrid of manual and self-evolving skill acquisition regimes—a key design axis for future self-improving agent architectures.

Conclusion

AgenticSTS formalizes a memory-typed, ablatable testbed for evaluating decision-time memory in long-horizon LLM agents, demonstrating that (1) a bounded contract with compositional context and type-decomposed retrieval is tractable, auditable, and minimizes unnecessary context growth; and (2) skill-layer augmentation produces the largest observed performance shift under study constraints. Whether such a design universally outperforms stateless or accumulating-context paradigms remains an open question, but the presented resource provides a rigorous foundation for systematic investigation.

Figure 7: Explicitly-structured prompts assembled for key agent decisions: left, combat planning; right, shop planning, illustrating the compositional retrieval architecture at inference time.