Reasoning Capabilities of Large Language Models on Dynamic Tasks

Abstract: LLMs excel on static benchmarks, but their ability as self-learning agents in dynamic environments remains unclear. We evaluate three prompting strategies: self-reflection, heuristic mutation, and planning across dynamic tasks with open-source models. We find that larger models generally outperform smaller ones, but that strategic prompting can close this performance gap. Second, an overly long prompt can negatively impact smaller models on basic reactive tasks, while larger models show more robust behaviour. Third, advanced prompting techniques primarily benefit smaller models on complex games, but offer less improvement for already high-performing LLMs. Yet, we find that advanced reasoning methods yield highly variable outcomes: while capable of significantly improving performance when reasoning and decision-making align, they also introduce instability and can lead to big performance drops. Compared to human performance, our findings reveal little evidence of true emergent reasoning. Instead, LLM performance exhibits persistent limitations in areas like planning and spatial coordination, suggesting that LLMs still suffer fundamental shortcomings that may not be fully overcome through self-reflective prompting alone. Reasoning is a multi-faceted task, and while methods like Chain-of-thought improve multi-step reasoning on math word problems, our findings using dynamic benchmarks highlight important shortcomings in general reasoning capabilities, indicating a need to move beyond static benchmarks to capture the complexity of reasoning.

• The paper demonstrates that advanced prompting methods such as Reflection, Oracle, and Planner significantly impact multi-step reasoning in LLM-driven agents across diverse tasks.
• The paper shows that restructuring rewards from sparse to dense signals improves agent alignment with task objectives more reliably than increasing prompt complexity.
• The paper identifies a 'knowing-doing gap' where LLMs can recite optimal solutions yet struggle to execute effective action trajectories in interactive, dynamic environments.

Reasoning Capabilities of LLMs on Dynamic Tasks

Introduction

The paper "Reasoning Capabilities of LLMs on Dynamic Tasks" (2505.10543) conducts a rigorous empirical investigation into the ability of open-source LLMs to act as autonomous agents in dynamic, interactive environments. The study scrutinizes both model scaling and prompting techniques on a set of reasoning-intensive tasks, focusing on the potential for in-context learning mechanisms—including self-reflection, heuristic mutation, and planning—to enable generalization and adaptation in the absence of parameter updates. The experimental analysis centers on SmartPlay, a suite of interactive tasks designed to probe agents' capacity for multi-step reasoning, planning, and spatial coordination.

Methodology

Agent Architecture and Prompting Strategies

Agents are implemented as LLM-driven policies operating in text-based simulated environments. At each timestep, the LLM receives a serialized prompt encoding the history of states, actions, and rewards, along with the current observation, environment manual, and set of legal actions. No fine-tuning is performed; all adaptation occurs via prompt updates.

Beyond the base setup, three augmentation modules are tested:

• Reflection: Per-step retrospective analysis to provide task-aligned feedback within the current episode.
• Oracle: Evolutionary refinement of textual heuristics at the episode level using a (1+1)-ES strategy.
• Planner: Forward simulation of multi-step action sequences with reward estimation for improved decision selection.

Models and Environments

Benchmarked models include Llama3-8B, Mistral-Nemo-12b, DeepSeek-R1-14b, and Llama3.3-70B, selected for their open-access architecture and broad parameterization. Tasks span Bandit, Rock Paper Scissors (RPS), Tower of Hanoi, and Messenger from SmartPlay, each targeting distinct dimensions of reasoning (e.g., exploration-exploitation, stochastic adaptation, multi-step planning, spatial and synonymic reasoning).

Results

Model Scaling and Prompt Engineering

Larger models demonstrate robust performance gains compared to smaller counterparts. Llama3.3-70B establishes state-of-the-art results on Bandit and Tower of Hanoi, with median scores approaching or surpassing smaller models by nontrivial margins. However, appropriately engineered prompting strategies can allow smaller models to match or even exceed baseline performance of larger models in certain environments. For example, Reflection + Oracle strategies close the performance gap on RPS and Messenger for smaller LMs.

Importantly, the use of advanced prompting techniques is a double-edged sword: while they elevate performance when task-appropriate reasoning emerges, they consistently introduce high variance. Instability is observed across independent runs, with prompting sometimes yielding substantial regression. For example, Reflection + Planner can reduce Bandit and Messenger task scores for Llama3.3-70B, underscoring the lack of robustness in in-context reasoning-driven adaptation.

Reward Restructuring

The authors find that transforming reward signals from sparse to dense markedly improves performance across agents, simplifying the task of aligning agent behavior with episodic objective functions. This finding suggests that reward signal engineering may be a more practical axis of progress than prompt optimization, especially for tasks with delayed or sparse feedback.

Task-level Analysis

• Bandit: Simpler models perform well with minimal prompting; intense reasoning degrades performance due to context bloat and distractor effects, as additional context dilutes relevant information.
• RPS: Larger models exhibit improved adaptation and probabilistic reasoning, leveraging planner modules to update strategies according to observed opponent biases.
• Tower of Hanoi: None of the architectures internalize task invariants sufficiently to consistently solve the puzzle; reflection modules facilitate marginal improvement, but both smaller and larger models frequently commit illegal moves, evidencing a failure to generalize reasoning about state constraints. Surprisingly, even random baselines occasionally outperform LLM agents in 3-disk settings.
• Messenger: Enhanced reward shaping and context simplification (removal of synonyms) improve object pickup and goal achievement rates, but agents routinely fail to demonstrate grounded spatial reasoning or robust object identification.

Theoretical and Practical Implications

Absence of Emergent Reasoning

Empirical findings indicate a lack of clear emergent reasoning even among the largest evaluated models. The supposed emergence of complex reasoning on static benchmarks does not translate to dynamic, multi-step tasks demanding spatial and temporal abstraction. The authors highlight a consistent "knowing-doing gap," where LLMs, despite being able to recite optimal solution sequences (e.g. the canonical Tower of Hanoi solution), cannot execute successful action trajectories when deployed as agents.

Prompt-based Self-Learning: Limits and Variance

Advanced prompting (reflection, mutation, planning) offers modest and highly variable performance improvements. Variance is particularly pronounced for smaller models, with overfitting to distracting contextual signals and overthinking detrimental to simple reactive decision-making. The Oracle (heuristic evolution) and Planner (lookahead) modules mitigate certain shortcomings but also introduce instability. This instability raises crucial questions regarding the use of purely in-context learning for lifelong adaptation in complex environments.

Benchmarking and Evaluation Methodology

The authors critique conventional static benchmarks (QA pairs, math problems) as insufficient for evaluating general reasoning. Dynamic tasks, which require continuous adaptation, spatial awareness, and sequential decision-making, are a more rigorous testbed and reveal persistent limitations not captured by aggregate accuracy or F1 scores.

Implications for Future Research

Recommendations include (i) integration of external persistent memory for meta-level recall, (ii) incorporation of explicit symbolic or programmatic components for verifiable reasoning, and (iii) inclusion of multimodal grounding to bridge the gap between text-based reasoning and embodied action. Furthermore, denser, task-aligned reward architectures are advocated as a tractable route to enhanced agent adaptation over prompt engineering alone.

Conclusion

The study provides a comprehensive evaluation of LLM agent capabilities in dynamic tasks, revealing the fundamental limitations of in-context adaptation via elaborate prompting. While prompt-based strategies can occasionally elevate performance, particularly for smaller models and in complex environments, they induce high variance and are prone to failure, especially in tasks demanding robust planning and spatial reasoning. The findings call into question the sufficiency of current benchmarks and advocate for a shift toward dynamic, interaction-rich evaluation protocols and agent architectures integrating persistent memory and symbolic reasoning. Methodologically, reward shaping proves more reliably beneficial than further prompt elaboration. Future work should focus on multimodal grounding and memory-augmented architectures to address the persistent knowing-doing and language-embodiment gaps exposed by this analysis.