Robot Prompt Generator

• Robot prompt generator is a system that converts human instructions into robot-executable forms like action code, PDDL, and workflows.
• It uses varied prompt modalities, including natural language, JSON schemas, code snippets, and visual overlays, to capture task structure and constraints.
• It integrates hierarchical planning, validation, and corrections to reliably coordinate decentralized control and sensorimotor tasks.

A robot prompt generator is a prompt-centric mechanism that translates human intent into robot-executable representations such as action code, behavior trees, PDDL problems, affordance masks, diffusion-conditioned policies, or end-to-end workflow artifacts. In current robotics literature, the term spans several distinct but related designs: hierarchical LLM planners that decompose natural-language missions into classical planning problems, in-context visuomotor policies that condition on a single demonstration, prompt-overlaid visual interfaces that specify contact geometry and motion, affordance-grounding pipelines that use VLM/LLM prompting to localize actionable object parts, and orchestration systems that turn one prompt into reproduction, evaluation, fine-tuning, or deployment workflows (Kawabe et al., 25 Feb 2026, Patel et al., 29 Jun 2026, Li et al., 4 May 2025, Tong et al., 2024, Jin et al., 12 May 2026). This suggests that the topic is best understood not as a single algorithmic family but as an interface pattern: prompts act as explicit carriers of task structure, environmental constraints, embodiment assumptions, and recovery logic.

1. Prompt modalities and representational scope

The prompt in a robot prompt generator is not restricted to free-form natural language. Across the literature, prompts appear as natural-language instructions, JSON schemas, Python-like program headers, template variables embedded in scripts, observation–action histories, behavior-tree fragments, single demonstrations, human demonstration videos, and key-frame visual overlays (Zhen et al., 2023, Singh et al., 2022, Obinata et al., 2024, Wang et al., 2023).

Prompt form	Representation in the system	Representative papers
Natural-language instruction	Task decomposition, planning, or workflow dispatch	(Kawabe et al., 25 Feb 2026, Pfitzer et al., 29 Sep 2025, Jin et al., 12 May 2026)
JSON / schema-constrained text	Directed-graph task expansion and parameter filling	(Zhen et al., 2023)
Pythonic program prompt	Code completion over robot primitives and objects	(Singh et al., 2022, Jin et al., 2023)
Template variables	Runtime slot filling for on-site uncertainty	(Obinata et al., 2024)
Demonstration as prompt	In-context action generation	(Patel et al., 29 Jun 2026, Zhu et al., 27 May 2025)
Visual prompt overlays	Contact pose and post-contact motion specification	(Li et al., 4 May 2025)
Observation–action history	Few-shot low-level feedback control	(Wang et al., 2023)

This breadth matters because different prompt forms encode different invariants. JSON schemas and program headers constrain syntax and admissible operators; demonstration prompts carry temporal and embodiment information; visual prompts expose contact geometry; template variables externalize uncertainty that cannot be resolved at planning time. A common misconception is that robot prompting is synonymous with “natural language to action.” The literature instead shows a heterogeneous prompt space in which text is only one carrier among several (Patel et al., 29 Jun 2026, Li et al., 4 May 2025, Wang et al., 2023).

In several systems, prompts are deliberately structured to reduce hallucination. Think_Net_Prompt uses JSON input and output schemas with fields such as "possible_subtasks", "subtask_parameters", and "possible_subtask_sequences" so that the LLM produces a path in a directed graph rather than unconstrained prose (Zhen et al., 2023). ProgPrompt, introduced by Singh et al., converts situated planning into a code-completion problem by enumerating imports for available primitives and an explicit objects = [...] list, thereby reducing the chance that the model invents unsupported actions or arguments (Singh et al., 2022). Cao and Lee’s behavior-tree formulation similarly constrains generation through a Phase–Step textual skeleton that mirrors a 3-layer behavior tree (Cao et al., 2023).

2. Formal substrates for robot prompt generation

Robot prompt generators differ most sharply in the formal substrate into which prompts are compiled. One major line targets symbolic planning. In the hierarchical multi-agent framework of “Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning,” the upper layer decomposes the natural-language instruction $u$, intermediate layers refine subtasks, and the leaf agents generate PDDL domain and problem specifications for a classical planner (Kawabe et al., 25 Feb 2026). Leaf agents output a domain file with operators such as move, pickup, and putdown, together with a problem file containing initial state $I$ and goal $G$; Fast Downward with the LAMA heuristic is then used as the planner, and a validator checks whether the generated plan reaches the goal (Kawabe et al., 25 Feb 2026).

A second line targets graph-structured task knowledge. Think_Net_Prompt models domain knowledge as a directed graph $G=(V,E)$, where task words are nodes, edges denote admissible subtask expansion, and a second relation $E_{\mathrm{seq}}$ constrains immediate succession in a valid subtask sequence (Zhen et al., 2023). The LLM operates over prompt fields that expose the graph’s local structure, then recursively expands non-terminal nodes into a task tree. Because the planning problem is factorized into $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$ and $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$, semantic decomposition is separated from machine-specific allocation (Zhen et al., 2023).

A third line uses executable control programs as the substrate. ProgPrompt encodes the action set $A$ as importable Python functions and the object set $O$ as an explicit scene list, then asks the LLM to complete the body of a new function corresponding to the requested task (Singh et al., 2022). RobotGPT also elicits executable Python routines, but treats LLM-produced code as demonstrations for subsequent policy learning rather than as a final controller (Jin et al., 2023). In both cases, code becomes the intermediate representation connecting prompt interpretation and environment execution.

Behavior trees provide another substrate. Cao and Lee’s Phase–Step prompt design maps each “Phase” to a second-layer Sequence node and each “Step” to a third-layer Action node, enabling the LLM to synthesize hierarchical task structure without a predefined fixed set of primitive tasks (Cao et al., 2023). The source behavior tree is retrieved by embedding similarity, and non-primitive steps are recursively expanded if their verbs fall outside a permitted verb list (Cao et al., 2023).

For decentralized multi-robot teams, Pfitzer et al. formalize a task as a DFA $M=(\mathcal H,\Sigma,\delta,h_0,\mathcal F)$, then distill LM-generated subtask logic into an RNN hidden state and condition a GNN controller on that hidden state and a language embedding (Pfitzer et al., 29 Sep 2025). The prompt is therefore compiled first into an automaton, then into a lightweight recurrent representation that supports decentralized, real-time control.

3. Hierarchical decomposition and runtime orchestration

Many robot prompt generators are explicitly hierarchical. The hierarchical LLM planner organizes logical agents into $I$0 layers, $I$1. Layer 0 acts as a global planner, intermediate layers refine and assign subtasks, and layer $I$2 translates those subtasks into PDDL specifications (Kawabe et al., 25 Feb 2026). The hierarchy is not merely organizational: the ablation study reports that removing it reduces overall success rate to $I$3, a drop of $I$4 percentage points relative to the full system (Kawabe et al., 25 Feb 2026). In that framework, meta-prompts $I$5 are shared across agents in the same layer, so prompt updates diffuse laterally among homogeneous specialists rather than remaining isolated (Kawabe et al., 25 Feb 2026).

Think_Net_Prompt also decomposes hierarchically, though in a graph-and-tree idiom rather than a layer-and-agent idiom. A Manager parses the free-form instruction into top-level task words, a Planner recursively expands each word into a subtree, and an Allocator binds executable leaves to physical robots or tools based on resource availability (Zhen et al., 2023). Here, the key design choice is decoupling: abstract semantics are resolved before embodiment-specific binding.

The remote life-support interface of “Remote Life Support Robot Interface System for Global Task Planning and Local Action Expansion Using Foundation Models” introduces a distinct form of hierarchy: global planning with unresolved template variables, followed by local action expansion at execution time (Obinata et al., 2024). A Global Task Planner outputs a robot action sequence that preserves variables such as @food@ or @drink@; when execution reaches an action containing such a variable, a Local Action Expander calls a Prompt Generator to build a VLM prompt, collects options from the scene, and presents them to the user via a feedback interface (Obinata et al., 2024). This design externalizes predictable uncertainty instead of forcing the initial prompt to resolve on-site details prematurely.

Nautilus extends the idea of prompt-driven hierarchy from task planning to robotics research workflows. A single prompt such as “Evaluate policy A with benchmark B” is parsed into an internal command, dispatched through a Guide layer to policy-generator, benchmark-generator, and optionally robot-generator subagents, then realized in chambered containers that obey typed contracts and communicate through a uniform WebSocket transport (Jin et al., 12 May 2026). In this setting, a robot prompt generator is not limited to generating a task plan; it can generate the entire experimental or deployment scaffold around a policy.

4. Validation, correction, and prompt optimization

A central theme in the literature is that prompt generation is coupled to verification. The hybrid PDDL framework explicitly treats the classical planner and validator as sources of failure signals. When a leaf sub-plan fails because the planner returns “no plan” or the validator detects an unmet precondition, the failure is reported back to the responsible agent, and a TextGrad-inspired update modifies either the local prompt $I$6 or an ancestor prompt (Kawabe et al., 25 Feb 2026). The update has an inner agent-level step and an outer meta-prompt step, aggregated across all agents in the same layer, yielding a two-level adaptation scheme inspired by MAML but operating through discrete text edits (Kawabe et al., 25 Feb 2026).

RobotGPT adopts a different correction loop. ChatGPT-generated code is executed step by step in PyBullet; syntax and runtime errors trigger a Code-Error-Catching module, task failure invokes an LLM-generated is_task_success() evaluator, and a corrector bot produces hints about logical errors such as wrong grasp order or incorrect placement offsets (Jin et al., 2023). The paper reports that $I$7 of buggy generations can be fixed in one or two such iterations (Jin et al., 2023). The resulting trajectories are then used to train a conventional agent via SDQfD with an Equivariant ASR backbone, replacing direct LLM execution with a learned policy (Jin et al., 2023).

PRAG formalizes validation as a two-stage acceptance test over procedurally generated tasks. Symbolic validation checks logical and operational consistency under action preconditions and postconditions, while physical validation samples scene states from predicate-defined volumes and rejects sequences that fail collision, reachability, or inverse-kinematics checks (Vavrecka et al., 12 Jul 2025). This is a prompt generator in a broader sense: it generates solvable long-horizon tasks rather than action sequences for a single task instance. The emphasis is again on pruning invalid combinations before they reach learning or execution.

Prompt quality can also be optimized before deployment. “Prompt Selection and Augmentation for Few Examples Code Generation in LLM and its Application in Robotics Control” expands a seed example library through a multi-stage augmentation scheme and then scores candidates by relevance, concept similarity, complexity, answer similarity, redundancy, and diversity (Wu et al., 2024). The selected examples are inserted into a Program-of-Thought prompt, and the paper reports a decrease of over $I$8 in the number of examples used in tabletop robotics, along with a $I$9 increase in successful task completions over Code-as-Policies (Wu et al., 2024).

Nautilus generalizes validation to an infrastructure level. Its runtime Sensors include a pre-action filter, render-time auditor, interface verification, a typed-contract spec-comparison gate, and a tiered smoke ladder comprising reset() and random-step tests, a 100-step rollout, and training-time checks such as finite and decreasing loss (Jin et al., 12 May 2026). In all of these systems, prompting is not treated as sufficient on its own; prompt generation is embedded in a monitored loop with explicit rejection and repair.

5. Prompt-conditioned sensorimotor control

A substantial branch of the field moves beyond symbolic plans and uses prompts to condition low-level or mid-level control policies. Behavior Prompting Policy formulates the problem as in-context adaptation from a single demonstration $G$0, with action generation given by $G$1 and no gradient updates at test time (Patel et al., 29 Jun 2026). The architecture has three stages: prompt chunking and attention pooling, prompt encoding by Transformer-decoder cross-attention, and action decoding by a diffusion U-Net with FiLM conditioning (Patel et al., 29 Jun 2026). The training objective is end-to-end behavior cloning, while prompt tokens can be cached once per rollout to reduce inference cost (Patel et al., 29 Jun 2026).

“Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt” uses a two-stage design. Stage 1 trains a Stable Video Diffusion-based cross-embodiment prediction model over human, gripper, and dexterous-hand videos; the CLS token of the DiT bottleneck becomes a task embedding $G$2 (Zhu et al., 27 May 2025). Stage 2 conditions a diffusion policy on both observation features and $G$3, while adding NT-Xent, prototypical cross-entropy, and prototype-level Siamese metric losses to align human and robot behaviors in a shared action space $G$4 (Zhu et al., 27 May 2025). The resulting policy can take a human demonstration video as a prompt and perform new tasks without any new teleoperation data and model finetuning (Zhu et al., 27 May 2025).

CrayonRobo makes the prompt explicitly geometric. Each key frame contains an RGB image with colored overlays: a blue dot for the desired 2D contact point, red and green lines for the 2D projections of the end-effector $G$5- and $G$6-axes, and an optional yellow line for the post-contact movement direction (Li et al., 4 May 2025). The backbone combines a CLIP visual encoder, a LLaMA model with LoRA, multimodal projection, and a policy head that predicts discretized 3D contact geometry and motion in textual form, using cross-entropy, orthogonality, and projection losses (Li et al., 4 May 2025). Sequential execution over a series of prompt-overlaid key frames decomposes long-horizon tasks into atomic contact-centric subgoals (Li et al., 4 May 2025).

Prompting can also target low-level locomotion. “Prompt a Robot to Walk with LLMs” conditions GPT-4 on a description prompt plus a rolling history $G$7 of observation–action pairs, then asks it to autoregressively output normalized joint targets at $G$8 while a PD controller tracks them at $G$9 (Wang et al., 2023). The prompt includes task description, input/output schema, joint order mapping, control-pipeline summary, and additional notes about normalization (Wang et al., 2023). Here the prompt is effectively a transient system-identification and control-context packet.

Perception-oriented prompting forms another subfamily. OVAL-Prompt uses a VLM prompt of the form “Please segment the <part_name> of the <object_name> in the image,” then queries an LLM to select which object and which part afford the target task; if the segmentation is empty or nonsensical, the system reprompts for alternate part names (Tong et al., 2024). The resulting part mask is converted into a 3D waypoint through centroid extraction, depth lookup, back-projection, and transformation into the robot base frame (Tong et al., 2024).

6. Empirical profile, limitations, and common misconceptions

Empirical results show that robot prompt generators can be effective, but the gains are highly contingent on representational structure and validation. On MAT-THOR, the hierarchical PDDL-integrated planner attains success rates of $G=(V,E)$0 on compound tasks, $G=(V,E)$1 on complex tasks, and $G=(V,E)$2 on vague tasks, improving over LaMMA-P by $G=(V,E)$3, $G=(V,E)$4, and $G=(V,E)$5 percentage points respectively; the ablation study attributes roughly $G=(V,E)$6, $G=(V,E)$7, and $G=(V,E)$8 percentage points to hierarchy, prompt optimization, and meta-prompt sharing (Kawabe et al., 25 Feb 2026). On DrawAnything-Sim, BPP achieves approximately $G=(V,E)$9 success on unseen drawings versus approximately $E_{\mathrm{seq}}$0 for Goal-Image and approximately $E_{\mathrm{seq}}$1 for ICRT; on DrawAnything-Real it achieves approximately $E_{\mathrm{seq}}$2 success versus approximately $E_{\mathrm{seq}}$3 for Goal-Image (Patel et al., 29 Jun 2026). On LIBERO-Gen, BPP reaches approximately $E_{\mathrm{seq}}$4 in Combination and approximately $E_{\mathrm{seq}}$5 in Chain, compared with approximately $E_{\mathrm{seq}}$6 and approximately $E_{\mathrm{seq}}$7 for language-only conditioning (Patel et al., 29 Jun 2026).

Prompt-driven geometric conditioning also yields strong results. In simulation, CrayonRobo reports average success $E_{\mathrm{seq}}$8 on seen and $E_{\mathrm{seq}}$9 on unseen categories with a suction gripper, while auto-prompts achieve $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$0, and prompt ablations show monotonic gains from position-only prompts to the full prompt set (Li et al., 4 May 2025). In the real world, tasks such as opening a trash can, opening a microwave, and wiping a table are reported in the $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$1 to $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$2 range overall, with sketch-drawn prompts generally outperforming automatically generated ones (Li et al., 4 May 2025). RobotGPT improves average task success from $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$3 for direct ChatGPT code generation to $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$4 for the trained agent, with simulation results of $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$5, $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$6, and $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$7 on easy, medium, and hard tasks respectively (Jin et al., 2023). For locomotion, the LLM walking controller achieves normalized walking time approximately $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$8 and success rate approximately $$f_1:\text{Instruction}\times\text{DomainKnowledge}\to\text{TaskTree}$$9 on flat-ground A1 trials with history length $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$0 (Wang et al., 2023).

Prompt generation has also been evaluated as task-bank synthesis. PRAG reports symbolic generation of sequences up to length $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$1, a physical validation success rate of approximately $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$2 for $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$3- to $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$4-step tasks, and generation of approximately $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$5 million symbolically valid sequences of length $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$6 in approximately $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$7 hours on a $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$8-core CPU plus $$f_2:\text{TaskTree}\times\text{RobotState}\to\text{Binding}$$9GPU cluster, of which approximately $A$0 million pass physics (Vavrecka et al., 12 Jul 2025). In the remote life-support interface, two real-world tasks on a PR2 are reported as $A$1 successful with average user-intervention time approximately $A$2, but the “buy” task also shows a VLM hallucination rate of $A$3, since $A$4 extracted options were spurious (Obinata et al., 2024). These results illustrate a recurrent pattern: prompt generation is often useful precisely when the system includes an independent mechanism for constraint checking, user confirmation, or downstream policy stabilization.

Several limitations recur across papers. LLM planners can hallucinate infeasible actions or fail on ambiguous long-horizon missions (Kawabe et al., 25 Feb 2026). Think_Net_Prompt reports limited complexity of task logic handling, ambiguity in part counts and precise assembly locations, and failures when a part is referenced by multiple synonyms (Zhen et al., 2023). Behavior prompting depends strongly on task diversity; under low-diversity training, prompt lookup is weaker, and adapting to entirely new action primitives remains challenging (Patel et al., 29 Jun 2026). The walking controller is sensitive to prompt wording and token-budget limits, and no hardware tests are reported (Wang et al., 2023). The multi-robot DFA–RNN–GNN framework can fail on out-of-distribution tasks not represented in the LM-generated automata bank, or when sparse atomic propositions never fire (Pfitzer et al., 29 Sep 2025). The remote life-support system still relies on human camera orientation and user selection in the loop (Obinata et al., 2024).

The literature also corrects three common misconceptions. First, robot prompt generation is not exclusively about text; prompts can be demonstrations, videos, overlaid geometry, or history buffers (Patel et al., 29 Jun 2026, Zhu et al., 27 May 2025, Li et al., 4 May 2025, Wang et al., 2023). Second, it is not necessarily an LLM-only paradigm; classical planners, validators, PD controllers, RL/IL policies, GNN controllers, and typed-contract middleware are often indispensable (Kawabe et al., 25 Feb 2026, Jin et al., 2023, Pfitzer et al., 29 Sep 2025, Jin et al., 12 May 2026). Third, prompt-based robotics is not uniformly training-free: some systems rely only on prompt engineering, but others require substantial offline learning, including diffusion-policy training, cross-embodiment video pretraining, or automata distillation (Zhu et al., 27 May 2025, Patel et al., 29 Jun 2026, Pfitzer et al., 29 Sep 2025).

Taken together, the current literature portrays the robot prompt generator as a unifying abstraction for prompt-mediated grounding rather than as a fixed architecture. Its strongest instantiations are hybrid systems in which prompts expose structure, external modules enforce validity, and learned or symbolic back ends convert that structured prompt into plans, policies, or workflows that respect embodiment and environment constraints.