BT-APE: A Computationally Light Backtracking Approach to Automatic Prompt Engineering for Requirements Classification

Abstract: LLMs are increasingly applied to requirements engineering (RE) tasks, yet the prompts guiding them are typically designed manually through trial and error, yielding inconsistent and suboptimal results. Automated prompt construction remains largely unexplored in RE, leaving its effectiveness unclear. To address this, we propose a lightweight Automatic Prompt Engineering approach, Backtracking APE (BT-APE), and apply it to requirements classification. We frame prompt design as an optimization problem, iteratively refining prompts via LLM-generated candidates, backtracking search, and dynamic example selection. Evaluating BT-APE on three benchmark datasets with five instruction-tuned LLMs, we compare it against four classical prompting baselines (zero-shot, few-shot, chain-of-thought, CoT+few-shot) and a state-of-the-art but resource-intensive APE baseline (PE2). BT-APE and PE2 achieve nearly identical accuracy, both substantially outperforming the classical baselines with large effect sizes; however, BT-APE imposes a far lighter computational footprint, consuming roughly 72% fewer input tokens and 66% less wall-clock time at equivalent accuracy, making it better suited to resource-constrained deployment. Our contributions are threefold: (i) a lightweight APE framework with an open interactive tool and replication package; (ii) the first systematic comparison of APE against classical prompting for requirements classification; and (iii) insights into how class definitions and prompt evolution affect performance.

• The paper introduces BT-APE, a backtracking-enhanced method that iteratively refines prompts for effective requirements classification.
• It employs a balanced, dynamic feedback strategy with majority voting to stabilize evaluations and reduce resource overhead.
• Experimental results demonstrate state-of-the-art performance with significant reductions in token usage and processing time.

BT-APE: A Lightweight Backtracking-Enhanced Automatic Prompt Engineering Approach for Requirements Classification

Introduction and Motivation

Prompt engineering is the predominant mechanism for tailoring LLMs to specialized domains, particularly in settings where annotated data is scarce and model retraining is infeasible. Requirements Engineering (RE)—with key tasks such as requirements classification—exemplifies this low-resource, high-stakes regime, where manually crafted prompts remain the de facto standard, yet lead to unsystematic, suboptimal, and irreproducible results. Existing literature has established the efficacy of LLMs on RE tasks using basic prompting strategies but leaves prompt optimization largely underexplored.

The paper "BT-APE: A Computationally Light Backtracking Approach to Automatic Prompt Engineering for Requirements Classification" (2607.00427) addresses this gap with BT-APE, a resource-efficient, mutation-oriented Automatic Prompt Engineering (APE) method framed as a bounded backtracking meta-search over prompt space. BT-APE systematically revises prompts by exploiting dynamic, balanced feedback, while mitigating evaluation noise and computational overhead—targeting deployment in resource-constrained, air-gapped, or privacy-sensitive engineering environments.

Figure 1: Overview of the BT-APE workflow applied to requirements classification, showing the sequential, iterative backtracking optimization of prompt definitions conditioned on LLM feedback.

Problem Formulation and Methodology

BT-APE operationalizes prompt engineering as a discrete optimization problem over prompt space. Given a dataset $D = (X, Y)$ and a target evaluation metric $f$, the objective is to find a prompt $p^*$ maximizing $f(\mathcal{M}(X, p), Y)$, where $\mathcal{M}$ is a frozen, instruction-tuned LLM.

The framework proceeds in three phases:

1. Initialization: The labeled data is partitioned into an example pool, a validation set, and a held-out test set. The initial prompt is scored on the validation set.
2. Iterative Refinement: At each step, the LLM proposes an improved prompt based on the current prompt and a balanced batch of four examples (covering both classes and error/correct cases), balancing exploration and exploitation. The candidate is evaluated via majority voting, and a ranked list of prompts is maintained to manage bounded backtracking if progress stalls.
3. Final Evaluation: Only the best prompt from the search trajectory is evaluated on the hold-out set, ensuring an unbiased final metric.

Distinctive aspects of BT-APE include:

• Explicit bounded backtracking: Only after $X$ consecutive non-improving iterations is the search trajectory reset to a previously high-ranking prompt, mitigating both myopic horizon effects and local optima.
• Balanced, dynamic example conditioning: Feedback batches are forcibly stratified, sidestepping class imbalance and error asymmetry pitfalls.
• Stable, low-variance evaluation: Each candidate prompt is scored with 3-run majority voting, reducing gradient noise and search instability.

  Figure 2: BT-APE optimization process, highlighting candidate prompt generation, validation set evaluation, majority-voting smoothing, and controlled backtracking.

Experimental Evaluation

BT-APE was evaluated across fifteen (dataset, LLM) pairs, spanning three canonical RE datasets (PROMISE, PROMISE-Refined, and SecReq) and five 7–8B parameter, instruction-tuned open LLMs (Qwen2-7B, Falcon3-7B, Granite-3.2-8B, Ministral-8B, LLaMA-3-8B). BT-APE was benchmarked against classical prompt engineering (zero-shot, few-shot, chain-of-thought, CoT+few-shot) and a state-of-the-art APE baseline (PE2 [ye2024promptengineeringpromptengineer])—chosen for its structural similarity and comparable compute budget.

Figure 3: Average F1 of five instruction-tuned LLMs across all prompting strategies and datasets; the big star indicates the top strategy per model.

Empirical results demonstrate:

• Superior performance: BT-APE and PE2 are statistically tied, each yielding substantially higher weighted F1 (wF1) than all classical baselines, with mean wF1 improvements ranging from +0.11 to +0.14 across baselines—effect sizes classified as large.
• Computational efficiency: BT-APE attains virtually identical accuracy to PE2 but reduces cumulative input tokens by 72% and wall-clock time by 66% (see Appendix), achieving each percentage point of wF1 at only 27–34% of PE2's resource budget.

Robustness to initialization was tested via informed (literature-derived) versus uninformed (minimal) class definitions, showing that BT-APE's iterative optimization neutralizes any initial advantage, converging to comparable final accuracy independent of seed prompt quality.

Analysis of Prompt Characteristics

Prompt evolution under BT-APE was subjected to linguistic profiling. Linear mixed-effects models reveal that high-performing prompts are:

• Concise: Shorter in both sentence and word count.
• Action-oriented: Higher verb density.
• Structurally explicit: More punctuation/organizational markers.
• Semantically focused: Lower lexical diversity, reduced syntactic complexity.

  Figure 4: Example BT-APE prompt, showing a fixed task-framing section plus an optimizable class definition region (highlighted), which is dynamically rewritten.

Semantic drift—large iterative changes in meaning—was positively correlated with performance gains, indicating that non-trivial prompt mutations, not merely surface rewordings, drive effectiveness.

Tooling and Reproducibility

To bridge research and practice, the authors provide an interactive GUI-based tool that enables practitioners to upload custom datasets, select LLM backends, configure all BT-APE and baseline strategies, and visualize prompt optimization trajectories, thus operationalizing the BT-APE framework without code.

Figure 5: Tool interface for dataset configuration, class distribution display, LLM selection, and hyper-parameter control.

Implications and Future Directions

BT-APE closes the gap between manual prompt engineering and more costly, brute-force APE approaches, offering theoretically sound and practically viable prompt optimization for RE contexts where training data is sparse and compute is limited. Its transferability to other classification tasks and domains remains promising, given its exclusive reliance on prompt text mutation and model feedback.

For theory, the results suggest that the main bottleneck in RE LLM application is not model architecture, but prompt optimization and task difficulty. The stability and efficiency of BT-APE further suggest that future APE work could benefit from more principled search control and error-driven, balanced batch conditioning.

Practically, BT-APE's resource profile supports RE deployments in privacy-sensitive corporate environments, where cloud APIs or massive retraining are infeasible. The insight that domain-knowledge in initialization is unnecessary under automated refinement also lowers the skill barrier for successful LLM adoption.

Conclusion

BT-APE demonstrates that single-trajectory, backtracking-enhanced APE methods can reproducibly yield state-of-the-art prompt optimization for requirements classification in LLMs, outperforming all classical prompt baselines and matching more resource-intensive methods at a fraction of the computational cost. Its robustness to prompt initialization and strong performance across models and tasks positions it as a default, efficient strategy for RE practitioners and a conceptual foundation for further research in domain-specialized prompt engineering.

Figure 6: Jaccard similarity analysis between LLM output and dataset ground truth, providing an empirical check against data leakage throughout experimental runs.