FAMOSE: A ReAct Approach to Automated Feature Discovery

Abstract: Feature engineering remains a critical yet challenging bottleneck in machine learning, particularly for tabular data, as identifying optimal features from an exponentially large feature space traditionally demands substantial domain expertise. To address this challenge, we introduce FAMOSE (Feature AugMentation and Optimal Selection agEnt), a novel framework that leverages the ReAct paradigm to autonomously explore, generate, and refine features while integrating feature selection and evaluation tools within an agent architecture. To our knowledge, FAMOSE represents the first application of an agentic ReAct framework to automated feature engineering, especially for both regression and classification tasks. Extensive experiments demonstrate that FAMOSE is at or near the state-of-the-art on classification tasks (especially tasks with more than 10K instances, where ROC-AUC increases 0.23% on average), and achieves the state-of-the-art for regression tasks by reducing RMSE by 2.0% on average, while remaining more robust to errors than other algorithms. We hypothesize that FAMOSE's strong performance is because ReAct allows the LLM context window to record (via iterative feature discovery and evaluation steps) what features did or did not work. This is similar to a few-shot prompt and guides the LLM to invent better, more innovative features. Our work offers evidence that AI agents are remarkably effective in solving problems that require highly inventive solutions, such as feature engineering.

• The paper introduces FAMOSE, an iterative ReAct-based method for automated feature discovery that leverages data-driven feedback loops.
• It achieves significant improvements with a mean ROC-AUC increase of 0.23% for classification and a 2.0% RMSE reduction for regression.
• It employs an mRMR post-selection step to prune redundant features, ensuring robustness, interpretability, and model transferability.

FAMOSE: A ReAct Approach to Automated Feature Discovery

Introduction and Motivation

Feature engineering is a persistent challenge and critical bottleneck in supervised learning, especially for tabular data, where domain-driven intuition and expertise are difficult to automate. While many traditional and LLM-based automated feature engineering (AutoFE) frameworks exist, they are generally constrained by an immense combinatorial feature space and a lack of iterative self-improvement. FAMOSE (Feature AugMentation and Optimal Selection agEnt) addresses these limitations by employing a ReAct-based agentic framework, empowering an LLM to iteratively generate, evaluate, and refine features in a data-driven, feedback-guided loop.

Figure 1: Illustration of the main AutoML paradigms, situating FAMOSE as a ReAct-based iterative feature engineering approach, contrasting classical and prior LLM-based methods by enabling adaptive trial-and-error cycles.

The central technical argument is that FAMOSE’s agent keeps track of past feature proposal outcomes within the LLM context window, enabling a “few-shot” style improvement where successive feature proposals are informed by empirical validation. This tightly-coupled reasoning and acting process closely simulates the workflow of an expert data scientist, and is distinct from prior one-shot LLM feature generation strategies ((2602.17641), Fig. 1).

FAMOSE Architecture and Methodology

FAMOSE implements a structured iteration, where in each round an LLM agent, instrumented with Python code execution and feature evaluation tools, proposes candidate feature transformations leveraging explicit schema metadata and previous evaluation results. Features are validated in held-out data, and only those achieving a statistically-significant improvement in predictive performance (e.g., 1% ROC-AUC or RMSE delta) are retained. This process continues for multiple rounds or until no further improvement is observed.

Figure 2: Example illustrating FAMOSE’s application, where iterative proposal and evaluation of features refines the feature set toward predictive optimality.

After feature proposal, an automated post-hoc validation step ensures that only genuinely performant features are selected, closing the loop on LLM hallucination or noisy self-reporting. Upon convergence, FAMOSE applies a minimal-redundancy, maximal-relevance (mRMR) feature selection strategy to prune the feature set for compactness and eliminate redundant signals.

Figure 3: Schematic of FAMOSE’s agentic workflow: iterative LLM-driven feature proposal, empirical validation, and mRMR post-selection.

Benchmarks are conducted on a broad suite of tabular datasets spanning classification and regression, using candidate LLMs (Sonnet 3.5 V2, Deepseek-R1) within the AWS Bedrock ecosystem.

Empirical Evaluation and Results

FAMOSE is comprehensively benchmarked against classical (AutoFeat, OpenFE) and advanced LLM-based (CAAFE, FeatLLM) feature engineering baselines on both classification (ROC-AUC) and regression (RMSE) metrics.

Classification

On large tabular classification tasks ($n > 10,000$), FAMOSE demonstrates a mean ROC-AUC increase of $0.23\%$ over state-of-the-art competitors. This advantage is most prominent in regimes where classical enumerative methods fail due to exponential blowup in feature proposals or memory constraints.

Regression

For regression, FAMOSE exhibits a mean 2.0% reduction in RMSE, outperforming both OpenFE and AutoFeat, which suffer both from scalability and generalization issues.

Across all tasks, FAMOSE exhibits high robustness: features generated for one model (e.g., XGBoost) transfer well to others (Random Forest, Autogluon), and performance is maintained across both proprietary and open-source LLMs.

Figure 4: Comparative analysis of the number of unique functions used in FAMOSE vs. CAAFE, demonstrating greater breadth in FAMOSE’s engineered transformations.

Figure 5: Function usage frequencies in CAAFE, highlighting a narrow transform profile dominated by a handful of operations.

Figure 6: FAMOSE’s function usage histogram evidences a more balanced and diverse set of transformations, supporting its empirical superiority.

Feature Diversity and Innovation

A core aspect of FAMOSE’s competitive edge arises from the diversity and contextual adaptivity of generated features. Unlike CAAFE, FAMOSE systematically explores a wider set of mathematical transformations, enabling discovery of domain-informed composite features that classical enumerative and one-shot LLM approaches rarely recover.

Figure 7: Proportion of tasks/folds where FAMOSE (vs. CAAFE) successfully contributed novel features, underlining its broader applicability.

Robustness and Ablations

Ablation experiments indicate that the explicit performance improvement goal ($\Delta$ROC-AUC or $\Delta$RMSE) is crucial—removing this leads to premature termination and reduced feature innovation. The addition of mRMR post-selection also plays a critical role in suppressing overfitting and distilling the gains of agentic exploration ((2602.17641), App.).

Interpretability and Theoretical Contributions

In addition to empirical advances, FAMOSE yields human-interpretable explanations for each proposed feature through the ReAct prompting paradigm. This inherently elevates the transparency of the feature engineering process, facilitating both model debugging and downstream trust in model-driven recommendations. The results support the hypothesis that agent-based LLM reasoning can augment classic “few-shot” learning with explicit trial-and-error guidance within automated data science workflows.

Of note, FAMOSE’s iterative data-informed reasoning directly counters the combinatorial and hallucination challenges endemic to earlier LLM and symbolic pipelines. Post-hoc evaluation and error-checking mechanisms further increase robustness and reconcile the limitations of LLM self-assessment.

Implications and Future Directions

FAMOSE marks a significant methodological step toward the full automation of high-value, innovative feature engineering for tabular ML:

• Practical Impact: The platform-agnostic architecture (LLM- and model-robust) makes it viable for production pipelines, scaling to large datasets where classical AutoFE fails.
• Theoretical Impact: Agentic LLMs, when tightly-coupled with empirical feedback, can traverse and innovate within exponentially-sized search spaces (such as feature engineering) without human-in-the-loop.
• Next Steps: Extending FAMOSE for multi-label tasks, incorporating domain-specific retrieval-augmented generation (RAG) to enhance generalization beyond pre-trained knowledge, and adapting the ReAct-agent framework to non-tabular modalities (e.g., structured vision features, sequence data).

Limitations

The main limitations are computational: the agentic interaction sequence incurs substantial token and inference costs, particularly for large LLMs. Performance is sensitive to the capabilities of the underlying LLM and degrades with smaller/less-informative LLMs. Tasks that require background knowledge absent from the LLM’s pretraining corpus could benefit from enhanced retrieval or injection of domain context. Current support is limited to standard classification and regression; coverage for more complex scenarios (multi-label, multi-modal) requires further engineering.

Conclusion

FAMOSE establishes a compelling empirical and theoretical paradigm for automated feature discovery rooted in agentic LLM reasoning under a ReAct protocol. It achieves top-tier performance in classification and regression, demonstrates robustness and transferability across models and LLMs, and significantly broadens the diversity and quality of engineered features. These advances have immediate implications for both industrial AutoML pipelines and the development of next-generation, fully automated machine learning workflows.

(2602.17641)