Heuristic Autoregressive Features

• Heuristic autoregressive features are hand-crafted, domain-specific variables used to capture structural properties and guide sequential predictions in models such as language and tree-structured systems.
• They are extracted via methods like sparse autoencoding and rule-based assignments, offering interpretable cues and measurable impacts on prediction performance.
• Empirical studies demonstrate their role in bias mitigation, improved variable selection, and enhanced generalization across language, phylogenetic, and time series models.

Heuristic autoregressive features are domain- or model-specific variables, often hand-engineered or algorithmically extracted, that capture meaningful structure in the input data or intermediate representations for use within autoregressive models. Their roles span next-token prediction in LLMs, structured probability assignment on combinatorial objects such as trees, and temporal-dependency selection in time series. Heuristic features may be syntactic, semantic, or structural, or may encode superficial cues. Their design and identification have deep implications for interpretability, bias, generalization, and performance. The following sections detail representative formulations, extraction methods, taxonomy, limitations, and research frontiers in heuristic autoregressive features across key machine learning settings.

1. Definition and Formalization

Heuristic autoregressive features are predefined or automatically extracted variables employed by autoregressive models to inform conditional predictions at each step. These features are termed "heuristic" when their design is not induced by end-to-end gradient-based training but instead rests on domain knowledge, shallow statistical indicators, or simple rules, as opposed to fully learned distributed representations.

In the context of neural LLMs, features can be extracted by fitting a sparse autoencoder (SAE) over hidden state activations $x\in\mathbb{R}^n$ at a given layer, yielding sparse feature activations $f\in\mathbb{R}^k$ via $f = \mathrm{ReLU}(W_e(x-b_d) + b_e)$, with $W_e$ and $b_e$ learned to reconstruct $x$ with an $\ell_1$ penalty encouraging most entries of $f$ to be exactly zero (Hanna et al., 2024). In classic probabilistic models over combinatorial structures (e.g., phylogenetic trees), hand-designed features such as "subsplit assignments" are used to index conditional probability tables (CPTs) that define the generative process over objects (Xie et al., 2023).

2. Taxonomy of Heuristic Autoregressive Features

Feature types depend on the domain and modeling objective, but a common organizing axis in neural sequence and structure models distinguishes:

• Syntactic or Structural Features: Empirical linguistic or domain-structural functions, such as detectors for grammatical subjects/objects (language modeling) or bipartition assignments in trees (phylogenetics).
• Shallow Heuristic Features: Surface cues or local patterns with tenuous high-level interpretability, e.g., function-word token positions or punctuation proximity in text; simple assignment frequencies or parent-child split pairs in structured discrete objects.

Representative shallow heuristic features identified in language modeling include:

Feature (Layer/Index)	Heuristic Description	Indirect Effect (IE, mean)	Δp(GP) when ablated
0/8234	"the" token detector	0.13	–0.02
4/14907	clause-end punctuation cue	0.21	+0.05
4/8505	noun-in-PP cue	0.19	–0.07

(Hanna et al., 2024)

In probabilistic modeling over trees, key hand-engineered features include:

• Subsplit assignments: Ordered bipartitions of clades encoding local topology.
• Subsplit support parameters: Scalar log-weights for both subsplits and their parent-child pairs.

(Xie et al., 2023)

3. Methods for Extraction and Application

Autoregressive models may utilize heuristic features at multiple levels:

• Neural LLMs: Applying sparse autoencoders to internal residual streams, one obtains a set of features $f_i(x)$ with interpretable mapping to either syntactic or heuristic functions. Attribution patching and integrated gradients allow decomposition of predictive logits as a sum over these features, elucidating their contribution at each step (Hanna et al., 2024).
• Tree-structured Models: Models like the Subsplit Bayesian Network (SBN) build the overall probability for a tree as a product of CPTs indexed by discrete subsplit features from pre-sampled trees. The CPTs are parameterized by softmax-normalized log-weights per observed split or split pair (Xie et al., 2023).
• Time Series with Joint Feature and Lag Selection: Hierarchical Bayesian models with spike-and-slab priors deploy binary inclusion indicators for covariates and autoregressive lags. The associated features correspond to selected variables and temporal lags, with inference performed via a two-stage MCMC algorithm separating indicator sampling from parameter updates (Manna et al., 12 Aug 2025).

4. Limitations and Interaction with Model Expressivity

Hand-engineered heuristic features present the following challenges:

• Coverage and Generalization: In models reliant on support extracted from pre-sampled data (e.g., sets of observed splits in trees), any unseen configuration receives zero probability, limiting coverage and generalization. Increasing sample size mitigates but does not eliminate this, and can result in computational overhead due to combinatorial explosion in CPT size (Xie et al., 2023).
• Bias and Spurious Correlations: Heuristics based on local patterns, such as function word or punctuation detectors in LLMs, can bias model predictions, especially in the absence of sufficient hierarchical signal. These biases manifest as premature prediction preferences in ambiguous syntactic contexts, later overridden if global structural features come online (Hanna et al., 2024).
• Parameter Efficiency: Maintaining explicit tables for all possible feature combinations (e.g., parent-child splits) quickly becomes infeasible for large, high-dimensional objects, necessitating regularization or pruning (Xie et al., 2023).

A plausible implication is that feature design must balance interpretability and structural faithfulness against efficiency and expressive coverage.

5. Empirical Findings and Case Studies

Notable empirical findings across domains include:

• LLMs: Both syntactic and heuristic features are simultaneously active in autoregressive transformer LLMs. Shallow heuristics, such as function word detectors, can have measurable effects on next-token predictions (e.g., IE≈0.13 for "the" detector), biasing toward certain interpretations before structural information is available. However, high-level syntactic detectors subsequently may override these biases (Hanna et al., 2024).
• Phylogenetic Inference: SBNs indexed by subsplit-based features suffer from "zero-support" for unseen splits and require pre-sampled tree libraries. By contrast, models like ARTree that replace heuristic features with learned GNN embeddings eliminate zero-support and generalize over continuous topology space (Xie et al., 2023).
• Time Series Variable Selection: Hierarchical spike-and-slab models jointly select covariate features and autoregressive lags, yielding high true positive and negative rates for selection (TPR≈0.92–0.98, TNR≈0.99), reduced mean squared prediction error, and robust performance even when candidate predictor dimension grows nearly exponentially in sample size (Manna et al., 12 Aug 2025).

Empirical findings confirm that both the inclusion and mitigation of heuristic features can markedly affect prediction quality, interpretability, and bias.

6. Mitigation Strategies and Future Directions

Research on mitigation and advancement of heuristic autoregressive feature usage highlights several strategies:

• Mechanistic Feature Analysis: Distinguishing between genuine hierarchical computation and shallow lexical heuristics using sparse autoencoding and attribution methods (Hanna et al., 2024).
• Architectural Reforms: Employing models with learned, context-sensitive embeddings (e.g., GNNs), which obviate the need for explicit hand-crafted features, offering full support and improved expressivity (Xie et al., 2023).
• Data Augmentation and Regularization: Breaking spurious lexical correlations through augmented data and regularizing against reliance on early-layer word detectors (Hanna et al., 2024).
• Model Design: Architectural incentives, such as gating or deep feature reuse, to encourage long-range feature utilization over local heuristics (Hanna et al., 2024).

A plausible implication is that as fully learned representations become tractable and data-efficient, reliance on fixed heuristic features may further diminish, especially in complex structured domains.

7. Interpretability and Model Selection

Heuristic features facilitate model interpretability by linking predictions to interpretable cues. In Bayesian time series methods, separation of variable and lag inclusion facilitates direct interpretation of model structure and promotes selection consistency under broad conditions (Manna et al., 12 Aug 2025). However, models that overfit to shallow heuristics can exhibit misleading or unstable attributions. The integration of mechanistic feature analysis and structured regularization is converging as a standard for robust interpretability studies.

In summary, heuristic autoregressive features remain central to both the analysis and construction of autoregressive models, though the trend is toward automated, expressive, and context-aware representations, accompanied by tools for causal feature attribution and mitigations for spurious heuristic reliance.