Foundation CAN Model: Pretrained Neural Architectures

• Foundation CAN Model is a class of pretrained neural architectures designed to process heterogeneous CAN telemetry data through unified tokenization and self-supervised learning.
• It adapts a BERT-style Transformer with custom discretization of continuous signals and encoding of discrete variables, enabling effective task-agnostic adaptation.
• The approach integrates causal abstraction and data-centric learning to support both robust signal representation and emerging intervention-aware reasoning frameworks.

A Foundation CAN Model denotes a class of pretrained neural architectures that bring the foundation model paradigm to the domain of Controller Area Network (CAN) data—multivariate, mixed-type vehicular telemetry streams essential for automotive, insurance, and industrial applications. These models employ large-scale unsupervised pretraining with a unified tokenization scheme over decoded CAN signals, enabling robust cross-task transfer and generalization analogous to advances in NLP and computer vision. The approach further extends to hybrid frameworks where causal abstraction and data-centric learning are integrated, paving a path toward models capable of both flexible representation and principled causal reasoning.

1. Formalization, Motivation, and Theoretical Underpinnings

Foundation CAN Models are grounded in the foundation model paradigm characterized by large-scale, self-supervised pretraining over heterogeneous unlabeled data and subsequent task-agnostic adaptation via lightweight output heads (Narayan et al., 2022, Esashi et al., 31 Jan 2026). In the context of CAN data, the model must operate over streams where each time-step comprises both continuous (e.g., velocity, acceleration) and discrete (e.g., gear, indicator) signals as well as symbolic context (e.g., trip ID).

The theoretical basis for extending foundation modeling to CAN data draws on the probabilistic view:

$$p(y \mid x; \theta) = \prod_{t=1}^{|y|} p(y_t \mid x, y_{<t}; \theta)$$

where $x$ is the tokenized signal sequence and $y$ the downstream prediction (Narayan et al., 2022, Esashi et al., 31 Jan 2026). Inspired by findings in causality-aware foundation models, there is increasing motivation to integrate structural causal representations ($\mathcal{M} = \langle U, V, F, P_U \rangle$) and do-calculus ($P(Y\mid do(X))$) (Willig et al., 2022), suggesting a future paradigm in which Foundation CAN Models not only generalize across tasks but also explicitly encode interventional and counterfactual dependencies.

2. Unified Tokenization and Data Representation

Foundation CAN Models require a specialized tokenization protocol to accommodate the structural and statistical idiosyncrasies of CAN telemetry. The process involves:

• Continuous Signal Discretization: All values are normalized to $[0,1]$ via empirical min–max scaling:

$$\tilde x_t = \frac{x_t - \min(x)}{\max(x) - \min(x)}$$

Subsequently, each feature $i$ is discretized into $B_i$ uniform bins determined by temporal dynamics:

$$\Delta_i = \mathbb{E}[|x_{t+1} - x_t|], \quad r_i = \frac{\Delta_i}{\max(x_i)-\min(x_i)}$$

Features with low $r_i$ receive finer binning.

• Discrete Variable Encoding: Enums like gear states are mapped one-to-one; context markers use meta-tokens such as <NEW_CAR>, <NEW_TRIP>, or <PAD>.
• Sequencing: At each 1 Hz time step, a <TS> token is prepended, followed by 44 feature tokens in canonical order; a 10-second window thus forms a 450-token sequence (Esashi et al., 31 Jan 2026).

This design supports joint modeling of heterogeneous signals and enables the transfer of NLP pretraining recipes to CAN data.

3. Model Architecture and Pretraining Objectives

A canonical Foundation CAN Model is instantiated as a BERT-style Transformer encoder with approximately 50 million parameters, composed of 9 layers, 670 hidden units, and 10 self-attention heads per layer (Esashi et al., 31 Jan 2026). Learned positional embeddings are used to encode sequence order.

The pretraining objective is Masked Signal Modeling: mask 15% of tokens per window (80% to <MASK>, 10% to random, 10% unchanged) and minimize the negative log-likelihood of the original token at each masked site:

$$\mathcal{L}_{\text{MLM}}(\theta) = -\sum_{i \in \mathcal{M}} \log P_\theta(t_i\mid\tilde T)$$

where $\mathcal{M}$ indexes masked positions in sequence $\tilde T$. No Next Sentence Prediction is used, diverging from BERT in favor of independence between windows.

This approach leverages ∼19 billion tokens (collected over 9 days from 10,000 vehicles) and produces rich representations adaptable to diverse tasks (Esashi et al., 31 Jan 2026).

4. Downstream Task Adaptation and Empirical Evaluation

Upon pretraining, Foundation CAN Models are fine-tuned via lightweight task-specific heads, with all backbone parameters updated ("full-parameter fine-tuning"). Two representative tasks include:

a) Binary Collision Detection: Classify the likelihood of a collision event in the next 10 seconds using weighted binary cross-entropy to address class imbalance. b) Point-of-Impact Multi-Class Classification: Eight-way softmax head for impact localization (Esashi et al., 31 Jan 2026).

Experimental results highlight:

• Comparable or improved performance to baselines (GLM for binary detection, 1D CNN for multi-class) especially in the multi-class scenario.
• Macro/Weighted F1 for Foundation CAN on point-of-impact classification: (28.3% / 33.1%, 28.2% / 34.9%, 27.0% / 32.6%); outperforming the CNN baseline (Esashi et al., 31 Jan 2026).
• Under extreme imbalance (100:1 negatives-to-positives), F1 drops, underscoring limitations of current pretraining corpus diversity and temporal windowing.

5. Integration of Causality and Hybrid Reasoning

While direct pretraining on CAN data yields robust signal representations, the capacity for genuine causal inference remains limited, as models are trained only on observational data. Insights from "Can Foundation Models Talk Causality?" (Willig et al., 2022) clarify several points:

• Foundation models encode "correlations on top of causation," capturing meta-level statements found in text but lacking grounding in actual interventional distributions.
• Prompt sensitivity, instability across formulation variants, and the inability to simulate $do$-interventions preclude true counterfactual reasoning (Willig et al., 2022).
• A future "Foundation CAN Model" (in the causality sense) would combine FM-based meta-level priors with statistical structure learning and explicit modules for simulating interventions and counterfactuals.

A table summarizing FM-causality junctions is provided:

Aspect	Foundation CAN (current)	FMs w/ Causality (aspirational)
Training Data	Decoded CAN, observational	Observational + Interventional
Causal Inference	Associative, non-causal	Intervention-aware, SCM-integrated
Stability	Prompt/embedding sensitive	Robust, standardized prompt schemes
Benchmarks	Task-specific metrics	Interventional/counterfactual suites

6. Design Challenges, Limitations, and Future Directions

Although Foundation CAN Models validate the extension of foundation modeling to structured, non-linguistic temporal domains, several challenges constrain their broader deployment:

• Generalization Gaps: The pretraining corpus (9 days) lacks sufficient environmental diversity, limiting model sensitivity to rare events and broader covariate shifts.
• Temporal Granularity: Current architectures (10 s at 1 Hz) cannot resolve longer-range dependencies or sub-second event dynamics.
• Tokenization Tradeoffs: Fixed binning may underserve low-variance critical features or overfragment high-variance signals.

Future improvements, as proposed, include:

1. Enlarging and diversifying CAN collections to capture richer event spectra.
2. Increasing signal sampling rates (to 5 Hz or higher) and extending context window lengths.
3. Exploring alternative pretraining objectives (e.g., next-signal prediction, contrastive learning).
4. Close integration with causal structure learning and symbolic reasoning engines (Willig et al., 2022).

A plausible implication is that the future "Foundation CAN" system may serve as a hybrid, combining statistical and causal representations to support not only robust task generalization but also explainable, intervention-aware decision-making pipelines in automotive and other sensor-driven domains.

7. Cross-Domain Implications and Research Trajectory

Foundation CAN Models demonstrate that the self-supervised, high-capacity paradigm of language foundation models can be fruitfully repurposed to dense, structured sensor time series (Esashi et al., 31 Jan 2026). This underscores a general trend toward universal multitask backbones, with domain adaptation and natural interfaces ("CAN" = C: scale and generality; A: adaptation; N: natural interfaces) as key principles (Narayan et al., 2022).

Emerging lines of research seek to close the gap between associative inductive biases and the operational requirements of causal (interventionist, counterfactual) reasoning, heralding new directions in robust, explainable AI for data-rich, safety-critical environments.