FlashInfer Trace Schema for LLM Kernels

• FlashInfer Trace Schema is a standardized, portable schema that unifies GPU kernel specification, benchmarking, and deployment for LLM inference systems.
• It clearly defines kernel semantics, interfaces, workloads, solutions, and evaluation metrics to ensure reproducible and rigorous performance validation.
• The schema enables automated integration and continuous improvement by supporting agent-generated implementations and immutable benchmarking records in production environments.

FlashInfer Trace Schema is a standardized, portable schema designed to unify the specification, benchmarking, and deployment of AI-generated GPU kernels within LLM inference systems. Developed as the foundational contract underpinning FlashInfer-Bench, this schema formalizes kernel descriptions, concrete workload instantiations, implementation records, and immutable evaluation reports, enabling rigorous correctness and performance validation, seamless integration of agent-generated solutions, and reproducible, dynamic deployment in large-scale serving engines (Xing et al., 1 Jan 2026).

1. Design Goals and Motivations

FlashInfer Trace was created to address the challenge of reliably integrating LLM-generated kernels into production LLM systems. Its primary motivations include ensuring that:

• Each kernel implementation is precisely defined with respect to semantics, interface, and supported shapes.
• The requirements for correctness, invocation, and deployment are machine-readable and enforceable.
• Benchmarking, reproducibility, and continuous improvement can be supported through a single schema shared by agents (generators), systems (deployers), and evaluators.
• The schema is minimal yet sufficient, supporting both human and LLM consumption via a fully specified JSON format.
• Automated processes such as leaderboard maintenance, benchmarking, and dynamic runtime dispatch (via flashinfer.apply()) are feasible without manual intervention.

A key implication is that the schema’s strict typing, controlled vocabularies, and inclusion of ground-truth reference semantics mitigate the risk of ambiguity, reward hacking, or silent deployment failures (Xing et al., 1 Jan 2026).

2. Schema Components: Formal Structure

The FlashInfer Trace object is a four-field JSON specification:

Trace ::= { Definition: ..., Workload: ..., Solution: ..., Evaluation: ... }

Each field captures a distinct pillar of the benchmarking and deployment lifecycle:

Component	Purpose	Key Fields
Definition	Describes kernel semantics and interface	name, op_type, axes, constraints, inputs/outputs, reference
Workload	Realizes a concrete test instance	uuid, axes (var values), inputs (scalar/random/safetensors)
Solution	Encapsulates an implementation and its metadata	name, spec (language, hardware, entry_point), sources, author
Evaluation	Immutable record of benchmarking outcome	status, environment, correctness metrics, performance, timestamp

This strict separation enables exhaustive tracking and communication of system requirements, ground-truth computation, agent-generated code, and evaluation artifacts (Xing et al., 1 Jan 2026).

3. Specification Details: JSON Structure and Controlled Vocabularies

Each Trace field is further defined with rigorously typed subfields and enumerations:

• Definition: Uniquely identified kernel, operation type (op_type ∈ {gemm, attention, gqa_paged, ...}), axes (type: const/var), explicit input/output specs (shape as symbolic axes, dtype), constraints on axis cardinalities, and a canonical reference implementation (e.g., PyTorch).
• Workload: Concrete values for variable axes, and input tensors specified as either random, scalar, or safetensors (with file path/tensor key).
• Solution: Implementation metadata, required language (triton, cuda, cutlass, tvm, ...), compatible hardware (e.g., "B200"), entry point (function locator), and full source code attached.
• Evaluation: Status (PASSED/FAILED), environment (hardware, library versions), correctness (elementwise or statistical criteria), performance metrics (latency, reference_latency, speedup_factor). Correctness mode may be deterministic (max_absolute_error, max_relative_error), low-precision (matched_ratio ρ), or stochastic (total_variation_distance).

A minimal GEMM Trace instance is illustrated in the cited data (Xing et al., 1 Jan 2026); see included concrete JSON examples.

4. Formal Metrics and Evaluation Mechanics

FlashInfer Trace encodes reference-based performance and correctness metrics, facilitating rigorous and uniform evaluation:

• Workload enumeration: For Definition $D$, $\mathcal{W}_D = \{w_1, ..., w_N\}$, where $w_i$ are concrete input/axis assignments.
• Performance: For Solution $s$ and Workload $w$, report $\mathrm{latency}(s, w)$, $\mathrm{latency}_{ref}(w)$, and $$\mathrm{speedup}(s, w) = \frac{\mathrm{latency}_{ref}(w)}{\mathrm{latency}(s, w)}$$.
• Correctness:
  • Deterministic kernels: $$\forall j: |y_{s,j} - y_{\mathrm{ref},j}| \leq \varepsilon_{\mathrm{abs}} + \varepsilon_{\mathrm{rel}}|y_{\mathrm{ref},j}|$$.
  • Low-precision: $$\frac{1}{|\textrm{elements}|} \sum_j \mathbf{1} (|y_{s,j} - y_{\mathrm{ref},j}| \leq \varepsilon_{\mathrm{abs}} + \varepsilon_{\mathrm{rel}}|y_{\mathrm{ref},j}|) \geq \rho$$.
  • Stochastic: $$\mathrm{TVD}(\hat f, q) = \frac{1}{2} \sum_k |\hat f_k - q_k| \leq \tau_{\mathrm{TVD}}$$.
• Leaderboard/benchmarking: The $\mathrm{fast}_p$ metric (from KernelBench):

$$\mathrm{fast}_p(s) = \frac{1}{N} \sum_{i=1}^N \mathbf{1}(\mathrm{correct}(s, w_i) = 1 \wedge \mathrm{speedup}(s, w_i) > p)$$

This formalism supports automated, immutable evaluation, and selection or rejection of candidate solutions before deployment (Xing et al., 1 Jan 2026).

5. End-to-End Agent–System Integration

FlashInfer Trace underpins the close-loop workflow for AI-driven system improvement and deployment:

• Agents receive the Definition (with reference) as a concrete specification for code generation.
• Agent-generated Solution objects must match the Definition schema and interface precisely.
• The system’s benchmarking harness materializes Workload instances, invokes the agent code, and compares outputs to the reference using the prescribed correctness mode.
• Evaluation records are produced automatically, guaranteeing scheme-conformance, ground-truth comparability, and reproducibility.
• The flashinfer.apply() dispatch mechanism indexes all Definition × Solution × Evaluation triplets, enabling runtime selection of the best validated kernel per axis/shape constraints—without code change in engines such as SGLang and vLLM.
• Immutable Evaluation records and controlled vocabularies guard against replay attacks, reward hacking, or specification drift.

This architecture ensures that every deployment is backed by a rigorously validated Trace record, and that benchmarking and leaderboard updates require no human intervention (Xing et al., 1 Jan 2026).

6. Practical Significance and Implications

The schema is designed to be both minimal and comprehensive: all the information necessary for kernel invocation, verification, benchmarking, and deployment is expressed in an interpretable and formal manner. A plausible implication is that this enables a virtuous cycle in which agents are continuously evaluated, compared, and substituted in production based on standardized, reproducible metrics.

The portability of the Trace schema allows for widespread adoption across different LLM-based systems, facilitating both the integration of AI-generated modules and the adoption of a community-maintained benchmarking corpus. This suggests a path toward vendor-independent, reproducibly auditable progress on AI-driven LLM infrastructure (Xing et al., 1 Jan 2026).