Papers
Topics
Authors
Recent
Search
2000 character limit reached

Sutra: Tensor-Op RNNs as a Compilation Target for Vector Symbolic Architectures

Published 20 May 2026 in cs.LG, cs.AI, and cs.PL | (2605.20919v2)

Abstract: Sutra is a typed, purely functional programming language whose compiled forward pass is a PyTorch neural network. The compiler beta-reduces the whole program -- primitives, control flow, string I/O -- to one fused tensor-op graph over a frozen embedding substrate. Rotation binding, unbind, bundle, polynomial Kleene three-valued logic, and tail-recursive loops all lower to tensor operations; the Kleene connectives are Lagrange-interpolated polynomials exact on the {-1, 0, +1} truth grid. Validation is one fact tested two ways. (1) The same program runs on four frozen embeddings spanning two modalities -- three text encoders (nomic-embed-text, all-minilm, mxbai-embed-large) and one protein LLM (ESM-2) -- and decodes bundles at 100% accuracy through width k=8 on every substrate, where the textbook Hadamard product has already collapsed (2.5% on mxbai-embed-large, 7.5% on all-minilm). (2) PyTorch autograd flows through the actually compiled graph: a fuzzy-rule classifier written in .su trains from random init (18.7 +/- 9.5%; chance = 20%, five classes) to 100.0 +/- 0.0% (three seeds) by backpropagating through the emitted graph, the symbolic source unmodified. A weighted variant additionally trains a scalar cosine gain and writes it back into the .su source as a numeric literal; recompiling reproduces the trained behaviour to ~2e-7 per logit, so the trained model is itself legible, recompilable code. The same artifact is therefore both a logic program and a trainable neural network.

Authors (1)

Summary

  • The paper introduces Sutra, a functional language that compiles to tensor-op graphs, unifying vector symbolic architectures with neuro-symbolic computing.
  • It details differentiable fuzzy logic gates and a compiler that translates high-level logic into smooth, branchless tensor operations using RNN-like constructs.
  • Empirical results show rotation binding achieving 100% decoding accuracy, outperforming Hadamard binding and ensuring end-to-end gradient flow.

Sutra: Compiling Typed Functional Programs to Tensor-Op RNNs for Vector Symbolic Architectures

Abstract and Motivation

"Sutra: Tensor-Op RNNs as a Compilation Target for Vector Symbolic Architectures" (2605.20919) introduces Sutra, a statically-typed, purely functional programming language whose compilation target is a PyTorch-based tensor computation graph, operating directly over vector embedding spaces. This approach unifies several strands in vector symbolic architectures (VSA), differentiable programming, and neuro-symbolic computing, offering a common substrate on which symbolic and neural operations coalesce. Sutra specifically addresses the transfer of VSA operations—traditionally derived under random hypervector assumptions—into the anisotropic, semantically meaningful embedding spaces produced by frozen language and protein models, and it systematizes this approach into a differentiable language and runtime.

Core Technical Contributions

Lagrange-Interpolated Fuzzy Logic Gates

Sutra proposes a construction of differentiable fuzzy logic connectives, using Lagrange interpolation to obtain smooth, polynomial operators that are exact on Kleene’s three-valued logic grid {−1,0,+1}\{-1,0,+1\}. These gates—AND, OR, NOT, NAND, NOR, XOR, XNOR—are C∞C^{\infty} and support end-to-end gradient flow, circumventing the nondifferentiability of standard t-norms at pivotal points. This choice ensures compatibility with modern autograd frameworks, essential for integrating logic programming structures within neural architectures.

Compilation to Substrate-Pure Tensor-Op Graphs

The compiler conducts full beta reduction over the source program, inlining and algebraically simplifying all constructs down to a straight-line tensor operation graph. Control constructs (conditionals, loops) are lowered to differentiable, branchless primitives—e.g., conditionals via soft-mux polynomials, and recursion via RNN-like self-halting cells observable in their runtime data flow. Role binding is realized via compile-time allocation of Haar-orthogonal rotation matrices, seeded by content hashes, ensuring invertibility and conditioning superior to Hadamard-like schemes in practical embedding spaces.

Tail-Recursive Loops and RNN Cells

Sutra supports tail-recursive function definitions which compile to recurrent neural (RNN) cells with substrate-side, soft-halt logic. Halt conditions are implemented with differentiable substrate operations (e.g., heaviside, monotonic cumulative sums) rather than explicit host-side branching, ensuring that loops maintain constant state width and are fully differentiable with respect to every parameter across all iterations.

Angular Hash Maps via Synthetic Dimension Rotation

The Sutra compiler reserves synthetic vector dimensions and implements key–value bindings as high-dimensional angular (rotation) hashes. Each role is mapped to a unique Haar-orthogonal rotation matrix, and the binding and unbinding operations correspond to applying and inverting these rotations, respectively. This is a novel binding implementation for VSAs over real-valued embeddings, shown to be robust to the noise and anisotropy of practical substrates.

Empirical Evaluation

Rotation Binding vs. Hadamard Binding

Extensive experiments benchmark rotation-based binding against standard Hadamard (elementwise multiplication) binding across multiple embedding substrates: three textual encoders (nomic-embed-text, all-minilm, mxbai-embed-large) and one protein LLM (ESM-2). Rotation consistently achieves 100% decoding accuracy in bundle width regimes where Hadamard binding collapses to near-chance performance (e.g., for k=8k=8, rotation is at 100% and Hadamard falls to 2.5%2.5\% on mxbai-embed-large). This holds across modalities, including the protein domain, establishing strong evidence for the universality of rotation-based binding on real-world embedding spaces.

Chained Bind/Unbind Noise Accumulation

The stability of recursive binding/unbinding chains is investigated, showing accurate retrieval up to depth L=2L=2, beyond which accuracy drops to chance as crosstalk noise accumulates. This bounds the practical compositional depth for records encoded with this paradigm, indicating that deeper record nesting is not currently viable without further innovation.

End-to-End Differentiable, Programmatic Training

Sutra compiles logic programs to differentiable tensor graphs that support standard PyTorch autograd. The paper demonstrates a fuzzy-rule classifier—written in Sutra source, compiled to PyTorch ops, then trained end-to-end via standard gradient descent—achieving 100%100\% classification accuracy from random initialization across multiple seeds. The system is further extended to allow learned scalar weighting of similarity scores, yielding models where all learned parameters can be hard-coded into legible, recompilable source files, with recompiled programs reproducing the trained behavior up to 2×10−72 \times 10^{-7} per logit—making trained models intrinsically interpretable.

Architectural and Implementation Properties

  • Substrate Agnosticism: Sutra programs can target any embedding model, including non-text domains, by recompiling source against the new embedding geometry. This generalizes the separation between logic and substrate found in classical VSA, allowing broad application across precomputed representation spaces.
  • Compile-Time Codebooks and Binding Primitives: String or sequence literals are embedded at compile time and mapped with a vector database using HNSW indices for efficient codebook lookup. Role rotation matrices are precomputed and cached, making runtime binding computationally optimal.
  • Type System and Surface Syntax: The language is statically typed with support for common primitives, with types directing synthetic layout allocation. However, type violations are warnings, not errors; the runtime enforces mathematical validity over strict static correctness.
  • Substrate-Purity Invariants: All primitives execute purely on the embedding substrate: no host-side control flow, no scalar extraction from vectors, and no reliance on Python outside the compiled tensor-graph.

Comparison to Prior Art

Sutra moves beyond existing VSA and HDC toolkits (e.g., TorchHD) by providing a compilation target that produces a single fused, differentiable tensor-op graph, eliminating host-language (Python) control flow and enabling efficient operator fusion. Compared to neuro-symbolic languages (Scallop, DeepProbLog, Logic Tensor Networks, NeurASP), Sutra treats the entire computation as continuous vector operations, with no intermediate symbolic discretization, differentiation by provenance annotations, or separation between perception and reasoning. The result is a fused artifact where the distinctions between "neural" and "symbolic" computation dissolve.

Limitations

Empirical results are measured on specific frozen embeddings, and the deep compositional regime remains out of reach due to noise accumulation in chaining binding/unbinding. Codebook size is O(vocabulary)O(\text{vocabulary}) and may present scaling limitations. No results on generalization or held-out splits are reported, and the substrate dependence of certain capacities is explicit. Bindings are not learned, but fixed by content hash, and the work provides no benchmark comparison against alternative neuro-symbolic stacks beyond capacity and gradient flow.

Implications and Future Work

Sutra establishes a clear path for logic-program-inspired, fully differentiable programming over learned, frozen vector spaces. The approach dissolves the barrier between symbolic programming and vector-based computation, providing a legible, retrainable, and optimizable pipeline. Practical implications include the possibility of compiling symbolic program specifications directly into neural architectures, supporting both transparent initialization and gradient-based optimization. The substrate-agnostic compiler opens the method to a wide range of domains, including molecular, textual, and potentially multimodal embedding spaces.

Future avenues include scaling to larger codebooks, investigating compositional depth through noise mitigation strategies, developing learned or semantic binding operators that adapt to the geometry of the substrate, and benchmarking Sutra programs on complex, held-out reasoning tasks, including those requiring generalization and abstraction over unseen compositional structures. Integration with dynamically learned representation models, rather than frozen embeddings, would further broaden scope and applicability.

Conclusion

Sutra represents an integrated approach to compiling logical and algebraic programs into substrate-pure, differentiable tensor computation graphs over arbitrary embedding spaces. It systematizes the choice and implementation of VSA primitives compatible with anisotropic, pretrained embeddings, and provides a compilation pipeline tying high-level, type-annotated functional code to efficient neural architectures. By solving for differentiable fuzzy logic gates, optimal binding mechanisms, and direct compositional embedding operations, Sutra lays the groundwork for interpretable, optimizable, and substrate-agnostic neuro-symbolic computing.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Overview

This paper introduces Sutra, a new programming language that turns your code into a trainable neural network. Instead of running on a normal computer CPU like most programs, Sutra programs run inside a “vector space” made by an existing model’s embeddings (for example, text embeddings from a LLM or protein embeddings from a biology model). The big idea is that the same thing you write as logical rules becomes a smooth, math-based system that can also learn.

What questions does the paper ask?

The paper focuses on two main questions:

  1. Which basic operations on embeddings are dependable enough to build a whole system out of them?
  2. What’s the best way to “bind” a role to a value inside these embeddings so we can store and later retrieve information without it getting scrambled?

How does Sutra work?

A language that becomes a neural network

When you write a Sutra program, the compiler simplifies and combines everything into one straight-line chain of tensor operations (like matrix multiplies and element-wise math) that PyTorch understands. There are no leftover if-statements or Python loops inside the final computation graph. This makes it both clean and trainable with standard deep learning tools.

Vectors and embeddings, in simple terms

An “embedding” is a way to turn things like words or protein sequences into points in a high-dimensional space. Think of each word as a unique dot in a big 3D-like space with hundreds of dimensions. Operations in Sutra move and combine these dots in consistent ways.

Binding and bundling: storing and retrieving information

  • Binding is like locking a value into a role using a special key. Sutra uses rotation binding: imagine spinning the value in the vector space by a unique angle pattern tied to the role. To unbind, you spin it back by the exact opposite pattern.
  • Bundling is like putting several items into one backpack by adding their vectors together and then re-normalizing, so you can carry them as a single vector.

Why rotation binding? The paper shows that older “multiply each dimension” binding (Hadamard product) breaks down on real embeddings from LLMs, because those embeddings aren’t spread out evenly. Rotation binding stays reliable longer and can decode correctly even when many items are bundled together.

Fuzzy logic that’s smooth and learnable

Logic usually has True or False. Fuzzy logic adds a middle value (Unknown). Sutra maps True to +1, Unknown to 0, and False to −1, and implements logic gates like AND, OR, and NOT as special polynomials. These polynomials:

  • Match classic three-valued logic exactly at those three points.
  • Are smooth everywhere else, so gradients can flow and the network can learn.

This means you can write rules like “A is true AND B is NOT true,” and under the hood it becomes math that works with neural network training.

Loops become RNN cells

If your program needs to repeat until something is done, Sutra compiles that loop into a small recurrent neural network (RNN) cell with a “soft halt” signal. Each step:

  • Checks if it should stop using a smooth threshold.
  • Either updates the state or freezes it. Because it’s smooth, you can still train through the loop using standard backpropagation.

Strings in and out using a codebook

You can write string literals in your Sutra code (like “blue” or “circle”). At compile time, the compiler embeds these once and stores them in a local codebook. At the end of a program, Sutra can convert a computed vector back to the nearest matching string, like translating a point in space back to “blue.”

What did the experiments show?

Here are the key results the paper reports:

  • Rotation binding beats Hadamard binding across multiple embedding models.
    • With bundles of 8 items, rotation binding decoded correctly 100% of the time on all tested substrates. Hadamard often collapsed: for example, 2.5% on mxbai-embed-large and 7.5% on all-minilm (both are text encoders).
    • Even at wider bundles (like 48 items), rotation still kept much higher accuracy than Hadamard.
  • It works beyond text. The same pattern held on a protein LLM (ESM-2), showing the approach is not limited to natural language.
  • Training flows through the compiled graph. A fuzzy-rule classifier written in Sutra (and compiled to PyTorch ops) trained from random guessing to perfect accuracy by backpropagating through the actual compiled graph—no hand-written reimplementation.
  • Trained parameters can be written back into readable code. In one test, the system learned a single scale factor that improves cosine similarity. After training, the learned number was written directly into the Sutra source as a literal. Recompiling reproduced the exact behavior. This means the “model” is also understandable code.

Why is this important?

  • It bridges logic and learning. Usually, logical rules and neural networks live in separate worlds. Sutra blends them: your logical rules compile into a smooth, trainable math graph.
  • It targets real embeddings. Many older vector methods assume random vectors. Sutra is designed for real embeddings from existing models, where vectors are not evenly spread out.
  • It simplifies engineering. You write rules and functions once. The compiler turns them into one fused, efficient computation that can run on CPU or GPU, and also supports training.
  • It keeps models legible. Because trained numbers can live as source code, the result isn’t a black box—it’s readable logic with learned constants.

Limitations and future work

To keep expectations realistic, the paper clearly lists limits:

  • Deeply nested records are fragile. If you repeatedly bind and bundle many levels, accuracy drops after a few levels when distractors are present.
  • Results depend on the embedding model. Different models vary in how well bundling and decoding work.
  • The codebook scales with vocabulary size. Very large vocabularies may need more advanced nearest-neighbor tricks.
  • Binding is fixed, not learned. Sutra uses rotation binding seeded by role names; learned binding strategies are future work.
  • Training demos are in-sample. The training experiments show gradient flow and in-sample success, but don’t claim generalization performance on held-out data.

Bottom line

Sutra shows that you can write a small, clean logic program and have it compile into a trainable neural network that runs directly on real embeddings. It gives you:

  • Reliable operations for storing and retrieving information in vector spaces.
  • Smooth fuzzy logic gates that work with gradients.
  • Loops that compile into RNNs.
  • A path to keep trained models readable as code.

In short, the same artifact is both a logic program and a neural network, opening up new ways to combine clear rules with learning.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Unresolved Knowledge Gaps, Limitations, and Open Questions

The following list synthesizes what remains missing, uncertain, or unexplored in the paper, framed as concrete directions future researchers could act on:

  • Depth of compositionality under noise
    • Retrieval degrades from 100% at L≤2L \le 2 to chance by L=8L=8 when chaining bind/unbind with bundling. What bundling/cleanup strategies (e.g., learned denoisers, sparse/weighted bundling, iterative cleanup memory) can extend reliable depth without sacrificing substrate purity?
  • Generalization across embedding substrates and model drift
    • Results are measured on four frozen encoders (three text, one protein). How do rotation-binding capacity and logic gate behavior transfer to other modalities (vision, audio), other text encoders, and evolving checkpoints? Can the system adapt online to representation drift (e.g., via learned whitening, centering, or adaptive gains)?
  • Theoretical guarantees for rotation binding in anisotropic spaces
    • Beyond empirical tables, there is no analysis of error bounds, capacity as a function of dimension and anisotropy, or conditions for near-orthogonality of rotated codes. Can one formalize retrieval error and interference for rotation+bundle under realistic covariance structures of modern embeddings?
  • Learned vs. fixed binding operators
    • Binding is fixed to hash-seeded Haar rotations. Can learned role transforms (constrained to be orthogonal or approximately so) improve capacity and robustness while retaining invertibility and substrate purity? What is the trade-off between learned binding and the legibility/compilability goals?
  • Role-hash collisions and determinism of rotation construction
    • The use of content hashes raises the risk of collisions and dependency on QR determinism across platforms. What is the collision rate at scale, how should collisions be handled, and how reproducible are rotations across devices, BLAS/LAPACK versions, and dtypes?
  • Scaling and engineering of the codebook layer
    • The codebook is O(∣V∣)O(|\mathcal{V}|) and relies on nearest-neighbor search. What are the latency/accuracy trade-offs for million-scale vocabularies, dynamic updates, and incremental compilation? How do HNSW parameters affect decode accuracy vs speed, and how to ensure persistence/consistency across compiles?
  • Robustness of nearest-string decoding
    • Nearest-neighbor decoding in anisotropic spaces may conflate semantically similar but distinct labels or fail with OOD inputs. What safeguards (margins, calibration, abstention) and isotropy corrections (e.g., all-but-the-top, whitening) improve reliability at the I/O boundary?
  • Properties of the Lagrange–Kleene polynomial gates off the grid
    • The connectives are exact on {−1,0,1}\{-1,0,1\} but uncharacterized off-grid. What are their Lipschitz constants, gradient magnitudes, and saturation behavior in deep compositions? How do they compare empirically/theoretically to smooth t-norms for trainability and numeric stability?
  • Differentiable halting and loop training
    • The soft-halt cell uses Heaviside→saturating accumulation with a host-side break, but loop training is not demonstrated. Which smooth approximations (e.g., sigmoid gates) enable stable gradient-based learning of halting conditions, and how does performance/gradient flow scale with long or nested loops?
  • Termination guarantees and failure modes
    • What prevents non-terminating loops when halt never saturates (e.g., due to numerical issues)? Are there compiler/runtime safeguards (max steps, adaptive thresholds) that preserve differentiability while guaranteeing termination?
  • Capacity and layout of the extended state vector
    • The synthetic block (e.g., 100 dims with 47 slots) is fixed a priori. How many concurrent variables/slots can be supported before interference appears, and what is the optimal semantic:synthetic allocation for different tasks and substrates?
  • Alternative bundling and cleanup mechanisms
    • Bundling is a normalized sum; no comparison to other VSA bundlers (e.g., majority, sparse, learned weights) or cleanup strategies. Which bundlers offer better noise properties in anisotropic spaces, and can cleanup be implemented substrate-purely without host lookup?
  • Formal semantics and correctness of compilation
    • There is no formal proof that beta-reduced programs and the emitted tensor graph are observationally equivalent, nor a denotational semantics for the language. Can one specify and verify correctness of the compilation pipeline and its algebraic simplifications?
  • Type system soundness and debugging ergonomics
    • Types are “opinionated not authoritarian,” allowing semantically meaningless but mathematically valid outputs. What static analyses or optional strict modes improve safety and debuggability without breaking differentiability or compilation?
  • Numerical precision and hardware portability
    • Effects of mixed precision, quantization, and kernel fusion on orthogonality (rotation matrices), decode accuracy, and gradient stability are not evaluated. How robust is Sutra to fp16/bf16 and torch.compile across GPUs/CPUs?
  • Benchmarks against external neuro-symbolic systems
    • There is no comparative evaluation on shared tasks (accuracy, sample efficiency, runtime) versus Scallop, DeepProbLog, LTNs, or HDC baselines. Which problem classes favor Sutra, and where does it underperform?
  • Task scale and complexity
    • Demonstrations use small codebooks and toy classifiers. What are the throughput, memory, and training characteristics on larger rule sets, deeper compositions, and real-world tasks (e.g., multi-hop QA, program induction, biosequence reasoning)?
  • Integration with trainable encoders
    • The substrate is frozen in all experiments. How does end-to-end training (fine-tuning the encoder jointly with the compiled graph) affect anisotropy, binding capacity, and rule fidelity? Can joint training maintain legibility and recompilability?
  • Cross-modal and multilingual generality
    • Only English text encoders and one protein LM are tested. How does the approach extend to images, audio, or multilingual embeddings with different anisotropy profiles and semantic structures?
  • Dynamic roles and maps at runtime
    • Role rotations are precomputed at compile time. How can one support roles created dynamically at runtime (e.g., data-dependent keys) without breaking substrate purity or incurring heavy on-the-fly QR costs?
  • Expressivity beyond Kleene three-valued logic
    • Only Kleene connectives are provided. How to extend to other many-valued logics (Łukasiewicz, product), quantifiers, and higher-order constructs while preserving differentiability and compilation to a fused graph?
  • Security and adversarial robustness
    • The system assumes benign inputs and codebooks. How vulnerable are nearest-neighbor decoders and fuzzy rules to adversarial perturbations in embedding space, and what defenses are compatible with the substrate-pure constraint?
  • Documentation of reproducibility knobs
    • Precise seeds, hash functions, QR initialization, and numeric tolerances that guarantee bitwise-stable compilation and training across platforms are not specified. What reproducibility protocol ensures identical emitted graphs and results?
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The following applications can be deployed now using the paper’s methods and validated regimes; each item notes sectors, likely tools/workflows, and key assumptions/dependencies.

  • Rule-as-neural-model classifiers (trainable fuzzy logic over embeddings)
    • Sectors: software, education, finance (risk triage prototypes), healthcare (non-clinical decision support prototypes)
    • Tools/workflows: author .su fuzzy rules using polynomial Kleene gates; compile to a PyTorch module; fit learnable prototypes and scalar gains by backprop through the emitted graph; optionally bake learned scalars back into source for legibility and code review
    • Assumptions/dependencies: frozen embedding substrate chosen and validated; class sets modest in size; training verified in-sample (generalization must be evaluated); anisotropy may require learned gain scaling; compliance demands external validation for regulated domains
  • Embedding-native record encoding/decoding with rotation binding (role–filler records, simple knowledge triples)
    • Sectors: software (search/indexing), knowledge management, education, bioinformatics (sequence annotation pipelines)
    • Tools/workflows: bind role and filler with Haar-orthogonal rotation; bundle normalized sums; unbind and nearest_string decode from a compile-time codebook; implement attribute/value records, shallow knowledge-graph lookups, slot filling
    • Assumptions/dependencies: single-cycle records; short chain depths (L ≤ 2) and moderate bundle widths (e.g., k ≤ 8 for 100% decode on tested substrates) to avoid crosstalk; codebook size modest for exact or HNSW-ANN decode
  • Neuro-symbolic post-processing for LLM/embedding pipelines (logic gating in vector space)
    • Sectors: software, education, enterprise NLP
    • Tools/workflows: use compiled fuzzy rules (e.g., AND of positive and NOT of negative similarity) to gate retrieval candidates, re-rank outputs, or enforce simple semantic constraints on embedding outputs; run end-to-end as a single fused PyTorch graph
    • Assumptions/dependencies: relies on frozen text embeddings (nomic-embed-text, all-minilm, etc.); anisotropy compresses cosine—support gain scaling; measurable rule performance on target corpus required
  • Substrate certification and capacity profiling (choose an embedding model and safe operating regime)
    • Sectors: MLOps, academia, platform teams
    • Tools/workflows: run the paper’s decode-accuracy and chain-length tests on candidate substrates to set bundle-width and nesting limits; auto-generate a “substrate profile” for project guidelines
    • Assumptions/dependencies: experiments reproduce on local hardware; profiles remain stable across model/version drift (monitor routinely)
  • Self-halting differentiable loops for finite-horizon controllers and iterative procedures
    • Sectors: software (iterative solvers), robotics prototypes, education
    • Tools/workflows: express loop bodies as straight-line tensor ops; halt via substrate truth-axis and cumulative saturation; deploy as RNN cells with O(1) state; trace and backprop in PyTorch
    • Assumptions/dependencies: loop termination conditions encoded as fuzzy truth; iteration counts moderate (autograd tape is O(N)); real-time robotics requires additional timing guarantees and safety validation
  • Cross-substrate reuse of the same program (text vs. protein embeddings)
    • Sectors: bioinformatics, multi-modal ML research
    • Tools/workflows: write one .su program; compile against different codebooks/substrates (e.g., ESM-2 for proteins) to run identical logic across modalities (e.g., motif/annotation lookups or simple sequence classifiers)
    • Assumptions/dependencies: per-substrate validation required; decode capacity varies by model; vocabulary size moderate
  • Model-as-code governance and auditability
    • Sectors: policy/compliance, MLOps, finance (model risk management)
    • Tools/workflows: treat trained scalar gains and prototypes as first-class source artifacts; bake scalars into .su; version-control the program and codebook; enable code review and change tracking for logic updates
    • Assumptions/dependencies: only scalar gains trivially bake back into code; embedding vectors are still parameters (store in codebook/artifacts); external validation and documentation remain necessary
  • Teaching and curriculum materials for neuro-symbolic computing
    • Sectors: academia, education technology
    • Tools/workflows: labs where students write rules that compile to differentiable graphs; visualize gradients through logic gates; compare Hadamard vs. rotation binding empirically
    • Assumptions/dependencies: classroom datasets small; CPU or modest GPU suffices; students install PyTorch and the Sutra toolchain

Long-Term Applications

These opportunities require further research, scaling, or engineering; each item includes sector links, possible products, and feasibility caveats.

  • Scalable neuro-symbolic retrieval and reasoning over large knowledge graphs
    • Sectors: enterprise search, data integration, software
    • Potential tools/products: “VSA-DB” that encodes entities/relations via rotation binding; compositional query operators in vector space; hybrid symbolic–vector stores
    • Dependencies/caveats: mitigate bundle crosstalk (larger d, error-correcting schemes, learned/semantic binding); ANN decode at scale; support deeper composition without accuracy loss
  • Regulated decision-support with interpretable, trainable logic
    • Sectors: healthcare (clinical decision support), finance (credit/scoring), public policy
    • Potential tools/products: rule authoring IDE with gradient-based calibration; automated documentation linking code to learned parameters; monitoring for substrate drift
    • Dependencies/caveats: rigorous external validation, fairness/causality analysis, post-hoc explanations; approval processes; generalization beyond in-sample demos
  • On-device/edge deployment via AOT compilation (beyond PyTorch)
    • Sectors: mobile, embedded systems, robotics
    • Potential tools/products: export compiled graphs to ONNX/XLA/TVM; precompute rotations; fuse kernels for low-latency inference
    • Dependencies/caveats: extend compiler backends; ensure numeric stability; manage codebook I/O on-device; real-time guarantees and safety certification for robotics
  • Multi-modal neuro-symbolic programs spanning text, vision, audio, and biology
    • Sectors: multimedia search, bio/med AI, digital asset management
    • Potential tools/products: one .su specification compiled against different substrates (CLIP, wav2vec2, ESM); cross-modal matching and rule-based routing
    • Dependencies/caveats: validate rotation binding and decode capacity across modalities; normalize anisotropy; unify codebook semantics across domains
  • Programmatic agents with trainable control logic (loops-as-RNN planning)
    • Sectors: autonomous systems, operations research, process automation
    • Potential tools/products: controllers whose guards are fuzzy rules over embeddings; learnable gains/prototypes for environment-specific adaptation; safety envelopes expressed as code
    • Dependencies/caveats: robust halting, stability and safety proofs; handling long horizons (tape/memory); richer I/O than current codebook abstraction
  • Database and analytics operators in vector-algebra form (vector relational algebra)
    • Sectors: data platforms, analytics
    • Potential tools/products: vectorized join/select via binding/unbinding; angular hash maps for composite keys; hybrid scalar–vector query planners
    • Dependencies/caveats: theoretical foundations for algebraic equivalences; performance vs. classic DBMS; precision/recall trade-offs with ANN indices
  • Learned or semantic binding operators
    • Sectors: ML research, software
    • Potential tools/products: substrate-adaptive binding that reduces crosstalk and anisotropy effects; role-specific learned transforms with invertibility constraints
    • Dependencies/caveats: preserve differentiability and reversibility; ensure cross-substrate generality; avoid overfitting to codebook statistics
  • Tooling ecosystem: IDEs, debuggers, and profilers for “logic-as-tensor-graph”
    • Sectors: developer tools, education
    • Potential tools/products: step-through evaluation on tensors; gradient-path visualizers through Kleene polynomials; static analysis for saturation/vanishing in rule chains
    • Dependencies/caveats: instrumentation for compiled graphs; UX for dual symbolic/numeric views; integration with existing ML IDEs
  • Standards for “substrate as architecture target”
    • Sectors: industry consortia, open-source
    • Potential tools/products: exchange formats for codebooks (.sdb), rotation caches, and substrate profiles; reproducibility guidelines for embedding-dependent programs
    • Dependencies/caveats: consensus on metadata (embedding model/version, d, anisotropy metrics); support across major embedding providers
  • Personal knowledge management and PKM agents operating in embedding space
    • Sectors: consumer productivity, enterprise knowledge
    • Potential tools/products: notebooks that bind notes into role-labeled bundles; fuzzy-rule retrieval and summarization; explainable filters
    • Dependencies/caveats: larger, dynamic codebooks; handling user drift (re-embedding, re-profiling); privacy and on-device processing

Notes on Feasibility and Dependencies (cross-cutting)

  • Substrate selection and drift
    • Behavior is embedding-model dependent; rotation binding capacity and decode accuracy vary by model and dimension; periodic re-profiling needed when models update.
  • Capacity limits
    • Validated for single-cycle records, short chains (L ≤ 2), and moderate bundle widths; deep nesting or wide bundles require new mitigation (e.g., higher dimensionality, denoising, error-correcting codes).
  • Anisotropy and scaling
    • Cosine compression reduces dynamic range; learned scalar gains help but may not fully resolve for all tasks; alternative similarity metrics or whitening may be required.
  • Codebook scale and ANN quality
    • Large codebooks necessitate approximate search (HNSW/FAISS); decode errors and latency must be managed; consistency of nearest_string with downstream expectations is critical.
  • Portability and runtime
    • Current target is PyTorch; broader deployment may need ONNX/XLA/TVM backends; ensure substrate purity and precomputed constants remain intact after conversion.
  • Governance and compliance
    • “Model as code” improves auditability, but regulated uses still require domain validation, bias assessment, and monitoring; compiling learned scalars back to source aids review but doesn’t replace model risk controls.
View Paper Prompt  View All Prompts 

Glossary

  • AOT compilation: Ahead-of-time compilation that transforms high-level models into optimized executable code before runtime. "AOT compilation of neural networks (TVM, XLA)"
  • argmax-cosine: Selecting the item with maximum cosine similarity, often used for nearest-neighbor decoding. "decoding by unbind + argmax-cosine against the full codebook."
  • anisotropy (embedding anisotropy): Non-uniform distribution of embeddings concentrated in a narrow cone, compressing cosine similarity ranges. "they are strongly anisotropic, concentrating in a narrow cone so cosine similarity is compressed and inflated even between unrelated items"
  • autograd: Automatic differentiation system that computes gradients through computational graphs. "PyTorch autograd flows through the actually compiled graph"
  • beta reduction: Functional-programming rewrite that applies functions to arguments by substituting bound variables. "The compiler beta-reduces the whole program"
  • bind (VSA binding): Operation combining a role and a filler into a single vector representation. "bind(role,\ filler)\ =\ R_role\ @\ filler"
  • bundle (VSA bundling): Superimposing multiple vectors and renormalizing to keep them retrievable. "bundle(x,y) = (x+y)/(\lVert x+y\rVert + \varepsilon)"
  • circular convolution: A VSA binding operator defined as convolution modulo the vector length. "Hyperdimensional computing’s textbook bind operators (Hadamard product, circular convolution) were derived assuming hypervectors"
  • codebook (vector codebook): Stored mapping from strings to their embeddings for compile-time I/O convenience. "a compile-time codebook (implemented with an embedded vector database, §3.5)"
  • cross-entropy: Loss function measuring divergence between predicted and target distributions. "cross-entropy loss falls to ≈ 0.43"
  • differentiable programming: Treating programs as differentiable functions to enable gradient-based optimization. "differentiable programming systems that treat programs as differentiable functions (JAX)"
  • egglog (e-graph simplifier): Equality-graph-based system used for algebraic simplification and common-subexpression elimination. "an egglog-based simplifier folds equivalent expressions"
  • embedding substrate (substrate): The target embedding space that programs operate in as their “architecture.” "beta-reduces to a substrate-pure tensor-op graph"
  • Futamura projections: Theoretical results about partial evaluation and specialization of interpreters/compilers. "partial-evaluation systems that specialize programs at compile time (the Futamura projections)"
  • Givens planes: Pairs of coordinates forming 2D rotation subspaces used to allocate slots in synthetic dimensions. "the remaining axes paired into 2D Givens planes for variable slots."
  • Gödel t-norm/t-conorm: Specific min/max-based fuzzy logic operators for conjunction/disjunction. "The min/max forms (the standard Gödel t-norm/t-conorm choice; Hájek 1998)"
  • Hadamard product: Elementwise multiplication used as a traditional VSA binding operator. "textbook bind operators (Hadamard product, circular convolution)"
  • Haar-orthogonal rotation: Random orthogonal matrices sampled with respect to the Haar measure, used for role rotations. "R_r = \mathrm{QR}(\mathrm{hash}(r)).Q) is Haar-orthogonal"
  • Heaviside (step function): Discontinuous function used to detect halting conditions in the loop cell. "heaviside → cumulative monotone halt → soft-mux state freeze"
  • HNSW index: Hierarchical Navigable Small World graph for fast approximate nearest-neighbor search. "implemented as an embedded vector database with an HNSW index"
  • hypervectors: High-dimensional vectors designed for VSA operations with specific statistical properties. "hypervectors drawn from a controlled random distribution"
  • knowledge compilation: Transforming logical knowledge into tractable representations for efficient inference. "knowledge compilation in symbolic AI (Darwiche {paper_content} Marquis 2002)"
  • Kleene three-valued logic: Logic with true (+1), unknown (0), and false (−1) truth values and associated connectives. "Kleene’s three-valued truth tables"
  • Lagrange interpolation: Polynomial construction method used to create smooth, exact-on-grid fuzzy logic operators. "Lagrange-interpolating each connective as a polynomial that is exact on the 3×33\times 3 Kleene grid"
  • MAP-VSA: A VSA scheme where binding is elementwise multiplication (Multiply/Add/Permute variants); here referring to elementwise product. "Hadamard binding is the textbook elementwise product (MAP-VSA)."
  • partial evaluation: Compile-time evaluation/specialization of program parts with known inputs. "partial-evaluation systems that specialize programs at compile time"
  • QR decomposition: Matrix factorization into orthogonal and upper-triangular components used to build rotations. "R_r = \mathrm{QR}(\mathrm{hash}(r)).Q"
  • rational-weight RNN: Recurrent neural network model with rational weights, known for specific computational properties. "the rational-weight RNN form whose computational power Siegelmann {paper_content} Sontag (1992) characterized."
  • rotation binding: Binding implemented as multiplication by a role-specific orthogonal rotation matrix. "Rotation binding works: each role gets a Haar-random orthogonal R_role seeded by hash(role)"
  • soft-halt RNN cell: Recurrent cell that accumulates a differentiable halt signal to freeze state without hard branching. "compiles to a soft-halt RNN cell with substrate-pure halt detection"
  • soft-mux: Differentiable conditional implemented as a soft mixture between branches. "conditionals lower to soft-mux polynomials"
  • substrate-pure tensor-op graph: A computation graph composed solely of tensor operations on the embedding substrate, with no host-side logic. "beta-reduces to a substrate-pure tensor-op graph"
  • tail recursion: Recursion where the final action is the recursive call, enabling loop-like compilation. "Tail recursion as the loop primitive."
  • t-norm losses: Loss formulations derived from t-norm fuzzy logic operators for learning logical constraints. "first-order logic compiled to t-norm losses"
  • Vector Symbolic Architectures (VSA): Algebraic frameworks for representing and manipulating symbols as high-dimensional vectors. "VSA is a family of algebraic frameworks for computing with high- dimensional vectors"
  • vmap (vectorized map): Transformation that vectorizes a function across a batch dimension, reusing the same compiled ops. "torch.vmap --- a transform that runs the same compiled ops with a batch axis"
  • weighted similarity (cosine gain): Scaling cosine similarity by a learned gain to counteract embedding anisotropy. "A weighted variant additionally trains a scalar cosine gain and writes it back into the .su source"
  • representation-degeneration effect: Phenomenon where learned embeddings collapse into a narrow cone, reducing expressivity. "the representation-degeneration effect; Gao et al. 2019, Ethayarajh 2019"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 157 likes about this paper.

Sign Up for Free