Neural Computation Without Slots: Steps Towards Biologically Plausible Memory and Attention in Natural and Artificial Intelligence

Abstract: Many models used in artificial intelligence and cognitive science rely on multi-element patterns stored in "slots" - dedicated storage locations - in a digital computer. As biological brains likely lack slots, we consider how they might achieve similar functional outcomes without them by building on the neurally-inspired modern Hopfield network (MHN; Krotov & Hopfield, 2021), which stores patterns in the connection weights of an individual neuron. We propose extensions of this approach to increase its biological plausibility as a model of memory and to capture an important advantage of slot-based computation in contemporary LLMs. For memory, neuroscience research suggests that the weights of overlapping sparse ensembles of neurons, rather than a dedicated individual neuron, are used to store a memory. We introduce the K-winner MHN, extending the approach to ensembles, and find that within a continual learning regime, the ensemble-based MHN exhibits greater retention of older memories, as measured by the graded sensitivity measure d', than a standard (one-neuron) MHN. Next, we consider the powerful use of slot-based memory in contemporary LLMs. These models use slots to store long sequences of past inputs and their learned encodings, supporting later predictions and allowing error signals to be transported backward in time to adjust weights underlying the learned encodings of these past inputs. Inspired by these models' successes, we show how the MHN can be extended to capture both of these important functional outcomes. Collectively, our modeling approaches constitute steps towards understanding how biologically plausible mechanisms can support computations that have enabled AI systems to capture human-like abilities that no prior models have been able to achieve.

• The paper's main contribution is the introduction of a K-winner MHN that uses a Hebbian update to achieve slot-free, distributed memory encoding with competitive retention.
• It demonstrates that distributed, sparse ensembles outperform traditional slot-based strategies by retaining older patterns more effectively while managing interference.
• The study extends MHN principles to transformer architectures, providing viable models for slot-free attention and continual learning.

Neural Computation Without Slots: Towards Biologically Plausible Memory and Attention

Introduction

The paper "Neural Computation Without Slots: Steps Towards Biologically Plausible Memory and Attention in Natural and Artificial Intelligence" systematically investigates how high-fidelity slot-based computation—central to digital architectures and LLMs—can be reinterpreted through mechanisms viable for biological brains. The work is motivated by limitations in slot-based representations: they assume a one-to-one mapping between items and dedicated neural resources, which is absent in neurobiology. The core contribution is to establish that distributed, slot-free mechanisms in neural networks, specifically through extensions to the Modern Hopfield Network (MHN), can approach or match the functionality of slot-centric systems, including continual memory formation and attention-driven credit assignment.

Slot-Based versus Distributed Memory

The canonical MHN stores each pattern using dedicated weights for an individual "memory" neuron, mimicking slot allocators in digital memory (Figure 1).

Figure 1: Equivalence of softmax-weighted associative memory retrieval and MHN-based retrieval—auto-associative MHN (A) vs. a hetero-associative MHN that mirrors transformer self-attention (B).

Although effective for auto-association and pattern completion, this paradigm fails to capture the distributed, overlapping ensembles observed in hippocampal coding. MHN’s localist mapping is contrasted against neuroscientific evidence favoring sparse, distributed encoding—where any one neuron participates in multiple memories, and memories are stored in graded fashion across many synapses.

The paper introduces the K-winner MHN, a sparse, distributed architecture where each input is encoded by updating the weights of the $K$ most responsive hidden units with a scalable Hebbian rule:

$$W_{ij} \leftarrow W_{ij} + \epsilon (x_j - W_{ij})z_iF_{ij}$$

with $\epsilon$ controlling learning rate, $z$ the k-winner-take-all response, and $F$ the synaptic mask enforcing sparsity. This modification achieves both finite capacity (resource-limited storage) and distributed, ensemble-based encoding.

Figure 2: Comparison between original MHN (localist), 1-winner MHN (slot replacement), and the general K-winner MHN, which enables ensemble encoding and sparse connectivity.

Memory Retention and Retrieval: Quantitative Analyses

Empirical evaluations reveal that the K-winner MHN consistently achieves superior retention of older patterns compared to the original, hard slot-replacement MHN—a gain measured both in raw retrieval rates and in signal detection sensitivity ($d'$).

Figure 3: Retrieval performance and $d'$ sensitivity for K-winner vs. original MHN as a function of memory age, under both perfect cues (left) and partial cues (right).

Memory retention curves exhibit slower exponential decay for the distributed K-winner model:

$\text{R.D.}(a) = C\, e^{-\beta (a-1)}$

where C and $\beta$ are network-dependent parameters. The trade-off between initial retrieval fidelity and longevity of stored patterns is characterized by manipulating fan-in $f$ and $K$: higher connectivity boosts immediate recall but increases interference, shortening retention span.

Figure 4: Retrieval accuracy and sensitivity for K-winner MHN with increased fan-in versus original MHN—performance tradeoffs for recent and older memories.

Further, with structured input distributions (e.g., tree-generating Chinese Restaurant Process, TGCRP), the K-winner model shows marked improvement over the original MHN in high-similarity, hierarchical datasets—mirroring robust human episodic memory generalization.

Figure 5: Sensitivity ($d'$) of K-winner vs. original MHN under hierarchically structured patterns for varying degrees of similarity.

Slot-Free Attention Mechanisms: MHN-Based Transformers

Extending distributed, slot-free concepts to temporal credit assignment and contextual attention, the paper develops slot-free analogues to transformer self-attention. Slot-based transformers retain past keys/values explicitly from long input sequences, essential for temporal error backpropagation.

Figure 6: Backpropagation through slot aggregation in transformers parallels backpropagation through time in recurrent architectures, highlighting the need for efficient, slot-free gradient transport mechanisms.

The proposed MHN-based transformer replaces direct slot storage by storing keys and values sequentially in fast weights, continuously refreshed with Hebbian updates. Several training variants are explored:

• Fixed $W_K$ MHN-Transformer: keys are static; queries and values are trained via backpropagation.
• MHN-Transformer: keys are trained via output-prediction loss, backpropagated through MHN retrieval.
• QK-MHN-Transformer: keys are aligned to queries directly using supervised gradient updates.

Adding input projection weights ensures robust distinctiveness of stored items, preventing memory overwriting and improving recall.

Figure 7: MHN-based transformer architecture—encoding item, key, and value representations into fast weights, facilitating slot-free credit assignment.

Transformer Task Performance and Emergent Representations

On a controlled Case Sequence Task, MHN-based transformers approach the accuracy (~100%) of baseline slot-based transformers when equipped with input projections and using query-key alignment for training (QK-MHN-Transformer).

Figure 8: Summary of model accuracy on the Case Sequence Task—QK-MHN-Transformer with input projections matches baseline transformer performance.

Analysis of learned matrices ($W_V$, $W_K$, $W_Q$) illustrates that MHN-based models develop semantically aligned query, key, and value representations, encoding abstract letter identities and associated cases.

Figure 9: Single-trial results from QK-MHN-Transformer—batch accuracy, $W_V$ encoding, query-key correlation ($W_Q^T W_K$), and key-key alignment ($W_K^T W_K$).

Empirically, both slot-based and MHN-based transformers first learn case-value mappings before developing aligned query-key structures, with the QK-alignment model exhibiting staged learning dynamics.

Figure 10: Learning dynamics in QK-MHN-Transformer—accuracy, loss, and emergence of semantically aligned representations during training.

Theoretical and Practical Implications

The findings demonstrate that distributed, slot-free memory and attention mechanisms can replicate and sometimes surpass the retention and generalization properties of slot-based architectures, provided appropriate trade-offs in connectivity and update strategy. Theoretically, this challenges the necessity of slot-based abstraction at the hardware/neural level; biologically plausible distributed systems can implement both working and episodic memory, credit assignment, and associative recall. Practically, this opens new avenues for designing artificial memory systems and neural architectures with robust continual learning and efficient resource utilization, using graded updates and sparse ensembles rather than explicit slot addressability.

For AI system implementation:

• Memory models: Multilayer Hopfield variants with configurable $K$, $f$, and $\epsilon$ allow tight control of accuracy-retention trade-offs, useful for long-term storage in continual learning.
• Attention architectures: Transformer analogues using dynamic, fast Hebbian weights can drastically reduce active resource needs, facilitating scalable deployment in neuromorphic or memory-constrained settings.

Future Directions

The paper specifies multiple areas for further investigation:

• Analytical characterization of parameter regimes yielding optimal retention versus generalization,
• Integration of preprocessing modules (e.g., stacked K-winner layers) for pattern separation,
• Biological mapping to hippocampal/dentate gyrus connections, paralleling introduced input projection matrices,
• Unification of long-term and context memory within slot-free distributed networks.

Exploration of these directions may yield foundational enhancements to both artificial and biological memory and attention modeling.

Conclusion

This work rigorously assesses the feasibility of biologically plausible, slot-free neural computation for memory and attention. Through theoretical analysis and empirical validation, it establishes that distributed MHN extensions and slot-free transformers are competitive with traditional slot-based counterparts, with distinct advantages in continual learning and memory retention. These results offer a robust conceptual and practical bridge between neuroscience and next-generation AI systems, demonstrating that “neural computation without slots” is not only viable, but often preferable.