SwiftMem: Query-Aware Memory System

• SwiftMem is a query-aware agentic memory system that delivers real-time, scalable retrieval for LLM agents using a multi-tiered indexing approach.
• It leverages temporal indexing and a semantic DAG-tag structure to drastically reduce retrieval complexity and improve cache locality through embedding co-consolidation.
• Performance evaluations on benchmarks like LoCoMo and LongMemEval_S show SwiftMem achieves orders-of-magnitude speedup while maintaining competitive retrieval accuracy.

SwiftMem is a query-aware agentic memory system designed for LLM agents to enable real-time, scalable retrieval of relevant past context and episodic information. It addresses the core bottleneck of existing agentic memory frameworks—namely, exhaustive filtering or full similarity search across all stored memory, which incurs $O(N_{\text{mem}})$ latency as the memory store grows. By introducing a multi-tiered index architecture that exploits both temporal and semantic locality, SwiftMem achieves provably sub-linear retrieval complexity while maintaining competitive retrieval accuracy on established benchmarks. The system also incorporates co-consolidation of embeddings based on tag-driven clustering to enhance hardware cache locality during similarity search. Performance evaluations demonstrate orders-of-magnitude speedup versus previous state-of-the-art systems, with minimal sacrifice in retrieval quality (Tian et al., 13 Jan 2026).

1. System Architecture and Motivation

The primary constraint in prior agentic memory systems is the linear search complexity across the memory corpus, leading to prohibitive search times (800 ms to multi-seconds with growing history) unsuited for real-time LLM agent interaction. Empirical observations show that queries exhibit strong temporal and semantic locality: most queries refer to recent or topically clustered episodes.

SwiftMem is engineered to leverage these observations with three principal design goals:

• Sub-linear retrieval complexity via index structures over both temporal and semantic axes.
• High retrieval quality, measured by LLM-judged semantic relevance and lexical overlap.
• Robust support for dynamism and growth via periodic memory reorganization.

The core indexing pipeline is organized into three tiers:

1. Temporal Index: Restricts queries to relevant time intervals in $O(\log N_{\text{mem}})$.
2. Semantic DAG-Tag Index: Routes queries through a hierarchical tag structure, exploiting semantic locality in $O(k \cdot (\log |V| + D_{\max}))$.
3. Embedding Index with Co-consolidation: Performs similarity search only within semantically and temporally filtered candidates, improving cache locality.

2. Temporal Indexing

The temporal index is defined as $\mathcal{T} = \{\mathcal{L}_u, \mathcal{M}\}$, where $\mathcal{L}_u$ is a sorted list of $(t_i, e_i)$ tuples for each user $u$ and $\mathcal{M}$ maps episode identifiers to user and timestamp metadata.

Temporal range queries are answered in $O(\log N_{\text{mem}})$ via binary search:

1. Locate lower and upper bounds for the interval $[t_{\text{low}}, t_{\text{high}}]$.
2. Return the set of episodes within the interval.

Insertion and maintenance are similarly efficient, supporting both single and multi-interval queries. This temporal layer ensures that time-sensitive queries are resolved without inspecting the entire memory, and it provides the first and often most significant reduction in candidate set size.

3. Semantic DAG-Tag Index

3.1 LLM-based Tag Generation

For each memory episode, an LLM is prompted to extract 3–8 normalized tags (multi-word, lowercase, with underscore separation) and parent-child relations, forming a directed acyclic graph (DAG) over the tags. If the LLM pipeline fails, fallback embedding-based keyword extraction is employed.

3.2 DAG-Tag Data Structure

The DAG $G = (V, E)$ assigns each tag node $v \in V$ with attributes $$(t, \mathcal{E}, \mathcal{P}, \mathcal{C}, \mathbf{e})$$:

• $t$: tag text
• $\mathcal{E}$: associated episodes
• $\mathcal{P}, \mathcal{C}$: parents and children
• $\mathbf{e} \in \mathbb{R}^d$: embedding vector

A specificity monotonicity theorem holds: along any path in the DAG, specificity $\mathcal{S}(v)$ strictly increases with depth.

3.3 Query-Tag Routing

Query processing follows these steps:

1. Embed the query $q$ to obtain $\mathbf{e}_q$.
2. Compute cosine similarity $s(q, t)$ to all tag embeddings.
3. Select top-$k$ tags maximizing $\sum s(q, t)$.
4. Expand each top tag through the DAG up to depth $D_{\max}$ to retrieve related tags and their associated episodes.

The complexity is $O(k (\log|V| + D_{\max}))$, with $|V|$ tags and expansion to $D_{\max}$ depth.

3.4 DAG Management

Tag insertion updates the DAG for every new episode in $O(\#\text{tags} \cdot \log|V|)$ time. Query retrieval aggregates episodes associated with each relevant tag in $O(k (\log|V| + D_{\max}) + |E_{\text{cand}}|)$.

4. Embedding-Tag Co-consolidation

4.1 Semantic Tag Clustering

To improve hardware cache locality in the final similarity search, SwiftMem periodically clusters the tag DAG based on connection patterns (DAG connectivity, episode co-occurrence, and connected components), forming clusters $C_i = (I_i, V_i, t_i, s_i)$:

• $I_i$: cluster ID
• $V_i \subseteq V$: cluster members
• $t_i$: centroid tag
• $s_i$: cohesion score (in $[0,1]$), e.g., $$s_i = \frac{\#\text{actual edges}}{|V_i|(|V_i|-1)/2}$$

4.2 Co-consolidation Procedure

A physical layout map $L$ records memory offsets for embedding blocks such that tag-clustered embeddings are contiguous. Consolidation is triggered when measured fragmentation or low cohesion indicates suboptimal cache use. The process is linear in index size ($O(|V| + \text{total episodes})$ per pass).

5. Indexing and Retrieval Algorithms

5.1 New Episode Indexing

Upon receipt of a new episode $e = (u, m, t, \vec{x})$:

1. Insert into the temporal index: $O(\log N_{\text{mem}})$.
2. Generate tags and relations via LLM: $O(1)$ LLM cost.
3. Update the DAG for each tag: $O(\#\text{tags} \cdot \log|V|)$.
4. Insert embedding: amortized $O(1)$ or $O(\log N_{\text{mem}})$.

Total indexing cost is $$O(\log N_{\text{mem}} + \#\text{tags} \cdot \log|V|)$$.

5.2 Query Retrieval

For a query $q$ and desired $k_q$ results:

1. Extract explicit time window if specified, restricting to $E_{\text{temp}}$ in $O(\log N_{\text{mem}} + |E_{\text{temp}}|)$.
2. Route via DAG-tag index for semantic candidates $E_{\text{semi}}$.
3. Candidate episodes $$E_{\text{cand}} = E_{\text{temp}} \cap E_{\text{semi}}$$.
4. Similarity search over $E_{\text{cand}}$: $O(|E_{\text{cand}}| \cdot \log N_{\text{mem}})$.
5. Return top-$k_q$.

Overall retrieval complexity is

$$O(\log N_{\text{mem}} + k(\log|V|+D_{\max}) + |E_{\text{cand}}| \cdot \log N_{\text{mem}})$$

which is sub-linear in corpus size due to aggressive candidate reduction.

6. Empirical Evaluation

SwiftMem's efficacy is demonstrated on the LoCoMo (10 dialogues, $\sim$24K tokens/dialogue, 1,540 queries) and LongMemEval_S (500 dialogues, $\sim$105K tokens/dialogue) benchmarks. Key metrics include LLM-Judge Score (GPT-4.1-mini), F1, BLEU-1, and search latency.

Performance Comparisons (LoCoMo, GPT-4.1-mini)

Method	LLM-Score	Search latency (ms)	Total (ms)
FullContext	0.723	–	5,806
LangMem	0.513	19,829	22,082
Mem0	0.613	784	3,539
RAG-4096	0.302	544	2,884
Zep	0.585	522	3,255
Nemori	0.721	835	3,448
SwiftMem	0.652	11	1,289

SwiftMem achieves a $47\times$ search speedup over Zep (522 ms → 11 ms) and $76\times$ over Nemori (835 ms → 11 ms), with $2.2\times$–$4.5\times$ lower total latency compared to RAG-4096 and FullContext, respectively.

Retrieval Quality

Method	LLM-Score	F1	BLEU-1
Nemori	0.792	0.519	0.445
SwiftMem	0.704	0.429	0.467

SwiftMem displays a minor reduction in semantic alignment relative to Nemori, but increases lexical precision (BLEU-1).

7. Trade-offs, Limitations, and Prospective Extensions

Trade-offs in SwiftMem's design include increased index maintenance overhead—LLM-driven tag generation and DAG updates amortize as writes accrue. The space requirement for multi-dimensional indices (tag embeddings, pointers, timelines) is higher than brute-force baselines.

Limitations arise from dependency on LLM-generated tag quality, which, if insufficiently granular or accurate, can diminish semantic routing efficacy. Fixed $k$ and $D_{\max}$ parameters may not optimize performance across all query distributions. Co-consolidation relies on effective scheduling; poor timing can yield suboptimal clustering.

Potential extensions include:

• Adaptive $k$ and $D_{\max}$ per query via uncertainty or importance estimation.
• Approximate nearest neighbor structures over tags to further minimize $|V|$ dependence.
• Hierarchical temporal trees for finer-grained time-based queries.
• Online DAG pruning/merging to control tag explosion at scale.
• Unified indices supporting heterogeneous memory types (procedural, resource).

SwiftMem's three-tier indexing (temporal, semantic, embedding) with periodic co-consolidation produces provable sub-linear retrieval and large practical speedups, with competitive accuracy on long-context evaluation settings (Tian et al., 13 Jan 2026).