Infini-gram Search Engine

• Infini-gram Search Engine is a high-performance system for exact n-gram and ∞-gram language modeling, utilizing suffix arrays and FM-index structures.
• It processes trillions of tokens efficiently, supporting precise probability evaluation and rapid substring queries for text analysis and neural LM augmentation.
• The architecture combines rigorous probabilistic frameworks with scalable indexing and compression techniques to enable effective data curation and anomaly detection.

Infini-gram Search Engine refers to a family of high-performance, large-scale systems for exact $n$-gram and $\infty$-gram language modeling and search, centered on the Infini-gram and Infini-gram mini engines. These systems modernize classical $n$-gram modeling to process up to trillions of tokens, supporting both efficient statistical language modeling (including arbitrarily unbounded $n$ contexts) and Internet-scale exact substring search. The Infini-gram architecture builds on full-text suffix arrays, while Infini-gram mini employs FM-index–based compression and search to extend practical scale to tens of terabytes and beyond. Both engines are motivated by the need for transparent text analysis, data curation, and neural LLM augmentation at previously infeasible scale, combining rigorous probabilistic frameworks with algorithmic innovations for efficient querying, storage, and deployment.

1. Core Data Structures and Algorithms

Infini-gram: Suffix Array Engine

The Infini-gram engine represents the tokenized corpus (5 trillion tokens) as a contiguous flat byte array, with each token assigned two bytes. All documents are delimited via a designated end-of-document marker (0xff 0xff). On this structure, a full suffix array (SA) is constructed—a dense array of length $N$ mapping each lexicographically ordered suffix to its byte offset within the corpus. The total storage is $7$ bytes per token: $2$ for the token array and $5$ for the suffix array pointer, resulting in a $35$ TB index for $5$ trillion tokens. Construction is parallelized: e.g., $1.4$ trillion tokens take $\sim 48$ hours on a $128$-CPU, $1$ TiB RAM cluster, while the full $5$T tokens index in $\sim 2$ days with $\sim 10$ TB SSD (Liu et al., 2024).

Given an $n$-gram $Q = x_1, ..., x_L$, retrieval is performed by binary search in the suffix array to identify interval $[\ell, r)$; query time is $O(L + \log N)$, and observed latency is dominated by $\sim \log N$ random-access disk operations, with count queries up to $|Q| = 1000$ feasible in $20$ ms even on $1.4$T tokens.

Infini-gram mini: FM-index Engine

Infini-gram mini compresses and indexes massive text collections using the FM-index, derived from the Burrows–Wheeler Transform (BWT) and sampled suffix/inverse suffix arrays. The input corpus $T$ (bytes of length $n$) is concatenated with unique separators, enabling document boundaries. The FM-index comprises:

• Sampled Suffix Array (SA/ISA): Stores only every $a$-th entry to reduce space; missing locations are recovered by iterating the LF-mapping.
• BWT with Huffman-Shaped Wavelet Tree: Stores the permuted text, compressed to $n H_0 + 2\sigma\log n$ bits, where $H_0$ is zeroth-order entropy (about $2.1$ bits/symbol).
• Operations: Primitive operations are find(Q) (SA interval for pattern $Q$ in $O(|Q| H_0)$ time), locate(i) (text location for a suffix, requiring $O(a H_0)$ LF steps), and `reconstruct(p,d)$(recover substring of length$d$at position$p$).</li> </ul> <p>Index files occupy only$0.44\times$the corpus size. For$46$TB of text, indexing completes in$50$days on a single node or$19$hours using$75$nodes in parallel; the index is$20.1$$TB (<a href="/papers/2506.12229" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Xu et al., 13 Jun 2025</a>).</p> <div class='overflow-x-auto max-w-full my-4'><table class='table border-collapse w-full' style='table-layout: fixed'><thead><tr> <th>Engine</th> <th>Index Structure</th> <th>Storage Overhead</th> <th>Search Latency</th> </tr> </thead><tbody><tr> <td>Infini-gram</td> <td>Full Suffix Array</td> <td>$$7$$bytes/token</td> <td>ms (length-dependent)</td> </tr> <tr> <td>Infini-gram mini</td> <td>FM-index (BWT+SA)</td> <td>$$0.44\times$$corpus</td> <td>s (text-length dependent)</td> </tr> </tbody></table></div><h2 class='paper-heading' id='probability-models-and-query-semantics'>2. Probability Models and Query Semantics</h2><h3 class='paper-heading' id='classical-and-44-gram-backoff'>Classical and$$\infty$-gram Backoff</h3> <p>Traditional$n$$-gram backoff models (e.g., Katz) interpolate between fixed$$n$and lower-order models. The$\infty$$-gram model, enabled by the Infini-gram engine, defines probability as:</p> <p>$$P_\infty(w_i \mid w_{1...i-1}) = \frac{\text{cnt}(w_{i-n+1...i-1} \circ w_i)}{\text{cnt}(w_{i-n+1...i-1})}$</p> <p>where$n$is maximal such that$\text{cnt}(w_{i-n'+1...i-1}) > 0$for$n' \le i$$. Backoff is invoked only when context is unseen; smoothing weights$$\alpha$$are unnecessary, ensuring immediate normalization:$$\sum_w P_\infty(w \mid h) = 1$$. For suffix array–based engines, this computes exactly via interval counts; for FM-index structures, the identical logic is applied using recursive pattern finding over compressed substrings.</p> <h3 class='paper-heading' id='query-api-and-interfaces'>Query API and Interfaces</h3> <p>Infini-gram exposes a REST JSON <a href="https://www.emergentmind.com/topics/geospatial-application-programming-interface-api" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">API</a>, supporting pointwise probability evaluation as well as full next-token distributions; example request for an$$\infty$-gram probability:</p> <p>

  POST /api/query { "type": "infgram_prob", "context_ids": [14, 502, 17, ... ], "token_id": 137 }

  </p> <p>Response includes counts, normalized probability, and &quot;effective n&quot; (context length used). Batch distribution queries return top-$K$next-token probabilities, with latency for$\infty$-gram distributions$\lesssim$ 200 ms on large-scale index shards.

  3. System Architecture and Engineering

  Infini-gram

  • Sharding and Parallelism: Both indexing and querying are performed in parallel, utilizing sharded suffix arrays across multiple disks and compute nodes for I/O efficiency.
  • Disk Layout: Token and suffix arrays reside on SSDs; queries are memory-mapped, minimizing in-RAM requirements.
  • Latency: On $1.4$T tokens and$8$-file shards, exemplary latencies are—count($n$-gram):$18$ms;$5$-gram distribution:$39$ms;$\infty$-gram probability (single token):$135$ms;$\infty$-gram distribution:$180$$ms.</li> </ul> <h3 class='paper-heading' id='infini-gram-mini'>Infini-gram mini</h3> <ul> <li><strong>Petabyte Scale Design:</strong> Text is sharded into$$\sim 600$–$700$$GB chunks. Construction is split into: (1) SA+BWT; (2) symbol counting; (3) wavelet tree construction; (4) SA sampling; (5) ISA sampling.</li> <li><strong>Engineering Optimizations:</strong> In-place streaming and multithreaded algorithms deliver$$18\times$speedup in construction,$3.2\times$$reduction in peak RAM. At inference time, indices are loaded in a memory-mapped, read-only fashion, ensuring sub-2 GB RAM use.</li> <li><strong>Distributed Query:</strong> Query performance (on GCP SSDs): single-token count ($$|Q|=1$) is$4$–$32$ms for$1$–$9$TB shards, but scales to several seconds for$|Q|=1000$$due to I/O for scattered compressed substrings.</li> </ul> <h2 class='paper-heading' id='empirical-performance-and-comparative-evaluation'>4. Empirical Performance and Comparative Evaluation</h2><h3 class='paper-heading' id='language-modeling'>Language Modeling</h3> <ul> <li><strong>Infini-gram ($$\infty$-gram LM):</strong> On held-out text,$5$-gram LM achieves$29\%$next-token accuracy;$\infty$-gram achieves$47\%$overall—rising above$75\%$for context$n \ge 16$and$80\%$$in sparse cases.</li> <li><strong>Neural LM Augmentation:</strong> Interpolating LLaMA-2$$13$B neural LM with$\infty$-gram reduces perplexity from$5.30$to$4.41$($-21\%$);$70$B LLaMA-2:$4.59$to$3.96$on$1.8$$T-token data, demonstrating nonparametric benefits (<a href="/papers/2401.17377" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Liu et al., 2024</a>).</li> </ul> <h3 class='paper-heading' id='benchmark-and-corpus-analysis'>Benchmark and Corpus Analysis</h3> <ul> <li><strong>Infini-gram mini Contamination Analysis:</strong> Benchmark contamination metric$$\eta$—fraction of overlapping$50$$-character substrings found in the training corpus—reveals “dirty” rates as high as$$27.7\%$(MMLU),$32.6\%$(ARC-Challenge), and$40.1\%$(SQuAD) on corpora up to$16.7$ TB. Most “dirty” entries are full Q&amp;A exact matches (up to $83\%$$).</li> <li><strong>Query Throughput:</strong> On medium shard sizes ($$\sim 1$–$9$TB), count queries for$|Q|=1$-$1000$take$4$ms–$25$s; document retrieval for spans up to$3000$bytes is$0.43$–$4.46$$s (<a href="/papers/2506.12229" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Xu et al., 13 Jun 2025</a>).</li> </ul> <h3 class='paper-heading' id='comparison-to-prior-systems'>Comparison to Prior Systems</h3><div class='overflow-x-auto max-w-full my-4'><table class='table border-collapse w-full' style='table-layout: fixed'><thead><tr> <th>System</th> <th>Indexable Tokens (approx.)</th> <th>Storage per Token or Corpus</th> <th>$$n$$Supported</th> <th>Query Latency</th> </tr> </thead><tbody><tr> <td>Google Books 5-gram</td> <td>$$5 \times 10^{11}$</td> <td>$24$GB for$5$-gram only</td> <td>$n=5$$</td> <td>—</td> </tr> <tr> <td>Suffix-tree LM</td> <td>$$9 \times 10^{9}$</td> <td>$63$GB RAM</td> <td>$n$$unbounded</td> <td>—</td> </tr> <tr> <td>Nearest-neighbor LM</td> <td>$$2.8 \times 10^{10}$</td> <td>$432$$TB (vector index)</td> <td>limited</td> <td>—</td> </tr> <tr> <td>Infini-gram</td> <td>$$5 \times 10^{12}$</td> <td>$35$TB</td> <td>$n=\infty$$</td> <td>ms</td> </tr> <tr> <td>Infini-gram mini</td> <td>$$4.6 \times 10^{13}$</td> <td>$20.1$TB ($0.44\times$)</td> <td>$n=\infty$$</td> <td>s</td> </tr> </tbody></table></div><h2 class='paper-heading' id='applications'>5. Applications</h2> <ul> <li><strong>Large-scale Text Analysis:</strong> Enables corpus inspection, such as quantifying$$n$$-gram frequencies and rare sequence identification.</li> <li><strong>Data Curation and Decontamination:</strong> Powering tools like SearchDoc for complex Boolean queries (CNF) over$$n$$-grams, supporting removal of toxic or sensitive material.</li> <li><strong>Neural LM Augmentation:</strong>$$\infty$$-gram model serves as a nonparametric memory, lowering neural model perplexity without GPU access.</li> <li><strong>Anomaly Detection:</strong> Agreement curve analysis between LMs and$$\infty$-gram models uncovers memorization and positional embedding artifacts in output from transformer-based models.
  • Web Interface and Programmatic API: Offers interactive and programmatic access for both general substring querying and benchmarking contamination rates (see https://infini-gram-mini.io/demo and https://api.infini-gram-mini.io) (Xu et al., 13 Jun 2025).

  6. Limitations and Future Directions

  Identified Limitations

  • Latency: Infini-gram (suffix array) supports millisecond latency; Infini-gram mini (FM-index) has seconds-level query time for long substrings or document retrieval due to decompression and scattered storage access.
  • Query Expressivity: Only exact, case-sensitive byte matching is supported; no semantic, fuzzy, or edit-distance–tolerant search.
  • Co-occurrence and Boolean Matching: Multi-pattern and co-occurrence search are inefficient, requiring serial location recovery in the suffix array.
  • Scalability: Petabyte-scale indexing is feasible but increases in random I/O and network sync demand further system optimizations.

  Prospective Improvements

  • Disk-page prefetching and batching of LF/rank operations to hide I/O latency.
  • Hybrid “hot index” caches for common substrings.
  • Support for multi-pattern and Boolean queries via auxiliary compressed posting lists.
  • Enhanced approximate/fuzzy search using generalized suffix automata or edit-distance–aware FM-index extensions.
  • Hardware offloading (GPU/FPGA) for intensive LF-mapping phases.

  7. Context and Significance in NLP

  Infini-gram and its successor Infini-gram mini mark a departure from traditional $n$-gram precomputation or neural-only retrieval by enabling exact, arbitrarily long pattern matching and corpus-scale likelihood estimation. This supports both theoretical investigation—such as revealing deficiencies in transformer positional encodings and benchmark contamination—and practical operations, including corpus curation and LLM debugging. The architectural shift to suffix and FM-index–based designs establishes new performance and scalability frontiers for nonparametric search engines and transparent statistical language modeling (Liu et al., 2024, Xu et al., 13 Jun 2025).