Scalable Text Classification

• Scalable text classification is a suite of techniques designed to assign categories to text data efficiently, even as document and class counts scale to millions.
• It leverages diverse methodologies—from fast linear baselines and neural fusion of embeddings to graph-based clustering—ensuring robust performance across diverse applications.
• Industrial and legal applications benefit through modular decompositions and domain-adaptive transformer fine-tuning, achieving significant speedups and improved precision.

Scalable text classification refers to algorithmic, architectural, and methodological innovations that enable efficient and accurate assignment of natural language inputs to categories, tags, or codes, even as the number of documents, classes, or features grows to millions or more. Recent developments span fast linear baselines, neural architectures, graph-based models, metaheuristic feature selection, embedding-centric workflows, multi-label legal and extreme classification, and resource-efficient inference leveraging conformal prediction and LLMs.

1. Algorithmic Foundations and Linear Models

Early approaches center on linear models over sparse features. FastText (Joulin et al., 2016) averages word and n-gram embeddings, applying a low-rank, fully linear classifier. The hashing trick fixes the dimension of feature vectors independently of vocabulary size or n-gram dictionary, while hierarchical softmax enables both training and inference in $O(h\log k)$ with $k$ classes. FastText achieves competitive accuracy to state-of-the-art CNNs for sentiment/tag tasks and processes $>10^6$ classes and $>10^9$ words orders of magnitude faster.

Sparse generative models such as multinomial Naive Bayes with inverted indices reduce per-document scoring to the nonzero features and their postings lists over classes (Puurula, 2016). Hierarchical and interpolated smoothing (Dirichlet, Jelinek-Mercer, and Pitman–Yor) further enhance performance; scalable MapReduce implementations enable linear scaling in documents and classes and order-of-magnitude efficiency over SVMs in the million-class regime.

2. Neural Feature Representations and Fusion

Distributed representations obtained by pooling skip-gram word embeddings (sum, average, max, concat) yield dense document vectors (Balikas et al., 2016). Fusing these with $\ell_2$-normalized TF-IDF vectors in a joint space boosts micro-F1 scores by $+2$–$3$ points on PubMed multi-label classification benchmarks (up to $10^4$ classes), with linear complexity in document and label counts. Feature extraction and SVM or logistic training scale well on CPU, and storing document embeddings on disk reduces training overhead.

In extreme multi-label setups, hybrid architectures combine semantic label clustering, deep contextual embeddings (BERT, XLNet, RoBERTa), and candidate pruning. X-Transformer (Chang et al., 2019) performs hierarchical label clustering and reduces the output head to $K\ll L$ dimensions. Fine-tuned transformer embeddings are matched to clusters, after which lightweight OVA rankers select from candidate labels. This pipeline maintains GPU feasibility for $L\sim 10^5$–$10^6$ and yields new state-of-the-art precision@1.

3. Graph-Based and Structured Methods

Graph neural networks (GNNs) address scalability and long tail issues in extreme multi-label classification (Zong et al., 2020, Ai et al., 2024). Label co-occurrence graphs are constructed and filtered, with labels grouped into semantic clusters by graph convolution with low-pass filters and mini-batch k-means (nonparametric optimization). Bilateral GIN branches (one with uniform, one with rebalanced sampling) decouple representation learning for major and tail labels, with adaptive fusion during training. At inference, only clusters relevant to the input are scored, and the final classifier is logistic regression per label within the selected cluster, yielding superior tail metrics and linear/sub-linear scaling in $L$.

Graph contrastive learning methods introduce cluster-refined negative sampling to avoid false negatives and over-clustering (Ai et al., 2024). Document graphs built from BERT representations, sparse PMI, and TF-IDF features are processed by GCNs. Clustering yields pseudo-labels, and negatives are sampled only across clusters. A self-correction mechanism identifies distant same-cluster nodes as additional negatives, further improving semi-supervised classification accuracy—demonstrated by consistent gains of $+0.4\%$–$1.3\%$ over prior GCL baselines on 20NG, R8, R52, Ohsumed, and MR.

Hierarchical taxonomy-aware capsule networks embed both document graphs and class taxonomies (Peng et al., 2019). Word-order-preserved graph-of-words are processed by attentive RCNN layers and capsule networks with dynamic routing. Taxonomy embeddings (random walks + skipgram) capture hierarchical label relations, and loss functions incorporate semantic label similarity for weighted margin optimization, yielding best Macro-F1/Micro-F1 on RCV1 and EUR-Lex versus flat, CNN/RNN, and SVM baselines.

4. Extreme Multi-Label Learning and Industrial Applications

Scaling to millions of classes and billions of samples has motivated modular decompositions (e.g., DeepXML/Astec (Dahiya et al., 2021)). Four sub-tasks—surrogate representation learning via residual connections and surrogate clustering, hard-negative mining via dual-graph HNSW indexes, feature transfer via residual blocks with spectral constraints, and final OVA classification on hard negatives—reduce training time and memory to $O(ND\log L)$ and support real-time, millisecond-scale inference per query. On Bing’s production datasets ($L$ up to $62$M), Astec delivers $5$–$30\times$ training speedup and significant real-world gains in revenue, click-through-rate, and coverage.

Concurrently, domain-adaptive transformer fine-tuning methods have established state-of-the-art for legal classification with thousands of labels (Shaheen et al., 2020). ULMFiT-style training (gradual unfreezing, discriminative layer-wise LRs, slanted triangular schedule), domain-specific language modeling, and iterative stratification for imbalanced multi-label corpora collectively yield micro-F1 $0.661$/$0.754$ for JRC-Acquis/EURLEX57K, surpassing SVM and BiGRU-attention baselines.

5. Metaheuristics and Feature Selection for High-Dimensional Data

Feature selection is critical when dimensionality is large ($>10^4$ features). Migrating Birds Optimization (MBO) with Naive Bayes (Kaya et al., 2024) is a population-based wrapper approach: after IG-based filtering ($96\%$ reduction), a flock of binary masks is refined via neighborhood flips and fitness (cross-validation accuracy). Reordering by best fitness maintains diversity, while parallelization of fitness computation is practical. Compared to IG+NB and PSO+NB, MBO-NB achieves $3$–$7\%$ higher accuracy and drastically reduced feature set size.

Unsupervised and semi-supervised extensions, such as mixed-topic link models (PMTLM, degree-corrected PMTLM-DC) (Zhu et al., 2013), integrate text and linkage for joint topic clustering and link prediction via scalable EM. Content and link likelihoods are co-optimized, and local search refines hard labels, with empirical state-of-the-art performance and $5$–$10\times$ speedups relative to variational RTM.

6. Embedding-Based and Conformal In-Context Classification

Few-shot, embedding-based classification is increasingly adopted for qualitative survey coding (Mjaaland et al., 27 Aug 2025). Category centroids from handfuls of support examples anchor the embedding space, with cosine similarity yielding assignment. Fine-tuning via contrastive losses (InfoNCE, margin-based) reshapes the space, especially for diffuse categories (“Other”), and Cohen’s Kappa $0.74$–$0.83$ closely matches human agreement. Computational cost is dominated by embedding extraction; classification scales linearly and batch embedding enables GPU parallelism.

Conformal In-Context Learning (CICLe) (Pantelidis et al., 5 Dec 2025) integrates a lightweight base classifier (Logistic + TF-IDF), conformal prediction for label space pruning, and LLM prompting for adaptive inference. Only the conformal set of candidate labels is presented to the LLM, reducing number of shots and prompt length by up to $34.45\%$ and $25.16\%$. Macro-F1 is competitive or superior to few-shot LLM baselines, especially for imbalanced tasks and resource-constrained settings; CICLe remains practical as class count or LLM size increases.

7. Evaluation, Limitations, and Best Practices

Empirical evaluation spans micro/macro-F1, Cohen’s Kappa, precision@k, nDCG@k, PSP for tail-sensitivity, and runtime/memory benchmarks. Typical findings: linear models and generative classifiers yield competitive accuracy in sparse high-class cardinality settings; neural and graph approaches dominate tail labels and overall metrics at extreme scale; metaheuristics boost efficiency for feature-rich data.

Limitations include:

• Reliance on well-curated pseudo-labels in clustering and embedding approaches.
• Computational and memory bottlenecks for fitness evaluation in wrapper methods.
• Exposure bias and inference inefficiency for rare/zero-shot labels in extreme multi-label settings.
• Sensitivity to clustering hyperparameters and graph construction heuristics.

Best practices entail hybrid feature engineering, batch-wise parallelism, label and negative mining for candidate pruning, adaptive loss and optimization regimes, stratified splitting for imbalanced data, and transparent sharing of code, centroids, and checkpoints for reproducibility.

Scalable text classification is a rapidly evolving field combining statistical, neural, metaheuristic, and structural innovations to enable automated, interpretable, and efficient categorization over vast, heterogeneous text corpora. The interplay of linear/sparse baselines, deep architectures, graph models, and resource-aware workflows ensures that new deployments can match the scale and diversity of modern information systems.