Cross-Script Handwriting Retrieval

• Cross-script handwriting retrieval is the automatic matching of handwritten word images across distinct scripts by mapping visual and textual inputs into a unified embedding space.
• Recent advancements employ explicit zone-wise segmentation and lightweight dual encoders to overcome handwriting variability and cross-lingual semantic gaps.
• Experimental results show hybrid models achieving up to 82.8% average accuracy, highlighting potential improvements through deep feature adaptation and refined mapping techniques.

Cross-script handwriting retrieval refers to the automatic retrieval of handwritten word images across different writing systems, where the query and target scripts may belong to distinct linguistic, phonetic, or visual domains. This capability is critical for digital archives and linguistic research, as it enables search and semantic linking in collections containing multiple scripts with limited annotated resources. Recent methodologies leverage cross-lingual embedding models and explicit zone-segmentation techniques to bridge visual and semantic gaps shaped by script variability, cursive writing, and resource constraints (Bhunia et al., 2017, &&&1&&&).

1. Core Principles and Problem Formulation

Cross-script handwriting retrieval operates on multilingual datasets $D = \{ (x_i, t_i, y_i, l_i) \}_{i=1}^N$, where $x_i$ denotes a cropped handwritten word image, $t_i$ its transcription, $y_i$ a semantic class identifier (language-agnostic), and $l_i$ the script label. The goal is to learn embedding functions $f_v$ (visual) and $f_t$ (textual) mapping inputs to a common $d$-dimensional normalized space $V$, s.t. corresponding words reside near each other independent of script. Retrieval is performed via nearest-neighbor search in $V$, where queries in script A (e.g., English) retrieve visually disparate words in script B (e.g., Chinese or Indic) (Chen et al., 16 Jan 2026).

This formulation directly addresses two challenges:

• Handwriting variability: Inter-writer differences, elastic distortions, and cursive overlaps generate considerable intra-script and cross-script variance.
• Cross-lingual semantic gap: Words encoding similar meanings can have entirely dissimilar glyph structures, making purely visual matching ineffective and OCR-based search pipelines vulnerable to cascading recognition errors.

2. Methodological Frameworks

Cross-script retrieval frameworks employ either explicit zone/character mapping or latent embedding models.

2.1 Zone-Wise Segmentation and Mapping

For scripts such as Bangla, Devanagari, and Gurumukhi, robust retrieval is facilitated by segmenting word images into three zones: upper (vowel marks above the “Matra” headline), middle (base consonants/conjuncts), and lower (vowel marks below). Each zone is processed independently:

• Middle zone PHOG features (168-dim) extracted over sliding windows, modeled with HMMs (8 states, 32 Gaussians per state).
• Upper/lower modifiers identified by skeleton analysis, mapped using RBF-kernel SVMs.

Source-target component mapping is achieved by majority-voting classifiers over isolated target samples, producing per-zone lookup tables (LUTs) (Bhunia et al., 2017).

2.2 Language-Agnostic Embeddings

Recent approaches introduce lightweight asymmetric dual-encoders:

• Text encoder: Pre-trained DistilBERT (66M parameters, lower layers frozen), 2-layer MLP to obtain $z_i$ (128-dim, $\|z_i\|_2=1$).
• Visual encoder: MobileNetV3-Small backbone (~1.2M parameters) with 2-layer MLP, yielding $v_i$ (128-dim, $\|v_i\|_2=1$).

Instance-level contrastive loss (InfoNCE) and class-level semantic consistency loss jointly align embeddings across scripts and modalities, abstracting away writer and script variation (Chen et al., 16 Jan 2026).

3. Mathematical Formulation

Modern embedding-based approaches formalize objectives as follows:

• Instance-level alignment: Symmetric contrastive loss

$L_{ITC} = \frac{1}{2}(L_{V2T} + L_{T2V})$

with

$$L_{V2T} = -\frac{1}{N}\sum_i \log \frac{\exp(v_i^\top z_i/\tau)}{\sum_j \exp(v_i^\top z_j/\tau)}$$

and analogous $L_{T2V}$, where $\tau$ is a learnable temperature.

• Semantic consistency alignment: For omni-modal batch $H = \{h_j\}$ define mask $M_{jk} = 1[y_j = y_k \land j \neq k]$,

$$L_{INV} = 1 - \frac{\sum_{j,k} M_{jk}(h_j^\top h_k)}{\sum_{j,k} M_{jk} + \epsilon}$$

Total loss:

$$L = L_{ITC} + \lambda L_{INV} \quad (\lambda = 0.5)$$

Zone-segmented frameworks quantify source-target script alignment via an entropy-based similarity score. For target class $X$, empirical recognition probabilities $P(X \to k)$ yield:

$H(X) = -\sum_{k=1}^K P(X \to k) \log_2 P(X \to k)$

Normalization:

$H_n(X) = H(X) / (1 + \log_2 K)$

Per-class similarity $S(X) = 1 - H_n(X)$, aggregated as

$S_{sim}(S,T) = \sum_{i=1}^M W(X_i) S(X_i)$

Relative similarity:

$S_{rel}(S,T) = S_{sim}(S,T) / S_{sim}(T,T)$

Empirically, higher $S_{rel}$ (e.g., Bangla–Devanagari ≈0.76) predicts higher cross-script retrieval accuracy (Bhunia et al., 2017).

4. Experimental Evaluation

Experiments employ both synthetic and real-world datasets:

• Synthetic pre-training: 262k word images (English, Chinese, Spanish).
• Real fine-tuning: IAM (English), HWDB1.0 (Chinese), synthetic Spanish (distinct fonts) (Chen et al., 16 Jan 2026).
• Indic scripts: Bangla, Devanagari, Gurumukhi—train/test splits: ~11k train / ~3.5k test per script (Bhunia et al., 2017).

Evaluation Metrics

• Acc@K (K=1,3,5), Mean Reciprocal Rank (MRR), and Normalized Edit Similarity (NES) for embedding-based and generative baselines.
• Word recognition (lexicon-based): Top-1 to Top-5 accuracy.
• Word spotting (lexicon-free): Precision-Recall curves, Mean Average Precision (MAP), global and per-keyword (“local”) (Bhunia et al., 2017).

Comparative Results

Method	In-Domain Acc@1	OOD Acc@1	Params (M)	Latency (ms)
EasyOCR	85.98%	60.44%	30.10	20.33
Qwen3-VL-4B (4B)	97.51%	84.44%	4437.8	18.21
Asymmetric dual-encoder	97.26%	86.05%	1.29	2.89

Cross-script OOD retrieval (six language pairs):

Method	en→zh	zh→en	zh→es	es→zh	es→en	en→es	Avg.
Random	0.34	0.26	0.29	0.30	0.40	0.30	0.32
SigLIP2Giant	36.89	6.71	8.26	29.45	52.59	31.58	27.55
GME-7B	42.05	57.36	44.63	32.26	50.42	30.62	42.89
Ours	73.55	84.96	83.88	90.36	90.98	73.66	82.80

Zone-segmented retrieval on Indic scripts yields Top-1 accuracy 57.5–61.1%, Top-5 up to 75.2%, MAP (word spotting) up to 67.5%, whereas traditional same-script training achieves Top-1 ≈80% and MAP ≈72–74% (Bhunia et al., 2017). This suggests cross-script systems can attain 60–75% of in-script performance, with accuracy modulated by $S_{rel}$.

5. Limitations and Open Challenges

Retrieval accuracy is sensitive to several factors:

• Zone segmentation failures: especially in cursive overlaps or absent Matra lines; such errors propagate through subsequent recognition.
• Limited class set: Exclusion of consonant conjuncts and complex ligatures simplifies the mapping, but restricts full-script generalization.
• LUT ambiguity: Majority voting may map multiple target classes to a single source label, causing word-level errors.
• Resource and parameter constraints: Conventional vision-LLMs are prohibitive for on-device inference; lightweight encoders enable efficient deployment but may underperform on rare or wild script samples.

A plausible implication is that hybrid architectures—combining explicit zone segmentation with deep cross-script embedding—could mitigate the failures of either approach alone.

6. Prospects and Future Directions

Ongoing research explores several enhancements:

• Hybrid approaches: Combining explicit, zone-segmented pipelines with latent embedding frameworks to recover from segmentation errors and exploit complementary strengths (Bhunia et al., 2017).
• Refined character mapping: Transitioning from majority-voted LUTs to context-sensitive or probabilistic alignment models.
• Extension to additional scripts: Incorporating Oriya, Assamese, Tamil, Arabic; leveraging multi-source fusion (training on several related scripts).
• Deep feature adaptation: Developing language-agnostic visual embeddings through self-supervised or semi-supervised learning, minimizing reliance on annotated examples (Chen et al., 16 Jan 2026).
• Hardware-aware deployment: Quantization and neuromorphic acceleration enable orders-of-magnitude efficiency improvements, facilitating archival and field applications where large VLLMs are impractical.

Continued empirical evaluation on diverse, real-world datasets—particularly for low-resource and rare scripts—remains essential for robust cross-script handwriting retrieval. Future frameworks should address fine-grained script structure, semantic bridging, and scalable deployment across heterogeneous archives.