Stroke-Order Salience in Handwriting

• Stroke-order salience is the critical importance of reconstructing stroke sequence, direction, and timing to enable accurate offline handwriting analysis.
• Advanced methods, including end-to-end models like TRACE and skeletonization-based pipelines, bridge the gap between offline and online handwriting recognizers.
• Empirical results indicate that recovering stroke order boosts symbol recognition and generative synthesis, enhancing applications such as signature verification and OCR.

Stroke-order salience refers to the critical role that the sequence in which pen strokes are made (their order, direction, and timing) plays in handwriting-centric tasks including offline handwriting recognition, signature verification, and handwriting synthesis. Offline data, such as scanned images of handwritten text, inherently lacks temporal dynamics, which are key discriminative features for both recognition performance and downstream generative tasks. Techniques that reconstruct or approximate stroke order from such static imagery enable the reuse of online-specific models, narrow the accuracy gap between offline and online systems, and facilitate high-fidelity writer style emulation. Salience in this context measures how much downstream task performance depends on fine-grained recovery or normalization of stroke-level ordering.

1. Methodologies for Stroke-Order Recovery

Two principal methodological paradigms have emerged for recovering stroke order from static handwriting images: end-to-end differentiable models and skeletonization-based oversegmentation pipelines.

• End-to-end models: TRACE (Trajectory Recovery by an Adaptively-trained Convolutional Encoder) is a convolutional-recurrent neural network (CRNN) that predicts relative stroke displacements and per-point sequence flags directly from an input gray-level image. The model integrates Dynamic Time Warping (DTW) to align its predicted trajectory with ground truth stroke sequences. TRACE processes input images of arbitrary width without any pre-processing (e.g., no skeletonization, binarization, or stroke exemplars) and requires no post-processing, yielding a fully differentiable stroke-trajectory prediction suitable for downstream sequential models (Archibald et al., 2021).
• Skeletonization and oversegmentation: In mathematical handwriting extraction, the leading approach involves adaptive binarization followed by the Wang–Zhang thinning algorithm to a one-pixel-wide skeleton. The skeleton is decomposed into segments and junctions, which are then merged via bottom-up clustering based on geometric continuity to form plausible stroke paths. This is followed by hierarchical grouping (XY-cut style) and local topological sort to impose reading and writing order suitable for online recognizers (Chan, 2019).

2. Alignment, Losses, and Training Regimes

Accurate stroke-order recovery depends on sequence alignment and loss design.

• Alignment: The core alignment mechanism in TRACE is DTW, which minimizes spatial $L_1$ or $L_2$ distances between predicted and ground truth stroke sequences, enforcing monotonic and continuous mappings between sequence elements.
• Loss components in TRACE:
  • Cumulative-sum $L_1$ loss applied to coordinate sequences, calculated after resampling the ground truth to the prediction length.
  • Cross-entropy losses on binary start-of-stroke (SOS) and end-of-sequence (EOS) signals, with class-balancing for the rare EOS events (Archibald et al., 2021).
• Adaptive ground truth: TRACE employs curriculum learning where the GT is adaptively updated by swapping adjacent strokes or reversing direction if it locally reduces the DTW loss. By softmax-weighted probabilistic selection according to strokewise DTW costs, the model relaxes strict ordering only where the image provides no disambiguation.
• Segment grouping and order normalization: In skeletonization-based systems, after path extraction, recursive projection and topological sorting provide a hierarchical grouping and canonicalization of stroke order (Chan, 2019).

3. Quantitative Impact on Recognition Tasks

Empirical results on both general handwriting and specialized mathematical OCR quantify the benefits of stroke-order recovery.

• Trace benchmarks on IAM-On:
  • Average DTW Loss (stroke-height normalized): $L_1=0.03060$, $L_2=0.02423$ under equidistant GT (TRACE-trained).
  • Nearest-Neighbor precision/recall: $0.01405{-}0.01751$ (L2 metric).
  • On pure image input (IAM-Off), rendering-based LPIPS similarity falls to $0.106$ (from $0.423$ baseline and $0.296$ without DTW loss).
  • No explicit velocity-RMSE is computed, as TRACE is trained on spatial sequences only (Archibald et al., 2021).
• Mathematical expression recognition:
  • Offline formulas recognized after stroke extraction: exact match rates of $58.22\%$ (CROHME 2014), $65.65\%$ (2016), $65.22\%$ (2019)—10–14 points below the underlying online engine.
  • Retraining an online model (TAP) on extracted strokes: test accuracy closes to within $0.6\%$ of the original online performance.
  • Structure-only rates (e.g., correct layout) show greater robustness to imperfect stroke order than symbol-labelling, suggesting stroke order primarily aids symbol segmentation and recognition (Chan, 2019).

4. Salience of Stroke Order in Downstream Applications

Stroke-order and timing are foundational for a range of handwriting-driven applications:

• Signature verification: Subtle differences in stroke sequencing and tempo allow discrimination between authentic and forged signatures.
• Handwriting synthesis: Models such as Graves’ LSTM+MDN require temporally ordered point sequences. When stroke trajectories recovered by TRACE are used as priming input, generated samples can mimic offline handwriting idiosyncrasies (loopiness, flourishes) with high fidelity; by contrast, synthesis models trained only on native online data fail to generalize when primed with offline-converted (TRACE) strokes (Archibald et al., 2021).
• Robustness and order normalization: Imposing a canonical stroke order makes downstream recognizers more invariant to writer-specific or idiosyncratic orderings, and in some cases, normalizing atypical styles may enhance recognition robustness (Chan, 2019).

5. Analysis and Implications of Stroke-Order Salience

Empirical and methodological findings indicate the following about stroke-order salience:

• Criticality for symbol recognition: Symbol-level recognition rates benefit substantially from access to temporal ordering (stroke sequence and direction), while structural parsing in mathematical OCR is less dependent on precise order, being more robust to errors in stroke segmentation or sequence.
• Model transferability: The main barrier to parity between offline and online handwriting recognition is the absence of sequential order. Once stroke order is reconstructed and normalized, the accuracy gap is effectively bridged.
• Curriculum learning for ambiguity: Allowing adaptive ground truth relaxes rigid ordering where images provide ambiguous cues (e.g., diacritical marks) and accelerates convergence to a stable, plausible stroke ordering. In TRACE, this reduces average NN loss by ≈20% and minimizes stroke swaps during training (Archibald et al., 2021).
• Resource efficiency: Both end-to-end and classical pipelines for stroke-order recovery are computationally feasible for on-device deployment, with stroke extraction time being negligible compared to recognition in practical pipelines (Chan, 2019).

6. Open Questions and Future Directions

Existing approaches mainly address stroke-order recovery in unconstrained handwritten text and mathematical expressions, but open questions remain:

• Ambiguity handling: There is an inherent limit to stroke-order recovery in cases where the image structure cannot unambiguously resolve order or direction (e.g., dots or diacritics applied out of lexical order).
• Rich temporal modeling: Most spatial-only models (e.g., TRACE) do not explicitly recover pen velocity or pause-gesture information. A plausible implication is that further integration of timing/velocity dynamics and richer online-to-offline modeling could improve downstream synthesis even further.
• Unified online/offline frameworks: The convergence in recognition performance after stroke-order normalization motivates unified architectures capable of flexibly processing both online and offline handwriting inputs.

Stroke-order salience thus occupies a central role in the accuracy, robustness, and generative capacity of handwriting-based systems, with its recovery from static images enabling the continued advancement and cross-pollination of online and offline recognition methodologies (Archibald et al., 2021, Chan, 2019).