Interactive Cytoarchitectonic Parcellation

• Interactive Cytoarchitectonic Parcellation Framework is a human-in-the-loop deep learning system that automates precise segmentation of brain histology images.
• It employs modular, multi-scale workflows combining U-Net/CNN and DINOv3 transformer models to efficiently process and analyze large-scale datasets.
• The framework enables real-time user-guided annotation and iterative model refinement, reducing manual effort while improving accuracy in brain atlas construction.

The Interactive Cytoarchitectonic Parcellation Framework encompasses a set of human-in-the-loop, deep learning-based algorithms and workflows for automated, high-resolution segmentation of brain cytoarchitectonic areas from histological image data. Cytoarchitectonics describes the microstructural organization and cellular arrangement within the brain, forming the anatomical foundation for region-level parcellation necessary for integrative, multi-modal neuroscience analyses. The framework leverages recent advances in convolutional neural network (CNN) architectures and vision transformers (ViTs), as well as interactive annotation interfaces, to enable efficient parcellation of large-scale brain datasets, even under constraints of sparse annotations and variable staining protocols. Two principal instantiations—the multi-scale U-Net/CNN pipeline (Schiffer et al., 2020) and the DINOv3-based interactive transfer learning approach (Zhang et al., 15 Jan 2026)—demonstrate complementary strategies for scaling cytoarchitectonic annotation and segmentation.

1. End-to-End Workflow Organization

The core of the framework consists of a modular, iterative pipeline designed to process multi-terabyte volumes of high-resolution, cell-body stained histological sections spanning entire brains.

Key workflow steps (Schiffer et al., 2020):

1. Acquisition of 2D histological section scans (up to 120,000 × 80,000 pixels at 1–2 μm resolution).
2. Sparse "observer-independent" annotations performed at intervals (~every 60th section) via Gray-Level Index (GLI) mapping and multivariate statistical testing (Mahalanobis distance, Hotelling t-test), demarcating area boundaries.
3. Subdivision of the image stack into reference-delimited intervals, with each interval [s₁, s₂] covered by its own local segmentation model $f_{s₁s₂}^a$ specific to area $a$.
4. Training of local CNN models per area and interval, using pairs of annotated images.
5. Segmenting all un-annotated sections in parallel using the trained models, followed by aggregation of the resulting per-section masks into contiguous 2D parcellation stacks.
6. Registration and projection of 2D segmentations into a 3D reference space (e.g., BigBrain), using precomputed 2D→3D transforms.
7. 3D post-processing: median filtering (11×11×11 voxels) and removal of spurious small components (<3³ voxels).
8. Outputting the final high-resolution 3D anatomical parcellation.

A variant leveraging the DINOv3 vision transformer (ViT) replaces CNN-based local models with a globally pretrained transformer encoder and a lightweight, user-trainable decoder operating on features drawn from multiple transformer layers (Zhang et al., 15 Jan 2026). This enables efficient transfer learning and rapid deployment to new datasets and stains.

2. Neural Network Architectures and Feature Extraction

Multi-scale U-Net Local Model (Schiffer et al., 2020)

The local segmentation model is a multi-scale U-Net comprising two parallel encoder branches (high-resolution and low-resolution), feeding into a shared decoder. The high-resolution encoder receives large, fine-grained patches (2025×2025 px at 2 μm/px) and applies max-pooling and convolutional blocks, while the low-resolution encoder processes coarser patches (682×682 px at 16 μm/px) using dilated convolutions. Three architecture variants are evaluated: HR (high-res only), LR (low-res only), and MS (multi-scale, using both encoders in parallel with skip connections), with MS exhibiting the best overall accuracy.

Regularization strategies include batch normalization, L₂ weight decay, and semantic segmentation into four classes (non-cortex, white matter, cortex, target area). Data augmentation involves both spatial (rotations) and photometric (intensity) transformations.

DINOv3 Multi-layer Feature Fusion and Decoder (Zhang et al., 15 Jan 2026)

The transformer-based approach employs a fixed, self-supervised DINOv3-B ViT encoder. The histology image is divided into non-overlapping $P\times P$ patches, which are embedded and passed through $L$ transformer blocks. Token maps from selected layers ($l=3,6,9,12$) are reshaped into spatial feature maps, projected to a uniform channel dimension ($C'=128$) via $1\times1$ convolutions, optionally refined by $3\times3$ local convolutions, and upsampled to a canonical resolution. The concatenation of upsampled maps yields a fused multi-scale feature tensor.

The lightweight segmentation decoder typically comprises two $3\times3$ Conv–BatchNorm–ReLU stages and a final $1\times1$ convolutional projection to $K$ area classes, with softmax normalization.

3. Interactive User-Guided Annotation and Real-Time Training

Interactive parcellation is achieved through a web-based (microdraw) or GUI (Napari) interface, supporting annotation with polygons, brushes, or sparse scribble masks. The workflow enables neuroscience experts to select reference sections, annotate boundaries, and initiate model training and inference from within a standard web browser.

Training leverages user-provided annotation masks:

• Loss is computed only on scribbled regions, using focal cross-entropy and soft-Dice objectives, with L₂ weight regularization and (optionally) a total variation penalty for spatial smoothness.
• Real-time responsiveness is enabled by cropping training patches to regions containing user scribbles, and fine-tuning only the weights of the segmentation decoder for a small number of epochs per interaction.
• Users may iteratively refine boundaries by submitting new annotations, shrinking or splitting reference intervals as needed; only affected intervals require model retraining.

The web interface streams segmentation overlays for validation, provides opacity/false-color controls, and supports batch and distributed computation via SSH-controlled HPC clusters (microdraw web stack).

4. Performance, Scalability, and Quantitative Evaluation

Compute and throughput requirements (Schiffer et al., 2020):

• CNN workflow: Each local model uses 4 × NVidia K80 GPUs (12 GB per GPU), 48 CPU threads, and 128 GB RAM.
• Training a local model requires ~70 minutes (batch size 64, 3000 iterations); segmenting ~120 sections takes ~30 minutes (~15 s per section at highest resolution).
• The framework is robust to artefacts; severely degraded sections are excluded during reconstruction and interpolated via Laplacian field methods.

Accuracy benchmarks:

• On BigBrain (18 areas), MS-U-Net achieves mean F₁ ≈ 0.72 (σ ≈ 0.18), outperforming HR-only (mean ≈ 0.57) and LR-only (mean ≈ 0.61) models.
• The transformer-based pipeline demonstrates substantial performance gains over nnU-Net: overall Dice coefficient (DSC) improvement from 0.425 (nnU-Net) to 0.639, and boundary error (HD95, ASSD) reductions of 50%–80% for rhesus macaque V1 laminar segmentation (1614 train/193 test 512×512 patches) (Zhang et al., 15 Jan 2026).

Evaluation Metrics: Precision, recall, Dice (F₁), Intersection-over-Union (IoU), Hausdorff Distance at 95% (HD95), and Average Symmetric Surface Distance (ASSD) are computed per class and overall. Only hold-out test sections not used during training are considered.

Example Results Table (excerpted from (Zhang et al., 15 Jan 2026)):

Class	DSC (Ours)	DSC (nnU-Net)	HD95 (Ours, μm)	HD95 (nnU-Net, μm)
L1	0.796	0.544	87.9	324.5
L2/3	0.733	0.530	251.0	508.0
...	...	...	...	...
overall	0.639	0.425	248.9	645.1

Performance is robust across diverse brains and staining protocols as long as minimal annotations are provided in the target domain.

5. Implementation and Deployment Guidelines

Deployment entails installing the front-end annotation server (atlas-UI/microdraw or Napari viewer) and ensuring SSH-based connectivity to suitable HPC or cluster hardware. The CNN branch requires distributed TensorFlow with Horovod, while the transformer-based decoder pipeline can leverage tightly tiled inference for interactive update.

• Python dependencies: TensorFlow (CNN pipeline), mpi4py, Flask, Napari, and, for transformer methods, PyTorch and AdamW optimizer.
• Data is exchanged primarily in TIFF/PNG (2D tiles), NIfTI (3D stacks), and metadata via JSON/APIs.
• Extensions to additional brain areas or staining protocols are achieved by annotating reference sections in the new modality and retraining the corresponding local models; no architecture change is necessary for differences in section thickness (adjust only the patch-size-to-resolution mapping).

A typical human-in-the-loop session involves user annotation, on-demand local model training (≤70 minutes/model for CNN branch; ≤15 epochs/interaction for transformer decoder), interactive inspection, and, when necessary, incremental annotation and retraining.

Example Pseudocode for DINOv3-based Interactive Parcellation (Zhang et al., 15 Jan 2026):

model.encoder = load_pretrained_DINOv3_B() freeze(model.encoder.parameters) model.decoder = LightDecoder(init=He) optimizer = AdamW(model.decoder.parameters, lr=5e-4, wd=1e-4) viewer = napari.Viewer() viewer.add_image(whole_slide) scribble_layer = viewer.add_labels(np.zeros_like(whole_slide[...,0]), name='scribbles') while not user_finished: wait_for_event('scribble_drawn') S_mask, Y_label = extract_scribbles(scribble_layer) patches, labels, masks = crop_patches(whole_slide, S_mask, Y_label, size=512) for epoch in range(1, fine_tune_epochs+1): for x_batch, y_batch, m_batch in DataLoader(patches, labels, masks): feats = model.encoder(x_batch) fused = fuse_features(feats) logits = model.decoder(fused) loss = compute_loss(logits, y_batch, m_batch) optimizer.zero_grad(); loss.backward(); optimizer.step() seg_map = tiled_inference(model, roi=whole_slide, tile_size=512, overlap=256) viewer.add_labels(seg_map, name='segmentation', opacity=0.5, blending='additive') if user_confirms(): break

6. Applications, Extensibility, and Impact

The framework supports the construction of high-fidelity, three-dimensional cytoarchitectonic atlases, facilitating quantitative neuroanatomical research, cross-brain comparisons, and multi-modal brain mapping. Its scalability and adaptability enable efficient mapping across stains, brains, and target structures. The integration of pretrained foundation models (DINOv3) and user-guided refinement substantially lowers the annotation burden, enabling generalization despite limited training data and heterogeneity in imaging.

A representative use case involves mapping the human visual cortex area hOc1 in BigBrain by annotating two reference sections, training a local model, applying segmentation to all intermediate sections, and, if necessary, refining results by splitting intervals and retraining (total hands-on annotation effort reduced from weeks to hours).

The interactive cytoarchitectonic parcellation framework demonstrates that a combination of observer-independent annotation, local or foundation-model-based segmentation networks, and web-based human-in-the-loop interaction constitutes an effective strategy for scalable, reproducible brain atlas construction (Schiffer et al., 2020, Zhang et al., 15 Jan 2026).