FCN Ensembles for Robust Image Segmentation

• FCN ensembles are techniques that aggregate multiple independently trained FCNs to improve segmentation accuracy, robustness, and generalization across heterogeneous datasets.
• They leverage diversity in training strategies—such as random initialization and varied loss functions—and mix architectures like U-Net, DeepLabV3⁺, and DenseNet to address segmentation challenges.
• These ensembles have demonstrated superior quantitative performance in tasks including WMH, polyp, and brain anatomical segmentation by effectively reducing variance and enhancing model resilience.

Fully convolutional network (FCN) ensembles are machine learning systems in which multiple independently trained FCN models are aggregated to produce robust, accurate, and reliable segmentation predictions for image analysis tasks. Originally proposed for biomedical image segmentation, these ensembles exploit stochasticity and architectural diversity to improve the stability and generalization of deep segmentation pipelines across heterogeneous datasets and acquisition protocols. Representative studies demonstrate that FCN ensembles not only achieve superior quantitative metrics compared to single models but also display resilience to data shift, class imbalance, and varied imaging protocols, establishing them as a state-of-the-art approach in semantic and anatomical segmentation tasks (Li et al., 2018, Nanni et al., 2021, 1901.01381).

1. FCN Ensemble Architectures and Backbones

Fully convolutional network ensembles are typically constructed by training multiple deep segmentation networks—often U-Net variants, DeepLabV3⁺, or DenseNet-derived architectures—on identical data with independent random initialization or using different loss functions and data orderings.

• 2D U-Net Variants: For white matter hyperintensities (WMH) in MR imaging, a 2D U-Net with 19 convolutional layers receives co-registered FLAIR and T1 slices as two-channel input. Contracting-path layers use larger (5×5) kernels initially and 3×3 in deeper stages; skip connections convey activations to symmetrically arranged transposed-convolution upsampling layers (Li et al., 2018).
• 3D U-Net and Multi-Stream Designs: For brain ROI segmentation, ensembles of 3D FCNs incorporate multi-encoding streams, processing both the target and multiple co-registered atlas image patches and labels, concatenated after early layers to preserve local and context information. The encoding-decoding stack with skip connections ensures both spatial and contextual feature retention (1901.01381).
• DeepLabV3⁺ and HarDNet Backbones: Diverse backbones, such as DeepLabV3⁺ (ResNet18/50/101 encoders, ASPP modules) and HarDNet-MSEG (harmonic dense connectivity for parameter efficiency), are combined within ensembles, leveraging backbone heterogeneity for increased diversity (Nanni et al., 2021).

No custom layers beyond these published backbones are required; diversity is introduced through stochastic training processes or loss function variation.

2. Diversity Mechanisms in FCN Ensembles

The effectivity of FCN ensembles derives from introducing diversity among constituent models:

• Random Initialization and Training Order: Independent models are trained from different weight initializations and shuffled data orders, with stochastic data augmentation further promoting divergence (Li et al., 2018).
• Loss Function Variation: Utilizing distinct loss functions—such as Dice loss, Structural Similarity Index Metric (SSIM) loss, Tversky loss, and composite losses (e.g., Dice+SSIM)—causes models to prioritize different error modalities and converge to different local minima (Nanni et al., 2021).
• Architectural Diversity: Mixing backbone architectures (e.g., combining DeepLabV3⁺ with HarDNet-MSEG) increases the representational variety across the ensemble.
• Atlas Guidance and Adaptive Patching: For multi-atlas-guided setups, each model is further individualized by selecting atlas patches by similarity for each ROI and varying patch sizes per region (1901.01381).

Diversity within the ensemble is essential, as it ensures that the aggregation of predictions can effectively reduce variance and correct for individual model errors.

3. Training Protocols and Data Augmentation

Robust training protocols are crucial for both per-model performance and the ensemble as a whole:

• Losses and Optimization: Dice loss is adopted to directly maximize overlap measures in class-imbalanced segmentation (e.g., WMH vs. background), while cross-entropy and composite losses are used for anatomical segmentation tasks (Li et al., 2018, 1901.01381, Nanni et al., 2021).
• Optimizers: Both stochastic gradient descent (SGD) and Adam have been effective, with learning rates set in the range of $2\times 10^{-4}$ (U-Net WMH) or $1\times 10^{-2}$ (DeepLabV3⁺ polyp segmentation) (Li et al., 2018, Nanni et al., 2021).
• Batch Size and Epochs: Batch sizes (compatible with hardware, e.g., 4–30) and epochs (20–50, based on validation loss stabilization) are chosen empirically (Li et al., 2018, Nanni et al., 2021).
• Data Augmentation: Heavy random rotation, shear, and scaling are employed to induce invariance to spatial and acquisition differences. For example, 10× augmentation in WMH segmentation led to significant reductions in overfitting and improved generalization, decreasing Hausdorff distance by ∼0.6 mm and increasing F1 by ∼5 % (Li et al., 2018). Standard flip and rotation augmentations are used for polyp and skin segmentation (Nanni et al., 2021).

Preprocessing commonly includes brain masking, Gaussian intensity normalization, and patch adaptation to anatomical context.

4. Ensemble Aggregation and Inference

• Probability Aggregation: Each independently trained FCN produces a per-pixel or per-voxel soft probability map. The ensemble aggregates these scores by averaging (element-wise mean across models), producing a consensus probability map (Li et al., 2018, 1901.01381, Nanni et al., 2021).
• Thresholding and Post-processing: For binary segmentation, a fixed threshold (usually 0.5) is applied to the aggregate probability map. For multiclass tasks, argmax over class probabilities is used. Anatomical outliers and isolated false positives are removed with postprocessing rules (e.g., discarding detections in the first/last 10% of slices) (Li et al., 2018).
• Multi-Atlas ROI Assignment: For each voxel, possible overlappings ROIs from different models are amalgamated by assigning the ROI with maximal weighted confidence (1901.01381).

No explicit weighting between models is applied during fusion; all networks contribute equally.

5. Quantitative Assessment and Impact of Ensemble Size

Empirical results verify the quantitative gains from ensemble methods:

Task & Dataset	Single Model Dice/F1	3-Model Ensemble Dice	5-Model Ensemble Dice	SOTA/Notes
WMH Segmentation (MICCAI 2017)	~78.3%	80.0%	Slightly higher	Best H95: 6.30mm, Precision: 84% (Li et al., 2018)
Polyp Segmentation (Bioimage)	≈0.808 (RN101)	0.834 (RN101, 10-nets)	0.843 (ELoss101, 10-nets)	HN&101: 0.852; TransFuse: 0.855 (Nanni et al., 2021)
Brain ROI Segmentation (PREDICT-HD)	0.917 (Single S-FCN)	-	0.922 (M-FCN, 3-nets)	JLF: 0.904; LiviaNET: 0.889 (1901.01381)

Ensemble size exerts a pronounced effect up to 3–5 models. In WMH segmentation, going from 1 to 3 U-Nets improves Dice by ∼1.7%, reduces variance by over 30%, and improves H95 by ~1 mm. Gains from 3 to 5 models are marginal, and computational costs rise (Li et al., 2018). In polyp/skin segmentation, 10-model ensembles demonstrate further marginal gains, with loss-function diversity providing additional improvements (Nanni et al., 2021).

6. Implementation Considerations and Practical Recommendations

• Scaling with Computation: Training a single FCN (e.g., U-Net for WMH) takes ∼3 hours for 50 epochs on a Titan-Xp GPU; inference requires ~8s per scan per model. For clinical workflows, ensemble size should be chosen based on practical GPU or CPU constraints (Li et al., 2018).
• Cross-Protocol Adaptation: Aggressive augmentation and per-scan normalization are advised for new protocols or scanners. Fine-tuning at least one model per new protocol enhances adaptation. Combining multiple input modalities (such as FLAIR+T1) consistently improves performance (Li et al., 2018).
• Loss and Hyper-parameter Choices: Dice loss is preferred for class-imbalanced targets. Monitoring validation loss curvature is essential for early stopping (Li et al., 2018). For loss-diverse ensembles, careful selection and tuning of distinct loss landscapes yield further small but measurable improvements (Nanni et al., 2021).
• Public Code Availability: Implementation details for both DeepLabV3⁺/HarDNet ensembles and their training/fusion are available at https://github.com/LorisNanni (Nanni et al., 2021).
• Atlas-based Models: For multi-ROI segmentation using atlas guidance, patch adaptation, dropout regularization, and Dice-based early stopping are key components of robust ensemble model training (1901.01381).

7. Applications and Outlook

FCN ensembles are widely adopted in the medical imaging domain for tasks such as WMH segmentation, polyp and skin lesion delineation, and multi-region brain anatomical labeling. In challenge and benchmark evaluations, ensemble approaches have demonstrated state-of-the-art performance (e.g., achieving first rank in the WMH Segmentation Challenge at MICCAI 2017 (Li et al., 2018), surpassing atlas-based and single CNN baselines). The reduction of variance, improved generalization across unseen scanners and protocols, and consistent performance gains corroborate the robustness of the ensemble methodology.

Extensions to diverse loss landscapes, architectural backbones, and domain-specific data augmentation schemes further broaden their applicability. These systems establish a methodological foundation for robust deployment in clinical and research settings, with the potential for further adaptation by integrating transformer-based models or deploying model pruning and knowledge distillation for resource-constrained environments (Li et al., 2018, Nanni et al., 2021, 1901.01381).