AI pipeline for accurate retinal layer segmentation using OCT 3D images

Abstract: Image data set from a multi-spectral animal imaging system is used to address two issues: (a) registering the oscillation in optical coherence tomography (OCT) images due to mouse eye movement and (b) suppressing the shadow region under the thick vessels/structures. Several classical and AI-based algorithms in combination are tested for each task to see their compatibility with data from the combined animal imaging system. Hybridization of AI with optical flow followed by Homography transformation is shown to be working (correlation value>0.7) for registration. Resnet50 backbone is shown to be working better than the famous U-net model for shadow region detection with a loss value of 0.9. A simple-to-implement analytical equation is shown to be working for brightness manipulation with a 1% increment in mean pixel values and a 77% decrease in the number of zeros. The proposed equation allows formulating a constraint optimization problem using a controlling factor {\alpha} for minimization of number of zeros, standard deviation of pixel value and maximizing the mean pixel value. For Layer segmentation, the standard U-net model is used. The AI-Pipeline consists of CNN, Optical flow, RCNN, pixel manipulation model, and U-net models in sequence. The thickness estimation process has a 6% error as compared to manual annotated standard data.