BOLDreams: Dreaming with pruned in-silico fMRI Encoding Models of the Visual Cortex

Abstract: In this article we use the Natural Scenes Dataset (NSD) to train a family of feature-weighted receptive field neural encoding models. These models use a pre-trained vision or text backbone and map extracted features to the voxel space via receptive field readouts. We comprehensively assess such models, quantifying performance changes based on using different modalities like text or images, toggling finetuning, using different pre-trained backbones, and changing the width of the readout. We also dissect each model using explainable AI (XAI) techniques, such as feature visualization via input optimization, also referred to as ``dreaming'' in the AI literature, and the integrated gradients approach to calculate implicit attention maps to illustrate which features drive the predicted signal in different brain areas. These XAI tools illustrate biologically plausible features that drive the predicted signal. Traversing the model hyperparameter space reveals the existence of a maximally minimal model, balancing simplicity while maintaining performance.