LLaMA-Reg: Using LLaMA 2 for Unsupervised Medical Image Registration

Abstract: Medical image registration is an essential topic in medical image analysis. In this paper, we propose a method for medical image registration using a pretrained LLM. We find that using the pretrained LLM to encode deep features of the medical images in the registration model can effectively improve image registration accuracy, indicating the great potential of the LLM in medical image registration tasks. We use dual encoders to perform deep feature extraction on image pairs and then input the features into the pretrained LLM. To adapt the LLM to our registration task, the weights of the LLM are frozen in the registration model, and an adapter is utilized to fine-tune the LLM, which aims at (a) mapping the visual tokens to the language space before the LLM computing, (b) project the modeled language tokens output from the LLM to the visual space. Our method combines output features from the fine-tuned LLM with the features output from each encoder layer to gradually generate the deformation fields required for registration in the decoder. To demonstrate the effectiveness of the large prediction model in registration tasks, we conducted experiments on knee and brain MRI and achieved state-of-the-art results.