Volumetric medical image segmentation via fully 3D adaptation of Segment Anything Model

The Segment Anything Model (SAM) exhibits exceptional generalization capabilities in diverse domains, owing to its interactive learning mechanism designed for precise image segmentation. However, applying SAM to out-of-distribution domains, especially in 3D medical image segmentation, poses challenges. Existing methods for adapting 2D segmentation models to 3D medical data treat 3D volumes as a mere stack of 2D slices. The essential inter-slice information, which is pivotal to faithful 3D medical image segmentation tasks, is unfortunately neglected. In this work, we present the 3D Medical SAM-Adapter (3DMedSAM), a pioneer cross-dimensional adaptation, leveraging SAM's pre-trained knowledge while accommodating the unique characteristics of 3D medical data. Firstly, to bridge the dimensional gap from 2D to 3D, we design a novel module to replace SAM's patch embedding, ensuring a seamless transition into 3D image processing and recognition. Besides, we incorporate a 3D Adapter while maintaining the majority of pre-training parameters frozen, enriching deep features with abundant 3D spatial information and achieving efficient fine-tuning. Given the diverse scales of anomalies present in medical images, we also devised a multi-scale 3D mask decoder to elevate the network's proficiency in medical image segmentation. Through various experiments, we showcase the effectiveness of 3DMedSAM in achieving accurate and robust 3D segmentation on both single-target segmentation and multi-organ segmentation tasks, surpassing the limitations of current methods.