Li, Xueyuan., Cui, Can., Deng, Ruining., Tang, Yucheng., Liu, Quan., Yao, Tianyuan., Bao, Shunxing., Chowdhury, Naweed Iffat., Yang, Haichun., & Huo, Yuankai. (2025).听.听Journal of Medical Imaging,听12(5), 57501.听
Recent advances in computational pathology鈥攖he use of computers to analyze tissue images鈥攈ave been driven by聽Vision Foundation Models (VFMs), particularly the聽Segment Anything Model (SAM). SAM, a type of VFM, can segment, or outline, cell nuclei using either prompts (zero-shot segmentation) or specialized cell-focused models, allowing it to work across many types of cells. However, general VFMs often struggle with fine-grained tasks, such as identifying specific nuclei subtypes or particular cells.
To address this, we developed the聽molecular empowered all-in-SAM model, which enhances SAM and VFMs for more precise pathology analysis. Our approach has three key components: (1)聽annotation, where molecular-informed guidance allows even non-experts to label images without detailed pixel-level work; (2)聽learning, where SAM is adapted with a SAM adapter to focus on specific cell types and biological features; and (3)聽refinement, which improves segmentation accuracy through molecular-oriented corrective learning.
Testing on both in-house and public datasets showed that all-in-SAM greatly improves cell classification, even when annotation quality varies. This approach reduces the workload for human annotators and makes precise biomedical image analysis more accessible, especially in resource-limited settings, supporting advances in automated pathology and medical diagnostics using VFMs.
Fig.听1
Overall idea of our work: this diagram illustrates the distinctions between our approach (bottom panel) and existing methods. (1)聽Traditional: expert annotators manually label cells using only PAS images. (2)聽MOCL: lay annotators provide pixel-level labels under the guidance of IF molecular images, followed by the application of deep learning for segmentation. (3)聽SAM-L: the SAM technique is utilized to expedite the annotation process, requiring only minimal (box) annotations. (4)聽All-in-SAM (our method): we integrate SAM in the annotation phase and adaptively fine-tune it during the training of the model.