A new explainable AI pipeline for cancer cell segmentation and classification for microscopy images

Abstract Background The development of novel therapeutic approaches to cancers requires significant time and labour, especially for uncommon cancers (e.g. of mesenchymal origin). In addition, these rare cancers typically receive less research attention but still need sufficient analysis. For example, a trustworthy analysis of tissue samples to identify cancer progression and cell response to candidate therapies is a task to develop certain treatments and drugs. Thus, this research work develops a new explainable Deep Learning pipeline for accurate, high-throughput and autonomous recognition of metastasising cells from large-scale fluorescence microscopy images, which can be used to evaluate cell response with low cost and high trustworthiness, and facilitate the therapeutic approaches development. Results The pipeline includes pre-trained and domain-generalised segmentation to identify single cells, training the latest self-supervised transformers: DINO-based models to provide normal or metastasising classification, and leveraging Explainable AI methods to produce biologically meaningful model explanations towards the trustworthiness of model predictions; this is crucial in future clinical applications. Compared to supervised learning-based models, DINO achieves the best predictability (over 97% F1-weighted score) when only training the classifier, and the most efficient downstream adaptation. Besides, all DINO-based models can classify successfully when finetuning the entire model (DINOv3 achieves the highest F1-weighted score with 98.35%). We have also used an additional cross-magnification test and a self- / weakly-supervised model comparison to discuss the generalisation ability and superiority of DINO-based models. Biologically accurate model explanations are produced via heatmaps and analysed through quantitative metrics and qualitative evaluation; the important cell regions in the model decision-making process align well with biological consensus. We also demonstrate that biologically meaningful cytoskeletal features can be detected and exploited by the developed pipeline for cancer cell classification, which facilitates the development of cytoskeleton-based diagnosis. Conclusion We develop a precise, efficient, and trustworthy pipeline that can facilitate the development of effective and bespoke cancer treatments via AI-assisted testing of the response of cancer cells to various treatments and drugs.