SG融合：基于Swin-Transformer和Graph卷积的多模式深神经网络，用于神经胶质瘤预后
SG-Fusion: A swin-transformer and graph convolution-based multi-modal deep neural network for glioma prognosis
影响因子:6.20000
分区:医学2区 Top / 计算机：人工智能2区 工程：生物医学2区 医学：信息2区
发表日期:2024 Nov
作者:
Minghan Fu, Ming Fang, Rayyan Azam Khan, Bo Liao, Zhanli Hu, Fang-Xiang Wu
摘要
从组织病理学图像和基因组数据中提取的形态学属性的整合在推进肿瘤诊断，预后和分级方面非常重要。组织病理学图像是通过对组织切片的显微镜检查获得的，为细胞结构和病理特征提供了宝贵的见解。另一方面，基因组数据提供了有关肿瘤基因表达和功能的信息。这两种不同的数据类型的融合对于获得对肿瘤特征和进展的更全面的理解至关重要。过去，许多研究依赖于单模式方法进行肿瘤诊断。但是，这些方法存在局限性，因为它们无法完全利用来自多个数据源的信息。为了解决这些局限性，研究人员转向了多型模式的方法，这些方法同时利用组织病理学图像和基因组数据。这些方法可以更好地捕获肿瘤的多面性质并提高诊断精度。尽管如此，现有的多模式方法在某种程度上已经过度简化了模态和融合过程的提取过程。在这项研究中，我们提出了一个双分支神经网络，即SG融合。具体而言，对于组织病理学方式，我们利用Swin-Transformer结构捕获局部和全局特征，并结合对比度学习，以鼓励模型辨别表示空间中的共同点和差异。对于基因组模态，我们基于基因功能和表达水平相似性开发了图形卷积网络。此外，我们的模型集成了一个交叉意见模块，以增强信息交互并采用基于差异的正则化来增强模型的概括性能。从癌症基因组图集上进行的神经胶质瘤数据集进行的验证明确表明，在生存分析和肿瘤分级中，我们的SG融合模型在生存分析和肿瘤分级中都优于单模式方法和现有的多模式方法。
Abstract
The integration of morphological attributes extracted from histopathological images and genomic data holds significant importance in advancing tumor diagnosis, prognosis, and grading. Histopathological images are acquired through microscopic examination of tissue slices, providing valuable insights into cellular structures and pathological features. On the other hand, genomic data provides information about tumor gene expression and functionality. The fusion of these two distinct data types is crucial for gaining a more comprehensive understanding of tumor characteristics and progression. In the past, many studies relied on single-modal approaches for tumor diagnosis. However, these approaches had limitations as they were unable to fully harness the information from multiple data sources. To address these limitations, researchers have turned to multi-modal methods that concurrently leverage both histopathological images and genomic data. These methods better capture the multifaceted nature of tumors and enhance diagnostic accuracy. Nonetheless, existing multi-modal methods have, to some extent, oversimplified the extraction processes for both modalities and the fusion process. In this study, we presented a dual-branch neural network, namely SG-Fusion. Specifically, for the histopathological modality, we utilize the Swin-Transformer structure to capture both local and global features and incorporate contrastive learning to encourage the model to discern commonalities and differences in the representation space. For the genomic modality, we developed a graph convolutional network based on gene functional and expression level similarities. Additionally, our model integrates a cross-attention module to enhance information interaction and employs divergence-based regularization to enhance the model's generalization performance. Validation conducted on glioma datasets from the Cancer Genome Atlas unequivocally demonstrates that our SG-Fusion model outperforms both single-modal methods and existing multi-modal approaches in both survival analysis and tumor grading.