白光内窥镜（WL）和弱放大内窥镜（WM）都是诊断胃肿瘤的重要方法。本研究构建了一个深度学习系统，名为ENDOANGEL-MM（多模式），旨在使用WL和WM数据进行胃肿瘤实时诊断。将同一病变的WL和WM图像合并成图像对。共使用了4201个图像，7436个图像对和162个视频进行模型构建和验证。构建了1-5个模型，包括两个单模式模型（WL，WM）和三个多模式模型（任务级，特征级和输入级的数据融合）。将模型在图像，视频和预期患者三个级别上进行测试。选择最佳模型以构建ENDOANGEL-MM。我们比较了模型和内镜医师之间的表现，并进行了一项诊断研究来探索ENDOANGEL-MM的辅助能力。第四个模型(ENDOANGEL-MM) 在五个模型中表现最佳。模型2在单模式模型中表现更好。在静止图像，实时视频和预期患者中，ENDOANGEL-MM的准确性高于模型2（86.54 vs.78.85％，P = 0.134; 90.00 vs.85.00％，P = 0.179; 93.55 vs.70.97％，P <0.001）。在WM数据(85.00 vs.71.67％，P = 0.002) 和多模式数据(90.00 vs.76.17％，P = 0.002)，模型2和ENDOANGEL-MM表现优于内镜医师，显著。在ENDOANGEL-MM的辅助下，非专家的准确性显着提高(85.75 vs.70.75％，P = 0.020)，并且与专家之间没有显着差异(85.75 vs.89.00％，P = 0.159)。特征级融合构建的多模式模型表现最佳。ENDOANGEL-MM能够准确识别胃肿瘤，在临床实践中发挥潜在作用。©2022作者独占许可权，授予国际胃癌协会和日本胃癌协会。

White light (WL) and weak-magnifying (WM) endoscopy are both important methods for diagnosing gastric neoplasms. This study constructed a deep-learning system named ENDOANGEL-MM (multi-modal) aimed at real-time diagnosing gastric neoplasms using WL and WM data.WL and WM images of a same lesion were combined into image-pairs. A total of 4201 images, 7436 image-pairs, and 162 videos were used for model construction and validation. Models 1-5 including two single-modal models (WL, WM) and three multi-modal models (data fusion on task-level, feature-level, and input-level) were constructed. The models were tested on three levels including images, videos, and prospective patients. The best model was selected for constructing ENDOANGEL-MM. We compared the performance between the models and endoscopists and conducted a diagnostic study to explore the ENDOANGEL-MM's assistance ability.Model 4 (ENDOANGEL-MM) showed the best performance among five models. Model 2 performed better in single-modal models. The accuracy of ENDOANGEL-MM was higher than that of Model 2 in still images, real-time videos, and prospective patients. (86.54 vs 78.85%, P = 0.134; 90.00 vs 85.00%, P = 0.179; 93.55 vs 70.97%, P < 0.001). Model 2 and ENDOANGEL-MM outperformed endoscopists on WM data (85.00 vs 71.67%, P = 0.002) and multi-modal data (90.00 vs 76.17%, P = 0.002), significantly. With the assistance of ENDOANGEL-MM, the accuracy of non-experts improved significantly (85.75 vs 70.75%, P = 0.020), and performed no significant difference from experts (85.75 vs 89.00%, P = 0.159).The multi-modal model constructed by feature-level fusion showed the best performance. ENDOANGEL-MM identified gastric neoplasms with good accuracy and has a potential role in real-clinic.© 2022. The Author(s) under exclusive licence to The International Gastric Cancer Association and The Japanese Gastric Cancer Association.