一、前言

本次组会阅读的文献为 Multi-modal Graph Learning for Disease PredictionSCI一区

二、Abstract

Benefifiting from the powerful expressive capability of graphs, graph-based approaches have been popularly applied to handle multi-modal medical data and achieved impressive performance in various biomedical（生物医学的） applications. For disease prediction tasks, most existing graph-based methods tend to defifine the graph manually based on specifified modality (e.g., demographic information), and then integrated other modalities to obtain the patient representation by Graph Representation Learning (GRL).

因为图的强表达能力，以图为基础的方法已经广泛地被运用去处理多模态医学数据并且在众多的生物医学应用上获得了极好的表现。对于疾病预测任务，大多数现有的以图为基准的方法都是在特定的模态上（例如，生物医学图信息）由人工来定义图，然后通过GRL整合其他模态来获得病人的表征信息。

However, constructing an appropriate graph in ad vance is not a simple matter for these methods. Meanwhile, the complex correlation between modalities is ignored. These factors inevitably（不可避免地） yield（导致） the inadequacy（不充足） of providing suffificient information about the patient’s condition for a reliable diagnosis. To this end , we propose an end-to-end Multi-modal Graph Learning framework (MMGL) for disease prediction with multi-modality. To effectively exploit the rich information across multi-modality associated with the dis ease, modality-aware （模态感知）representation learning is proposed to aggregate（聚合） the features of each modality by leveraging（利用） the correlation and complementarity between the modalities.

然而，通过这些方法提前建造一个合适的图并不是一件简单的事。同时，模态之间的复杂关系被忽略了。这些因素不可避免地导致无法提供关于病人的情况以进行合理的诊断。最后，我们提出了一个端对端的多模态图学习框架用于多模态的疾病预测（MMGL）。为了有效地提取到与疾病关系密切的多模态信息，我们还提出了模态感知表征学习通过利用模态之间的相关性和互补性来聚合每一个模态的特征。

Furthermore, instead of defifining the graph manually, the latent （潜在的）graph structure is captured through an effective way of adaptive（自适应） graph learning. It could be jointly（联合） optimized（优化） with the prediction model, thus revealing the intrinsic （内在的）con nections among samples. Our model is also applicable to the scenario（场景） of inductive（归纳） learning for those unseen data. An extensive group of experiments on two disease predic tion tasks demonstrates that the proposed MMGL achieves more favorable performance.

此外，我们通过一种有效的方式即自适应图学习来捕捉潜在的图结构并非人工定义图结构。它可以被嵌入到预测模型中，由此可以表现出样本之间的内在联系。我们的模型同样被运用在那些看不见的归纳学习场景。两种疾病预测表现出我们提出的MMGL获得了不错的表现。

三、Introduction and Related work

Although the above methods have achieved remarkable performance, three key issues remain to be further considered with respect to the graph-based methods in disease prediction tasks, and even in some other biomedical-related aspects:

尽管上述的方法已经获得了比较好的表现，三个关于在疾病预测任务和其他生物医学方向的图基准方法的核心问题仍然需要被考虑到：

(i) Insuffificient inter-modal relationship mining（挖掘）. Each modality provides different information for the diagnosis of a disease, which explicitly（确实） is complementary but also redundant（冗余）. However, both concatenation [20], [23], [24] and intra-modal attention mechanism（机制） [21], [22] adopted in previous studies are hard to capture the latent inter-modal correlation, which may cause the learned representation to be biased towards a single modality. In addition, the general multi-modal shared representation learning methods merely focus on capturing the commonalities（共性） between modalities, while the dissimilarities（不同性）between modalities are ignored, possibly resulting in the lack of complementary information.

第一点是不充分的内在模态关系挖掘。每一个模态为疾病诊断提供了不同的信息，这些信息确实是互补的但却有些冗余。然而，运用在之前研究中的连接法和模态间注意力机制在捕捉潜在模态间联系是困难的，这些方法可能会导致学习到表征信息趋向于单个模态。除此之外，大多多模态表征学习方法仅仅是捕捉模态间的共性信息，模态之间的不同性缺被忽略，很有可能导致互补性的不足。

(ii) Hand-designing（手工设计） the graph adjacency matrix in a multi-stage framework. Both existing single-graph based methods [11], [16], [25], [26] and multi-graph based methods [20], [22], [27] construct the graph through hand-designed similarity（相似性） measures, which inevitably require careful tuning（调整） and are thus diffificult to generalize to downstream tasks. Meanwhile, the training of the several parts, such as multimodal representation learning, graph construction, and prediction, are independent（独立的） of each other in a multi-stage framework. Such practice not only weakens the integrality of the model, but also leads to suboptimal（次优）performance in downstream tasks. A better approach is to learn a graph in an adaptive way, which has been studied in GNNs to some extent [28]–[30]. But currently, less focus has been put on the graph structure learning in the biomedical fifield [23].

第二点是手工设计的多阶段邻接矩阵框架。现有的基于如下方法的单图结构和多图结构

都通过一种手工设计的相似性方法，这些方法不可避免地需要仔细的调整并且困难地去概括下游任务。同时，在多阶段框架中不同部分的训练是独立的，例如模态表征学习，图结构和预测。这种方式不仅仅削弱了模型之间的完整性，同时也导致了在下游任务中的次优表现。一种自适应方式去学习一个图已经某种程度上在GNNs有所体现。但是目前来说，在生物医学领域的图结构学习很少。

(iii) Hard applicable to inductive learning. For the ap proaches based on spectral（光谱） graph convolution like [11], [16], [24], it’s hard for them to generalize to unseen samples. Besides, to accommodate the setting of inductive learning, it is also essential but cumbersome（麻烦） for multi-graph based methods [20], [22], [27] to measure the relationship of unseen samples on each graph.

第三点是很困难地被运用在归纳学习中。对于基于类似如下的谱图卷积方法来说，概况看不见的样本是困难的。此外，为了容纳一系列归纳学习，基于如下方法去测量每一个图之间的看不见的样本是必要的但缺很麻烦。

To address the issues mentioned above, we concentrate in this paper on graph learning for disease prediction with multi- modality, and the main contributions can be highlighted in the following aspects:

为了上述提出的问题，我们在这篇文章中主要研究基于图学习的多模态疾病预测，主要贡献有如下几个方面：

1、We propose a Multi-modal Graph Learning model (MMGL) for disease prediction with multi-modality, which is applicable to the scenarios of inductive learning.

1、我们提出了针对多模态疾病预测提出了MMGL，它可以被运用在归纳学习场景。

2、To characterize a patient with multi-modality, the pro posed modality-aware representation learning (MARL) obtains not only the modality-shared（模态共享） representation serv ing as commonality（共性）, but also the modality-specifified rep resentation that is patient-sensitive（患者敏感） as complementary.

2、为了描述一个多模态的病人，提出的MARL不仅将模态共享表征服务作为共性，同时将患者敏感的模态具体表征作为互补性。

3、To reveal the intrinsic relations among patients, an adaptive graph learning (AGL) is proposed to obtain a la tent graph structure to match flexibly for GNN-based downstream tasks. Furthermore, the unified（统一） modeling of MARL and AGL can be jointly optimized in an end-to- end way, facilitating more effificient training and inductive testing.

3、为了显示病人的内在联系，我们提出了AGL来获得潜在的图结构以灵活匹配以GNN为基础的下游任务。此外，MARL和AGL的统一模型可以以一种端到端的方式来嵌入以促进更有效的训练和归纳验证。

4、Compared to the state-of-the-art approaches, the comparable even signifificant improvement on two disease datasets indicates the advantages of our MMGL in terms

of disease prediction tasks. Meanwhile, the visualization of contribution score reflected by the obtained depen dencies among multi-modality also provides a modal- explainable decision support for doctors in real medical applications and inspiration for disease research.

4、与SOTA方法比较，在两个疾病数据集上的可比甚至重大改进表明MMGL的优势在疾病预测任务上。同时，多模态独立性的可视化贡献得分也为医生在真实医学应用和疾病研究提供了模型可解释性决策。

四、Methodology

Architecture(建筑) of three types of multi-modal shared representation learning. (a) Directly concatenation, (b) Intra-modal attention based weighted fusion. (c) Our modality-aware representation learning. The (a) and (b) fusions have only one interactive （交互）operation for different modal features, which proceed （继续）at the end of the module. In contrast（相反）, our module has more interactive operations through multi-modal attention for features from different modalities.

三个部分的多模态共享表征学习。（a）直接拼接（b）基于权重融合的模态间注意力（c）我们的模态感知表征学习。（a）和（b）仅是一种不同模态特征的交互操作，运用在单元末尾。相反，我们的单元通过不同模态特征的多模态注意力有更多的交互操作。

The architecture overview of our MMGL. The multi-modal features X is first embedded into the modality-specified representation space and the modality-shared representation space through the modal-aware representation learning. Then an adjacency matrix A for X is learned based on the adaptive graph learning. Finally, we could obtain the prediction results through a GNN based on A and H, where H = Concat(H sh, H sp).

我们MMGL的框架图。多模态特征X首先被嵌入到模态指定表征空间并且模态共享表征空间会通过模态感知表征学习。然后一个基于X的邻接矩阵A会在自适应图学习中被学习到。最终，我们通过基于A和H的GNN获得预测结果。H是Concat(H sh, H sp).

五、Conclusion

In this paper, we propose a multi-modal graph learning framework named MMGL for disease prediction. To capture the shared and complementary information among multi- modality, we propose modal-aware representation learning to simultaneously obtain the modality-specifified representa tion and the modality-shared representation considering inter- modal correlations. Furthermore, a lightweight （轻量）adaptive graph learning is proposed to reveal the intrinsic relations among subjects, which could construct an optimal graph structure for downstream tasks. Meanwhile, MMGL could be jointly optimized in an end-to-end way, which enables more efficient training and inductive testing. Our ongoing（不断的） research work will extend our MMGL to unified（统一的） graph learning for incomplete data and more biomedical tasks.

在这篇文章中，我们提出了一个多模态图学习框架(MMGL)用于疾病预测。为了捕捉到多模态直接的互补性和共享性，考虑到内在模态交互，我们提出了模态感知表征学习来同时获取模态指定表征和模态共享表征。此外，我们还提出了一个轻量的自适应图学习来获取任务之间的内在联系，这个方法可以为下游任务构造一个优化图结构。同时，MMGL可以以一种端到端的方式被嵌入，这种方式确保了更有效的训练和归纳测试。我们的后续研究将会将我们的MMGL扩展到在不完整数据和更多生物医学任务的统一的图学习上。

1.14组会 ------ Multi-modal Graph Learning 论文精读相关推荐

MindSpore Graph Learning
技术背景 MindSpore Graph Learning是一个基于MindSpore的高效易用的图学习框架.得益于MindSpore的图算融合能力,MindSpore Graph Learning能 ...
Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings
摘要:在本文中,我们提出了一个端到端的图学习框架,即迭代深度图学习(IDGL),用于联合和迭代地学习图结构和图嵌入.IDGL的关键理论是基于更好的节点嵌入来学习更好的图结构,反之亦然(即基于更好的图结 ...
Paddle Graph Learning (PGL)图学习之图游走类deepwalk、node2vec模型[系列四]
Paddle Graph Learning (PGL)图学习之图游走类模型[系列四] 更多详情参考:Paddle Graph Learning 图学习之图游走类模型[系列四] https://aist ...
论文阅读笔记：Low-Rank Tensor Graph Learning for Multi-view Subspace Clustering(LRTG)
@[TOC](Low-Rank Tensor Graph Learning for Multi-view Subspace Clustering(LRTG)) ★论文笔记-Low-Rank Tenso ...
[万字综述] 21年最新最全Graph Learning算法，建议收藏慢慢看
今天小编给大家带来了一篇极全的2021最新图学习算法综述.该综述不仅囊括了目前热门的基于深度学习的图学习方法,还全面介绍了其它三个大类:基于图信号处理的方法.基于矩阵分解的方法.基于随机游走的方法.因 ...
【GCN-RS】对比学习SGL：Self-supervised Graph Learning for Recommendation (SIGIR‘21)
Self-supervised Graph Learning for Recommendation (SIGIR'21) 还是何向南组提出的图自监督学习框架,对用户-物品的二部图上的节点.边做drop ...
文献学习(part88)--Graph Learning for Multiview Clustering
学习笔记,仅供参考,有错必纠文章目录 Graph Learning for Multiview Clustering 摘要 Introduction Related Work GRAPH LEARN ...
文献记录(part19)--Cross modal similarity learning with active queries
学习笔记,仅供参考,有错必纠关键词:主动学习;跨模态相似学习;度量学习文章目录 Cross modal similarity learning with active queries 摘要简介 ...
【论文】最新图学习推荐系统综述 | Graph Learning Approaches to Recommender
0 前言还记得在知乎上回答过两个关于图的相关问题: 问题1:有必要学习图论么? 当时我的回答是:广义上来讲,任何数据在赋范空间内都可以建立拓扑关联,也就是可以构成graph,所以学习图论很有用. ...

1.14组会 ------ Multi-modal Graph Learning 论文精读