검색 상세

Attention Mechanism and Spectral Domain Insights on Message Passing for Graph Attention Network

초록/요약

Graph Neural Networks (GNN) have become a powerful tool for analyzing and processing graph-structured data, finding widespread applications across various domains. Unlike traditional Convolutional Neural Networks (CNN) designed for grid-like data, GNN are more adept at leveraging topological structures to handle non-Euclidean data. In GNN, message passing is a fundamental mechanism that enables nodes to update their features by aggregating information from their neighbors. This dissertation aims to address two core research objectives: (1) In existing research, message passing typically relies on learnable weights obtained through training, as demonstrated in studies like GAT and FAGCN, which turns it into a "black box" and limits the interpretability of GNN. Our research directly computes attention coefficients using neighborhood information from the graph, proving their effectiveness in message passing, and (2) GNNs inherently function as low-pass filters. In our study, we design independent dual-channel spectral graph neural networks under a multi-perspective framework, with one channel dominated by high-frequency and the other by low-frequency information. This design overcomes performance bottlenecks in current methods and enhances the interpretability of network structures. To achieve these goals, this dissertation focuses on structure information and spectral domain GNN designs, systematically improving GNN performance across three interrelated dimensions: 1. Exploration of Neighborhood Information in Graph Data: Conventional approaches often use learnable parameters throughout the message passing process. In contrast, our research uses the degrees of 1-hop and 2-hop neighbors as the raw data for computing attention coefficients in message passing. By evaluating the significance of each neighbor within its local neighborhood, we calculate attention coefficients between each node and its neighbors. Based on this specialized attention mechanism, we propose the Local Neighborhood Graph Attention Layer (LNGAL) and the Local Neighborhood Graph Attention Network (LNGAT). Experiments on various graph datasets show that LNGAT achieves notable improvements, highlighting the crucial role of topological structure in graph data. As the attention coefficients in LNGAT are directly computed from structural information, the model requires fewer parameters and converges faster. 2. Adaptive High and Low Frequency Dual-Channel Spectral Domain Exploration: Traditional attention coefficients are positive, effectively making GNNs behave as low-pass filters. We propose the Negative Graph Attention Network (NGAT), which sets all attention coefficients to negative values and demonstrates performance gains on heterophilic graph datasets. Building upon this, we propose the High-frequency and Low-frequency Dual-channel Graph Attention Network (HLGAT), where message passing occurs through two separate channels—one dominated by high-frequency components and the other by low-frequency components. The aggregation coefficients in the high-frequency channel are negative, while those in the low-frequency channel are positive. The final node representation is obtained by aggregating information from both frequency domains. This dual-channel approach enhances frequency representation, leading to substantial performance improvements on multiple heterophilic graph datasets. 3. Integration of Neighborhood Information with Spectral-domain GNN: To fully leverage neighborhood information and spectral-domain GNNs, we propose the Fourier-Wavelet Dual-View Residual Graph Convolutional Network (FWGCN). To capture the influence of neighborhood information on node representations, we incorporate multi-hop neighborhood information as network input. The network architecture combines the Fourier and wavelet bases to design a residual network, where the Fourier basis excels at extracting global features and the wavelet basis captures local features. Moreover, FWGCN incorporates distinct high-frequency and low-frequency channels in both views, thereby improving the model’s capacity to capture multi-scale and multi-frequency features. Experimental results demonstrate that FWGCN consistently outperforms other models, achieving leading performance across various graph datasets. The proposed models—LNGAT, HLGAT, and FWGCN—exhibit outstanding performance across a wide range of tasks and datasets, marking significant progress toward developing more robust and interpretable GNN architectures.

more

목차

1 Introduction 1
1.1 Motivation 1
1.2 Contributions of the Dissertation 4
1.3 Overview of the Dissertation 6
2 Background 8
2.1 Graph-Structured Data Overview 8
2.2 Current State of Research on Graph Neural Networks 11
2.2.1 Spatial Domain Graph Neural Network 11
2.2.2 Spectral Domain Graph Neural Network 13
2.3 Concepts and Formulations of Spectral Domain Graph Neural Network 15
2.4 Why Spectral Domain Graph Neural Network 17
2.5 Summary of Chapter 19
3 LNGAL: Attention coefficients can be calculated instead of being trained 20
3.1 Introduction 21
3.2 Related works 23
3.2.1 Graph Convolutional Neural Networks 23
3.2.2 Graph Attention Networks 24
3.3 Methodology 26
3.3.1 Notations 26
3.3.2 Local Neighborhood Graph Attention Layer (LNGAL) 26
3.3.3 The architecture of Multi-scale Network 30
3.4 Experiments 32
3.4.1 Datasets and Implementation Setting 32
3.4.2 Performance on Classification 33
3.4.3 Evaluation and Analysis of LNGAL 34
3.4.4 Convergence Speed Evaluation 36
3.4.5 Node Feature Embedding Visualization 36
3.4.6 Ablation Study 38
3.4.7 Impact of Local Neighborhood Depth 39
3.4.8 Effect of Hyperparameter on Neighbor Information Integration 40
3.4.9 Ablation of Architecture of Network 40
3.5 Summary of Chapter 41
4 HLGAT: A daul-channel way to combine high-frequency and low-frequency 42
4.1 Introduction 43
4.2 Related Works 47
4.2.1 Introducing Attention Mechanisms into GNNs 47
4.2.2 Homophilic vs. Heterophilic Graphs 49
4.3 Preliminaries 50
4.3.1 Graph Signal Filtering 50
4.3.2 Graph-based Information Aggregation 52
4.4 Proposed Models 54
4.4.1 Negative Graph Attention Network (NGAT) 54
4.4.2 High-frequency and Low-frequency Dual-channel Graph Attention Network (HLGAT) 56
4.5 Experiments 60
4.5.1 Datasets 60
4.5.2 Experimental Setup 61
4.5.3 Evaluation of Our Proposed Models 63
4.5.4 Ablation Evaluation of Learnable Representation Integration 65
4.5.5 Impact of Stacking HLGAT Layers 66
4.5.6 Edge Coefficient Visualization 68
4.5.7 Effect of Activation Function Slopes a and b on Performance 69
4.5.8 Influence of Normalization on Coefficients 70
4.5.9 Evaluation on Homophilic Graph Datasets 71
4.6 Summary of Chapter 73
5 FWGCN: A residual GCN framework with Fourier and Wavelet based graph convolutional network modules 75
5.1 Introduction 76
5.2 Related Works 79
5.2.1 Fourier-based Graph Neural Network 79
5.2.2 Wavelet-based Graph Neural Network 80
5.3 Proposed Method: FWGCN 80
5.3.1 Fourier-based and Wavelet-based Graph 82
5.3.2 Dual-channel Fourier-based Graph Structure 83
5.3.3 Dual-channel Wavelet-based Graph Structure 86
5.3.4 Feature Aggregation 88
5.4 Experiments 88
5.4.1 Datasets 88
5.4.2 Parameter Setup 88
5.4.3 The performance of FWGCN 89
5.4.4 Ablation Experiments: Effect of Structure 91
5.4.5 Ablation experiment: the impact of Hop 93
5.4.6 Ablation experiment: Fusion method 93
5.4.7 Visualization of FWGCN 94
5.5 Summary of Chapter 96
6 Conclusions 97
6.1 Conclusion 97
6.2 Further Works 98

more
반출 Meta View 목록