X-LogSMask: Expand Transformer for Graph-Structured Data

Abstract: Transformers have become general-purpose architectures, but their all-to-all self-attention is poorly matched to graph data, whose interactions are sparse, structured and multi-scale. Existing Graph Transformers address this mismatch through structural encodings, hybrid message-passing modules or learned attention constraints, often introducing additional complexity and limited interpretability. Here we introduce X-LogSMask, an explainable multi-head logarithmic structural mask that injects symmetrically normalized graph topology directly into attention logits. The logarithmic transform converts structural connectivity into a topology-aware gating signal, suppressing unsupported node interactions while preserving feature-dependent attention. By assigning different powers of the normalized adjacency matrix to different attention heads, X-LogSMask gives each head a defined structural radius and supports multi-hop information propagation within a single layer. We further show that a standard Transformer encoder can be interpreted as one-step message passing on a complete graph, motivating X-LogSMask as a topology-constrained alternative to unrestricted self-attention. Across 20 node-, edge- and graph-level benchmarks, Transformers equipped with X-LogSMask achieve state-of-the-art performance on 13 datasets and remain competitive in a lightweight one-layer configuration. These results show that simple, interpretable structural masks can make self-attention an effective graph-learning operator without changing the Transformer architecture. The code is available at https://github.com/LiLeyan-0120/X-LogSMask.

• The paper introduces X-LogSMask, which injects explicit structural priors into Transformer attention using a logarithmically transformed, symmetrically normalized adjacency matrix.
• It assigns different matrix powers to distinct attention heads, capturing multi-hop dependencies and enabling interpretable, multi-scale receptive fields without deep stacking.
• Empirical results demonstrate state-of-the-art performance on various graph tasks, outperforming baselines in efficiency and accuracy across node, edge, and graph-level benchmarks.

X-LogSMask: Explainable Multi-Head Logarithmic Structural Mask for Transformer-Based Graph Learning

Context and Motivation

Transformers have demonstrated substantial efficacy as general-purpose architectures across various domains, including NLP, vision, and time series. However, their canonical self-attention mechanism, which allows unconstrained all-to-all interactions, lacks alignment with the sparse, irregular, and multi-scale dependencies intrinsic to graph-structured data.

Prior approaches for graph representation learning have primarily relied on GNNs, which integrate graph topology through localized message passing. Recent advances in adapting Transformers to graph data involve hybridization with GNN modules, incorporation of structural biases via positional encodings, or the explicit constraint of attention mechanisms. These adaptations tend to increase architectural complexity and often obscure the interpretability and explicit control of the structural bias.

X-LogSMask: Methodological Contributions

The central contribution of the paper is the introduction of X-LogSMask—a modular, additive, and interpretable structural masking approach that injects topological priors directly into the Transformer attention mechanism without altering its core architecture.

Figure 1: Transformer for all kinds of data.

Structural Masking Mechanism

X-LogSMask constructs a mask from the symmetrically normalized adjacency matrix of the input graph, incorporating self-loops. A logarithmic transform is then applied to compress the dynamic range, yielding an attention-logit bias:

$$\mathbf{M}_{ij} = \log(\tilde{\mathbf{A}}_{ij} + \epsilon)$$

where $\tilde{\mathbf{A}}$ is the normalized adjacency matrix and $\epsilon$ ensures numerical stability. This mask is added directly to the query-key attention energy before softmax. By exponentiating, the structural mask effectively acts as a multiplicative gating term, suppressing unsupported node interactions while preserving sensitivity to feature-derived attention.

Explainable Multi-Head Design

To encode multi-hop dependencies within a single Transformer layer and offer explicit interpretability, X-LogSMask assigns different matrix powers of the normalized adjacency to different attention heads. Thus, each head has a defined receptive field radius (from local to higher-order neighborhoods), and multi-scale graph context is captured in parallel. There is no need for deep stacking as in traditional GNNs.

Figure 2: Model architecture and structural masking mechanism of X-LogSMask, including head-wise assignment of multi-hop adjacency powers for multi-scale receptive fields.

Theoretical Underpinnings

The paper establishes that a standard Transformer encoder can be interpreted as a message-passing neural network (MPNN) on a fully connected graph, with every node communicating with every other. Without explicit structural bias, this induces non-discriminative, topology-free aggregation.

By incorporating X-LogSMask, the propagation pattern transitions from indiscriminate full-graph communication to topology-constrained message passing. The multi-hop head-wise decomposition associates the model’s attention patterns with structurally explainable paths and radii, facilitating both interpretability and efficient long-range dependency capture.

Empirical Evaluation

Extensive benchmarks on node-, edge-, and graph-level tasks (20 datasets total) assess the effectiveness of X-LogSMask. Notably, X-LogSMask achieves state-of-the-art (SOTA) performance on 13 datasets, and the results highlight several strong and contradictory claims relative to prior work:

• On graph-level benchmarks, X-LogSMask exhibits consistent SOTA performance, with the most pronounced gains on protein and molecular datasets (e.g., PROTEINS, 80.63% vs. 75.77% for the previous best).
• On edge-level tasks, both full and 1-layer variants of X-LogSMask outperform all non-structurally-masked baselines by a substantial margin (e.g., in link prediction, >20 MRR point improvement on Cora; in edge regression, an order-of-magnitude reduction in MAE/RMSE on epic-games-plr).
• A single X-LogSMask layer frequently matches or exceeds performance of much deeper Transformer or GNN architectures, indicating successful receptive field expansion via head-wise specialization.

  Figure 3: Average ranks of X-LogSMask variants across task families, illustrating top-tier relative performance for both full and one-layer configurations.

  

  Figure 4: Layer-wise attention decomposition on PROTEINS, demonstrating that raw attention energy encodes node-attribute similarity, X-LogSMask injects edge structure, and their sum couples content and topology with head-specific multi-hop focus.

Ablation and Interpretability

Ablation studies confirm that symmetric normalization and the logarithmic masking transformation are critical for stable and effective graph learning. The multi-head masking scheme enhances interpretability (assigning distinct hop radii to each head) and can substitute for architectural depth in several cases, reducing over-smoothing and over-squashing commonly observed in deep GNNs.

Visualization of the decomposed attention scores reveals that low-order heads focus on local graph structure, while higher-order heads attend to longer-range context within a single layer. During training, the attention pattern becomes progressively more GNN-like at deeper layers, further validating the coupling of structural and feature-based inductive biases.

Implications and Prospective Directions

Practically, X-LogSMask offers a drop-in, architecture-conservative extension for Transformer models that require graph sensitivity, with negligible increase in computational overhead. Its core design—injecting structured, multi-scale masks into attention logits—may generalize to grid (vision) and sequential (temporal) domains, as preliminary CIFAR-10 experiments suggest accuracy improvements over vanilla ViT.

Theoretically, the approach bridges the gap between MPNN theory and the flexibility of full self-attention, offering explicit and interpretable control over model inductive bias.

Future research directions include:

• Scaling X-LogSMask to ultra-large and dynamic graphs leveraging sparse attention or approximate masking schemes.
• Systematic investigation of optimal head allocations and structural powers for different graph types.
• Application to temporal and heterogeneous graph representation, or in multi-modal structured data integration.
• Integration with neural architecture search for automated selection of mask granularity and head distribution.

Conclusion

X-LogSMask provides a concise and effective mechanism for incorporating explicit, interpretable, multi-scale structural priors into Transformer-based graph learners. By leveraging symmetrically normalized, logarithmically transformed adjacency-derived masks and headwise specialization, the approach achieves SOTA performance on diverse graph benchmarks and enables efficient, shallow model configurations. The paradigm has broad implications for structurally-biased attention in deep learning and provides a theoretically sound and practically versatile foundation for future graph-aware architectures.