Differentiable Directed Acyclic Graphs (D-DAG)

• Differentiable Directed Acyclic Graph (D-DAG) is a framework that relaxes the acyclicity constraint into a differentiable penalty, enabling continuous optimization of DAG structures.
• It integrates linear and nonlinear models using analytic, log-determinant, and power-series constraints to enforce the DAG condition while scaling to high dimensions.
• The approach underpins advances in causal discovery, ensemble consensus, nonparametric methods, and dynamic graph extensions, yielding improved structural recovery and computational efficiency.

A Differentiable Directed Acyclic Graph (D-DAG) is a theoretical and algorithmic framework enabling the learning of DAG structures from observational data via continuous, gradient-based optimization. Unlike classical combinatorial approaches, D-DAGs encode the acyclicity constraint as a differentiable function of the real-valued adjacency matrix, rendering the structure search tractable for high-dimensional problems and amenable to modern automatic differentiation. The approach underpins a spectrum of recent advances in causal discovery, ranging from linear and nonlinear models, analytic acyclicity constraints, ensemble consensus, and dynamic graph extensions, to nonparametric function classes and scalable optimization schemes.

1. Differentiable Structure Learning of DAGs

D-DAG frameworks cast the search for acyclic graph structures as a constrained optimization problem in the continuous space of real-weighted adjacency matrices. For observed data $D \in \mathbb{R}^{n \times d}$ on $d$ variables $X_1,\dots,X_d$, one posits a structural equations model:

$$X_j = f_j(X; \theta_j) + \epsilon_j, \quad j=1,\ldots,d,$$

where $f_j$ is a potentially nonlinear function parameterized by $\theta_j$. The existence and strength of an edge $X_i \to X_j$ are encoded in a real matrix $A \in \mathbb{R}^{d \times d}$, with $A_{ij}$ parameterizing the effect.

In the linear NOTEARS paradigm, $f_j(X; \theta_j) = \sum_{i=1}^d A_{ij} X_i$, enabling a compact data fit $X \approx X A$ (Berrevoets et al., 2022). For nonlinear, neural or kernel-based settings, $A_{ij}$ gates respective input pathways in more expressive mappings (Lachapelle et al., 2019, Liang et al., 2024).

The key innovation is to relax the combinatorial acyclicity constraint by enforcing a smooth penalty function $h(A) = 0$ that characterizes the space of DAG adjacency matrices, facilitating direct optimization with standard gradient-based algorithms.

2. Acyclicity Constraints: Analytic Foundations and Implementation

A central challenge is the continuous characterization of acyclicity. Classical approaches use the matrix-exponential trace function:

$$h_{\exp}(A) = \operatorname{tr}\left( e^{A \odot A} \right) - d,$$

where $\odot$ denotes element-wise multiplication and $h_{\exp}(A) = 0$ if and only if $A$ encodes a DAG (Berrevoets et al., 2022, Fan et al., 2022). Variations employ analytic functions of the entrywise-squared adjacency, generalizing $h$ to:

$h_f(B) = \operatorname{tr} f(B \odot B) - c_0 d,$

for analytic $f(x) = c_0 + \sum_{i=1}^\infty c_i x^i$ with $c_i > 0$ (Zhang et al., 24 Mar 2025). This family encompasses the exponential, log-determinant, and polynomial powers. The log-det form,

$$h_{\text{ldet}}^s(A) = -\log \det(s I - A) + d \log s,$$

provides favorable gradient properties and numerical stability near the DAG boundary (Liang et al., 2024).

Advanced formulations (e.g., Truncated Matrix Power Iteration, TMPI) replace decaying coefficients of high-order powers with geometric or uniform weighting, preventing vanishing gradients and enhancing constraint expressiveness:

$$h_{\text{geo}}(A) = \sum_{k=1}^d \operatorname{tr}(A^k)$$

with truncation at order $k$ determined by spectral norm thresholds (Zhang et al., 2022).

Efficient evaluation schemes exploit power series doubling and automatic differentiation for scalable training (Zhang et al., 24 Mar 2025, Zhang et al., 2022). All gradient terms can be computed in closed-form or via standard autodiff frameworks.

3. Differentiable Optimization Objectives and Algorithms

The D-DAG empirical loss typically comprises a data-fitting term $F(A; D)$, sparsity regularization (e.g., $\ell_1$ penalty $\|A\|_1$), and an acyclicity constraint implemented via an augmented Lagrangian:

$$L(A; \alpha, \rho) = F(A; D) + \lambda \|A\|_1 + \alpha h(A) + \frac{1}{2} \rho h(A)^2,$$

with periodic updates to multipliers $\alpha$ and the penalty $\rho$ (Berrevoets et al., 2022).

Extensions for nonlinear generative models include neural parameterizations (GraN-DAG) or kernel-based representers (RKHS-DAGMA), with the acyclicity constraint applied to derived (e.g., derivative-norm or path-strength) adjacency matrices (Lachapelle et al., 2019, Liang et al., 2024). Nonlinear models admit highly expressive function classes while preserving the continuous and acyclic structure via the differentiable penalty.

Training algorithms interleave unconstrained gradient-based minimization (e.g., L-BFGS-B, Adam) with constraint updates, optionally employing fast matrix methods for high-dimensional settings (Zhang et al., 24 Mar 2025, Zhang et al., 2022). Large-scale or stochastic variants employ projection-based schemes substituting expensive spectral penalties with approximate adjacency projections to the closest DAG (e.g., $\psi$DAG) (Ziu et al., 2024).

4. Extensions: Ensembles, Nonparametric Methods, and Dynamic Graphs

Ensemble Learning and Transportability

The D-Struct architecture extends D-DAGs to handle multi-source or resampled datasets by learning an ensemble $\{A^{(k)}\}_{k=1}^K$ of adjacency matrices, regularizing them toward consensus:

$$L_{\text{MSE}}(A^{(k)}) = \|A^{(k)} - \bar{A}\|_2^2, \quad \bar{A} = \frac{1}{K} \sum_{k=1}^K A^{(k)},$$

with $\bar{A}$ detached from gradient flow. This transportability constraint—absent in vanilla NOTEARS—ensures structure learned on one dataset generalizes to others in the same domain, and achieves substantial empirical improvements in structural Hamming distance and agreement (Berrevoets et al., 2022).

Nonparametric and Kernel-Based Models

RKHS-DAGMA operationalizes nonparametric SEMs in reproducing kernel Hilbert spaces. Edge presence is inferred via empirical norms of partial derivatives, forming an adjacency $W_{kj} = \|\partial_{x_k} f_j\|_n$, and acyclicity is enforced via the log-determinant penalty on $W \odot W$ (Liang et al., 2024). The extended representer theorem justifies finite-basis kernel expansions incorporating kernel derivatives.

Dynamic Graphs

GraphNOTEARS (D-DAG for dynamic graphs) jointly learns contemporaneous and time-lagged dependencies in vector autoregressive settings with dynamic node features and adjacency matrices. The acyclicity constraint is imposed only on intra-slice coefficients, leveraging the directed nature of lagged effects (Fan et al., 2022).

5. Algorithmic Complexity, Scalability, and Practicalities

Matrix exponential and log-determinant based acyclicity constraints incur $O(d^3)$ cost per iteration, becoming costly at scale. TMPI and related series-doubling algorithms reduce per-iteration complexity to $O(d^3 \log k)$ (Zhang et al., 24 Mar 2025, Zhang et al., 2022). Projection-based methods ($\psi$DAG) further lower the barrier by interleaving cheap SGD steps and $O(d^2)$ DAG projections via topological sorting and active edge masking, enabling structure learning at tens of thousands of nodes (Ziu et al., 2024).

Theoretical error bounds and gradient control offer guarantees on constraint approximation and convergence to stationary points, consistent with standard smooth nonconvex optimization guarantees.

6. Theoretical Properties and Empirical Performance

The analytic characterization of DAGs via trace, determinant, and power-series constraints establishes exact equivalence to acyclicity for the respective class of real matrices (Berrevoets et al., 2022, Zhang et al., 24 Mar 2025). Closure properties under operator functionals (differentiation, summation, multiplication) enable principled construction and stacking of new constraints (Zhang et al., 24 Mar 2025).

Empirically, D-DAG approaches (and their variants) yield substantial gains over combinatorial or greedy alternatives:

• In both linear and nonlinear SEMs, finite-radius or high-order geometric constraints (TMPI, higher-order analytic penalties) systematically lower structural Hamming distance compared to exponential (NOTEARS) or binomial surrogates, often by factors of $2$–$3$ (Zhang et al., 2022, Zhang et al., 24 Mar 2025).
• Transportable learners (D-Struct) cut SHD by 20–40% and converge up to $10\times$ faster than single-dataset NOTEARS-MLP (Berrevoets et al., 2022).
• Nonparametric and RKHS-based D-DAGs exhibit stability, edge accuracy, and competitive or superior recovery on real causal networks (e.g., Sachs protein signaling) (Liang et al., 2024).
• Stochastic projected algorithms ($\psi$DAG) achieve comparable structure recovery at $10\times$ the scale and order-of-magnitude reduction in runtime (Ziu et al., 2024).

7. Mathematical Innovations and Connections

Recent work has established a deep correspondence between analytic function-based matrix criteria for acyclicity (including the exponential, log-determinant, and power series families), the nilpotency of adjacency matrices, and structural properties of the underlying graph (Zhang et al., 24 Mar 2025, Zhang et al., 2022). For efficiency, the field leverages convergence of infinite series, truncation with rigorous error control, and functional closure properties.

Ensemble consensus (D-Struct), kernel-representer-based sparsity (RKHS-DAGMA), projection via Hodge decompositions (DAG-NoCurl), and stochastic approximation with DAG projections ($\psi$DAG) represent methodologically diverse but mathematically unified innovations under the D-DAG paradigm, all centered on tractable differentiable parameterizations of acyclic graph structure and rigorous empirical validation.