Path-Length Regularization

• Path-length regularization is a method that penalizes cumulative path properties in models by using algebraic (path-norm) or probabilistic (optimal transport) measures to induce sparsity and control exploration.
• In GFlowNets, it quantifies the divergence between successive forward policies, balancing mode-seeking generalization with increased sample diversity by adjusting the regularizer's sign and magnitude.
• In feedforward and ReLU networks, algebraic path-norms reformulate weight regularization into convex problems that promote group sparsity and efficient optimization.

Path-length regularization refers to a family of methods that penalize or constrain the cumulative properties of paths in a model—such as in neural networks or graph-structured policies—by introducing an explicit regularizer that measures aggregate properties along those paths. This approach leverages either algebraic (path-norm) or probabilistic distances (e.g., optimal transport on policy flows) to induce desirable inductive biases such as sparsity, generalization, convexity, or controlled exploration.

1. Path-Length Regularization in Generative Flow Networks

In the context of Generative Flow Networks (GFlowNets), path-length regularization is formulated as a principled penalty between the forward policy distributions at successive states along trajectories in a directed acyclic graph. For a complete trajectory $\tau = (s_0 \to \cdots \to x \to s_f)$, the flow $F(\tau)$ defines forward and backward policies:

$$P_F(s'\mid s)=\frac{F(s\to s')}{F(s)}, \qquad P_B(s\mid s')=\frac{F(s\to s')}{F(s')}.$$

The core regularization is based on a directed distance $$d(s,s') = -\log \max_{\pi:s\to\cdots\to s'}P(\pi\mid s)$$, where $P(\pi\mid s)$ is the (forward or backward) path probability. The regularizer between two successive states $s, s'$ quantifies an optimal transport (OT) distance between their children’s forward distributions:

$$\mathrm{OT}_{\mathbf C}(\alpha, \beta) = \min_{\pi\in\Pi(\alpha,\beta)} \sum_{i,j} \pi_{ij}\,C_{ij}$$

where $\alpha$ and $\beta$ are the forward probabilities over children of $s$ and $s'$, and $C_{ij}$ is the directed distance between child pairs. The total path-length regularizer over $\tau$ is the sum over each step:

$$\mathcal{L}_{\rm OT}(\tau) = \sum_{t=0}^{n-1} \mathrm{OT}_{\mathbf{C}_t}\bigl(P_F(\cdot\mid s_t),\,P_F(\cdot\mid s_{t+1})\bigr)$$

Minimizing this term promotes mode-seeking and generalization (i.e., flow is concentrated along short, high-probability paths), while maximizing it increases exploration and sample diversity by encouraging divergence between successive forward policies. The tradeoff between exploration and generalization in GFlowNets can therefore be directly controlled by the sign and magnitude of the path-length regularizer (Do et al., 2022).

2. Algebraic Path-Norms in Feedforward and Parallel Neural Networks

In multilayer perceptrons (MLPs) and parallel deep ReLU architectures, path-length regularization often takes the explicit form of a path-norm. For an MLP with weights $W_1, \cdots, W_K$, the $1$-path-norm is

$$P_1(\mathbf{W}) = \sum_{p\in\text{paths}} \prod_{(k,i\to j)\in p} |W_k[j, i]| = \mathbf{1}^\top |W_K|\,\cdots\,|W_1|\,\mathbf{1}$$

This summation, across all input-output paths, aggregates the products of absolute weight magnitudes along each path (Biswas, 2024). In parallel ReLU networks, a path-norm can involve $\ell_2$ aggregates over all possible paths:

$$R(\theta) := \sum_{k=1}^{K} \sqrt{\sum_{j_1, \dots, j_L} (\|w_{1k,j_1}\|_2^2 \prod_{\ell=2}^{L} w_{\ell k, j_{\ell-1}j_\ell}^2)}$$

This regularization introduces both convexity and sparsity due to the norm structure, allowing the reformulation of non-convex training objectives as convex programs with group-sparsity (Ergen et al., 2021).

3. Computational Strategies and Efficient Approximations

Path-length regularizations involving OT distances can be computationally demanding. In GFlowNets, closed-form solutions are available when the cost matrix $\mathbf C$ is diagonal (e.g., in settings without action decomposition and with aligned action labels), reducing the computational complexity. When the closed form is unavailable or the action space is large, an efficient upper bound involving (cross-)entropy and log-probability is used:

$$\mathrm{OT}\big(P_F(\cdot|s), P_F(\cdot|s')\big) \le \mathbf{H}(P_F(\cdot|s), P^*_B(\cdot|s)) - \log P_F(s'|s) + \mathbf{H}(P_F(\cdot|s'))$$

This bound can be evaluated in $O(d)$ per edge, where $d$ is the support size, making it practical for large-scale problems (Do et al., 2022).

For algebraic path-norms in overparameterized networks, architecture-specific simplifications (such as in PSiLON Nets with $L_1$-weight normalization) enable $O(K)$-time evaluation by collapsing the path products:

$$P_1(\mathbf{W}) = \|\mathbf{g}_K\|_1 \prod_{k=1}^{K-1} |g_k|$$

where $g_k$ are shared layer scaling parameters (Biswas, 2024).

4. Convexity and Group Sparsity via Path-Norm Regularization

In parallel ReLU networks, path-norm regularization fundamentally alters the optimization landscape by making the training objective convex after appropriate reparameterization. The resulting convex objective

$$\min_{z,z'} L(\tilde X(z-z'), y) + \frac{\beta}{\sqrt{m_2}}(\|z\|_{F,1}+\|z'\|_{F,1})$$

incorporates a group-sparsity-inducing block norm $\|\cdot\|_{F,1}$, which penalizes whole groups (corresponding to hyperplane-arrangements or subnetworks) and yields parsimony in high dimensions (Ergen et al., 2021).

These convex formulations guarantee global optimality and allow efficient algorithms by leveraging low-rank approximations to the data and exploiting combinatorial structure in the underlying hyperplane arrangements. Polynomial-time solvers are possible for fixed rank, and empirical results confirm that convex path-norm solutions are both group-sparse and competitive in accuracy on benchmarks.

5. Exploration, Generalization, and Optimization Dynamics

Path-length regularization directly mediates the exploration-generalization tradeoff by shaping the geometry of the flow or the function class. In GFlowNets:

• Minimizing path-length regularization ($\lambda>0$) focuses flow onto shorter, high-probability paths, promoting mode-seeking and improved generalization in low-dimensional or structured target distributions.
• Maximizing the same term ($\lambda<0$) forces forward policies to diverge, increasing path-entropy and thus enhancing sample diversity and novelty without catastrophic loss in reward (Do et al., 2022).

In conventional MLPs and ResNets, empirical results show that 1-path-norm regularization not only sharpens the empirical-to-generalization transition but also yields high near-sparsity (with effective pruning without loss of accuracy) and increased optimization stability compared to standard $L_2$ weight decay, especially in the small data regime (Biswas, 2024).

6. Implementation Practices and Empirical Results

Path-length regularization is typically integrated additively into the training loss:

$$\mathcal{L}_{\mathrm{reg}}(\mathbf{W}) = \mathcal{L}_0(\mathbf{W}) + \lambda\,R(\mathbf{W})$$

where $R(\mathbf{W})$ denotes the path-norm or OT-based regularizer, and $\lambda$ controls the tradeoff. In GFlowNets, practical implementations alternate between evaluating the exact OT regularizer, its closed or upper-bound form, and trajectory subsampling ("Dropout OT") for further efficiency (Do et al., 2022). In path-norm-regularized feedforward and residual networks, algorithmic schemes enable smooth transitions to exact sparsity via layerwise parameterization and pruning procedures (Biswas, 2024).

Empirical studies consistently show that path-length regularization provides marked improvements in:

• Convergence speed and optimization for convexified ReLU networks (Ergen et al., 2021)
• Generalization and expressivity in overparameterized residual architectures (Biswas, 2024)
• Mode discovery and diversity control in GFlowNet-based compositional generation tasks (Do et al., 2022)

7. Broader Methodological Connections

Path-length regularization unifies concepts from optimal transport, entropy-regularized mirror descent, and convex analysis. In online graph traversal, entropic regularizers assign potentials over distributions on evolving trees, yielding path selection strategies that are provably $O(k^2)$-competitive, leveraging depth-weighted entropy to smooth path choices and prevent over-concentration under adversarial uncertainty (Bubeck et al., 2022).

In summary, path-length regularization provides a flexible and theoretically grounded framework for controlling the geometry, convexity, and expressivity of model classes across generative, supervised, and online learning scenarios. Both analytical and empirical evidence supports its effectiveness in inducing sparsity, enhancing generalization, and managing trade-offs between exploration and exploitation.