On the Expressiveness of Multi-Neuron Convex Relaxations

Abstract: To provide robustness guarantees, neural network certification methods heavily rely on convex relaxations. The imprecision of these convex relaxations, however, is a major obstacle: even the most precise single-neuron relaxation is incomplete for general ReLU networks, a phenomenon referred to as the single-neuron convex barrier. While heuristic instantiations of multi-neuron relaxations have been proposed to circumvent this barrier in practice, their theoretical properties remain largely unknown. In this work, we conduct the first rigorous study of the expressiveness of multi-neuron relaxations. We first show that the ``$\max$'' function in $\mathbb{R}^d$ can be encoded by a ReLU network and exactly bounded by a multi-neuron relaxation, which is impossible for any single-neuron relaxation. Further, we prove that multi-neuron relaxations can be turned into complete verifiers by semantic-preserving structural transformations or by input space partitioning that enjoys improved worst-case partition complexity. We also show that without these augmentations, the completeness guarantee can no longer be obtained, and the relaxation error of every multi-neuron relaxation can be unbounded. To the best of our knowledge, this is the first work to provide an extensive characterization of multi-neuron relaxations and their expressiveness in neural network certification.