Scene Structure Guidance Network: Unfolding Graph Partitioning into Pixel-Wise Feature Learning

Abstract: Understanding the informative structures of scenes is essential for low-level vision tasks. Unfortunately, it is difficult to obtain a concrete visual definition of the informative structures because influences of visual features are task-specific. In this paper, we propose a single general neural network architecture for extracting task-specific structure guidance for scenes. To do this, we first analyze traditional spectral clustering methods, which computes a set of eigenvectors to model a segmented graph forming small compact structures on image domains. We then unfold the traditional graph-partitioning problem into a learnable network, named \textit{Scene Structure Guidance Network (SSGNet)}, to represent the task-specific informative structures. The SSGNet yields a set of coefficients of eigenvectors that produces explicit feature representations of image structures. In addition, our SSGNet is light-weight ($\sim$ 56K parameters), and can be used as a plug-and-play module for off-the-shelf architectures. We optimize the SSGNet without any supervision by proposing two novel training losses that enforce task-specific scene structure generation during training. Our main contribution is to show that such a simple network can achieve state-of-the-art results for several low-level vision applications. We also demonstrate that our network generalizes well on unseen datasets, compared to existing methods which use structural embedding frameworks. We further propose a lighter version of SSGNet ($\sim$ 29K parameters) for depth computation, SSGNet-D, and successfully execute it on edge computing devices like Jetson AGX Orin, improving the performance of baseline network, even in the wild, with little computational delay.