Differentiable Programming: Efficient Smoothing of Control-Flow-Induced Discontinuities

Abstract: We want to obtain derivatives in discontinuous program code, where default Algorithmic Differentiation may not perform well. Specifically, we consider discontinuities induced by control flow statements, where meaningful derivatives should ideally be capable of representing the resulting kinks in the trajectory. To achieve this, one can interpolate the trajectory at the control flow statements before taking the derivative. We formulate a method to efficiently interpolate between all boundaries induced by control flow in program code. Theoretically, code can be viewed as a series of piecewise continuous functions applied in succession. These functions are nested inside one another and result in a function composition with several cases. We interpret this function composition as a tree and devise a heuristic to identify paths that are relevant to the interpolation. This allows us to conceive a language that smoothly interpolates control-flow statements automatically and efficiently, making it fully differentiable.