Fundamental Limits of Distributed Linearly Separable Computation under Cyclic Assignment

Abstract: This paper studies the master-worker distributed linearly separable computation problem, where the considered computation task, referred to as linearly separable function, is a typical linear transform model widely used in cooperative distributed gradient coding, real-time rendering, linear transformers, etc. %A master asks $\Nsf$ distributed workers to compute a linearly separable function from $\Ksf$ datasets. The computation task on $\Ksf$ datasets can be expressed as $\Ksf_{\rm c}$ linear combinations of $\Ksf$ messages, where each message is the output of an individual function on one dataset. Straggler effect is also considered, such that from the answers of any $\Nsf_{\rm r}$ of the $\Nsf$ distributed workers, the master should accomplish the task. The computation cost is defined as the number of datasets assigned to each worker, while the communication cost is defined as the number of (coded) messages that should be received. The objective is to characterize the optimal tradeoff between the computation and communication costs. The problem has remained so far open, even under the cyclic data assignment.Since in fact various distributed computing schemes were proposed in the literature under the cyclic data assignment, with this paper we close the problem for the cyclic assignment. This paper proposes a new computing scheme with the cyclic assignment based on the concept of interference alignment, by treating each message which cannot be computed by a worker as an interference from this worker. Under the cyclic assignment, the proposed computing scheme is then proved to be optimal when $\Nsf=\Ksf$ and be order optimal within a factor of $2$ otherwise.