Heterogeneous Dual-Core Overlay Processor for Light-Weight CNNs

Abstract: Light-weight convolutional neural networks (CNNs) have small complexity and are good candidates for low-power, high-throughput inference. Such networks are heterogeneous in terms of computation-to-communication (CTC) ratios and computation patterns between layers, especially for different layer types. Yet, existing AI processors either use homogeneous processing elements (PEs), resulting in low runtime PE efficiency, or run different layers on heterogeneous PEs in sequential, introducing resource redundancy. This paper proposes a heterogeneous dual-core architecture (dual-OPU), where one core is optimized for regular convolution layers and the other for depthwise convolution layers. PEs are homogeneous with each core. To make full use of dual-core parallelism, we develop a scheduling algorithm to concurrently execute layers for different input images on dual-core and balance parallel workload. Meanwhile, we automatically tune the PE number for a core and tune the input size for each PE to maximize throughput. Compared with a single-core processor with the same area for a single network, heterogeneous dual-OPU on average improves runtime PE efficiency and throughput by 11% and 31%, respectively. For a workload of multiple networks, dual-OPU improves average throughput by 11% compared with the state-of-the-art processors scaled to the same area. To the best of our knowledge, it is the first in-depth study on the heterogeneous dual-core processor for light-weight CNNs.