Regularization and Optimization strategies in Deep Convolutional Neural Network

Abstract: Convolution Neural Networks, known as ConvNets exceptionally perform well in many complex machine learning tasks. The architecture of ConvNets demands the huge and rich amount of data and involves with a vast number of parameters that leads the learning takes to be computationally expensive, slow convergence towards the global minima, trap in local minima with poor predictions. In some cases, architecture overfits the data and make the architecture difficult to generalise for new samples that were not in the training set samples. To address these limitations, many regularization and optimization strategies are developed for the past few years. Also, studies suggested that these techniques significantly increase the performance of the networks as well as reducing the computational cost. In implementing these techniques, one must thoroughly understand the theoretical concept of how this technique works in increasing the expressive power of the networks. This article is intended to provide the theoretical concepts and mathematical formulation of the most commonly used strategies in developing a ConvNet architecture.