[1911.09464v1] Quantization Networks
This work focused on interpreting and implementing low-bit quantization of deep neural networks from the perspective of non-linear functions

Abstract Although deep neural networks are highly effective, their high computational and memory costs severely challenge their applications on portable devices. As a consequence, low-bit quantization, which converts a full-precision neural network into a low-bitwidth integer version, has been an active and promising research topic. Existing methods formulate the low-bit quantization of networks as an approximation or optimization problem. Approximationbased methods confront the gradient mismatch problem, while optimization-based methods are only suitable for quantizing weights and could introduce high computational cost in the training stage. In this paper, we propose a novel perspective of interpreting and implementing neural network quantization by formulating low-bit quantization as a differentiable non-linear function (termed quantization function). The proposed quantization function can be learned in a lossless and end-to-end manner and works for any weights and activations of neural networks in a simple and uniform way. Extensive experiments on image classification and object detection tasks show that our quantization networks outperform the state-of-the-art methods. We believe that the proposed method will shed new insights on the interpretation of neural network quantization. Our code is available at https://github.com/aliyun/ alibabacloud-quantization-networks.
‹Figure 1: Non-linear functions used in neural networks. (Introduction)Figure 2: The relaxation process of a quantization function during training, which goes from a straight line to steps as the temperature T increases. (Related Work)Figure 3: The distribution of full-precision parameters in ResNet-18. (a)(b)(c) are the distributions of parameters of the first convolution layers from Block2 to Block4 before quantization training. (Image Classification)Figure 4: The gap between the training model and testing model along with the training process for ResNet-18 {−4, +4}. The gap between training and testing model converges when learning proceeds. (Ablation Experiments)

Figure 5: The training error curve and the training/validation accuracy curve for AlexNet quantization (left to right: T = 5/10/20 ∗ epoch). Similar curves are observed for T = 1/30/40 ∗ epoch, we do not show here because of the limit of space. (Ablation Experiments)›