[1910.10853] Circulant Binary Convolutional Networks: Enhancing the Performance of 1-bit DCNNs with Circulant Back Propagation
Our extensive experiments demonstrate that CBCNs have superiority over stateof-the-art binary networks, and obtain results that are more close to the full-precision backbone networks ResNets and WRNs, with a storage reduction of about 32 times

Abstract: The rapidly decreasing computation and memory cost has recently driven the
success of many applications in the field of deep learning. Practical
applications of deep learning in resource-limited hardware, such as embedded
devices and smart phones, however, remain challenging. For binary convolutional
networks, the reason lies in the degraded representation caused by binarizing
full-precision filters. To address this problem, we propose new circulant
filters (CiFs) and a circulant binary convolution (CBConv) to enhance the
capacity of binarized convolutional features via our circulant back propagation
(CBP). The CiFs can be easily incorporated into existing deep convolutional
neural networks (DCNNs), which leads to new Circulant Binary Convolutional
Networks (CBCNs). Extensive experiments confirm that the performance gap
between the 1-bit and full-precision DCNNs is minimized by increasing the
filter diversity, which further increases the representational ability in our
networks. Our experiments on ImageNet show that CBCNs achieve 61.4% top-1
accuracy with ResNet18. Compared to the state-of-the-art such as XNOR, CBCNs
can achieve up to 10% higher top-1 accuracy with more powerful representational
ability.

‹Figure 1. Circulant back propagation (CBP). We manipulate the learned convolution filters using the circulant transfer matrix, which is employed to build our CBP. By doing so, the capacity of the binarized convolutional features are significantly enhanced, e.g., robustness to the orientation variations in objects, and the performance gap between the 1-bit and full-precision DCNNs is minimized. In the example, 4 CiFs are produced based on the learned filter and the circular matrix. (Introduction)Figure 2. Circulant Binary Convolutional Networks (CBCNs) are designed based on circulant and binary filters to variate the orientations of the learned filters in order to increase the representational ability. By considering the center loss and softmax loss in a unified framework, we achieve much better performance than state-of-theart binarized models. Most importantly, our CBCNs also achieve the performance comparable to well-known full-precision ResNets and WideResNets. The circulant binary filters are only shown for demonstrating the computation procedure, which are not saved for testing. (Introduction)Figure 3. Illustration of the circulant transfer matrix M for K = 8. The center position stays unchanged, and the remaining numbers are circled in a counter-clockwise direction. Each column of M is obtained from m0 with a rotation angle ∈ {0◦ , 45◦ , ..., 315◦ }. It clearly shows that a circulant filter explicitly encodes the position and orientation. (Methodology)Figure 4. CiF and CBConv examples for K = 4 orientations (0◦ , 90◦ , 180◦ , 270◦ ) and H = 3. (a) The generation of a CiF and its corresponding binary CiF based on a learned filter and M. To obtain the 4D CiF, the original 2D H×H learned filter is modified to 3D by copying it 3 times. (b) CBConv on an input feature map. Note that in this paper, a feature map is defined as 3D with K channels, and these channels are usually not the same. (Circulant Filters (CiFs))Figure 5. Three approximations of the sign function for its gradient computation. (a) The clip function and its derivative in [2]. (b) The piecewise polynomial function and its derivative in [9]. (c) Our proposed function and its derivative. (Circulant Back Propagation (CBP))Figure 6. Network architectures of ResNet18, XNOR on ResNet18 and CBCN on ResNet18. Note that CBCN doubles the shortcuts. (Datasets and Implementation Details)Figure 7. Training and Testing error curves of CBCN and XNOR based on WRN40 for the CIFAR10 experiments. (Rotation Invariance)Figure 8. Training and Testing error curves of CBCN and XNOR based on the ResNet18 backbone on ImageNet. (Accuracy Comparison with State-of-the-Art)