We are developing inference machine learning algorithms that are light-weight, which would ensure a real-time and distributed autonomous drone-based sensing. In order to address that, we focused on removing redundancy in the adaptation and learning process to aggressively reduce the computational and storage costs while maintaining the algorithmic performance, such as the sensing and detection accuracy. Low-precision deep neural network (DNN) training has gained tremendous attention as reducing precision is one of the most effective knobs for boosting DNNs’ training time/energy efficiency. Based on that, we attempt to explore low-precision training from a new perspective as inspired by recent findings in understanding DNN training.
We are focusing on using modern deep neural networks (DNNs) to build data-driven inference solutions that can be applied to the gas-sensing drone application. This however comes at a prohibitive training cost due to the required massive training data and parameters, limiting the development of the highly demanded DNN-powered intelligent solutions for numerous applications. To achieve that, we leverage recent findings in understanding DNN training in order to provide a new perspective of low-precision training. We therefore propose a new training solution and validate it using multiple datasets and while considering different inference models.
Low-precision deep neural networks (DNN) have been proven to be very efficient in reducing precision is one of the most effective knobs for boosting DNNs’ training time/energy efficiency. Based on that, and inspired by recent findings in understanding DNN training, we conjecture that DNNs’ precision might have a similar effect as the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efficiency of DNN training. Specifically, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identified using a simple precision range test within the first few training epochs. Extensive simulations and ablation studies on five datasets and eleven models demonstrate that CPT’s effectiveness is consistent across various models/tasks (including classification and language modeling). Furthermore, through experiments and visualization we show that CPT helps to (1) converge to a wider minimum with a lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efficiency of DNN training.
Publications
Fu, Yonggan; Guo, Han; Li, Meng; Yang, Xin; Ding, Yining; Chandra, Vikas; and Lin, Yingyan, “CPT: Efficient Deep Neural Network Training via Cyclic Precision,” In proceedings International Conference on Learning Representations (ICLR 2021), (Spotlight paper: ~ Top 3%).