TOT-Net: An Endeavor Toward Optimizing Ternary Neural Networks
Nazari, N.; Loni, M.; Salehi, M.E.; Daneshtalab, M.; Sjodin, M. (2019). TOT-Net: An Endeavor Toward Optimizing Ternary Neural Networks. Proceedings - Euromicro Conference on Digital System Design, DSD 2019. 305−312.10.1109/DSD.2019.00052.
Nazari, N.; Loni, M.; Salehi, M.E.; Daneshtalab, M.; Sjodin, M.
Proceedings - Euromicro Conference on Digital System Design, DSD 2019
Journal of Functional Foods
1.1. Teadusartiklid, mis on kajastatud Web of Science andmebaasides Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index, Emerging Sources Citation Index ja/või andmebaasis Scopus (v.a. kogumikud)
University of Tehran; Mälardalens högskola
© 2019 IEEE. High computation demands and big memory resources are the major implementation challenges of Convolutional Neural Networks (CNNs) especially for low-power and resource-limited embedded devices. Many binarized neural networks are recently proposed to address these issues. Although they have significantly decreased computation and memory footprint, they have suffered from accuracy loss especially for large datasets. In this paper, we propose TOT-Net, a ternarized neural network with [-1, 0, 1] values for both weights and activation functions that has simultaneously achieved a higher level of accuracy and less computational load. In fact, first, TOT-Net introduces a simple bitwise logic for convolution computations to reduce the cost of multiply operations. To improve the accuracy, selecting proper activation function and learning rate are influential, but also difficult. As the second contribution, we propose a novel piece-wise activation function, and optimized learning rate for different datasets. Our findings first reveal that 0.01 is a preferable learning rate for the studied datasets. Third, by using an evolutionary optimization approach, we found novel piece-wise activation functions customized for TOT-Net. According to the experimental results, TOT-Net achieves 2.15%, 8.77%, and 5.7/5.52% better accuracy compared to XNOR-Net on CIFAR-10, CIFAR-100, and ImageNet top-5/top-1 datasets, respectively.
activation function | convolutional neural networks | optimization | ternary neural network