## Ternary NNetworks

2016 [Ternary weight networks](https://arxiv.org/pdf/1605.04711.pdf) Zhang, Liu

2018 [TRAINED TERNARY QUANTIZATION](https://arxiv.org/pdf/1612.01064.pdf#page10) Zhu, Han, Mao, Dally

2017 - a video lecture by Han [Harware and Software Co-design](https://www.youtube.com/watch?v=eZdOkDtYMoo)

## Motivation

Take a look at Fig. 1 in (Zhang,Liu). You can see that the accuracy of **BNN saturates at some level** which is noticeably lower than for full precision and Ternary NNs.

A conclusion - probably, **Binary NNs are not good enough for some essential reasons** (not clear structural reasons?)

## but 2-bit (weights and activation) cells are good enough

- 2019 [Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)](https://arxiv.org/abs/1807.06964)



> Collapsing the weights and the unlimited range of ReLU-generated activations into four discrete bins, as required for 2-bit inference computations, **causes large accuracy losses for deep learning inference**.
> ...
> Combining the PACT and SAWB advances allows us to perform deep learning inference computations w**ith high accuracy down to 2-bit precision.**

a short summary from IBM - 2018 [**Highly Accurate Deep Learning Inference with 2-bit Precision**](https://www.ibm.com/blogs/research/2019/04/2-bit-precision/)

>The proposed scheme achieves zero accuracy degradation for AlexNet
> quantized down to 2-bits for weights and [activations without requiring any increase in the network size](https://en.wikipedia.org/wiki/Activation_function)

hardware [9.1 A 7nm 4-Core AI Chip with 25.6TFLOPS Hybrid FP8 Training, 102.4TOPS INT4 Inference and Workload-Aware Throttling](https://ieeexplore.ieee.org/abstract/document/9365791)

- particular techniques

2017 [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://jmlr.csail.mit.edu/papers/volume18/16-456/16-456.pdf) - Hubara, Matthieu Courbariaux,etc

1-bit weight and 2-bit activation achieves decent accuracy

2018? [Two-Step Quantization for Low-bit Neural Networks](https://openaccess.thecvf.com/content_cvpr_2018/papers/Wang_Two-Step_Quantization_for_CVPR_2018_paper.pdf) Wang,Hu, etc

## Questions

- can VMM (vector-matrix multiplication) via analog crossbar circuitry be extended to **2-bit cells easily?**

- is the building **microcontroller** on stt-mram bits a good idea? Stochastic computing will be required here to improve signal-noise ratio

- is it possible to build VMM via crossbar circuitry on a standard DRAM?