ICML 2023 · PMLR 202:28351–28388
Haotong Qin, Mingyuan Zhang, Yifu Ding, Aoyu Li, Zhongang Cai, Ziwei Liu, Fisher Yu, Xianglong Liu
Published paper | PDF | arXiv | Citation
BiBench benchmarks operator-level network binarization across tasks, architectures, robustness, training cost, theoretical complexity, and real hardware. It helps select and evaluate 1-bit methods under practical accuracy and deployment constraints.
The study covers 8 binarization algorithms, 9 datasets (including corruption evaluation), 13 architectures, 2 deployment libraries, and 14 hardware platforms (Sections 3–4; Table 1). The selected algorithms are BNN, XNOR-Net, DoReFa-Net, Bi-Real Net, XNOR++, ReActNet, ReCU, and FDA. Tracks span 2D/3D vision, language, and speech; the benchmark is not a universal ranking of all binary architectures.
Selected aggregate efficiency metrics from Table 3. These summarize the benchmark's evaluated configurations; they are not the speed of one model on one device. See Section 3 for metric definitions and Appendix B, Tables 17–20, for device/model/batch-specific latency.
| Operator algorithm | Theoretical speedup (×) | Theoretical compression (×) | Hardware speedup (×) | Hardware compression (×) |
|---|---|---|---|---|
| BNN | 12.60 | 13.27 | 5.45 | 15.62 |
| Bi-Real Net | 12.37 | 13.20 | 5.45 | 15.62 |
| ReActNet | 12.26 | 13.20 | 4.89 | 15.52 |
| XNOR-Net | 12.26 | 13.20 | Unsupported | Unsupported |
| XNOR++ | 12.26 | 13.16 | Unsupported | Unsupported |
“Unsupported” corresponds to False in the paper under the tested deployment libraries, not an impossibility of implementing the algorithm. Compression measures parameter storage, not peak activation memory.
- Task dependence: rankings vary across image classification, detection, point clouds, language, and speech; ImageNet performance alone is insufficient for selecting a binarizer (Section 5.1.1; Table 2).
- Architecture dependence: direct operator binarization is especially challenging for the evaluated Transformers compared with CNNs/MLPs (Section 5.1.2; Table 2; Appendix A.7).
- Training cost matters: soft gradient approximations can increase training time even when inference is binary (Section 5.2.1; Table 3).
- FLOPs do not equal latency: theoretical gains exceed measured hardware gains in the aggregate results (Sections 5.2.2–5.2.3; Table 3).
- Deployment compatibility matters: inference-library support depends on operator design and hardware (Table 4; Appendix B, Tables 17–20).
- BiBench: benchmark implementations and experiment configurations.
- Libraries: supporting implementations for the evaluated tasks.
- Configurations, training entry, and evaluation entry: task-specific experiment settings and command-line options.
The former external documentation site is currently unavailable; the repository entries above provide direct access to the maintained code and configurations.
Use the task-specific configurations and compare against the corresponding full-precision baseline. Report the algorithm, architecture, dataset, training settings, library, hardware, and batch size alongside results. The published tables above have not been rerun as part of this documentation update.
Please cite the published paper below. Open paper versions are linked at the top of this README.
@inproceedings{qin2023bibench,
title = {{BiBench}: Benchmarking and Analyzing Network Binarization},
author = {Haotong Qin and Mingyuan Zhang and Yifu Ding and Aoyu Li and Zhongang Cai and Ziwei Liu and Fisher Yu and Xianglong Liu},
booktitle = {Proceedings of the 40th International Conference on Machine Learning},
year = {2023},
series = {Proceedings of Machine Learning Research},
volume = {202},
pages = {28351--28388},
publisher = {PMLR},
url = {https://proceedings.mlr.press/v202/qin23b.html}
}