This repository includes the official implementation of our paper “PACEAttention: Principled and Adaptive Feature Compression-Expansion Grounded in the Geometry of
Inspired by the geometric interpretation of the gradients of the maximal coding rate reduction (
Deep neural networks are often powerful but difficult to interpret, making it unclear why they work well or how their architectures should be designed. In this work, we propose PACEAttention, a new attention mechanism motivated by geometric principles from representation learning theory.
Our method entails feature learning as a progressive process, where same-category feature representations are compressed into compact subspaces, while cross-category representations are expanded. This process is guided by the intrinsic structure of the data. The resulting model, PACENet, is both principled and efficient, and produces competitive performance on several classification benchmarks. This design also makes PACENet easier to interpret, with different parts of the model naturally focusing on different image regions and fine details, even when trained using standard supervised learning.
