SimCC.md

SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation

Keywords:

Human Poes Estimation, 2D Heatmap, Coordinate Classification

Background:

the long-standing quantization error problem in the 2D heatmap-based methods leads to several well-known drawbacks：

1. The performance for the low-resolution inputs is limited；
2. To improve the feature map resolution for higher localization precision, multiple costly upsampling layers are required;
3. Extra post-processing is adopted to reduce the quantization error.

Motivations:

Aim to explore a brand new scheme, called SimCC, which reformulates HPE as two classification tasks for horizontal and vertical coordinates.

Contributions:

• propose a coordinate classification pipeline for human pose estimation called SimCC, reformulating the problem as two classification tasks for horizontal and vertical coordinates. SimCC serves as a general scheme and can be easily applied to existing CNN-based or Transformer-based HPE models.
• SimCC achieves high efficiency by omitting the extra time-consuming upsampling and post-processing in heatmap-based methods. In particular, applying SimCC reduces over 55% GFLOPs of SimBa-Res50 and achieves higher model performance than heatmap-based counterpart.

How the SimCC work:

The SimCC uniformly divides each pixel into several bins, thus achieving sub-pixel localization precision and low quantization error.

Benefiting from that, SimCC can omit additional refinement post-processing and exclude upsampling layers under certain settings, resulting in a more simple and effective pipeline for HPE.

Details:

Step1:

SimCC firstly employs a Convolutional Neural Network (CNN) or Transformer-based backbone to extract keypoint representations.

Step2:

Given the obtained keypoint representations, SimCC then performs coordinate classification for vertical and horizontal coordinates independently to yield the final predictions.

To reduce the quantitaion error, SimCC uniformly divides each pixel into several bins, which achieves sub-pixel localization precision.

Differents:

The different from heatmap-based approaches which may introduce multiple deconvolution layers, SimCC only needs two lightweight classifier heads (i.e. only one linear layer for each head).

Pipeline:

Backbone:

Given an input image of size H ×W ×3, SimCC employs either CNN-based or Transformer-based network as the backbone to extract n keypoint representations for n corresponding keypoints.

Head:

horizontal and vertical classifiers (i.e., only one linear layer for each classifier) are appended after the backbone to perform coordinate classification, respectively. For the CNN-based backbone, simply flatten the outputted keypoint representations from (n, H′, W ′) to (n, H′ × W ′) for classification.

Coordinate classification:

To achieve classification, they propose to uniformly discretize each continuous coordinate value into an integer as class label for model training: $c_x ∈ [1, N_x], c_y ∈ [1, N_y]$, where $N_x = W · k$ and $N_y = H · k$ represent the number of bins for horizontal and vertical axes, respectively. $k$ is the splitting factor and set as $≥ 1$ to reduce quantization error, resulting in sub-pixel localization precision. To yield the final prediction, SimCC performs vertical and horizontal coordinate classification independently based on the $n$ keypoint representations learnt by the backbone. Concretely, given $i^{th}$ keypoint representation as input, the $i^{th}$ keypoint predictions $o^i_x$ and $o^i_y$ are generated by the horizontal and vertical coordinate classifiers, respectively. In addition, Kullback–Leibler divergence is used as loss function for training.

Label smoothing:

adopt Label smoothing for SimCC, which is called equal label smoothing in this paper. However, equal label smoothing punishes the false labels indiscriminately, which has ignored the spatial relevance of adjacent labels for the task of human pose estimation. A more reasonable solution is supposed to encourage the model to work in this way: the closer the output category is to the groundtruth, the better. To address this issue, they also explore to use Laplace or Gaussian label smoothing, resulting in smoothed labels following corresponding distribution. Unless noted otherwise, SimCC is used as the abbreviation for the variant with equal label smoothing .

Experiment:

1. Heatmap-based approaches rely seriously on post-processing for refinement, which brings extra computational cost and complicates the whole process;

2. The proposed SimCC works well without any refinement post-processing, leading to a more simple and efficient scheme compared to heatmap-based methods.

Ablation Study

Splitting factor k:

The splitting factor $k$ controls the how many bins per pixel in SimCC. Specifically, the larger $k$ is, the smaller the quantization error of SimCC is. Nevertheless, model training becomes more difficult when $k$ increases. Hence, there is a trade-off between the quantization error and the model performance. We test $k ∈ {1, 2, 3, 4}$ based on SimpleBaseline and HRNet under various input resolutions. model performance tends to increase first and then decrease as $k$ grows. For HRNet-W32, the recommended settings are $k = 2$ for both $128×128$ and $256×192$ input size. For SimBa-Res50 , the recommended settings are $k = 3$ and $k = 2$ for $128×128$ and $256×192$ input size, respectively.

Upsampling modules:

Upsampling modules are usually computational costly and substantially slow down the network’s inference speed, however, indispensable for heatmap-based methods. Hence, it’s of practical significance to explore if applying SimCC can reduce the dependence of upsampling modules in HPE. Notice that the upsampling modules1 adopted in SimpleBaseline is independent to the backbone and thus can be easily removed.

compared to heatmap, SimCC allows one to remove the costly deconvolution layers of SimpleBaseline, resulting in consistent computational cost reduction across various input resolutions.

Label smoothing:

Label smoothing is a commonly used strategy to improve generalization for the task of classification. To investigate its effect on our proposed method, we train SimpleBaseline-Res50 based on SimCC with various label smoothing strategies: {w/o, equal, Gaussian, Laplace}. Table 5 demonstrates that label smoothing strategy does make a difference. Therefore, a promising way to further improve SimCC may be replacing the heuristic label smoothing strategy in a self-adaptive way. Further discussion is out the scope of this paper and we regard it as future work.

Limitation and Future Work

SimCC introduced in this paper works under the setting of top-down human pose estimation. When it comes to bottom-up multi-person pose estimation, the presence of multiple people brings the identification ambiguity. Potential future work is to introduce extra embeddings in a similar way to AE , in order to address the matching problem between candidate coordinate x and y values.