tr_hash_sensor_fusion.md

TR-HASH Sensor Fusion

TRHashSensorFusionClassifier is the compact multimodal HAR model prepared for the CUHK-X Small Model Track. It predicts one of 40 activities from any available combination of depth, infrared, thermal, IMU, mmWave radar, and skeleton streams.

Architecture

• separate lightweight tokenizers for visual clips, numeric sensor sequences, and joint trajectories;
• a fixed 288-token fusion sequence with learned modality and local-position embeddings;
• six Transformer blocks using eight stored TR-HASH experts and top-2 routing;
• routing identities defined by (modality, local token position) rather than input content;
• explicit sample-level modality masks for missing sensors;
• 40-class activity head.

The default model contains approximately 7.95M parameters and occupies about 31.8 MB with FP32 weights, below the competition's 100 MB limit.

Input contract

from complexity.generative.sensor_fusion import (
    TRHashSensorFusionClassifier,
    TRHashSensorFusionConfig,
)

model = TRHashSensorFusionClassifier(TRHashSensorFusionConfig())
output = model(
    {
        "depth": depth,        # [B, 3, T, H, W]
        "ir": ir,              # [B, 1, T, H, W]
        "thermal": thermal,    # [B, 3, T, H, W]
        "imu": imu,            # [B, T, 45]
        "radar": radar,        # [B, T, 16]
        "skeleton": skeleton,  # [B, T, 17, 3]
    },
    labels=labels,
)

The local CUHK-X loader normalizes the source schemas to five IMU devices with nine motion values each, 16 per-frame radar statistics, and 17 three-dimensional skeleton joints. Visual resolution and sequence duration may vary; each tokenizer adaptively emits a fixed number of tokens.

Local dataset and cross-subject split

Extract the licensed multi-volume HAR.zip archive locally. Point the trainer at either the directory containing HAR/data or directly at HAR/data. The first process builds a lightweight JSON manifest so the other DDP ranks do not rescan hundreds of thousands of files.

The default validation people are users 8, 9, 23, and 24. They never appear in the training split. Override them explicitly with --validation-users when running a published competition protocol.

Eight-GPU training

torchrun --standalone --nproc_per_node=8 \
  -m complexity.generative.sensor_fusion.training \
  --data-root /workspace/datasets/CUHK-X/extracted \
  --output artifacts/tr_hash_cuhkx_sensor_fusion_v1 \
  --optimizer musgd \
  --epochs 50 \
  --batch-size 2 \
  --eval-batch-size 2 \
  --workers 4 \
  --precision bf16 \
  --require-fused-cuda

--batch-size is per GPU. The trainer uses MuSGD, a separate learning-rate group for routed expert parameters, inverse-square-root class weighting, distributed non-padding validation, top-1/top-5/macro accuracy, and safetensors model weights. It saves optimizer, scheduler, epoch, exact batch cursor, and one RNG state per rank.

Resume with the same architecture, preprocessing, world size, batch size, and total epoch target:

torchrun --standalone --nproc_per_node=8 \
  -m complexity.generative.sensor_fusion.training \
  --data-root /workspace/datasets/CUHK-X/extracted \
  --output artifacts/tr_hash_cuhkx_sensor_fusion_v1 \
  --resume artifacts/tr_hash_cuhkx_sensor_fusion_v1/step_0000500 \
  --optimizer musgd \
  --epochs 50 \
  --batch-size 2 \
  --eval-batch-size 2 \
  --workers 4 \
  --precision bf16 \
  --require-fused-cuda

This first CUHK-X baseline trains all parameters. LoRA is intentionally not used: at roughly 8M parameters, the visual, inertial, radar, skeleton, fusion, and classification layers all need to learn the task jointly.

Official Kaggle submission

Extract small_model_track_test.zip locally, then run inference from the selected checkpoint. test.csv is the only source of test paths; the command refuses a populated prediction column and never reads sample_submission.csv.

torchrun --standalone --nproc_per_node=8 \
  -m complexity.generative.sensor_fusion.submission \
  --checkpoint artifacts/tr_hash_cuhkx_sensor_fusion_v1/best \
  --data-root /workspace/datasets/CUHK-X/test-extracted \
  --test-csv /workspace/datasets/CUHK-X/small-model-track/Small-Model-Track/Testing/test_file/test.csv \
  --output artifacts/tr_hash_cuhkx_sensor_fusion_v1/submission_best.csv \
  --batch-size 8 \
  --workers 2 \
  --require-fused-cuda

Rank zero gathers and sorts all 405 predictions, verifies the exact official path order and class range, writes path,prediction, and stores a compressed submission_best.logits.npz. Keep the logits: they allow checkpoint ensembles without decoding the multimodal test clips again.

CUHK-X data must remain local. Its license forbids redistributing or mirroring the raw data and derived data shards. Source code, configuration, model weights, and aggregate evaluation results can be published separately.