This project explores self-supervised learning of object interactions from raw temporal trajectories using a Transformer-based architecture. The goal is to learn meaningful representations of interaction dynamics without any manual labels.
Understanding object interactions from motion trajectories is a fundamental problem in computer vision and robotics. Traditional approaches rely on handcrafted features, which often fail to generalize.
This project investigates whether a self-supervised Transformer can learn disentangled representations of interaction types directly from temporal data.
We propose a pipeline consisting of:
- Temporal trajectory input (synthetic multi-object scenes)
- Data augmentation for self-supervised learning
- Transformer encoder for sequence modeling
- VICReg-based loss function to prevent representation collapse
Input: Temporal Object Trajectories (Ball, Box, Cylinder positions over time) │ ▼ Data Augmentation (two stochastic temporal views) │ ├───────────────┐ ▼ ▼ View 1 View 2 │ │ ▼ ▼ Transformer Encoder (shared weights) │ │ ▼ ▼ z₁ ∈ ℝ⁶⁴ z₂ ∈ ℝ⁶⁴ └───────────────┘ │ ▼ VICReg Objective ┌──────────────────────────┐ │ Invariance loss │ │ Variance regularization │ │ Covariance regularization│ └──────────────────────────┘ │ ▼ Learned Embedding Space
We evaluate learned representations using clustering (DBSCAN) and compare them with handcrafted temporal features (baseline method).
- The model successfully separates interaction types:
- Chain interactions
- Pass-by interactions
- Independent motion
Self-supervised Transformer embeddings show clear cluster separation compared to baseline temporal features, indicating that the model learns meaningful interaction structure without supervision.
src/ Core modules (model, dataset, loss) experiments/ Training & embedding extraction results/ Trained model & embeddings archive/ Old experiments (not required for reproduction)
Sara Haghverdi Mamaghani