act-slim

act-slim is a deployment-oriented implementation of Action Chunking with Transformers (ACT) for a dual-arm upper-body robot with stereo RGB-D sensing. It is a general algorithm framework covering the complete path from HDF5 demonstrations to offline evaluation, shadow execution, and bounded real-hardware command execution.

The repository intentionally contains no proprietary robot SDK, laboratory network configuration, device credentials, or device-specific safety values. Its version-bound backend architecture keeps device-specific alignment, commissioning records, and proprietary transport private while exposing the reusable contract in docs/hardware_backend.md.

Project status

Stage	Status	Evidence in this repository
Training	Implemented	HDF5 dataset, normalization, ACT/CVAE loss, checkpoint and resume
Offline evaluation	Implemented	Recorded-episode normalized action error
Shadow execution	Implemented	Read-only backend handshake and real-time prediction loop
Command execution	Implemented with fail-closed gates	Exact hardware-profile match, explicit limits, finite step count and backend stop
Real-hardware integration	Supported	Version-bound external backend, identity/profile gates and commissioning protocol
Device-specific transport	Private-capable	Installed separately through package.module:factory

Hardware support is scoped to an exact backend/profile pair. Additional robot types use the same public algorithm path and repeat the commissioning sequence in docs/hardware_validation.md.

Reference robot platform

The default platform is a dual-arm upper-body robot with:

• a stereo RGB-D camera providing left/right RGB views and a depth map registered to the left view;
• a seven-value Cartesian end-effector/gripper state for each arm;
• an optional two-degree-of-freedom pan/tilt neck.

The public hardware contract preserves the depth observation and the handshake validates it when supplied. The default ACT policy currently consumes the two RGB views only; depth is available to deployment-specific validation and future model variants, but is not silently injected into the RGB policy.

Model

The default RGB policy includes:

• left/right RGB cameras and a shared ResNet18 backbone;
• DETR-style sine position encoding and Transformer encoder/decoder;
• a CVAE latent encoder;
• continuous 6D rotation representation;
• action chunks, padding masks, state/action normalization and ACT losses;
• temporal aggregation for overlapping inference chunks;
• right-only, left-only, bimanual, and optional neck state/action layouts.

Gaze conditioning and point-cloud policies are deliberately outside the scope of this package.

Installation

Python 3.10 or newer is required.

python -m pip install -e ".[dev]"
python -m pytest

PyTorch installation varies by CUDA platform. For GPU training, install the appropriate PyTorch build before installing this package.

The public mock backend can exercise the adapter boundary without hardware:

python scripts/handshake_hardware.py \
  --backend act_slim.mock_backend:create_backend \
  --image-width 64 --image-height 48 --samples 2

Data contract

Episode files are HDF5 files. The default loader expects:

• left_img and right_img: (time, height, width, 3);
• filtered arm state fields such as left_f_state and right_f_state, with raw left_state and right_state as fallbacks;
• filtered gripper/hstate fields, with raw hstate fields as fallbacks;
• head_state when neck state or action is enabled.

The final frame is not used as an observation because it has no future action. Images are converted to RGB when data.bgr: true, resized, scaled to [0, 1], and ImageNet-normalized.

Keep private demonstrations under data/; the directory is ignored by Git.

Training

python scripts/train.py \
  --config config/act_right.yaml \
  --output checkpoints/act_right

Resume from a checkpoint:

python scripts/train.py \
  --config config/act_right.yaml \
  --output checkpoints/act_right \
  --resume checkpoints/act_right/latest.pt

The default aligned model is intentionally large. For a from-scratch run set pretrained_backbone: false and train_backbone: true; freezing a randomly initialized backbone is fast but does not train useful visual features.

Offline evaluation

python scripts/evaluate.py \
  --checkpoint checkpoints/act_right/latest.pt \
  --episode data/example.h5

The command reports normalized one-step action L1 error. Held-out task metrics and deployment commissioning evidence are recorded separately.

External hardware backend

Install a private or third-party adapter separately, then identify its factory as package.module:factory. The adapter returns the public HardwareState type and is responsible for all device SDK calls, coordinate conversion, joint-space enforcement, and emergency-stop integration.

Create local deployment files:

cp config/hardware_profile.example.yaml config/hardware_profile.local.yaml
cp config/realtime_safety.example.yaml config/realtime_safety.local.yaml

Both local files are ignored by Git. Fill them only with values approved for the deployed robot and adapter version.

Run a bounded read-only handshake first:

python scripts/handshake_hardware.py \
  --backend private_adapter.runtime:create_backend \
  --backend-config config/backend.local.yaml \
  --hardware-profile config/hardware_profile.local.yaml \
  --samples 10 \
  --timeout 60

Run shadow execution. No command method is called unless --execute is present:

python scripts/run_realtime.py \
  --checkpoint checkpoints/act_right/latest.pt \
  --backend private_adapter.runtime:create_backend \
  --backend-config config/backend.local.yaml \
  --hardware-profile config/hardware_profile.local.yaml \
  --safety-limits config/realtime_safety.local.yaml \
  --max-steps 100

After the deployment checklist passes, command mode additionally requires an exact hardware-profile match, explicit safety limits, and a finite step count:

python scripts/run_realtime.py \
  --checkpoint checkpoints/act_right/latest.pt \
  --backend private_adapter.runtime:create_backend \
  --backend-config config/backend.local.yaml \
  --hardware-profile config/hardware_profile.local.yaml \
  --safety-limits config/realtime_safety.local.yaml \
  --max-steps 10 \
  --execute

The application-level checks do not replace certified controller safety, joint-space limits, collision handling, an operator, or a physical emergency stop.

Repository hygiene

The default ignore rules exclude demonstrations, checkpoints, local deployment profiles, credentials, generated build metadata, editor history, vendor trees, and external SDKs. Before making a fork public, scan both the working tree and the complete Git history; deleting a secret from the latest commit does not remove it from earlier commits.

License

Released under the MIT License.