Skip to content

[RL][Phase 4] Implement the native C++ trainer #1074

Description

@utilForever

Parent: #1056
Epic: #1052

Goal

Train the shared policy-value model from replay samples using the LibTorch C++ frontend.

Scope

  • Load mini-batches of observation, policy target, and outcome
  • Compute policy loss, value loss, and the minimal documented regularization
  • Run optimizer steps
  • Save model weights and runtime-local training state as needed
  • Report only essential losses and step counts

Non-goals

  • Distributed training
  • Mixed precision
  • Hyperparameter search
  • Python compatibility for optimizer state

Acceptance criteria

  • Forward, backward, zero-grad, and optimizer steps run on CPU.
  • Loss is finite and decreases on a tiny repeated batch.
  • Checkpointed model weights reload correctly.
  • Invalid shapes and non-finite samples fail before updating parameters.

Verification

Add one focused C++ tiny-batch overfit test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-rlCategory: Search, self-play, reinforcement learning, and training.P-importantPriority: Other work depends on this, or it is low-level and critical.T-featureType: New capability or supported behavior.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions