Skip to content

[RL][Phase 4] Bind the C++ model and implement the Python trainer #1075

Description

@utilForever

Parent: #1056
Epic: #1052

Goal

Train the same C++ policy-value module from Python without defining a second model.

Scope

  • Bind the C++ torch::nn::Module as a Python torch.nn.Module-compatible object
  • Expose forward, parameters, device movement, and weight save/load
  • Implement the minimal Python training loop with torch.optim
  • Consume the same replay sample contract as C++

Non-goals

  • A pure-Python model implementation
  • torch.compile, DDP, or AMP
  • A second checkpoint format

Acceptance criteria

  • Python autograd populates gradients on the C++ model parameters.
  • A Python optimizer updates those parameters.
  • Loss decreases on the same tiny repeated batch used by C++.
  • The Python trainer saves weights loadable by C++.

Verification

Add one focused pytest training and weight-round-trip smoke test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-rlCategory: Search, self-play, reinforcement learning, and training.P-importantPriority: Other work depends on this, or it is low-level and critical.T-featureType: New capability or supported behavior.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions