feat: Phase 46 — RLAIF (Reinforcement Learning from AI Feedback)

[aarambh-darshan]

Summary

Implements Phase 46 — RLAIF (Reinforcement Learning from AI Feedback) — a third alignment signal, alongside GRPO (v1 §11, verifier-based) and DPO (v2 §28, human-preference-based). A frozen judge model scores pairs of self-sampled completions, automatically generating preference data that feeds the existing DPO training pipeline unchanged — useful for open-ended quality dimensions where neither a hard verifier nor a static human preference dataset is available.

RLAIF is deliberately architected as a data-generation front end, not a new training objective: dpo_loss (v2 §28) is byte-for-byte unchanged. The output is (chosen, rejected) pairs in the exact {prompt, chosen, rejected} JSONL schema DpoDataset::from_jsonl already consumes.

This faithfully implements the Phase 46 spec already authored in ROADMAP_V4.md (lines 659–719), ARCHITECTURE_V4.md §60, and SELF_LEARNING_V4.md §46.

What's new

crates/aarambh-studio-finetune/src/rlaif.rs (new module)

• RlaifConfig (serde, Default, validate): n_candidates (4), candidate sampling temperature/top-k/top-p/max-tokens, judge max-tokens, bias_discard (false), agreement_margin (0.1), max_pairs_per_prompt, base seed, judge prompt template.
• JudgeGenerator trait — deliberately free of aarambh-studio-inference types so the finetune crate (Layer 4) does not depend on the inference crate (Layer 5), mirroring Phase 45's CompletionVerifier layering. generate_verdict(judge_prompt, max_tokens) takes an already-built judge prompt so the finetune crate owns the template logic.
• CandidateSampler trait — abstracts v1 §12's N-completion sampling pattern (sample N candidates with seeds base + i).
• JudgeVerdict / JudgeChoice (A/B/Tie) / parse_judge_verdict — robust JSON parser; malformed JSON, unknown preferred values, or non-finite margins all fall back to a neutral Tie with margin 0.0, discarded downstream rather than trusted at face value.
• BiasCorrectedPair / AgreementLevel / judge_pair_both_orderings / resolve_preference — position-swap bias correction: every pair is judged twice, in both A/B and B/A orderings. Judges have a documented first-position bias; when the two orderings agree, the pair is emitted at weight 1.0 (or down-weighted by margin when below agreement_margin); when they disagree, the pair is down-weighted to DISAGREEMENT_WEIGHT (0.25) using the more-confident ordering's verdict, or discarded entirely (--discard-disagreements) or when the disagreement is ambiguous (equal margins). Ties are discarded.
• generate_rlaif_dataset — the main entrypoint: sample N candidates per prompt, form all C(N, 2) pairs, judge both orderings, resolve preferences, return Vec<DpoExample> + RlaifSummary.
• write_preference_jsonl — writes the exact {prompt, chosen, rejected} schema DpoDataset::from_jsonl consumes.
• RlaifPair carries a provenance: "rlaif_judge" marker (§46's vocabulary) for downstream replay analysis.
• 16 unit tests (4 roadmap-named acceptance tests + 12 supporting).

CLI: finetune rlaif subcommand (new)

The InferenceEngine implementations of JudgeGenerator/CandidateSampler live in the CLI binary (aarambh-studio/src/cmd/finetune.rs: InferenceJudge, InferenceSampler), alongside Phase 45's MathVerifierAdapter — preserving the Layer 4/5 architectural boundary. The subcommand wires policy + judge engines, supports self-judging (--judge defaults to --base), and feeds the generated JSONL into the unmodified finetune dpo pipeline.

finetune rlaif --base <policy> [--judge <judge>] --prompts <jsonl> --output <jsonl> \
  [--n-candidates N] [--temperature] [--top-k] [--top-p] [--seed] \
  [--max-new-tokens] [--judge-max-tokens] [--bias-threshold] \
  [--discard-disagreements] [--max-pairs]

Supporting files (new)

• configs/rlaif_smoke.toml — CPU smoke training config (tiny Shakespeare, 8 steps) that produces a checkpoint the smoke script runs RLAIF against (policy == judge, self-judging).
• scripts/phase46_smoke.sh — runs the 16 rlaif unit tests, trains a tiny checkpoint, generates a preference-pair JSONL via finetune rlaif --n-candidates 2, verifies the JSONL is valid DPO schema, feeds it into the unmodified finetune dpo pipeline (1 step), verifies the new flags appear in finetune rlaif --help, and writes a scorecard to artifacts/phase46_rlaif_smoke.json.
• docs/phase46_rlaif.md — dedicated Phase 46 runbook (mirrors docs/phase45_test_time.md structure).

Doc updates

• ROADMAP_V4.md — Phase 46 checkboxes flipped [ ] → [x] + status blockquote.
• ARCHITECTURE_V4.md §60 — added an "Implementation (Phase 46, v4.0.0-alpha.6)" subsection.
• SELF_LEARNING_V4.md §46 — added a "Status: Verified for v4.0.0-alpha.6 (Phase 46)" blockquote.
• CHANGELOG.md — added [4.0.0-alpha.6] section.
• README.md — bumped version reference to 4.0.0-alpha.6, added Phase 46 / RLAIF to the v4 arc description and the Fine-tuning capabilities row.
• Cargo.toml — workspace version bumped to 4.0.0-alpha.6.
• .github/workflows/ci.yml — CLI smoke step now exercises finetune rlaif --help.

Minimal change to existing code

• DpoTrainer.train_loader field widened from private to pub(crate) (aarambh-studio-finetune: dpo.rs) so the RLAIF integration test in rlaif.rs can pull one batch and prove the pairs feed through the unmodified train_step. Not part of the public API; dpo_loss, DpoDataset, DpoTrainer::new, and run_dpo_from_config are byte-for-byte unchanged.

Key design invariants (per the roadmap)

1. Offline-only — RLAIF is a data-generation front end for DPO, not a new training objective. dpo_loss (v2 §28) is byte-for-byte unchanged.
2. DPO-schema-identical output — {prompt, chosen, rejected} JSONL, consumed by the unmodified DpoDataset::from_jsonl / run_dpo_from_config.
3. Position-swap bias correction — every pair is judged in both A/B and B/A orderings; disagreements are down-weighted (default) or discarded, never silently trusted.
4. Text-only — multimodal RLAIF is future work, not a half-implementation.
5. Layer 4/5 boundary preserved — the finetune crate does not depend on the inference crate; InferenceEngine impls live in the CLI binary (mirrors Phase 45's CompletionVerifier/MathVerifierAdapter).
6. Zero unsafe, #![deny(missing_docs)] on the finetune crate.
7. No new crate — RLAIF is a new module in the existing aarambh-studio-finetune crate; release audit stays at 20 packages.
8. "Measured, not assumed" — the win-rate acceptance test asserts only a non-negative delta vs the 0.5 random-chance baseline, not an improvement (same discipline as every v3/v4 alignment phase).

Acceptance tests (4 roadmap-named, all pass)

Test	Gate
position_swap_disagreement_is_downweighted_not_silently_trusted	disagreement weight < 1.0 (down-weighted, not trusted); bias_discard discards
rlaif_generated_pairs_match_existing_dpo_pair_schema_exactly	output is {prompt, chosen, rejected} and round-trips through DpoExample + JSONL
rlaif_preference_pairs_fed_into_unmodified_dpo_pipeline_train_successfully	generated pairs → DpoDataset::from_examples → real DpoTrainer::train_step (finite loss)
rlaif_dpo_run_reports_non_negative_win_rate_delta_on_preference_eval_task	measured win-rate ≥ 0.5 baseline (non-negative delta), not asserted improvement

CI gates (all green)

• ✅ cargo fmt --all --check
• ✅ cargo check --workspace --all-targets --locked (all 20 crates at 4.0.0-alpha.6)
• ✅ cargo clippy --workspace --all-targets --locked -- -D warnings -D clippy::undocumented_unsafe_blocks
• ✅ cargo test -p aarambh-studio-finetune --lib (68 tests: 16 RLAIF + 52 existing)
• ✅ scripts/phase28_release_audit.sh (20 packages, version 4.0.0-alpha.6)
• ✅ Shell syntax check (bash -n on all scripts/*.sh)
• ✅ CLI smoke (aarambh-studio finetune rlaif --help surfaces all flags)
• ✅ scripts/phase46_smoke.sh end-to-end (16 tests + tiny train + RLAIF generate + DPO pipeline + CLI help + scorecard)

Milestone

RLAIF-generated preference pairs, fed through the existing (unmodified) finetune dpo pipeline, produce a checkpoint whose held-out preference win-rate (v2 §28's eval task) is reported against the pre-RLAIF baseline — an honest delta, not a claimed win, consistent with every other "measure, don't assume" phase since v2 §17.