fix(phase-10/10): perplexity was different on every run

[thejesh23]

What this PR does

The evaluation lesson's perplexity numbers changed on every run, despite being explicitly seeded.

Kind of change

• Fix to an existing lesson

Checklist

• Code runs without errors with the listed dependencies
• No comments in code files
• One lesson per commit
• Tested locally / code output matches what docs/en.md claims

Phase / lesson

Phase 10 · 10-evaluation

Detail

token_log_probs_simulated seeds from the text precisely so the same text yields the same simulated log-probs, and STEP 3 compares Strong / Medium / Weak on that basis. But hash() of a str is salted per interpreter process, so seeding from the content — whose only purpose is stability — produced different numbers each run.

Three consecutive runs of the unmodified file:

Strong model (quality=0.9): perplexity = 1.20
Strong model (quality=0.9): perplexity = 1.16
Strong model (quality=0.9): perplexity = 1.13

Seeded from hashlib.sha256 instead. Two call sites use the same pattern and both are updated.

After — three runs, byte-identical output:

$ for i in 1 2 3; do python3 main.py | md5; done
c574f26767da2f723522af74a3436b92
c574f26767da2f723522af74a3436b92
c574f26767da2f723522af74a3436b92

The Strong < Medium < Weak ordering the lesson relies on is preserved: 1.13 / 1.44 / 2.52.

[coderabbitai]

📝 Walkthrough

Walkthrough

The evaluation helpers now derive NumPy seeds from SHA-256 digests of their text or prompt inputs instead of Python’s hash, making results stable across runs.

Changes

Deterministic evaluation seeding

Layer / File(s)	Summary
SHA-256 seed updates phases/10-llms-from-scratch/10-evaluation/code/main.py	Adds hashlib and uses SHA-256-derived seeds in token_log_probs_simulated and demo_model_random.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

• rohitg00/ai-engineering-from-scratch#355: Replaces Python hash-based seeding with stable digest-derived seeding in a related evaluation utility.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main fix: nondeterministic perplexity across runs.
Description check	✅ Passed	The description accurately describes the seeding bug, the SHA-256 fix, and the deterministic output change.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@phases/10-llms-from-scratch/10-evaluation/code/main.py`:
- Line 1: Add the required 4–6 line educational comment header at the top of
main.py before import hashlib, citing the lesson’s docs/en.md path and the
applicable canonical sources. Keep the existing import and implementation
unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: a29a3ed3-4113-40df-a0a8-d82938a18e52

📥 Commits

Reviewing files that changed from the base of the PR and between 7157ca7 and 0b99402.

📒 Files selected for processing (1)

• phases/10-llms-from-scratch/10-evaluation/code/main.py