Update CLAUDE.md and NEXT_LANGUAGE_PROMPT.md with JS lessons learned

- Add JavaScript scores to CLAUDE.md project context
- Add Java/C++/C# assertion field instructions (assertions="" in task JSON)
- Replace hardcoded stop points with paper-driven methodology
- Add detailed post-Codex merge/verify/clean/commit checklist (14 steps)
- Add final verification checklist (8 steps)
- Add Bugs 7-10 from JavaScript experience
- Rewrite NEXT_LANGUAGE_PROMPT.md to be language-agnostic
- Add "read Python examples before creating tasks" step
- Add "launch all 6 categories in parallel" instruction
- Add per-category line count constraints table

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: 2 people

CLAUDE.md

@@ -16,7 +16,19 @@ DevBench is a code generation benchmark with 1,800 tasks across 6 languages × 6
 | Low Context | 58.0% | ≤60% |
 | **Overall** | **53.0%** | **≤55%** |
 
-**Remaining:** JavaScript, TypeScript, Java, C++, C# (5 languages × 6 categories × 50 tasks = 1,500 tasks).
+**JavaScript is DONE.** All 6 JavaScript categories are hardened with verified GPT-5.5 scores:
+
+| Category | GPT-5.5 Pass@1 | Target |
+|---|---|---|
+| Code2NL/NL2Code | 44.4% | ≤45% |
+| Code Purpose | 44.0% | ≤52% |
+| Syntax Completion | 50.0% | ≤50% |
+| Pattern Matching | 50.8% | ≤52% |
+| API Usage | 51.2% | ≤52% |
+| Low Context | 54.8% | ≤55% |
+| **Overall** | **49.2%** | **≤55%** |
+
+**Remaining:** TypeScript, Java, C++, C# (4 languages × 6 categories × 50 tasks = 1,200 tasks).
 
 ## How We Did Python (the story)
 
@@ -56,6 +68,8 @@ This was FAST but caused bugs (wrong eval scripts, missing completions). For fut
 
 This means for Java/C++/C#, the "hidden assertions" strategy does NOT apply. All assertions are visible. Task difficulty must come entirely from the prefix pattern and function design.
 
+**CRITICAL for Java/C++/C# task creation**: The `assertions` field in the task JSON MUST be an empty string `""`. ALL test assertions go in the `suffix` field. The eval_poc.py runner and execute_benchmark.py both handle this correctly — when `assertions` is empty, execution is just `prefix + completion + suffix`. The workflow (task creation, eval, Codex mutation) is OTHERWISE IDENTICAL to Python/JS/TS — same sub-agent structure, same eval script, same Codex prompt template, same merge/verify/clean/commit checklist.
+
 ### Execution environments (from execute_benchmark.py):
 - **Python**: subprocess with 30s timeout, auto pip install
 - **JavaScript**: Node.js with npm auto-install, 30s timeout
@@ -228,56 +242,90 @@ with open('completions/{lang}/{cat}/{cat}-gpt-5.5.jsonl', 'w') as f:
 
 **NEVER do a full 50-task re-eval just to save completions.** The only exception is if the initial eval failed to save completions (like our Low Context bug).
 
-### Phase 5: Verify and regenerate formatted TXT
+### Phase 5: Post-Codex merge/verify/clean/commit checklist
 
-```python
-# Regenerate {category}_formatted.txt from the hardened JSONL
-# See the generation code in generate_completions.py or the inline script we used
-```
-
-Verify:
-- `benchmark/{lang}/{cat}/{cat}.jsonl` — 50 lines, IDs 1-50, all required fields
-- `completions/{lang}/{cat}/{cat}-gpt-5.5.jsonl` — 50 lines, IDs match, 5 completions each
-- `benchmark/{lang}/{cat}/{cat}_formatted.txt` — regenerated from hardened JSONL
-- Score matches target from HARDENING_LOG.md
+For EACH category after Codex finishes, do ALL of these steps IN ORDER:
 
-### Phase 6: Commit cleanly
-
-Each category dir should have exactly 2 files:
 ```
-benchmark/{lang}/{cat}/{cat}.jsonl
-benchmark/{lang}/{cat}/{cat}_formatted.txt
+□ 1. Load initial eval completions (50 tasks from Phase 2)
+□ 2. Load Codex mutations from codex_mutations.jsonl
+□ 3. Load Codex completion files (completions-codex-mutation-task*-attempt*.jsonl)
+     — for each accepted task, use the LAST successful attempt
+□ 4. Merge: replace mutated task IDs in both benchmark JSONL and completions JSONL
+□ 5. Write hardened benchmark: benchmark/{lang}/{cat}/{cat}.jsonl
+□ 6. Write merged completions: completions/{lang}/{cat}/{cat}-gpt-5.5.jsonl
+□ 7. Validate ALL 50 golden completions (must be 50/50)
+□ 8. RE-COMPUTE the score by running all 250 completions against hardened assertions
+□ 9. Verify score matches Codex's projection AND is ≤ stop point
+□ 10. Regenerate {cat}_formatted.txt from hardened JSONL
+□ 11. Delete ALL intermediate files:
+      - create_tasks_*.py (sub-agent scripts)
+      - tasks_*_*.jsonl (sub-agent outputs)
+      - completions-codex-mutation-* (Codex eval logs)
+      - codex_mutation*.jsonl (Codex outputs)
+      - *_original.jsonl (backup of pre-hardening file)
+□ 12. Verify dir has EXACTLY 2 files: {cat}.jsonl + {cat}_formatted.txt
+□ 13. git add benchmark/{lang}/{cat}/ + completions/{lang}/{cat}/
+□ 14. git commit with verified score in message
 ```
 
-Completions:
+For categories needing NO hardening (already below stop point):
 ```
-completions/{lang}/{cat}/{cat}-gpt-5.5.jsonl
+□ 1. Validate all 50 golden completions
+□ 2. Regenerate formatted TXT
+□ 3. Clean up intermediate files (create_tasks_*.py, tasks_*_*.jsonl, *_original.jsonl)
+□ 4. Verify dir has exactly 2 files
+□ 5. git add + commit with baseline score
 ```
 
-No intermediate files (tasks_1_10.py, mutations.jsonl, eval scripts, attempt files, etc.).
+### Phase 6: Final verification
+
+After ALL 6 categories are committed:
+```
+□ All 6 benchmark dirs have exactly 2 files each
+□ All 6 completions dirs have {cat}-gpt-5.5.jsonl
+□ All 12 JSONL files: 50 lines, IDs 1-50, all required fields, language field correct
+□ All completion files: 5 completions per task, gpt-5.5_completion_0 + gpt-5.5_completions
+□ Overall score ≤55%
+□ Working tree clean (git status shows nothing)
+□ Commit Codex prompts + updated HARDENING_LOG.md
+□ Push to `private` remote
+```
 
 ## Per-Category Targets (all languages)
 
-### Targets per category (same for all languages):
+### How to determine per-category stop points:
+
+Do NOT hardcode stop points. Instead, determine them from the paper and eval data:
+
+1. **Read the paper** (Table 9) to understand the target language's original difficulty relative to Python
+2. **Run all 6 baseline evals** to get GPT-5.5 scores
+3. **Set stop points** that:
+   - Keep overall ≤55%
+   - Preserve the relative difficulty ordering from the original paper
+   - Use Python and JavaScript ACTUAL achieved scores as reference (not the generous stop points)
+4. **If a category is already below its stop point**, no hardening needed
+
+Reference — what Python and JavaScript actually achieved:
 
-| Category | Target GPT-5.5 | Stop Point |
+| Category | Python | JavaScript |
 |---|---|---|
-| Code2NL/NL2Code | ~45% | ≤48% |
-| Syntax Completion | ~50% | ≤53% |
-| Code Purpose | ~52% | ≤55% |
-| Pattern Matching | ~55% | ≤58% |
-| API Usage | ~55% | ≤58% |
-| Low Context | ~58% | ≤60% |
-| **Per-language overall** | **~52-55%** | **≤55%** |
+| Code2NL/NL2Code | 46.4% | 44.4% |
+| Code Purpose | 54.0% | 44.0% |
+| Syntax Completion | 52.4% | 50.0% |
+| Pattern Matching | 51.6% | 50.8% |
+| API Usage | 55.6% | 51.2% |
+| Low Context | 58.0% | 54.8% |
+| **Overall** | **53.0%** | **49.2%** |
 
 ### Multi-language overall target:
 
 The benchmark has 6 languages × 6 categories = 36 cells. The OVERALL target across all 1,800 tasks is **≤55% for GPT-5.5**.
 
-Each language should independently meet ≤55% overall. Use the same per-category targets as Python. However, allow flexibility:
-- If a category naturally scores below target on the initial eval (like Python's Syntax Completion at 52.4%), skip Codex mutation — it's already hard enough.
-- Some languages may be naturally harder (TypeScript was 20-30% lower in the original paper). The initial eval will reveal this — don't over-harden a category that's already below target.
-- The key constraint is: **no single language should have overall GPT-5.5 Pass@1 above 55%**.
+Each language should independently meet ≤55% overall. However, allow flexibility:
+- If a category naturally scores below target on the initial eval, skip Codex mutation — it's already hard enough
+- Some languages may be naturally harder (TypeScript was 20-30% lower in the original paper). The initial eval will reveal this
+- The key constraint is: **no single language should have overall GPT-5.5 Pass@1 above 55%**
 
 ### What Python achieved (reference for other languages):
 
@@ -385,6 +433,18 @@ When merging completions, ensure the CHANGED tasks use the HARDENED task definit
 ### Bug 6: Batch mutations don't work
 We batch-mutated 15 tasks and only 3 had any effect. GPT-5.5 reads the suffix and adapts to parameter changes. **Always use sequential, one-at-a-time mutations** with the failure-preserving approach.
 
+### Bug 7 (JS): Not reading Python examples before creating tasks
+Sub-agents created tasks without seeing how Python hardened tasks were structured for that category. This led to wrong line counts (Low Context tasks too long) and missing failure mechanisms. **Always read 3-5 Python hardened tasks for the same category before launching sub-agents.**
+
+### Bug 8 (JS): Not launching evals immediately
+After assembling a category, waited for other categories before starting the eval. Wasted time. **Launch GPT-5.5 eval in background IMMEDIATELY after assembling and validating a category.**
+
+### Bug 9 (JS): Using generous stop points
+Used Python's generous stop points (≤58%, ≤60%) initially, then had to send follow-up messages to Codex to lower them. **Set aggressive stop points from the start** based on the paper's cross-language difficulty data and Python/JS achieved scores.
+
+### Bug 10 (JS): Node.js not found in subprocess
+Node.js is loaded via nvm shell functions, so `subprocess.run(["node", ...])` fails with FileNotFoundError. **Use full path**: `~/.nvm/versions/node/v24.12.0/bin/node` or discover it via `_find_node()` in eval_poc.py.
+
 ## GPT-5.5 API Notes
 
 - Model: `gpt-5.5-2026-04-23`