JS hardening complete: add Codex prompts, update hardening log

JavaScript overall GPT-5.5 Pass@1: 49.2% (target ≤55%)

Per-category scores:
- Code2NL/NL2Code: 44.4% (9 Codex mutations)
- Code Purpose: 44.0% (no hardening needed)
- Syntax Completion: 50.0% (3 Codex mutations)
- Pattern Matching: 50.8% (15 Codex mutations)
- API Usage: 51.2% (20 Codex mutations)
- Low Context: 54.8% (10 Codex mutations)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: 2 people

CODEX_PROMPT_JS_API_USAGE.md

@@ -0,0 +1,178 @@
+# Task for Codex: Harden DevBench JavaScript API Usage
+
+## Context
+
+DevBench is a code generation benchmark paper rejected from ICLR 2026 (ceiling effect: 84.8% Pass@1). Resubmitting to NeurIPS 2026 with harder tasks. Python is done (53% overall for GPT-5.5). Now doing JavaScript.
+
+The benchmark uses fill-in-the-middle (FIM) code completion:
+- Each task has: `prefix`, `golden_completion`, `suffix`, `assertions`
+- The model sees: `prefix + #TODO: You Code Here + suffix`
+- The model does NOT see `assertions` (hidden tests)
+- Execution runs: `prefix + model_completion + suffix + assertions`
+- Pass@1 with n=5 samples
+
+## Goal
+
+**Bring GPT-5.5 Pass@1 on JavaScript API Usage to ≤58%** (currently 91.2%).
+
+- Current total points: 4560 (out of 5000)
+- Target: ≤2900 points (58%)
+- Need to reduce by: 1660 points
+- If each conversion takes a 100% task to 0%: need ~17 conversions
+- There are 44 tasks at 100% — plenty of candidates
+
+This is the HARDEST category to harden — 91.2% baseline means GPT-5.5 handles Node.js APIs very well. You'll need aggressive mutations.
+
+## MANDATORY: Read these files first
+
+1. **`HARDENING_LOG.md`** — FULL methodology. Pay special attention to:
+   - Round 6 (API Usage Codex report) — CSV dialect mechanism
+   - Round 6B (API Usage continuation) — docstring precision mechanism
+   - "What worked best" and "What did not work" from both rounds
+2. **`benchmark/javascript/api_usage/api_usage.jsonl`** — Current 50 JavaScript API Usage tasks
+3. **`eval_poc.py`** (at repo root) — Evaluation script
+4. **`.env`** — API keys
+
+## What Worked for Python API Usage (from HARDENING_LOG.md)
+
+### Mechanism 1: CSV dialect edge cases (Round 6)
+- Custom `csv.writer` with specific delimiter, quotechar, escapechar combinations
+- GPT-5.5 used `lineterminator=''` which changed quoting behavior
+- 3 tasks accepted (0/5, 2/5, 0/5)
+
+### Mechanism 2: API-specific docstring precision (Round 6B — MORE ROBUST)
+- Completion slot is a function DOCSTRING
+- Suffix contains the implementation using specific stdlib APIs
+- Hidden assertions require precise documentation of API params, return types, side effects, exceptions
+- GPT-5.5 wrote generic "Return the SHA-256 hex digest..." and omitted details
+- 5 tasks accepted (all 0/5)
+
+**For JavaScript, the docstring approach needs adaptation:**
+- JS doesn't have Python's `.__doc__` attribute
+- Use `.doc` property assignment: `myFunc.doc = 'precise description...'`
+- Or use a `describe*()` function that returns the description string
+- Or use `Function.prototype.toString()` which includes JSDoc comments inside the function body
+
+### What didn't work:
+- `base91_encode`: GPT-5.5 solved 5/5 in API Usage context
+- Explicit Decimal docstring: GPT-5.5 scored 4/5 when prefix directly said "add a docstring"
+
+## Current GPT-5.5 Scores
+
+```
+100% (44 tasks): 1,2,3,5,6,7,10,11,12,13,14,15,16,17,18,19,20,21,22,23,25,26,27,28,30,31,32,33,34,35,36,37,38,39,40,41,42,44,45,46,47,48,49,50
+80%  (1 task):   24
+60%  (1 task):   43
+20%  (1 task):   4
+0%   (3 tasks):  8,9,29
+```
+
+Overall: 91.2% (4560/5000). Target: ≤58% (2900/5000).
+
+### Analysis of 0% tasks (what's working):
+- Task 8: SafeEmitter with error handling + emit override + maxListeners — multi-invariant pattern
+- Task 9: Raw deflate (`deflateRawSync`/`inflateRawSync`) — API edge case (windowBits: -15)
+- Task 29: fs.mkdtempSync — specific API usage pattern
+
+### Analysis of 100% tasks:
+Most are straightforward Node.js API usage that GPT-5.5 handles well. Need aggressive mutations using BOTH mechanisms: API edge cases AND docstring precision.
+
+## Your Task
+
+**Replace 100% tasks ONE AT A TIME.** This category needs ~17 conversions — the most of any category. Be efficient.
+
+### Recommended approach (mix of mechanisms):
+- **8-10 encoding/escaping tasks**: Custom base encoders with reversed alphabets (proven 0% in Pattern Matching and Low Context). These work in ANY category when wrapped appropriately.
+- **4-5 API docstring precision tasks**: Completion is a `.doc` string or `describe*()` return that must precisely document Node.js API behavior
+- **3-4 API edge case tasks**: Specific Node.js API usage where GPT-5.5 gets edge cases wrong
+
+### JavaScript encoding task template for API Usage context:
+
+Frame the encoding as "API Usage" by having the prefix establish a pattern of using Buffer/crypto APIs:
+
+```javascript
+// PREFIX: Show 2-3 examples of encoding functions using Node.js Buffer API
+const STANDARD58 = '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz';
+const REVERSED58 = STANDARD58.split('').reverse().join('');
+
+function hexEncode(data) {
+    return Buffer.from(data).toString('hex');
+}
+
+function leadingZeroBytes(data) {
+    let count = 0;
+    for (const b of data) { if (b !== 0) break; count++; }
+    return count;
+}
+
+// GOLDEN: base58Encode using Buffer API with reversed default
+function base58Encode(data, alphabet = REVERSED58) {
+    if (typeof data === 'string') data = Buffer.from(data);
+    let n = BigInt('0x' + (data.length ? Buffer.from(data).toString('hex') : '0'));
+    const base = BigInt(alphabet.length);
+    const out = [];
+    while (n > 0n) { out.push(alphabet[Number(n % base)]); n = n / base; }
+    return alphabet[0].repeat(leadingZeroBytes(data)) + out.reverse().join('');
+}
+
+// SUFFIX: assert custom != standard
+```
+
+### Process for each task:
+1. Pick a task ID from the 100% list
+2. Design a new task using one of the three mechanisms
+3. Write JSON with all required fields
+4. Validate with Node.js
+5. Eval against GPT-5.5 (n=5)
+6. If pass@1 ≤ 40%: SUCCESS
+7. If pass@1 ≥ 80%: try different approach (up to 3 attempts)
+8. Update running score
+9. **Stop when overall ≤ 58%**
+
+### MANDATORY: Save completions for every evaluated task
+
+```bash
+echo '{TASK_JSON}' > /tmp/codex_eval_task.jsonl
+EVAL_MODEL=gpt-5.5-2026-04-23 \
+EVAL_FILE=/tmp/codex_eval_task.jsonl \
+SAVE_COMPLETIONS=1 \
+COMPLETIONS_OUT=/Users/adarshkumarappan/Impt/devbench/benchmark/javascript/api_usage/completions-codex-mutation-task{ID}-attempt{N}.jsonl \
+/Users/adarshkumarappan/miniconda3/envs/devbench/bin/python -u /Users/adarshkumarappan/Impt/devbench/eval_poc.py
+```
+
+### Evaluation setup:
+- Model: `gpt-5.5-2026-04-23` (no temperature)
+- Use `max_completion_tokens` not `max_tokens`
+- n=5, Pass@1 = c/n
+- Eval script: `/Users/adarshkumarappan/Impt/devbench/eval_poc.py`
+- Node.js: `/Users/adarshkumarappan/.nvm/versions/node/v24.12.0/bin/node`
+- Conda: `/Users/adarshkumarappan/miniconda3/envs/devbench/bin/python`
+- **ALWAYS use absolute paths**
+
+### Important constraints:
+- Tasks must be solvable from visible context (FAIRNESS)
+- Suffix must contain signals for correct behavior
+- Tasks should fit "API Usage" category: testing Node.js built-in module usage
+- Golden completions: 3-15 lines
+- Node.js standard modules only
+- ONE task at a time, track running score
+- **Pre-compute all expected values** — do NOT guess encoding outputs
+
+### What to save:
+1. Append new tasks to: `benchmark/javascript/api_usage/codex_mutations.jsonl`
+2. Save per-task completions (see MANDATORY section)
+3. Save scores to: `benchmark/javascript/api_usage/codex_mutation_scores.jsonl`
+4. Update `HARDENING_LOG.md` with JS API Usage results
+5. Do NOT assemble final file
+
+## [CODEX: ADD YOUR OWN INSIGHTS HERE]
+
+Before starting:
+1. Analyze the 3 tasks that score 0% — understand what mechanisms work
+2. Look at the 44 tasks at 100% — understand why they're easy
+3. Plan which tasks to replace and with which mechanism
+4. Since you need ~17 conversions, efficiency matters — start with proven encoding tasks
+
+## Stop condition
+
+**Stop when running_score ≤ 58%.** Do NOT continue past this. Do NOT re-eval all 50.

CODEX_PROMPT_JS_CODE2NL.md

@@ -0,0 +1,192 @@
+# Task for Codex: Harden DevBench JavaScript Code2NL/NL2Code
+
+## Context
+
+DevBench is a code generation benchmark rejected from ICLR 2026 (ceiling effect). Resubmitting to NeurIPS 2026. Python is done (53% overall for GPT-5.5). Now doing JavaScript.
+
+FIM code completion format:
+- Model sees: `prefix + #TODO: You Code Here + suffix`
+- Model does NOT see `assertions` (hidden tests)
+- Execution: `prefix + model_completion + suffix + assertions`
+- Pass@1 with n=5 samples
+
+## Goal
+
+**Bring GPT-5.5 Pass@1 on JavaScript Code2NL/NL2Code to ≤45%** (currently 62.4%).
+
+- Current total points: 3120 (out of 5000)
+- Target: ≤2250 points (45%)
+- Need to reduce by: 870 points
+- If each conversion takes a 100% task to 0%: need ~9 conversions
+- There are 28 tasks at 100% — plenty of candidates
+
+## MANDATORY: Read these files first
+
+1. **`HARDENING_LOG.md`** — Full methodology. Pay special attention to:
+   - Round 7 (Code2NL Codex report) — docstring precision mechanism
+   - The exact failure pattern: GPT-5.5 returns empty completion or `pass` instead of precise docs
+2. **`benchmark/javascript/code2NL_NL2code/code2NL_NL2code.jsonl`** — Current 50 JavaScript tasks
+3. **`eval_poc.py`** (at repo root) — Evaluation script
+4. **`.env`** — API keys
+
+## What Worked for Python Code2NL (from HARDENING_LOG.md — achieved 46.4%)
+
+**Docstring precision gaps** — the ONLY mechanism that worked:
+- Completion slot is immediately after a function/method header, before the executable body
+- The body is visible in the suffix and shows specific API behavior
+- Hidden assertions check that `.__doc__` contains specific keywords
+- GPT-5.5 returns empty completion or generic "Processes input" instead of precise docs
+
+**For JavaScript, the docstring approach uses:**
+1. **`.doc` property**: `myFunc.doc = 'precise description...'` — assertions check `myFunc.doc.includes('keyword')`
+2. **`describe*()` function**: returns description string — assertions check return value
+3. **NL2Code**: prefix has natural language spec, completion is code implementation with subtle behavioral requirements
+
+### What the 17 existing 0% tasks prove works in JS:
+- Tasks 1-8: `.doc` property and `describe*()` approaches — GPT-5.5 writes generic descriptions missing specific keywords like "mutating", "WeakSet", "circular", "leading", "undefined"
+- Tasks 21-22, 24-28, 30: doc property and describe functions requiring precise API documentation — GPT-5.5 omits details about LRU promotion, bottom-up traversal, deref(), non-enumerable properties
+- Task 40: Complex template engine NL2Code — GPT-5.5 can't coordinate pipe filters + double-brace escaping
+
+## Current GPT-5.5 Scores
+
+```
+100% (28 tasks): 9,10,11,12,13,14,15,17,18,19,20,23,29,31,32,33,34,36,37,38,39,41,42,44,46,47,49,50
+80%  (3 tasks):  16,45,48
+60%  (1 task):   43
+20%  (1 task):   35
+0%   (17 tasks): 1,2,3,4,5,6,7,8,21,22,24,25,26,27,28,30,40
+```
+
+Overall: 62.4% (3120/5000). Target: ≤45% (2250/5000).
+
+## Your Task
+
+**Replace 100% tasks ONE AT A TIME with harder Code2NL/NL2Code tasks.**
+
+### Recommended mechanism mix (for 9 conversions):
+- **5-6 doc/describe tasks**: Completion is a `.doc` property string or `describe*()` return value. Assertions check for specific keywords that GPT-5.5 omits from generic descriptions.
+- **3-4 NL2Code tasks**: Prefix has detailed natural language spec, completion is code with subtle behavioral requirements that differ from standard implementations.
+
+### Doc/describe task template:
+
+```javascript
+// PREFIX:
+function stablePartition(arr, predicate) {
+
+// GOLDEN_COMPLETION (doc property):
+    stablePartition.doc = 'Stably partition array in place by predicate. ' +
+        'Items where predicate returns true move to front, preserving relative order ' +
+        'within both groups. Mutates the input array. Predicate is called exactly once ' +
+        'per item. Returns the split index.';
+
+// SUFFIX (implementation visible to model):
+    const kept = [];
+    const rest = [];
+    for (const item of arr) {
+        (predicate(item) ? kept : rest).push(item);
+    }
+    const idx = kept.length;
+    kept.push(...rest);
+    arr.length = 0;
+    arr.push(...kept);
+    return idx;
+}
+
+// ASSERTIONS (hidden):
+const assert = require('assert');
+assert(stablePartition.doc.includes('in place'));
+assert(stablePartition.doc.includes('Mutates'));
+assert(stablePartition.doc.includes('exactly once'));
+assert(stablePartition.doc.includes('split index'));
+```
+
+### Keywords that GPT-5.5 consistently OMITS (use these in assertions):
+- "mutate" / "in place" / "modifies" (for mutation operations)
+- Specific API method names (e.g., "WeakSet", "deref", "bisect")
+- "exactly once" / "at most once" (for call count guarantees)
+- Exception/error behavior ("throws TypeError", "throws RangeError")
+- Return value details ("returns the split index", "returns undefined")
+- Side effect details ("appends to audit log", "updates counter")
+- Edge case behavior ("empty input returns", "null treated as")
+
+### NL2Code task template:
+
+```javascript
+// PREFIX (natural language spec):
+// Implement slugify(str) that:
+// - Converts to lowercase
+// - Replaces spaces with hyphens
+// - REMOVES non-alphanumeric chars (does NOT replace with hyphens)
+// - Collapses consecutive hyphens
+// - Trims leading/trailing hyphens
+
+// GOLDEN_COMPLETION:
+function slugify(str) {
+    return str.toLowerCase()
+        .replace(/\s+/g, '-')
+        .replace(/[^a-z0-9-]/g, '')  // remove, not replace
+        .replace(/-+/g, '-')
+        .replace(/^-|-$/g, '');
+}
+
+// SUFFIX:
+const assert = require('assert');
+assert.strictEqual(slugify('Hello World'), 'hello-world');
+
+// ASSERTIONS (hidden — test the subtle behavior):
+assert.strictEqual(slugify('a!!!b'), 'ab');  // NOT 'a-b'
+```
+
+### Process for each task:
+1. Pick from the 100% list
+2. Design using doc/describe or NL2Code mechanism
+3. Write JSON with all fields
+4. Validate with Node.js
+5. Eval against GPT-5.5 (n=5)
+6. If pass@1 ≤ 40%: SUCCESS
+7. If pass@1 ≥ 80%: try different (up to 3 attempts)
+8. Update running score
+9. **Stop when overall ≤ 45%**
+
+### MANDATORY: Save completions
+
+```bash
+echo '{TASK_JSON}' > /tmp/codex_eval_task.jsonl
+EVAL_MODEL=gpt-5.5-2026-04-23 \
+EVAL_FILE=/tmp/codex_eval_task.jsonl \
+SAVE_COMPLETIONS=1 \
+COMPLETIONS_OUT=/Users/adarshkumarappan/Impt/devbench/benchmark/javascript/code2NL_NL2code/completions-codex-mutation-task{ID}-attempt{N}.jsonl \
+/Users/adarshkumarappan/miniconda3/envs/devbench/bin/python -u /Users/adarshkumarappan/Impt/devbench/eval_poc.py
+```
+
+### Evaluation setup:
+- Model: `gpt-5.5-2026-04-23` (no temperature)
+- `max_completion_tokens` not `max_tokens`
+- n=5, Pass@1 = c/n
+- Eval script: `/Users/adarshkumarappan/Impt/devbench/eval_poc.py`
+- Node.js: `/Users/adarshkumarappan/.nvm/versions/node/v24.12.0/bin/node`
+- Conda: `/Users/adarshkumarappan/miniconda3/envs/devbench/bin/python`
+- **ALWAYS use absolute paths**
+
+### Important constraints:
+- Doc tasks: completion is DOCUMENTATION, not code
+- NL2Code tasks: completion is CODE from a spec, with subtle behavioral traps
+- Do NOT use encoding tasks — wrong category
+- Golden completions: 3-12 lines
+- Node.js standard modules only
+- ONE task at a time
+
+### What to save:
+1. Append new tasks to: `benchmark/javascript/code2NL_NL2code/codex_mutations.jsonl`
+2. Save per-task completions
+3. Save scores to: `benchmark/javascript/code2NL_NL2code/codex_mutation_scores.jsonl`
+4. Update `HARDENING_LOG.md`
+5. Do NOT assemble final file
+
+## [CODEX: ADD YOUR OWN INSIGHTS HERE]
+
+Analyze the 17 existing 0% tasks. What keywords do their assertions check for? Which doc/describe patterns are most effective? Use that to design your replacements.
+
+## Stop condition
+
+**Stop when running_score ≤ 45%.** Need ~9 conversions.