Skip to content

Create instruction.txt in evaluation sandboxes - #1

Open
sail-amir wants to merge 1 commit into
waynchi:mainfrom
sail-amir:fix-evaluation-instruction-txt
Open

Create instruction.txt in evaluation sandboxes#1
sail-amir wants to merge 1 commit into
waynchi:mainfrom
sail-amir:fix-evaluation-instruction-txt

Conversation

@sail-amir

Copy link
Copy Markdown

Summary

This PR writes each problem's instruction to instruction.txt when creating an evaluation sandbox.

Some JavaScript/React tests read ../instruction.txt directly. The current sandbox setup writes files such as original_code.jsx, implementation1.jsx, and tests/test_code.test.js, but does not write instruction.txt, so these tests can fail before any assertions run.

Affected example

Problem 83 fails during Jest setup because its test reads:

const instruction = fs.readFileSync(path.join(__dirname, '../instruction.txt'), 'utf8').trim();

Before this fix, the failure is:

ENOENT: no such file or directory, open '/root/editbench_sandboxes/question_83/instruction.txt'
Tests: 0 total
No test results found at /root/editbench_sandboxes/question_83/test_results.json

This affects the full translated problem family because all variants share the same pair_id and test structure:

pair_id: c7f2ad26-828e-4bc0-a2d1-ec8171ff195a

83   english   javascript/react   adjust width according to content
212  chinese   javascript/react   根据内容调整宽度
309  russian   javascript/react   регулируйте ширину в соответствии с содержимым
414  polish    javascript/react   dostosuj szerokość do zawartości
519  spanish   javascript/react   ajustar el ancho según el contenido

Fix

Create instruction.txt in each question sandbox from question["instruction"].

The change is intentionally small:

with open(curr_dir / "instruction.txt", "w") as f:
    f.write(question["instruction"])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant