Consider pinning public-skill context in benchmark metadata

[neubig]

Context

During a local SWE-bench smoke evaluation on an HPC/Apptainer setup, I temporarily added an environment variable to disable load_public_skills() so the run only reflected the benchmark prompt, tools, model checkpoint, and dataset instance.

That local change was not kept for upstreaming because public skills are part of the current OpenHands benchmark harness and are standardized through OpenHands/extensions.

Reproducibility question

For controlled model comparisons, it may still be useful to make the skill set explicit in run metadata, because public skills are loaded from external extension content rather than being fully described by the benchmark repo commit alone. If the extension ref changes, two runs using the same benchmark commit/model/dataset could differ in available skill context.

Possible directions

• Record the resolved EXTENSIONS_REF and loaded skill names in benchmark metadata.
• Add documentation recommending a pinned EXTENSIONS_REF for benchmark papers or checkpoint comparisons.
• Consider an explicit opt-out only for ablation/control runs, if maintainers think that is appropriate.

Non-goal

This is not proposing to disable public skills by default. The issue is only to decide whether benchmark metadata/config should make the loaded public skill context more explicit for reproducibility.