Fix Getting Started snippets to work on a fresh clone#15
Merged
celiawaggoner merged 8 commits intoMay 27, 2026
Merged
Conversation
The README's "Run linters" snippet invoked `uv run mypy .`, which scans `tools/` and trips on a duplicate-module error (`tools/run_autolint.py` script vs `tools/run_autolint/` package both resolving as `run_autolint`). `tools/run_checks.sh` and CI both run `uv run mypy src tests`, so align the README with that. Also swap the `pytest` snippet to `pytest -k "not agentic"` — the agentic example tests require Docker, which isn't running in a fresh clone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The README's "Run an evaluation" snippet uses --model openai/gpt-5-nano,
and CONTRIBUTING.md:290 recommends openai/gpt-5-nano as the dev model,
but the openai SDK isn't installed by `uv sync`. Fresh-clone users hit:
ERROR: OpenAI API requires optional dependencies. Install with:
pip install openai
Adding openai to the dev group keeps it out of the eval's runtime deps
(downstream eval consumers don't pull dev deps) while making the
documented dev workflow work first try.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
richardhuaaa
changed the title
The Quickstart jumped from `cp -r src/examples/simple_qa src/my_eval` straight to `inspect eval my_eval/my_task`, but `my_task` doesn't exist inside the freshly-copied directory — the @task function is still named `simple_qa`, and `__init__.py` still imports from `examples.simple_qa`, so even `my_eval/simple_qa` would dispatch back to the example package. Add an explicit rename step covering the module file, the task function, and the __init__.py import so step 5's command actually resolves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Quickstart used literal `my_eval`/`my_task` names which weren't obviously placeholders. Switch to `<eval_name>`/`<task_name>` for clarity (consistent with the Getting Started snippet above). Also add the distribution name step: `[project] name` must match the eval package name, otherwise inspect_ai's task resolver can't map the entry-point package to an installed distribution and the `<eval_name>/<task_name>` CLI form silently fails. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ItsTania
reviewed
May 26, 2026
ItsTania
reviewed
May 26, 2026
ItsTania
reviewed
May 26, 2026
Co-authored-by: Tania <120768997+ItsTania@users.noreply.github.com>
Co-authored-by: Tania <120768997+ItsTania@users.noreply.github.com>
Co-authored-by: Tania <120768997+ItsTania@users.noreply.github.com>
ItsTania
reviewed
May 26, 2026
ItsTania
approved these changes
May 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains
Description
Fixes so the Getting Started and Quickstart snippets work on a fresh clone.
Linter command.
uv run mypy .fails on a duplicate-module error.tools/run_checks.shand CI useuv run mypy src tests; align the README with that.Test command.
uv run pytestincludes agentic example tests that need Docker. Switch touv run pytest -k "not agentic".Provider SDK. The quickstart uses
--model openai/gpt-5-nanobut theopenaiSDK isn't installed byuv sync. Addopenaito thedevdependency group.Quickstart task resolution. The
<eval_name>/<task_name>CLI form requiresinspect_aito map the entry-point package to an installed distribution. This only works when[project] nameinpyproject.tomlmatches the eval package name. Document this step and the post-copy rename. Use<eval_name>/<task_name>placeholders for clarity.Verified locally
Checklist
🤖 Generated with Claude Code