Skip to content

Update evaluation command in README#17

Closed
ItsTania wants to merge 5 commits into
mainfrom
Update-Docs
Closed

Update evaluation command in README#17
ItsTania wants to merge 5 commits into
mainfrom
Update-Docs

Conversation

@ItsTania

@ItsTania ItsTania commented May 26, 2026

Copy link
Copy Markdown
Collaborator

Currently - the template makes a few ridged assumptions.

For example with [project] \n name = "eval_name" in the .toml file and scr/"eval_name". If eval_name is not the same the docs are no longer accurate:

With Generality-Labs#1 :

@task calls get_installed_package_name(), which only resolves to a package prefix when an installed distribution exists by that name. If [project] name is e.g. "lab-bench" but the package folder is lab_bench_2, no distribution matches the import name, so the task is registered under its bare name (lab_bench_2) instead of the namespaced form (lab_bench_2/lab_bench_2).

A long-term solution would be to simplify this template - for now, this PR updates docs

Note: if loading through the full file path e.g src/lab_bench_2/lab_bench_2.py@lab_bench_2 is fine then we can remove all of the hassle of registering tasks etc etc.

Checklist

  • Are you adding a new eval?
  • Does this change affect existing eval(s)?
  • Does this change affect how future contributors write or submit evaluations (e.g. new required fields, changed tooling, updated conventions)?

@ItsTania

Copy link
Copy Markdown
Collaborator Author

@ItsTania ItsTania closed this May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant