readme: report the per-genus average and stop calling the split stratified - #10
Merged
Merged
Conversation
…ified Two claims in the README did not survive a check against eval/eval_bioclip_vs_orchid_clip.py. The holdout was described as "stratified". It is not: split_bucket() is an md5 of source_id giving a uniform bucket, val is everything under --val-fraction (0.02) capped at --max-val (4000), and the balancing flag --cap-per-species defaults to 0. Calling it stratified claims the one design choice that would have evened out the composition. The headline led with the per-image top-1 (+3.8 pp) while making a long-tail claim. "top1" in the harness is correct1 / n, so Ophrys -- 2,754 of 4,000 images, and the genus that gains least at +2.8 pp -- dominates it. Averaged over the 14 genera the harness reports (n >= 20, 3,937 of 4,000 images) the lift is +7.6 pp: 0.768 -> 0.844. The honest number is the better number, and it is the one that actually supports the claim being made. Also states what the previous text omitted: three of the fourteen genera regress (Cymbidium -6.0, Laelia -4.2, Encyclia -1.0 pp). Numbers derived from the per-genus values behind assets/plotly/orchidclip_per_genus.html; the weighted mean over those 14 genera reproduces the published 0.873/0.911. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ydeaSFAE8vThCNTYKTi41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The site page was corrected on 2026-08-08; this README still carried both of the claims that correction removed. A retraction has to reach every place the claim was stated.
1. The holdout is not stratified
README.mdsaid "a stratified 4,000-image holdout". Fromeval/eval_bioclip_vs_orchid_clip.py:split_bucket()ismd5(source_id)→ a uniform bucket--val-fraction(default0.02), capped at--max-val(default4000)--cap-per-species— the balancing flag — defaults to0, and its own help text reads "If >0, tail-balance the val set"Calling it stratified claims precisely the design choice that would have evened out the composition. Now described as a hash-partitioned random 2% split that inherits the domain's imbalance.
2. The headline used the average that hides the tail
"top1"in the harness iscorrect1 / n— per image. Ophrys is 2,754 of 4,000 images (69%) and is the genus that gains least (+2.8 pp), so it dominates that mean. The README led with it (+3.8 pp) while making a long-tail argument.Averaged over the 14 genera the harness reports (
MIN_PER_GENUS = 20; 3,937 of 4,000 images):+3.8 pp per image → +7.6 pp per genus. The honest number is the better number, and it's the one that actually supports the claim.
3. Three genera regress, which went unmentioned
Cymbidium −6.0, Laelia −4.2, Encyclia −1.0 pp — all well-sampled genera BioCLIP 2 already did well on. "Gains landing exactly where they should" was doing some work there. Now stated.
Provenance
Per-genus values taken from the data behind
assets/plotly/orchidclip_per_genus.html(14 genera, n ≥ 24). The macro mean over them is 0.7677 / 0.8436; the n-weighted mean over the same 14 reproduces the published 0.873 / 0.911, which is the consistency check that the two columns come from one run.No numbers were changed — one was added, and the one already there is now labelled with what it averages over.
🤖 Generated with Claude Code