fix(autograder): grade comment-stripped SQL so guide comments don't skew scores#6
Merged
Merged
Conversation
… and hidden keywords don't skew scores
The Week 9 autograder had the same file_is_filled() blanket-TODO bug, but Week 10
needs a deeper fix: the dbt scaffold ships "-- TODO:" guide comments *inside* the
starter models, and those comments contain gradeable keywords ("add JOIN",
"add GROUP BY", the output column names). So two failure modes existed:
1. A correct submission that leaves the guide comment in place was scored as an
empty stub (file_is_filled failed on the word TODO). Verified: a fully-correct
solution that keeps the comments scored 55/FAIL under the old grader.
2. Naively relaxing file_is_filled alone let the *bare scaffold* pass at 62,
because the pattern greps matched the JOIN/GROUP BY/column keywords sitting
inside the TODO comments.
Fix:
- file_is_filled() strips SQL line-comments and only flags a line whose content
*begins* with TODO (an unreplaced markdown/YAML placeholder).
- New sqlgrep() helper matches patterns against comment-stripped SQL; every .sql
content check (staging filters, tip_pct, mart JOIN/GROUP BY, columns, refs,
singular-test ref) now goes through it, so keywords hidden in comments no
longer count as real SQL.
Verified: bare scaffold 62->48 (now correctly FAILS), a correct comments-kept
solution 55->100 (now correctly PASSES). YAML/markdown checks unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tartup Same load-time permissions bug as the other assignment repos: the grade job forced contents:none, but the reusable auto-grade.yml needs contents:read. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Week 10 autograder (
.hyf/test.sh) has the samefile_is_filled()blanket-"TODO" gate as Week 9 — but Week 10 needs a deeper fix, because its dbt scaffold ships-- TODO:guide comments inside the starter models, and those comments contain gradeable keywords ("add JOIN", "add GROUP BY", the six output column names).Two failure modes result:
Correct work scored as a stub. A student who writes the model below the guide comment and leaves the comment in place was failed by
file_is_filled(it rejects any file containing "TODO"). Verified: a fully-correct solution that keeps the comments scores 55/FAIL under the current grader (fct_daily_borough_stats.sql: still contains TODO stubs,singular test: still a TODO stub).Naively relaxing the gate lets the empty scaffold pass. Just fixing
file_is_filled(the Week 9 approach) makes the bare scaffold score 62 and pass, because the pattern greps (grep JOIN "$mart") match the JOIN / GROUP BY / column-name keywords sitting inside the TODO comments.Fix
file_is_filled()strips SQL line-comments and only flags a line whose content begins with TODO (an unreplaced markdown/YAML placeholder).sqlgrep()helper matches patterns against comment-stripped SQL. Every.sqlcontent check now routes through it: stagingsource(),IS NOT NULL,fare_amount >= 0,tip_pct; martref()to both,JOIN,GROUP BY, output columns; and the singular-testref(). Keywords hidden in comments no longer count as real SQL. YAML/markdown checks are unchanged.Verified (both sides)
So it stops penalizing left-in guide comments and stops the empty scaffold from passing on comment text. There are currently no open student PRs on this repo, so no live submissions are affected; this hardens the template before the cohort forks it.
🤖 Generated with Claude Code