fix(api): /data/search dumps full course/prof catalog regardless of matview freshness - #187
Draft
jerryzhou196 wants to merge 1 commit into
Draft
fix(api): /data/search dumps full course/prof catalog regardless of matview freshness#187jerryzhou196 wants to merge 1 commit into
jerryzhou196 wants to merge 1 commit into
Conversation
The search dump INNER JOINed aggregate.course_rating / aggregate.prof_rating, which are views over materialized views (materialized.course_rating / prof_rating). Those matviews only refresh via triggers on review and section_meeting writes, never on plain course/prof inserts. When Hasura is up but the DB is freshly/partially seeded, the matviews are stale or empty, so the INNER JOIN silently drops most rows and the endpoint returns a tiny dump with a 200. Use LEFT JOIN so the live course/prof tables are the source of truth, and COALESCE(filled_count, 0) so rating counts default to 0 when the matview lacks a row. Also check rows.Err() after each iteration loop so a mid-stream pgx error surfaces as a 5xx instead of being returned as partial success. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
/data/searchreturns a tiny (or empty) dump — with a200— when Hasura is up but the database is freshly or partially seeded, instead of the full catalog (regtest/api/dump.jsexpectscourses.length > 7000).Root cause
Both queries in
flow/api/data/data.goINNER JOINedaggregate.course_rating/aggregate.prof_rating. Those are views over materialized views (materialized.course_rating,materialized.prof_rating), which only get refreshed by triggers onreviewandsection_meetingwrites (seerefresh_course_rating/refresh_section_meetingin the init migration) — never on plaincourse/profinserts.So when the base tables are seeded but no review/section write has fired a refresh, the matviews are stale/empty and the
INNER JOINsilently drops the missing rows. The handler also never checkedrows.Err(), so a mid-stream pgx error would be returned as partial success too.Fix
INNER JOIN→LEFT JOINonaggregate.course_rating/aggregate.prof_ratingso the livecourse/proftables are the source of truth for which entries appear in the dump.COALESCE(filled_count, 0) AS review_countso the rating count defaults to0when the matview has no row yet (also avoids a NULL scan into anint).rows.Err()after each iteration loop so a streaming error surfaces as a 5xx instead of a silent truncated200.Rating counts can still read
0until the matviews are refreshed, but the catalog is now complete regardless of refresh state.Test plan
go build ./.../go vet ./api/data/(passes locally)/data/searchagainst a seeded-but-unrefreshed DB and confirm fullcourses/profscountsregtest/api/dump.jsstill passes against a fully seeded DB🤖 Generated with Claude Code