feat: classify and exclude bot traffic at click ingestion#31
Merged
Conversation
Bot detection belongs at ingestion, where the full request context exists and the classification can be persisted once for every consumer to read — rather than re-deriving it from a stored user-agent at read time. - click_events: add is_bot (NOT NULL DEFAULT false) + bot_reason columns, with a partial index for the common human-only analytics filter. Idempotent ALTER for existing databases. - lib/bot-detection.ts: classify each click via the isbot user-agent database, the HTTP method (HEAD/OPTIONS are probes, never human), and an optional trusted upstream bot signal (honored only when TRUST_EDGE_BOT_HEADER=true). Keep-and- flag, not drop — the row is recorded and flagged, preserving an audit trail. - Wire classification into both click-recording paths (redirect + SDK resolve). - Analytics: exclude is_bot rows from all click counts. - scripts/backfill-bot-flags.ts: one-time, keyset-paginated backfill that flags historical rows from their stored user-agent (SIT-298). Verified: 139 unit tests pass (incl. classifier cases) and an integration check confirms the schema applies, flags persist, and analytics excludes bots.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Detects bot/automated traffic at the moment a click is recorded and persists the classification on the
click_eventsrow, so analytics can exclude it consistently. Detection belongs at ingestion - that's the only point with the full request context (user-agent, method, headers), and classifying once means every consumer reads one answer instead of re-deriving it from a stored user-agent.Changes
click_eventsgainsis_bot BOOLEAN NOT NULL DEFAULT falseandbot_reason VARCHAR(16)[1], plus a partial index (WHERE is_bot = false) for the common human-only analytics filter [2]. IdempotentALTERfor existing databases [3]; backward compatible (legacy rows default tofalseuntil the backfill runs).lib/bot-detection.ts: classifies via (1) an optional trusted upstream signal - honored only whenTRUST_EDGE_BOT_HEADER=true[1], (2) HTTP method (HEAD/OPTIONSare probes/prefetch, never human), (3) theisbotuser-agent database [2]. Keep-and-flag, not drop - the row is still recorded and flagged, preserving an audit trail and enabling "% automated" reporting.is_botrows from all click counts [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11].scripts/backfill-bot-flags.ts(npm run backfill:bot-flags[2]): one-time, keyset-paginated backfill that flags historical rows from their stored user-agent [1].Behavior change (release note)
Analytics click counts now exclude detected bots - reported numbers will drop to the human-only figure. Run the backfill once to apply this retroactively to existing data.
Verification
HEAD/OPTIONS, edge signal, empty UA).true/ua, human→false/null), and the analytics filter excludes bots.