Multi-destination: click-to-WhatsApp ads, the encoded ref, and ad attribution - #244
Merged
Conversation
… map
vlab is taking over the ad -> stratum join from the dotted ref string that
rides to the survey platform inside every message. This is A1 + A2 of
planning/ad-id-attribution.md: capture the id Facebook returns when an ad is
created, and freeze what that ad means into a new ad_attributions table.
Purely additive. No existing study's ad creation behaviour changes -- make_ref
and create_creative's ref emission are untouched, because changing a creative
triggers ad rewrites across every live study on the next reconciliation run.
A1, capture:
- Instruction gains an optional, defaulted `provenance` field, so every
existing construction site is unchanged (guarded by a test).
- GraphUpdater.execute returns (report, created_id) instead of dropping the
SDK's return value. It is the only moment vlab learns the ad id.
- ad_dif/adset_dif take an optional provenance lookup keyed by
(adset name, ad name) == (stratum id, creative name) and stamp it onto ad
creates. Instruction generation stays pure; the write lands in
run_instructions, in the imperative shell.
- creative_metadata is extracted out of create_creative so the frozen blob and
the ref are computed from one expression and cannot drift.
A2, the table:
- devops/migrations/20260816000000_add_ad_attributions.{up,down}.sql, plus the
matching declaration in devops/helm/migrations/init.sql.
- No TTL and no FK to studies: a cascading delete is still a delete path, and
respondents keep arriving from ads that reconciliation has deleted, via
reshared page posts. The row must outlive the ad.
- Writes are ON CONFLICT DO NOTHING, so a re-run can never overwrite the
snapshot with metadata from an edited conf.
The metadata blob is `{"creative": name, **md}` -- what the ref would have
carried -- not stratum.metadata, which omits `creative` and `form`. Getting
that wrong does not error, it miscounts: the stratum matches nobody and the
optimizer reallocates budget away from it. A round-trip test against a
reimplementation of fly's own ref parser is the guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
Adds documentation/ad-attributions.md (cross-component: the data path from ad_provenance through to the row, the three invariants and the failure each prevents, and why network is the ad network rather than the messaging channel) and an "Ad-ID attribution" section to adopt/README.md (app-level: where the plumbing lives and which tests cover which half). Both reference planning/ad-id-attribution.md for the full design. That file is still untracked in the main worktree while it is being revised; the reference points at where it will land. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
Phases A5-A7 of planning/ad-id-attribution.md. vlab now consumes the ad_attributions mapping written in A1/A2: an event carries the ad that recruited its respondent, and swoosh resolves stratum variables through the frozen mapping row instead of the dotted ref. A5 -- first-class ad id on the event: - InferenceDataEvent gains typed AdID/AdNetwork fields, not reserved keys in User.Metadata. A map key would be a fly-shaped convention every future connector has to imitate with nothing enforcing it -- the same mistake as the dotted ref, a convention smuggled inside an untyped blob. - No migration. inference_data_events stores the whole event as one JSON blob in `data`, and the new fields are omitempty, so existing rows are unchanged. - The fly connector maps platform -> ad network. Messenger and WhatsApp ads are both Meta ads in one id namespace, so both are "facebook": this is the ad network, not the messaging channel. A6 -- the "ad" extraction location: - User-declared, same ExtractionConf shape as every other location. vlab derives nothing. - No fallback to location "metadata". swoosh recomputes a study's whole history every run, so a fallback would let an existing study's conf swap silently re-attribute its back-catalogue through a path its events cannot satisfy. New studies only. - The mapping is loaded once per study in swooshStudy and passed to Reduce as plain data. Reduce has no pool, so a per-event query is unrepresentable rather than merely avoided. Per-study, so a foreign ad id misses instead of importing another study's strata. A7 -- the three-way split: - attributed / organic (no ad id; counted, warning) / unmapped (ad id with no mapping row; counted, severity error). Classified once per event, not per conf, so one organic arrival is not multiplied by the conf count. - Unmapped is self-healing: inserting the missing row retroactively fixes prior runs, and the dashboard's recency window ages the old error out. Also adds the completeness check as pure functions plus a warning: a stratum targeting a variable no extraction conf supplies can never match, so it counts zero and the optimizer moves its budget away. Warns rather than raises -- the predicate has never been measured against existing studies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
Extends documentation/ad-attributions.md to cover both halves end to end: the fly connector's platform -> ad network derivation, location "ad" and why a fallback to "metadata" would be a bug, where the database touch lives and why it cannot move into the RetrieveFunc, the attributed/organic/unmapped split with its self-healing property, and the completeness check. Adds an "Ad-ID attribution" section to inference/README.md covering the same ground at app level, placed after Architecture Overview. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
C1 of planning/ad-id-attribution.md. Until now an ad-derived variable could only be declared by hand-editing conf JSON, which defeats the point of the declared-not-derived design: vlab deliberately does not infer these confs, so the form is the only way a researcher can state one. - Adds `ad` to the fly source's location options, labelled "Ad (which ad recruited them)" rather than in terms of ad ids or the mapping table. - `ad` gets aggregate "first", like metadata. The old condition read `location === "metadata" ? "first" : "last"`, so any new location silently became "last"; it is inverted now, so only `variable` -- the one location whose value can meaningfully change over time -- takes the later value. - `metadata` and `ad` are both keyed lookups with no response path, so the `isMetadata` flag becomes `isKeyedLocation`. The concept is "keyed, not response-pathed", expressed once instead of as scattered location checks. - Switching to a keyed location now resets `functions` to the identity select. Without it, a conf built as `variable` with path "response" and then switched to `ad` would keep trying to select "response" out of a bare metadata value and fail extraction for every event. The Qualtrics form already did this for metadata; here it covers both keyed locations. - The key placeholder asks for a stratum metadata key when the location is ad, since that is what the study's ads were actually built with. The Qualtrics/Typeform form deliberately does NOT offer `ad`: only the fly connector populates an event's ad id, so the option there would let someone configure a variable that silently yields nothing forever -- exactly the quiet miscount this design exists to prevent. Its location list is extracted into its own module so the two can never be merged by accident, and a test asserts the Qualtrics list has no `ad`. Logic lives in pure modules with no React dependency, following the existing forms/variables/extract.ts precedent, so it is testable in isolation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
A3 of planning/ad-id-attribution.md.
GET /{org_id}/studies/{slug}/ad-attributions.csv, on the same org/study routing
and auth as the other study endpoints.
The frozen metadata blob is flattened into columns under its own key names, so
the user story is one line: left-join your survey export on `ad_id` and your old
metadata columns come back, named as they always were. That is only true because
the blob is key-for-key the dict the dotted ref used to carry -- the phase-1
invariant is what makes this export honest rather than approximately right.
Shape: ad_id, network, <frozen metadata keys>, created.
- Columns are the union across rows in first-seen order, not the first row's
keys. Keys are uniform within a study in practice, but a conf edit mid-flight
leaves rows frozen under two shapes, and append-only means both survive.
- A metadata key colliding with ad_id/network/created is emitted as
metadata_<key>. Two columns of the same name is the kind of thing nobody
notices until the analysis is already wrong.
- Deleted ads are included, deliberately. Respondents keep arriving from ads
reconciliation has removed, via reshared page posts, so a CSV of only live ads
would silently lack rows the researcher needs -- and those respondents would
look unattributed. Nothing in the read path filters on liveness.
Rendering is a pure function over already-fetched rows; the route is the only
part that touches HTTP or the database.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
Extends documentation/ad-attributions.md with the two things that make the feature reachable by a researcher rather than only by an engineer: how to declare location "ad" in the dashboard (and why the option is fly-only), and the CSV export with its left-join user story. Records that nothing between the form and swoosh constrains `location` -- it is a bare string in the dashboard's TypeScript, in the Go API's opaque conf storage and in Python's ExtractionConf -- so the form's dropdown and swoosh's getRetrieveFunc are the only two places that enumerate the allowed values. Also notes the absence of a dashboard download button and why the API-key path suits the export's primary use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
Eight commits across two repos referenced this file while it existed only as an untracked file in the main worktree. Commit it alongside the code that implements it so those references resolve. Records the design as built, including the two reversals made during implementation: extraction confs are user-declared rather than derived, and `location: "ad"` is for new studies only with no fallback to metadata -- swoosh recomputes every study from scratch on every run, so a fallback would let an existing study's conf swap silently re-attribute its entire back-catalogue through a path its events cannot satisfy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
A8. vlab could not create CTWA ads at all: create_creative branched only on Messenger, App and Web, and nothing set an autofill message. Greenfield, so there is no legacy CTWA behaviour to preserve. The design is driven by one empirical constraint. A CTWA referral carries no advertiser-settable `ref` -- url_tags was measured not to reach WhatsApp -- so fly recovers the shortcode from the ad's autofill text, matched against an anchored full-match pattern (WHATSAPP_ENTRY_REF in fly's event-normalizer.js). Verified against that regex directly: - make_ref's output can NEVER match. The pattern anchors on `form.` and make_ref leads with `creative.`, so it fails whatever the values are. This is structural, not a character-set problem, so WhatsApp needs its own form-first serialisation: whatsapp_autofill. - Every token is [A-Za-z0-9_-]. Real production stratum values -- "Bauchi State", "Like Parents", "South East", "Static English - Girls" -- all fail, raw and percent-encoded alike, since `%` is not in the class either. A failure here is silent: Meta delivers the text intact (dots and spaces both survive autofill_message.content, measured), fly's pattern rejects it, no conversation_started is derived, and the arrival lands in FALLBACK_FORM -- a real survey, so the respondents look like completions. The VIR-19 shape. So: shortcode-only by default, full metadata opt-in per destination, and both validated at config time rather than at ad creation. - FlyWhatsAppDestination, shaped after FlyMessengerDestination minus button_text plus include_metadata_in_ref. `type` is a Literal because its required fields are a strict subset of the Messenger destination's, and without a discriminator pydantic's smart union could resolve a Messenger destination to it. - The shortcode is validated on the destination itself, since even the default token is `form.<shortcode>`. - Full-ref metadata is validated on StudyConf, the first point where the destination and the strata meet -- they are separate confs, POSTed independently, so no per-conf validator can see both. It fails closed: the study creates no ads rather than ads that recruit into the fallback. - creative_metadata and destination_shortcode treat both fly destinations alike, so the frozen ad_attributions blob is identical across channels and a study on location "ad" reads the same keys either way. Creative shape (CTA, link, autofill welcome message) follows what adopt/scripts/ctwa_probe.py measured against live Meta ads. No url_tags. Dashboard UI for the destination is deferred: another agent holds uncommitted work in forms/destinations/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
Documents FlyWhatsAppDestination in documentation/ad-attributions.md (the cross-component view) and adopt/README.md (the app view). The load-bearing part is the finding, not the feature: fly's WhatsApp entry pattern anchors on `form.` and admits only [A-Za-z0-9_-] tokens, so make_ref's output can never match structurally, and roughly half of the production stratum values on record are undeliverable even in the form-first serialisation -- percent-encoding included, since `%` is not in the class either. A ref that fails is not an error anywhere: Meta delivers it intact, fly rejects it, and the respondent lands in FALLBACK_FORM looking like a completion. That is why full-ref mode is opt-in, why validation is at config time, and why it fails closed. Also records what is not built: the dashboard form for the destination, and the adset-level promoted_object.whatsapp_phone_number Meta requires for a WHATSAPP destination_type ad set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
A9. FlyWhatsAppDestination could not produce a working ad: adset_instructions set promoted_object only for AppDestination, and Meta rejects a WHATSAPP destination_type ad set without one. A conf class that looks finished and cannot create an ad is worse than an absent one. - whatsapp_phone_number on FlyWhatsAppDestination, required rather than optional. Meta treats it as optional and falls back to the Page's "primary" number, but many-numbers-to-one-Page is documented and supported, so an org running several would silently recruit into whichever one that happens to be. - Validated at config time: digits only (the promoted-object reference types it as a numeric string while credentials store the display form), within E.164's 7..15. There was no phone normaliser in this repo to reuse; the digit-strip shape is what ctwa_probe.py measured. - destination_type is checked, not overridden. A WhatsApp destination on a MESSENGER adset produces a valid creative and a valid promoted_object and an ad that never reaches WhatsApp. Overriding would change what every existing study sends, so this raises instead. Fixes the latent bug planning/click-to-whatsapp-ads.md explicitly said to fix rather than inherit. promoted_object is an adset field while destinations are per-creative, and the app branch read destinations[0] under a standing `# TODO: assert all destinations are the same` -- so a stratum mixing an app creative with any other kind published half its ads under the wrong promoted object, silently. adset_promoted_object now checks agreement and raises only on genuine ambiguity: strata whose creatives all want none, which is every Messenger and Web study, produce None exactly as before, mixed or not. probe.py held a second copy of that same branch, so it would have built a different adset than production sends for WhatsApp -- and the probe exists to report what production sends. It now calls the shared function. Reconciliation safety, asserted directly rather than argued: promoted_object is not in COMPARED_ADSET, so update_adset neither compares it nor includes it in update params. A live adset without one is not rewritten when we start sending one, and an unchanged study still produces zero instructions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
Records A9 in documentation/ad-attributions.md and adopt/README.md: why whatsapp_phone_number is required on the destination when Meta treats it as optional, why the Page id comes off the creative template, and why destination_type is checked rather than derived. The part worth reading is the reconciliation argument. promoted_object becoming non-None for a whole destination type is the kind of change that can make every existing adset look drifted, and the reason it cannot is that promoted_object is absent from COMPARED_ADSET -- so update_adset neither compares it nor sends it, and it rides only on creates. Also records the destinations[0] bug that adset_promoted_object replaces, and that probe.py no longer carries a second copy of it. Corrects a wrong detail in the adset_promoted_object docstring: the old code took whatever the stratum's first creative wanted, not whatever sorted first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
A4, the last lever. Until now `location: "ad"` worked but ads still shipped vlab's whole stratum vocabulary into fly on every message, so nothing was actually decoupled -- the ad id duplicated the ref rather than replacing it. FlyMessengerDestination gains include_metadata_in_ref, named to match the WhatsApp destination's field because it is one concept; the channels differ only in default. Messenger defaults True, the historical behaviour. Off, the ad emits `form.<initial_shortcode>` on both carriers -- url_tags and the welcome message's quick-reply payload -- because a respondent can arrive by either and two different refs would mean one ad describing two different people. The mode selects a serialisation and nothing else. creative_metadata is untouched and still returns the complete dict regardless of mode, because that same dict is what ref_metadata freezes into ad_attributions.metadata, and for a shortcode-only study that frozen blob is the only attribution it will ever have. Had the mode leaked into creative_metadata, such a study would freeze rows holding nothing but `form`, every location "ad" conf would resolve to nothing, every stratum would count zero, and the optimizer would reallocate on empty data -- silently and unrecoverably, since the blob is never refreshed. There is a test pinning that both modes freeze identical blobs, and another on ad_provenance one layer up. Flipping a live study changes the creative, so update_ad rewrites that study's ads on its next run. That is intended and contained: each study reconciles from its own conf, and the flip is an in-place ad *update* against the same ad id -- not a delete and recreate -- so existing ad_attributions rows stay valid and the study's past respondents remain attributable. Both properties are asserted. Web and App stay on full refs. Neither type has an initial_shortcode, because their URL or deeplink already points at a specific survey, so routing is not a job the ref does for them; making them shortcode-only would mean inventing a conf field for a token neither needs. The equivalent decoupling for a web platform is capturing the ad id from the ad URL, which is separate work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
Unrequested, and the coordinator may want it dropped -- but A4 is what creates this failure, so shipping the lever without the guard felt incomplete. Turning include_metadata_in_ref off only works if the study also reads the ad -> stratum mapping through a `location: "ad"` extraction conf. One without the other leaves the study with no attribution at all: the ref no longer carries the stratum and nothing looks the ad up, so every stratum counts zero and the optimizer reallocates on empty data. Same silent shape as an unmapped ad, reached from the opposite direction. Warns rather than raises, on the same reasoning as the completeness check: a study recruiting uniformly, with no question_targeting, needs no stratum attribution and is entitled to a thin ref. Covers WhatsApp destinations too, whose default is already thin -- a CTWA study that never declares ad-location confs has no attribution and should hear about it. Documents A4 in documentation/ad-attributions.md and adopt/README.md, and marks the plan complete except the WhatsApp destination's dashboard form. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
D1 + D2, plus the creative-name extension. D2. quote() never escapes `.` `-` `_` `~` -- they sit in urllib's _ALWAYS_SAFE and safe='' does not override it (asserted, not assumed). Two of those corrupt a ref: `.` is the separator, so a dotted token silently mis-pairs everything after it, and `~` is outside fly's WhatsApp alphabet so it fails the gate. ref_value encodes exactly those two and deliberately leaves `-` and `_`, which are both separator-safe and inside the alphabet. The creative name was worse: interpolated with no quote() at all, so strictly more exposed than the values. And more damaging -- a dotted *value* shifts the pairs after it and mis-attributes, while a dotted *name* shifts `form` too and misroutes the respondent into a different survey. unicef-immunization-kyrg ran `*.png` creative names for ~9h in Jan 2023 and only timing caught it. Every segment now goes through ref_value: name, keys and values. Purely prophylactic -- the production measurement found zero affected studies, current or historical. No remediation, no migration. Containment is asserted rather than argued: refs for values without `.` or `~` are byte-identical (the recorded production values all come out unchanged), the frozen ad_attributions blob still holds raw values, and the Facebook ad *name* stays the raw creative name -- encoding it would orphan every live ad and mint new ids, the stranding failure A4 guards against. The phase-1 test that pinned the corruption now asserts the closed round trip, still through a _parse_ref that mirrors fly's getMetadata rather than inverting make_ref. D1. fly widened its entry gate to accept percent-encoded octets (fly@feature/ad-id-attribution 37e1e06e -- verified against the source; my mirrored copy had gone stale). Deliverability is now judged on the encoded form, which takes the recorded production values from 5 of 9 to 9 of 9. The only residual is `/`, which quote() keeps literal by default. The shortcode keeps the narrow alphabet on purpose. Values are only ever carried by an ad, but a shortcode is shareable by design -- someone texts `form.<shortcode>` into WhatsApp by hand, and a hand-typed space is a literal space. It has to be typeable, not merely encodable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
Documents D1/D2 in documentation/ad-attributions.md and adopt/README.md. The part worth reading is the severity asymmetry, which is not visible in the code: a dotted metadata *value* shifts the pairs after it and mis-attributes the respondent, while a dotted *creative name* shifts `form` too and misroutes them into a different survey. The name was also the least protected of the three ref contributors -- interpolated with no quote() at all. Records that the fix is prophylactic (zero affected studies across every conf revision; no corruption signature over 17.8M response rows), what deliberately does not change (the Facebook ad name, the frozen blob), the re-measured WhatsApp deliverability of 5-of-9 to 9-of-9, why the shortcode still keeps the narrow alphabet, and the deploy ordering: Messenger is safe either way, only the WhatsApp gate needs fly's widened pattern deployed first. Also corrects two now-stale claims elsewhere in the docs: the A8 statement that percent-encoding rescues nothing, and the top-level claim that make_ref and create_creative's ref emission are untouched -- A4 and D2 both changed them, each contained per-study. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BqFepoVJPvP2CsmKHVGnCf
…stination Two changes, the first a prerequisite for the second. destination_type was ONE string on the recruitment conf, consumed by every ad set of every arm. Derive it instead from each stratum's actual (creative, destination) pairs, beside the promoted_object agreement check that already existed: destination_type_for(destination) in study_conf, and adset_destination_type(pairs, recruitment_default) in marketing. That closes two silent misroutes -- MESSAGING_MESSENGER_WHATSAPP with only Messenger destinations, and the same with only WhatsApp destinations -- both of which built happily and stranded one arm's respondents in FALLBACK_FORM, where they look like completions rather than errors. It also makes channel expressible per arm, so a DestinationRecruitmentExperiment can finally run a real Messenger-vs-WhatsApp experiment. Measured against production study_confs (2026-08-17) before landing: 110 studies derive exactly what they already store, 5 legacy WEB/WEBSITE studies have destinations that imply nothing and keep their stored value verbatim, and the only 2 studies where the derivation differs both ended in April 2024. destination_type is absent from COMPARED_ADSET, so it rides only on ad-set creates and cannot rewrite anything live -- now asserted directly. FlyMultiDestination is a third destination type (type: "multi"), not a platforms list on a merged class. Its creative emits all three carriers: the Messenger token on url_tags and in a quick-reply payload, and the WhatsApp token as an autofill_message, with the last two sharing one page_welcome_message blob byte-identical to ctwa_probe.py's welcome_combined. optimization_goal is validated to be CONVERSATIONS at config time. The type is GATED OFF behind ADOPT_ENABLE_MULTI_DESTINATION. Its Messenger arm is measured against real Meta delivery; its WhatsApp arm has never been observed, and if the symmetry we infer is wrong every WhatsApp respondent lands silently in the fallback survey. documentation/multi-destination-ads.md carries the measurement procedure and the empty result log that clears it; ctwa_probe.py gains --multi-fallback whatsapp to make that arm reachable by preview without a second device or a live activation. Invariants asserted directly rather than inferred: the frozen ad_attributions blob is identical across all three fly destination types for identical strata, existing Messenger/Web/App creatives and ad-set instructions are byte-identical, ad names are unchanged, and both tokens round-trip through their own parsers to the same metadata dict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without these, both destination types are reachable only by hand-POSTing a
study conf, which makes them unusable by the researchers they exist for.
WhatsApp.tsx is the one that pays off immediately: FlyWhatsAppDestination is
proven end to end against real Meta delivery and is not gated. Multi.tsx is
built too, and its option is labelled "not yet enabled" rather than hidden --
hiding it would make the capability undiscoverable and leave no explanation,
whereas leaving it selectable means the save fails with the gate's own
message, which names the measurement and the variable.
The metadata text box's parse rule is extracted into additionalMetadata.ts
and unit tested rather than copied a third time. Its real job is the
half-typed case: `{"wave"` is not parseable but is a normal thing to have
mid-keystroke, and treating it as a cleared field would wipe a saved value.
Messenger.tsx keeps its inline copy for now -- that file has uncommitted
changes on another branch, so it is left alone rather than churned.
include_metadata_in_ref is deliberately not a form field. It defaults off,
its token is respondent-visible and respondent-editable, and turning it on
can make a study's refs unparseable by fly.
The two repos share no schema -- the form builds a plain object and adopt
parses it -- so test_study_conf.py gains a "dashboard contract" section
asserting that the exact shapes Destination.tsx's emptyStates produce parse
into the right classes, and that the type literals still match what the
pydantic union discriminates on. Those literals are load-bearing: both
classes need a discriminator because their required fields overlap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records what is left between the gated code and a study a researcher can run,
and two things found while writing it that were not in the original brief.
fly never stamps ad_id: zero references in replybot/lib, dean, message-worker,
botserver or devops/migrations, while vlab's connector documents AdID as a
first-class column on fly's responses view. adFields("") returns empty, so the
miss is not even counted as unmapped -- it is invisible. Since
include_metadata_in_ref defaults off on both WhatsApp and multi, a
default-configured study of either type routes correctly and attributes nobody.
That corrects an earlier claim that single-destination WhatsApp was usable end
to end: routing is measured, attribution is not.
Nothing alerts on FALLBACK_FORM arrivals, which is the terminus of every failure
mode in this project and the reason VIR-19 ran four days undetected. That lands
as Phase 0, ahead of everything, and is worth doing whether or not multi ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The producer half of the format fly already decodes
(fly@feature/recruitment-arrival-health 341be39a). fly could decode an
encoded ref; nothing produced one.
Why the format exists. A thin ref routes but carries no attribution, so
attribution falls to ad_id -- and ad_id reaches only ~31% of Messenger ad
entrants, because Meta simply does not send the referral webhook for the
rest. That is not a bug vlab can fix. The encoded ref carries the join key
on a carrier vlab authors instead (the quick-reply payload, the WhatsApp
autofill), so it reaches everyone.
r.<base64url(v1 | len(shortcode) | shortcode | token)>
ref_encoding.py owns the wire format and says why each choice is what it
is -- length-prefixed rather than delimited because a delimiter is a
character a shortcode might one day contain, and that failure would be a
silent mis-route.
Three ref modes, not two. `include_metadata_in_ref` is a bool and cannot
express a third state, so `ref_mode` supersedes it on the three fly
destinations via a shared mixin. The bool is still accepted and still
resolves to exactly today's behaviour per channel, so no stored conf
changes and no existing study is migrated; setting both to contradictory
values raises rather than picking a winner.
Two properties the design turns on, both tested:
- the token is deterministic. Reconciliation compares creatives and the
ref is part of one, so a random token would rewrite every ad in the
study on every run, forever, while spending money.
- one ad, one token, every carrier. A multi destination's two arms are
two grammars over the same facts; different tokens would give one ad
two attribution identities and let Meta's arm choice decide which one
a respondent got.
ad_provenance now carries `ref_token` and refuses to generate instructions
for a campaign whose ads collide on one -- the last moment before the ads
exist and are spending. A collision attributes respondents to the wrong
row, which is a wrong answer rather than a missing one, and nothing
downstream can see it.
Golden vectors in test_ref_encoding.py were verified against fly's shipped
decoder, byte for byte, including a multi-byte shortcode. Regenerating
them to make a future change pass would break every live encoded ad; bump
ENCODED_REF_VERSION instead.
687 passing (+40), mypy clean on the changed files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The encoder from 9bd5ac8 mints a token and ships it; nothing recorded what it meant. ad_attributions is keyed (network, ad_id), so until now every lookup went through the one identifier Meta may not send. `ref_token` on ad_attributions, both schema paths as the table's own note requires. NULL is the normal case and it means something specific -- this ad's ref carries no token, because its destination is not in ref_mode "encoded". Not a gap in the record, hence no NOT NULL and no default. The two keys are never a fallback for one another. Which one attributes a respondent is fixed when the ad is built. Choosing at read time by whichever lookup happened to hit would make a genuine miss indistinguishable from a mechanism switch. No index, deliberately -- and the first draft had one until CockroachDB refused to build it in the same transaction as the ALTER, which was worth the interruption: every read of this table is already per study (get_ad_attributions loads the whole set and swoosh joins in memory), so nothing ever filters on the token and the index would only have cost writes. Not UNIQUE either: uniqueness is asserted at instruction-generation time where the failure is still a fixable config error, whereas a constraint would abort an INSERT after the ad exists and is spending, and this table is append-only precisely so a write cannot fail a run. The CSV export gains the column, because for an encoded study that column *is* the join key -- fly exposes it as the response metadata key `vt`. Without it the export's one-sentence promise would be false for exactly the studies the format was built for. Written with .get rather than [], so a provenance dict predating the column still writes: the ad exists on Facebook either way, and a missing row cannot be recovered, so refusing the write is strictly worse than an unattributable ad. 691 passing (+4). The token is covered by the same append-only freeze as the rest of the row. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… field
The read side of the encoded ref. An ExtractionConf gains one new field:
mapping: "raw" the value read IS the answer (default, and what
the empty string means, so every conf ever
written keeps meaning what it meant)
mapping: "ad_table_lookup" the value read is an opaque token; the answer is
a stratum variable off the frozen
ad_attributions row that token identifies
`location` is unchanged and stays `metadata` | `variable`. The token lives in
metadata, so reading it is an ordinary metadata read -- which is why "ad" was
never really a location, and is now removed. A conf still declaring it gets an
error naming its replacement rather than silently resolving to nothing.
ref_token supersedes ad_id as the join. They are the same shape (opaque ad
identifier -> frozen row -> stratum metadata) and differ only in the carrier:
ad_id rides Meta's referral webhook, which Meta sends for only ~31% of
Messenger ad entrants, so the other 69% could never be joined. The token rides
the ref itself, a carrier vlab authors, so it reaches everyone. ad_id stays
captured on the event and on the row -- fly's recruitment-health alerting gates
on it -- but nothing joins on it and there is no ByAdID index.
No runtime mechanism selection, deliberately: the mechanism is declared in the
conf, fixed at config time. A token that matches no row is `unmapped`, never a
quiet retry against ad_id -- a fallback would make a genuine miss
indistinguishable from a study part-way through switching mechanisms.
The token's metadata key is conf-declared too (`key`), never hardcoded: fly
stamps `vt` by convention and the conf says so. For a lookup, `name` does
double duty -- the output variable name AND the key into the frozen row.
Also unquotes the token before joining: metadata values are JSON, so it
arrives quoted while ref_token comes out of a text column bare. Joining the raw
bytes would miss every time, on a value that looks right in every log line.
The three-way split is unchanged in shape, rebased on token presence rather
than ad_id: attributed / organic (expected, warns) / unmapped (always a bug,
errors), classified once per event.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Python's half of the contract swoosh now reads.
ExtractionConf gains `mapping` ("raw" | "ad_table_lookup", defaulting to raw so
every conf ever written keeps meaning what it meant), and rejects the removed
`location: "ad"` with an error naming its replacement.
That rejection fails closed, deliberately, following the precedent of
check_whatsapp_refs_are_deliverable. swoosh no longer resolves an "ad" conf at
all, so a study still declaring one already produces no variable, matches no
stratum, counts zero and has its budget reallocated on empty data -- silently.
Refusing to load the conf stops that study creating ads until someone fixes it,
which is strictly better than letting it recruit people it cannot attribute.
Two predicates, both warning rather than raising:
- thins_its_ref_without_reading_the_mapping, reworked off `location: "ad"` and
onto the mapping. Unchanged in what it is about: a study that stops shipping
its stratum vocabulary while nothing looks the token up has no attribution.
- disagreeing_token_keys (new): one respondent has one token in one place, so a
source's lookup confs must agree on which metadata key it arrives under.
Confs on another key attribute nobody, and silently -- a token that is not
there looks exactly like an organic arrival. swoosh takes the first key it
finds; this is where anyone gets told a guess was needed.
Both warn because a raise would stop ad reconciliation outright, which is a
heavier consequence than the miscount being warned about, and because swoosh
carries on regardless -- a raise here would make its tolerance unreachable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ocation
The "Ad (which ad recruited them)" option moves off `location` and onto a new
`mapping` select, shown only for a metadata read. `location: "ad"` is gone from
the form entirely -- it joined on ad_id, which is superseded.
The two prompts change with it, because both fields are contextual to the
mapping and getting them backwards is the easy mistake:
key for a lookup this addresses the TOKEN ("usually: vt"), not the stratum
variable, which is what it used to mean under location "ad"
name for a lookup this is the stratum variable AND the output name, since it
does double duty
Switching a conf away from metadata resets the mapping to raw. Without it a
conf could end up `variable` + `ad_table_lookup`, whose `key` swoosh would read
as a declaration of where the token lives -- classifying every respondent in
the study against the wrong metadata key, and reporting them organic, which
does not alarm.
The fly-only guard reworks rather than disappears: the locations are now the
same two on both forms, and what stays per-source is the mapping. Qualtrics and
Typeform export an empty mappingOptions -- exported precisely so the guard test
asserts the absence rather than the module merely not mentioning it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A conf declaring `mapping: "ad_table_lookup"` on `location: "variable"` was incoherent but tolerated, and the tolerance was dangerous. swoosh reads a lookup conf's `key` as the declaration of WHERE THE TOKEN LIVES and picks the first such conf to classify a whole source -- so one stray conf would have every respondent in the study checked against the wrong metadata key. Finding no token there is indistinguishable from an organic arrival, so it would not alarm; it would just miscount. Rejected at config time (pydantic), and refused loudly by getRetrieveFunc rather than degraded to a silent raw variable read. `is_ad_table_lookup` / isAdTableLookup now check the location too, so the property everything branches on is true by construction rather than by trusting the validator ran. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rewrites the read-side half of the attribution docs around the mapping concept. documentation/ad-attributions.md: the join-key comparison (ad_id reaches ~31% of Messenger ad entrants, the token ~100%), the `mapping` field and what `key` and `name` each mean under it, the conf-declared token key, fly's ownership of `vt`, why there is still no fallback, the JSON-unquoting the join depends on, the three-way split rebased on token presence, the reworked dashboard form, and the three config-time checks. The write side is untouched -- it did not change. inference/README.md, adopt/README.md, dashboard/README.md: the same rework at per-app altitude. multi-destination-ads.md: attribution holds under either key; the encoded ref does not rescue the case where Meta replaces the compose text. planning/multi-destination-rollout.md: Blocker 2 is resolved, and by a different mechanism than the plan assumed. fly does stamp ad_id now, but measurement put its Messenger reach at ~31%, so it could never have closed the blocker alone. The reasoning is kept and the conclusion marked stale rather than deleted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`location: "ad"` never shipped in any study, so there was nothing to reject. Removes the pydantic validator, the explicit case in getRetrieveFunc, and both tests. It is now just an unknown location and gets the same error a typo does. Docs updated to match: the config-time checks are two, not three. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records what was built against planning/encoded-ref-attribution-plan.md, and supersedes it as the entry point. Carries forward the invariants, and adds the errata (§5): the `location: "ad"` validation was built then removed once it was confirmed never live, and the `variable` + `ad_table_lookup` hole was not in the original design. The release engineering is the part worth reading twice. Two problems found while checking it: the pinned vlab-migrations image is two migrations stale and may mean prod has no ad_attributions table at all, and toixo-staging pins no migrations image, so staging cannot run the hook today. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FlyMultiDestination no longer refuses to construct. ADOPT_ENABLE_MULTI_DESTINATION, multi_destination_enabled(), the multi_destination_must_be_enabled validator and the three tests pinning them are gone, along with the now-unused `os` import. The gate encoded a real, still-true asymmetry: the Messenger arm is measured against live Meta delivery (ad 120254903561240150, 2026-08-17), the WhatsApp arm is inferred from it by symmetry and has never been observed. That knowledge is not deleted with the gate -- it moves to where someone configuring a multi destination will meet it: FlyMultiDestination's docstring, adopt/README.md, documentation/multi-destination-ads.md 4, and the dashboard form copy. What changes is who carries the risk. A shut gate made the failure impossible and the feature unusable; documentation makes it possible and visible. If the symmetry inference is wrong, a multi ad's WhatsApp arrivals land on FALLBACK_FORM and look like completions, so the 4.5 result log stays the thing that settles it and the first multi study's arrivals want watching. adopt 694 passed, 1 skipped. dashboard 142 passed, tsc clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HooztAJBGuViFhXTbFcxBj
✅ Deploy Preview for vlab-dashboard ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
nandanrao
added a commit
that referenced
this pull request
Aug 20, 2026
The multi-destination stack (#244). adopt gains FlyWhatsAppDestination and FlyMultiDestination, the encoded-ref encoder and the ad_attributions writes; swoosh gains the ref_token join behind the `mapping` conf field. Both images verified pullable by scripts/release.sh before this bump. Rolls one live pod: vlab-conf-dashboard is a Deployment on the same *vadopt anchor, so the study-conf API restarts. That is the point of it -- it is what parses the new destination types. replicaCount is 1, so there is a brief gap. The four adopt cronjobs and swoosh pick the new image up on their next scheduled run. The migration hook re-runs and is a no-op: prod is already at 20260818000000. Claude-Session: https://claude.ai/code/session_01HooztAJBGuViFhXTbFcxBj Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
29 commits, four groups. The migration half already shipped in #242/#243 —
ad_attributionsexists in prod withref_token, empty and unread.156fc855→b24471d1ad_attributionswrites, CSV export, the Go joinfce38e68→63b248eaFlyWhatsAppDestination, adsetpromoted_object, the CTWA ref grammar,FlyMultiDestination9bd5ac8e→f1c1f78cref_tokenpersistence, swoosh'smappingjoin, the dashboard form8402d55fGreen
go test ./... -p 1tsc --noEmitcleanMerges clean against
main.What goes live on merge
Both
type: "whatsapp"andtype: "multi"destinations become configurable.The
ADOPT_ENABLE_MULTI_DESTINATIONgate is removed in8402d55f— deliberately,so the feature is usable. Two things are then possible that were not before, and
both need someone to actively configure a destination:
live delivery (ad
120254903561240150, 2026-08-17); the WhatsApp arm isinferred from it by symmetry and has never been observed. If that inference is
wrong, those arrivals land on
FALLBACK_FORMand look like completions.documentation/multi-destination-ads.md§4.5's result log is still empty andis what settles it.
include_metadata_in_refdefaults off forboth types, so a default-configured study routes correctly and counts zero for
every stratum.
thins_its_ref_without_reading_the_mappingwarns; it does notrefuse.
The knowledge the gate carried now lives in
FlyMultiDestination's docstring,adopt/README.md,documentation/multi-destination-ads.md, the rollout plan andthe dashboard form copy.
Deploy
adopt v0.1.77 → v0.1.78,swoosh v0.1.9 → v0.1.10, thenhelm upgrade. Thisrolls 5 adopt cronjobs and swoosh, unlike the schema-only step. The React
dashboard ships via Netlify on this merge, independent of Helm.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HooztAJBGuViFhXTbFcxBj