From e442c6d25dcd93973ae17bf40047befaaa12c373 Mon Sep 17 00:00:00 2001 From: cbwinslow Date: Sat, 10 Oct 2026 03:14:12 +0000 Subject: [PATCH] docs: adopt research-center direction (vision brief, data audit, epics 12-19) Operator approved the 2026-10-10 vision brief and sprint change proposal: report cards and publishable findings for any office-holder. Amends the AGENTS.md/v1-scope rules on trades, news-as-evidence and scorecards (no opaque score, no verdict labels), replaces Epic 7 with Epics 14/16/17/18, and records the keep-all-PUMS decision. No code or data changed. --- AGENTS.md | 18 ++-- .../brief-OpenDiscourse-2026-10-10/.memlog.md | 26 ++++++ .../addendum.md | 71 ++++++++++++++ .../brief-OpenDiscourse-2026-10-10/brief.md | 77 +++++++++++++++ _bmad-output/planning-artifacts/epics.md | 32 ++++++- .../prds/prd-opendiscourse-2026-09-14/prd.md | 18 +++- .../sprint-change-proposal-2026-10-10.md | 93 +++++++++++++++++++ .../specs/spec-opendiscourse/v1-scope.md | 8 +- docs/PROJECT-STATE.md | 10 ++ docs/data-audit-2026-10-10.md | 81 ++++++++++++++++ 10 files changed, 418 insertions(+), 16 deletions(-) create mode 100644 _bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/.memlog.md create mode 100644 _bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/addendum.md create mode 100644 _bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/brief.md create mode 100644 _bmad-output/planning-artifacts/sprint-change-proposal-2026-10-10.md create mode 100644 docs/data-audit-2026-10-10.md diff --git a/AGENTS.md b/AGENTS.md index 45b3484..eaee473 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,12 +30,18 @@ Start from `_bmad-output/specs/spec-opendiscourse/SPEC.md` and - Do not name-match people. Federal person *joins* need BioGuide (Epic 3). FEC/disclosure/elections politician joins stay gated by `person_join` in `inventory/sources.yaml`; call `identitygate.require_person_join` before promoting - person-keyed rows. Do not start - Epic 7 (FEC-native/crime staging, elections) in v1. -- Do not invent news, stocks, or corruption scores as schema domains. - Politician scorecards are allowed later only as derived `mart` outputs over - evidence-backed `core`/`fact` rows (SPEC non-goals); never an opaque single - "corruption score". + person-keyed rows. The money, outcomes, disclosures and statements epics + (14, 16, 17, 18; they replace the old Epic 7) open only after Epic 10 Story 10.6 + (`slice_proven`), and each source first needs a written contract (coverage, + licence, size) and a capacity preview. Operator decision 2026-10-10. +- No market-price data and no news product. Members' STOCK Act trades and other + required disclosures are an allowed source domain. GDELT/news is allowed only as + evidence that a statement was reported (link plus short quote, lower trust). + Report cards are derived `mart` outputs over evidence-backed `core`/`fact` rows, + built from versioned metrics that carry a reliability tier (established, + estimated, exploratory). Never an opaque single score and never verdict labels + such as "bribed", "insider trading" or "lie": show timelines and both items. + Vision: `_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/`. - Never commit secrets or `.env`. Capacity gate fails closed on unknown size. - `dlt` writes `stage` only, never `core`/`fact`. diff --git a/_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/.memlog.md b/_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/.memlog.md new file mode 100644 index 0000000..b3c45bf --- /dev/null +++ b/_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/.memlog.md @@ -0,0 +1,26 @@ +--- +topic: OpenDiscourse vision: free local political research center +updated: 2026-10-10T03:01 +--- + +- (decision) Must-have core tracks: legislation+votes (1), bill/law text (2), outcomes by place (5), report cards/summary sheets (8) +- (decision) Very important, harder inference: donations/lobbying (3), federal awards/regulation beneficiaries (4), politician trades from STOCK Act disclosures (6) +- (decision) Track 7 statements vs votes is wanted and is the key differentiator; hard +- (decision) Outcomes scope: all of them (income, crime, standard of living, housing, immigration, taxes, health...) at district, county, state +- (decision) Audience: researchers plus ordinary readers; free and local; OpenStates/think-tank/NGO equivalent +- (assumption) Do not start over: identity, provenance, legislation, Epic 10 district slice are the foundation +- (assumption) Language rules: timelines and evidence not legal conclusions; consistency check not 'lie'; many explained metrics not one corruption score; misandry needs measurable definition +- (event) Operator picked option A: write vision brief before data audit; AGENTS.md rule changes only after operator sees brief +- (decision) First user is the operator as researcher; design so an organization (think tank) could run it; output is published findings (blog/Substack), not a voter-facing app +- (decision) Point-in-time report cards wanted: store every fact with an as-of (knowledge) date so a card can be regenerated for any past date; design now, build later +- (decision) Report card contents: party loyalty with dissent reasons where available, attendance, ideology position from voting history, donor breakdown and concentration, tenure and prior offices, district change since election, consistency over tenure (flip-flops), ethics actions, committee memberships, donor links to other candidates +- (decision) Statements sources: Congressional Record floor speeches first; GDELT news-quoted statements as lower-trust reported statements; social media later and only if legally and cheaply available +- (decision) Credibility rule: honest metrics; every metric labelled established/estimated/exploratory with method and inputs; deservedly right, not pretending +- (decision) Do not hand-label members as fascist/communist/etc.; use empirical measures (Voteview DW-NOMINATE already loaded) plus issue-level measures +- (direction) Operator stated strong prior views and wants report cards to 'prove' them; brief requires the tool to be able to contradict operator priors: hypotheses and metrics defined before results are viewed; null results published +- (direction) Operator limits: do not dox anyone, do not break laws +- (assumption) Donor 'country of origin' is mostly not in FEC data (foreign nationals barred); use employer, occupation, state, PAC type, concentration instead; foreign influence via FARA/LDA is a separate source +- (assumption) Illegal immigration per district is not directly measured; use CBP encounters, modelled unauthorized-population estimates (state level), foreign-born share (ACS), and sanctuary-policy lists with stated definitions +- (event) Operator approved brief; operator clarified they are nonpartisan, earlier rhetoric not a policy input; objectivity principle accepted; addendum with sources/metrics/methods written +- (decision) Add: LDA filings link bills to lobbyists; CES for district opinion; CAP topic codes for flip-flops; district change is context not credit; donor grouping is a separate labelled estimated layer +- (decision) Scope widened: any office-holder (senators, president, governors, state legislators via OpenStates #102), one person-office-term model; local officials need a source; courts out diff --git a/_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/addendum.md b/_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/addendum.md new file mode 100644 index 0000000..a7ef6ca --- /dev/null +++ b/_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/addendum.md @@ -0,0 +1,71 @@ +--- +title: OpenDiscourse vision brief addendum (metric, source and method catalog) +created: 2026-10-10 +status: draft +--- + +Reliability tiers: **E** established (published method, official data), **S** estimated +(modelled or probabilistic joins), **X** exploratory (needs validation before publishing). + +## 1. Additional sources (beyond what is loaded) + +| Purpose | Source | Notes | +|---|---|---| +| Floor speeches, dissent explanations | Congressional Record (GovInfo CREC) | Official, structured by speaker; start here for track 7 | +| Hearings, witnesses, markups | GovInfo CHRG, committee sites | Who testified, which industries | +| Ethics actions | House Ethics Committee, Office of Congressional Conduct, Senate Ethics statements | Small, public, official | +| Trades and assets | House Clerk periodic transaction reports, Senate disclosures | Values are ranges, not exact | +| Lobbying | Lobbying Disclosure Act database | **Filings name specific bills**: direct link bill -> lobbyist -> client | +| Foreign agents | DOJ FARA eFile | The honest route to "foreign influence", instead of donor country | +| Federal money | USAspending, SAM.gov, grants.gov | Awards by place and recipient | +| Rules and beneficiaries | Federal Register, regulations.gov (comments) | Who commented, who gained | +| Waste and fraud | Payment-accuracy (improper payments) and Inspector General reports | The only official measures of waste | +| Constituent opinion by district | Cooperative Election Study (open, Harvard Dataverse) | Lets us compare votes with what the district says it wants | +| Elections | MIT Election Data + Science Lab results | Needed for close-election comparisons and tenure | +| Alternative ideology | Bonica DIME / CFscores (donor-based), Volden-Wiseman legislative effectiveness | Cross-check on DW-NOMINATE | +| Topic taxonomy | Comparative Agendas Project codes | Standard way to say which bills are the same issue (needed for flip-flops) | + +Outcomes by place (track 5), each by county/district where published: +income and poverty (Census SAIPE, ACS), jobs and wages (BLS LAUS, QCEW; BEA regional), taxes (IRS SOI by +ZIP/county), housing cost (HUD fair-market rents, FHFA house price index, ACS rent burden), +homelessness (HUD count), crime (FBI NIBRS; coverage gaps must be flagged, not zero-filled), +health (CDC PLACES/WONDER, CMS), education (NCES), environment (EPA), mobility (Opportunity Insights), +immigration (CBP encounters, DHS yearbook, immigration court backlog, ACS foreign-born share), +disasters (FEMA). Approval ratings: pollster licences must be checked first. + +## 2. Additional metrics for the card +- Bipartisanship (share of cosponsors and votes across parties; published method). +- Legislative effectiveness (bills advanced per stage versus expected for seniority). +- Vote alignment with district opinion (CES) and with the president. +- Absences split by reason where recorded, and by whether they were decisive. +- Donor concentration: top-donor share, small-dollar share, in-state share, industry mix (E/S). +- Same donor, many candidates (S, because individual donors have no stable id). +- Trades timed against committee activity and votes (timeline, not verdict). +- Money received by the district from federal awards (E). +- Consistency over time at topic level (X until a human-checked sample passes). +- Statement-to-vote consistency (X; published only with a measured accuracy rate). + +## 3. Methods and cautions +1. **District change is context, not credit.** A member's effect on local crime or income is small and + confounded. Compare the district to peer districts and the national trend, label it context, and + use difference-in-differences or close-election comparisons only for specific, stated questions. +2. **Donor matching is probabilistic.** FEC individual donors have no stable id. The rule "never match by + name" protects politician joins; donor grouping must be a separate, labelled (S) layer with its + error rate measured. +3. **Language models must be measured before they are trusted.** Hand-label a sample (around 200), + measure agreement and error, and publish the rate. Keep the extracted claim, its source passage and + the model/version. +4. **Versioned methodology.** Every metric has a definition, version and changelog; a card records + which versions produced it. Hypotheses are committed before results are viewed. +5. **Bitemporal facts.** Store when a fact was true and when we learned it, so cards can be rebuilt + for any date. Redistricting makes district-based tenure comparisons need the Epic 10 boundary years. +6. **Corrections policy and legal review** before publishing named-person grades. +7. **Scope: every office-holder, one model.** Cards are keyed by person, office and term, with the + outcome geography taken from the office (district, state or nation): U.S. representatives, senators, + president and vice president, governors and other statewide officers, and state legislators. Metrics + are defined per office type (legislators vote; governors sign, veto and appoint). The first card is a + U.S. House member because that data is ready; state legislators come with the approved OpenStates + promotion (issue #102), not after everything else. Local officials (mayors, councils) are not in the + OpenStates snapshot and need a separate source. Courts are out of scope for now. + Cross-office history (state legislator who became a member of Congress) joins only on reviewed + identifiers, never on name. diff --git a/_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/brief.md b/_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/brief.md new file mode 100644 index 0000000..bf8230b --- /dev/null +++ b/_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/brief.md @@ -0,0 +1,77 @@ +--- +title: OpenDiscourse vision brief +status: approved-by-operator +created: 2026-10-10 +updated: 2026-10-10 +supersedes_in_part: ../brief-opendiscourse-2026-09-14/brief.md +--- + +# OpenDiscourse: a free, local political research center + +## What it is +A research workbench that does for one researcher, or a small organization, what +OpenStates and think tanks do: join official records on **people, votes, bill text, +money, places and time**, keep the evidence behind every number, and produce +**findings and politician report cards that can be published** (blog, Substack). +The reader is the researcher. The audience is whoever the researcher publishes to. +This replaces the September line "not a consumer app"; it is still not a voter-facing app. + +## Why +Think-tank-grade analysis is paywalled or one-sided. The pieces (votes, text, donors, +district statistics) are public, but they sit in separate systems with different +identifiers and dates. Linking them correctly is the product. + +## Tracks and priority +- **Core (must):** 1 legislation and votes; 2 bill and law text; 5 outcomes by place + (income, housing, jobs, crime, taxes, health, population, immigration-related measures); + 8 report cards and summary sheets. +- **Important, weaker inference:** 3 donations and lobbying; 4 federal awards and + regulation beneficiaries; 6 members' trades from required disclosures. +- **Hard, wanted:** 7 what members say versus how they vote. Sources, in order: floor + speeches in the Congressional Record; news-reported quotes via GDELT (marked as + reported, lower trust); social media only if legally and cheaply obtainable. + +## Who gets a card +Any office-holder in the data: U.S. representatives and senators, president and vice president, +governors and other statewide officers, and state legislators (via OpenStates). One card model, with +metrics chosen by office type. The first published card is a U.S. House member. + +## Report card (target content) +Party loyalty (with the member's stated reason for dissent where one exists); attendance; +position on a voting-derived left-right scale and on issue-specific scales; donor +breakdown (size mix, top-donor concentration, industry, state, other candidates the same +donors back); tenure and prior offices; committees; ethics actions; district change since +first elected (income, jobs, crime, housing, population, spending received); consistency of +positions over time with flagged reversals; each with its inputs one click away. + +## Principles +1. **Evidence first.** Every number traces to an official file, checksum and run. +2. **Honest grading.** Each metric is labelled *established*, *estimated* or *exploratory*, + with method and known limits shown. Uncertain metrics may be published only as labelled. +3. **Findings can go against the researcher.** Hypotheses and metric definitions are written + before results are viewed; null and contrary results are published too. This is what makes + a "deservedly right" claim credible. +4. **Timelines, not verdicts.** The tool shows "trade, then committee vote" or "donation, + then bill"; it does not label "bribed", "insider trading" or "lie". It labels + "statement inconsistent with recorded vote" with both items shown. +5. **Many explained metrics, never one opaque corruption score.** +6. **Time travel.** Facts carry an as-of date so a card can be regenerated for any past date. +7. **People joins by identifier only** (BioGuide), never by name. No doxxing: no home + addresses, family finances or non-public data. Follow the law and each source's terms. +8. **Build the foundations now, decide the analysis later.** Linkages and workflows first; + language-model extraction is added once its accuracy has been measured. + +## Non-goals (for now) +Market-price data; news as an editorial product; automatic "truth" verdicts; voter-facing app; +claims of intent about a person; local-government officials and courts (until a source is chosen); any ingest of data that cannot be tied to a source. + +## Success signals +- A researcher can regenerate any member's card as of any past date, and each figure drills to its sources. +- For the 119th House: cards exist with at least attendance, party loyalty, vote-based ideology, + donor concentration and district change, each labelled by reliability. +- A published finding can be re-derived by another person from the repository alone. + +## Open +How to measure dissent reasons and topic-level flip-flops reliably; which approval-rating and +crime sources are usable and licensed; a measurable definition for any "bias" research; +legal review before publishing named-person grades. diff --git a/_bmad-output/planning-artifacts/epics.md b/_bmad-output/planning-artifacts/epics.md index cf0752f..c724534 100644 --- a/_bmad-output/planning-artifacts/epics.md +++ b/_bmad-output/planning-artifacts/epics.md @@ -253,6 +253,8 @@ member attribution remains behind `person_join`. ## Epic 7 — v1.1 money, elections, crime +**Superseded 2026-10-10** by Epics 14, 16, 17 and 18 below (sprint-change-proposal-2026-10-10). + Do not start in v1. Politician *joins* still need Epic 3 / CAP-4. FEC-native and crime-native staging are not blocked on BioGuide; open this epic only when v1 spine + Epic 8 are in place. Existing `stage.fec_row` (~102M rows) @@ -332,11 +334,35 @@ manifest; passes the 9.1 harness. Spec `spec-9-5-billstatus-connector.md`. Built (`research-db sync-billstatus`, `ingestion/billstatus.py`, `providers/govinfo.py`, `providers/paced.py`). Does not remove the old fixed-path loaders; that is the next story. +## Epics 12-19 — research center (approved 2026-10-10; each needs its own spec before work starts) + +Source: `sprint-change-proposal-2026-10-10.md`, brief and addendum in `briefs/brief-OpenDiscourse-2026-10-10/`. +Stories are not written yet. A row here is not authorization to ingest. + +- **Epic 12 Time-aware facts and method registry:** as-of (knowledge-date) facts; versioned metric + definitions with reliability tiers; hypothesis log; text-model evaluation harness. (AD-12, AD-13, AD-14) +- **Epic 13 Politician profile and vote metrics, any office:** person-office-term model for + representatives, senators, president, governors, state legislators; depends on OpenStates promotion #102 + for state offices. +- **Epic 14 Outcomes by place and opinion:** district ACS (Story 10.4), ACS PUMS all years (1- and 5-year) typed + with weights and replicate weights, labour, tax, housing, crime, health, immigration measures, + Cooperative Election Study, election results. +- **Epic 15 Text analytics:** policy topic taxonomy, chunks and embeddings (ADR first), text reuse, + accuracy-measured claim extraction. +- **Epic 16 Money in and out:** FEC typed cycles and identity bridge (#103, #104), donor grouping (estimated), + Lobbying Disclosure Act, FARA, USAspending, rules and comments. +- **Epic 17 Disclosures and ethics:** House and Senate disclosures, ethics actions, timeline views. +- **Epic 18 Statements:** Congressional Record speeches, GDELT reported quotes, consistency check; social media + only if legally and cheaply available. +- **Epic 19 Report cards and publishing:** card generator, as-of regeneration, methodology and corrections + pages, findings export (AD-15), legal review gate before named-person grades. + +Order: Epic 10 to `slice_proven`, then 12, 13, 19 (first card, U.S. House), then 16, 14, 17, 18, 15. + ## Later (not started; do not begin without a spec) -- **Scorecards** (CAP-9, reserved; needs its own spec before it is added to - SPEC.md): derived `mart` outputs over - evidence-backed rows; transparent indicators; no opaque corruption score. +- **Scorecards** (CAP-9): now Epics 12, 13 and 19 (2026-10-10); needs its own spec before it is + added to SPEC.md; derived `mart` outputs; transparent, tiered indicators; no opaque score. - **Text/NLP/vectors:** keep bill text as immutable artifacts and `core.document` now; embeddings, summaries, and kNN only after chunks exist (ADR first). - Crime data (Epic 7), FRED depth. diff --git a/_bmad-output/planning-artifacts/prds/prd-opendiscourse-2026-09-14/prd.md b/_bmad-output/planning-artifacts/prds/prd-opendiscourse-2026-09-14/prd.md index 6768814..078811e 100644 --- a/_bmad-output/planning-artifacts/prds/prd-opendiscourse-2026-09-14/prd.md +++ b/_bmad-output/planning-artifacts/prds/prd-opendiscourse-2026-09-14/prd.md @@ -33,11 +33,15 @@ The product is the **standardizing surface**, not another one-off downloader. - Join a member to votes, districts, and (v1.1) donations/disclosures via BioGuide, never name-matching. - Add a new government source without forking `cli.py` / `plans.py`. +- (2026-10-10) Publish findings and politician report cards for any office-holder + (representatives, senators, president, governors, state legislators), regenerable + for any past date, each figure traceable to source. See the 2026-10-10 vision brief. ### 2.2 Non-Users (v1) -Casual news readers, campaign operatives wanting opposition research scores, -quant traders (CFA owns markets). +Casual news readers and a voter-facing app (readers consume published findings, not +the tool), campaign operatives wanting opposition research scores, quant traders +(CFA owns markets). ### 2.3 Key User Journeys @@ -150,8 +154,9 @@ only. `[SOURCE: 2026-09-17 schema review absorb.]` - News, Epstein, or stocks as first-class schema domains. Existing `core.instrument` / `fact.market_bar` are empty compatibility tables, not a license to ingest prices. -- Corruption/integrity **scores** as a schema domain or opaque single score (keep - evidence). Evidence-linked scorecards may be added later as derived marts. +- Corruption/integrity **scores** as a schema domain, opaque single scores, or verdict + labels (bribed, insider trading, lie). Report cards are derived marts over versioned, + tiered metrics (2026-10-10). - Meltano/Singer as the foundation; Qdrant/Pinecone; FastAPI CRUD. - OpenSpec or Spec Kit beside BMAD. - Playwright as the default test stack. @@ -188,6 +193,11 @@ FRED branch. counts match, every offered field is on a checklist, and the live database agrees with `inventory/progress.yaml`. Validates the 2026-09-25 operator goal. +- **SM-5 (2026-10-10):** A report card regenerates for any past date and every figure + drills to its source file, run and checksum. +- **SM-6 (2026-10-10):** Every published metric carries a definition version and a + reliability tier (established, estimated, exploratory). + **Counter-metric SM-C1:** Number of datasets registered. Do not optimize catalog size. diff --git a/_bmad-output/planning-artifacts/sprint-change-proposal-2026-10-10.md b/_bmad-output/planning-artifacts/sprint-change-proposal-2026-10-10.md new file mode 100644 index 0000000..096ecf2 --- /dev/null +++ b/_bmad-output/planning-artifacts/sprint-change-proposal-2026-10-10.md @@ -0,0 +1,93 @@ +# Sprint Change Proposal: from warehouse to research center (report cards and publishable findings) + +Date: 2026-10-10. Status: **approved by operator 2026-10-10** (rule amendments applied). Scope class: **Major** (PRD, epics, architecture), +implemented as a **direct adjustment** (add epics, no rollback). Epic 10 and Story 10.3 stay on the path. + +Inputs: `briefs/brief-OpenDiscourse-2026-10-10/` (brief + addendum), `docs/data-audit-2026-10-10.md`. + +## 1. Issue summary +The approved PRD (2026-09-14) calls the user "a researcher/operator, not a consumer app" and treats scorecards, +stocks and money as later or out of scope. The operator has now set the goal: a free local political research +center that produces **published findings and politician report cards** across eight tracks. Core: votes, +bill text, outcomes by place, report cards. Important: donations/lobbying, federal awards, members' trades. +Hard: statements versus votes. The current plan builds the data spine well but has no epic for +report cards, time-travel facts, metric definitions, NLP evaluation, or publishing, and several rules +forbid parts of the vision. + +## 2. Impact +**Epics.** Epic 10 unchanged, proceeds 10.3 to 10.6. Epic 7 ("v1.1 money, elections, crime", stories TBD) is +replaced by Epics 14, 16, 17 below. Epic 5 (marts) is absorbed into Epics 13 and 19. Epic 9 (ingestion contract) is +the template every new source uses. +**PRD.** Section 2.2 non-users, section 5 non-goals, section 6 MVP, section 7 metrics need edits (see 4). +**Architecture.** Four new decisions (AD-12 to AD-15, see 4). No change to AD-1 to AD-11. +**Code.** Existing code unaffected. New: as-of columns on new fact tables, metric registry, evaluation harness. +**Rules (`AGENTS.md`, `v1-scope.md`).** Three amendments need operator approval (see 5). + +## 3. Recommended approach +Direct adjustment: keep the identity, provenance and legislation foundation; finish the district slice; then build +the **first report card** from data already loaded before widening any source. Reason: it proves the end-to-end +path (as-of facts, metric definitions, evidence drill-through, publishing) early and shows what the data can +honestly support. Effort is large but incremental; risk is scope growth, mitigated by the gates below. + +## 4. Detailed change proposals + +### New epics (each needs its own spec before starting; a roadmap row is not authorization) +| Epic | Name | Tracks | Contents | +|---|---|---|---| +| 12 | Time-aware facts and method registry | foundation | As-of (knowledge-date) columns and view pattern; versioned metric definitions with reliability tier (established / estimated / exploratory); hypothesis log committed before results; evaluation harness for text models | +| 13 | Politician profile and vote metrics, any office | 1, 8 | One person-office-term card model (representatives, senators, president, governors, state legislators); office-specific metrics (votes for legislators; signs, vetoes for governors); tenure, prior offices, committee history; attendance; party loyalty; bipartisanship; vote-based ideology (Voteview); legislative effectiveness | +| 14 | Outcomes by place and opinion | 5 | District ACS (Story 10.4), PUMS all years typed with weights (see 6), LAUS, QCEW, BEA, IRS SOI, HUD, FHFA, NIBRS, CDC PLACES, immigration measures, Cooperative Election Study, election results | +| 15 | Text analytics | 2 | Policy topic taxonomy (Comparative Agendas), bill chunks and embeddings (ADR first), text reuse, accuracy-measured claim extraction | +| 16 | Money in and out | 3, 4 | FEC typed cycles and identity bridge (issues #103, #104), donor grouping layer (estimated), Lobbying Disclosure Act, FARA, USAspending, rules and comments | +| 17 | Disclosures and ethics | 6 | House/Senate disclosures, ethics actions; timeline views, no verdict labels | +| 18 | Statements | 7 | Congressional Record speeches, GDELT reported quotes (lower trust), consistency check with measured accuracy; social media only if legally and cheaply available | +| 19 | Report cards and publishing | 8 | Card generator, as-of regeneration, methodology and corrections pages, evidence drill-through, findings export, legal review gate | + +### Architecture additions (ADRs to write) +- AD-12 bitemporal facts (valid time and knowledge time). +- AD-13 metric registry: definition, version, inputs, reliability tier; a card records the versions used. +- AD-14 model-output rule: any text-derived claim stores the passage, model and version, and has a published error rate. +- AD-15 publishing exports: Postgres remains the record; Parquet/CSV extracts are derived and rebuildable. + +### PRD edits +- 2.1/2.2: add "publishes findings and report cards"; keep "not a voter-facing app". +- 5 Non-goals: replace "scorecards later" with "no single opaque score; no verdict labels such as bribed or lie". +- 7 Metrics: add SM-5 (first card regenerates for any past date and every figure drills to source), SM-6 (every + published metric carries a tier and version). + +### OpenStates promotion is part of this plan, not later +The snapshot holds about 22,700 people, 1.19 M vote events and 1.56 M bills across all states, with 254 +executive-office membership records (not yet confirmed to be all governors). The already-approved +programme (issues #100 to #105) promotes this into owned tables. Epic 13 depends on #102 for state +legislators and governors; it does not wait for the other epics. + +### First report-card slice (the next milestone after Epic 10 slice_proven) +Epic 12 (minimum) then Epic 13 then Epic 19 (minimum), for the 119th House: tenure, attendance, party loyalty, +vote-based ideology, committees, district context. Money, outcomes beyond Census, trades and statements follow in +the order 16, 14, 17, 18, 15. + +## 5. Rule amendments for operator decision (not applied) +1. `AGENTS.md`: "Do not invent news, stocks, or corruption scores as schema domains" becomes: member disclosures + (STOCK Act trades) are an allowed source domain; market prices stay out; GDELT news is allowed only as reported + statement evidence; scorecards are derived marts with versioned metrics, no single opaque score. +2. `AGENTS.md` and `v1-scope.md`: "Do not start Epic 7" becomes: Epics 14, 16, 17, 18 open only after Epic 10 + `slice_proven` and the gates below. +3. `v1-scope.md` "Not a product domain" section: update to match item 1. + +## 6. Sequencing and gates +1. Story 10.3 now, then 10.4 to 10.6 (unchanged). +2. Parallel, not blocking: Postgres query tracking and tuning (needs restart); PUMS typed load design (ADR-0006 + style, all years 1- and 5-year, 80 replicate weights, measured size first; no drop until reconciled). +3. Epics 12, 13, 19 (first card). Gate: operator reviews a real card before anything is published. +4. Then 16, 14, 17, 18, 15. Gate for each: source contract (coverage, licence, size), capacity preview, and for + person-linked data the identity gate. Legal review before any named-person grade is published. + +## 7. Handoff +Architect: ADRs AD-12 to AD-15. PM: PRD edits and Epic 12, 13, 19 specs. Developer: continue Story 10.3. +Scrum master: rerun sprint planning after approval. Success: the first card exists, regenerates for a past date, +and the operator can trace each number. + +## 8. Operator decisions needed +- Approve this proposal (yes / revise). +- Approve the three rule amendments in section 5 (they change what the project forbids). +- Confirm the card sections in the first-card slice. diff --git a/_bmad-output/specs/spec-opendiscourse/v1-scope.md b/_bmad-output/specs/spec-opendiscourse/v1-scope.md index 5c01960..62713e4 100644 --- a/_bmad-output/specs/spec-opendiscourse/v1-scope.md +++ b/_bmad-output/specs/spec-opendiscourse/v1-scope.md @@ -59,9 +59,11 @@ bounded BLS). ## Not a product domain -News, stocks/CFA, Epstein-as-schema, corruption scores as a schema domain. -Scorecards over evidence-backed rows are allowed later as derived marts (SPEC -non-goals); not part of the v1 ingest spine. +News as a product, market prices/CFA, Epstein-as-schema, corruption scores as a +schema domain, single opaque scores, verdict labels. Allowed (operator, 2026-10-10): +members' required disclosures (STOCK Act trades), news only as reported-statement +evidence, and report cards as derived marts over versioned, tiered metrics; these +open after Epic 10 `slice_proven` (see sprint-change-proposal-2026-10-10). `core.instrument` / `fact.market_bar` are retained empty compatibility tables; no market-price ingest. diff --git a/docs/PROJECT-STATE.md b/docs/PROJECT-STATE.md index 1b5b3ce..5912c2f 100644 --- a/docs/PROJECT-STATE.md +++ b/docs/PROJECT-STATE.md @@ -1,5 +1,15 @@ # Project state and handoff +**Product direction set (2026-10-10).** The operator approved +`_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/` (vision brief and addendum): +a free local political research center that produces published findings and politician report cards, +with eight tracks (1 votes, 2 bill text, 5 outcomes by place, 8 report cards are core). The data audit is +`docs/data-audit-2026-10-10.md`. Decision: keep all ACS PUMS (all years, 1- and 5-year) as typed tables with +weights and replicate weights; nothing is dropped until the typed load reconciles. Story 10.3 is specified in +`_bmad-output/implementation-artifacts/spec-10-3-division-boundary-crosswalk.md` (draft, awaiting go). +Postgres query tracking and tuning are approved and need a restart at a quiet moment. No project rules +(`AGENTS.md`) have been changed yet. + **District programme approved; Story 10.2 live validation in progress (2026-10-05).** The operator approved `_bmad-output/specs/spec-district-linked-context/`. `stories.yaml` is the executable Epic 10 queue and diff --git a/docs/data-audit-2026-10-10.md b/docs/data-audit-2026-10-10.md new file mode 100644 index 0000000..92ed96a --- /dev/null +++ b/docs/data-audit-2026-10-10.md @@ -0,0 +1,81 @@ +# Data audit, phase 1: what we hold versus what the report card needs + +Date: 2026-10-10. Read-only. Measured against +`_bmad-output/planning-artifacts/briefs/brief-OpenDiscourse-2026-10-10/` (brief + addendum). +Counts are planner estimates from `pg_class.reltuples`, not exact. Nothing was changed or deleted. +Verdicts are proposals for operator sign-off. I did not verify that every staging table can be +rebuilt from retained files; that check must come before any wipe. + +## 1. Database size: 536 GB total + +| Table | Size | Rows | What it is | Proposed verdict | +|---|---:|---:|---|---| +| `stage.acs_pums_record` | 286 GB | 144 M | Half-loaded person-level census microdata, raw JSON | **Trim** (see 3a) | +| `fact.acs_bulk_estimate` | 105 GB | 281 M | ACS table values, one row per value, state/county scope | **Replace** by packed table after checks (ADR-0006) | +| `stage.fec_row` | 74 GB | 102 M | FEC files as raw JSON, unverified legacy | **Replace** with typed cycle tables, then drop (3b) | +| `stage.cbp_row` | 22 GB | 29 M | Business-patterns staging; loaded into `fact.business_pattern` (12 GB) | Trim staging once the fact table is verified | +| `core.geography_boundary` | 11 GB | 377 K | District/county/etc shapes | Keep; consider simplified copy for display | +| `stage.tiger_feature` | 11 GB | 374 K | TIGER staging used by 10.2 validation | Keep until Story 10.3 done, then trim | +| `stage.acs_bulk_row` | 5.5 GB | 5.6 M | ACS staging | Trim after packed load verified | +| Legislation and votes (core/fact) | about 9 GB | | bills, text, actions, votes (7.2 M), sponsors | **Keep** (core of cards) | + +About 388 GB (72%) is staging or superseded data. The report-card core (people, votes, bills, text, +committees) is under 10 GB. + +## 2. Report-card sections versus holdings + +| Card section | Needed data | Have | Gap | +|---|---|---|---| +| Identity, tenure, offices | people, memberships, terms, IDs | 12,770 people, 45,535 memberships, BioGuide | Pre-Congress offices (prior positions) | +| Attendance, party loyalty | roll calls, member votes | 23,359 roll calls, 7.2 M votes, 108-119; party totals | Reasons for dissent (floor speeches) | +| Ideology | votes | Voteview 51 K members, 113 K roll calls | Issue-level scales (need topic taxonomy) | +| Committees | assignments | 559 committees, 3,895 seats (current only) | Historical seats | +| Bill and law text | text, summaries, subjects | 135 K text versions (113-119), 208 K summaries, 2.1 M subject rows | 108-112 text; amendments text | +| Consistency over time | topic labels + votes | subjects exist | Topic taxonomy, human-checked sample | +| Ethics actions | committee reports | none | Source to build | +| Donors | FEC cycles 2000-2024 | 102 M raw rows, unverified, not linked to people | Typed model, candidate-committee bridge | +| Lobbying, foreign agents | LDA, FARA | none | Sources to build | +| Trades and assets | House/Senate disclosures | none | Source to build | +| District change | place-time outcomes | ACS (state/county), population estimates, DHC 2020, business patterns, FRED macro | District-level ACS, jobs (LAUS/QCEW), crime, housing, health, taxes, immigration measures | +| Money into district | awards | none | USAspending | +| Constituent opinion | CES | none | New source | +| Statements | CREC speeches, GDELT | none (OpenStates snapshot has no federal speeches) | Sources to build | + +## 3. Notes behind the verdicts +a. **ACS person microdata.** The published district tables already provide district income, housing, +nativity and similar measures. Person-level records matter only for custom tabulations (they are keyed to +PUMAs, not districts) and do not feed any card section directly. The 136 GB of source files stay +retained. The 286 GB raw-JSON staging can be dropped, with the decision recorded, and a typed reload +done only if a research question needs it. +b. **FEC.** The approved plan already replaces `stage.fec_row` with typed, cycle-partitioned facts after a +four-file pilot. Dropping the raw JSON before the typed pilot passes would remove the only copy of the +parsed rows; the source ZIPs remain retained, so it is rebuildable. +c. **Margins of error.** Keep them only for curated metrics that get compared across districts; drop them +from other packed tables. + +## 4. Sources to add, in priority order (for the cards) +1. FEC typed cycles + candidate-committee bridge (donors, concentration). +2. Lobbying Disclosure Act (bill-level links) and FARA. +3. USAspending (money into districts). +4. Outcome measures at district/county: BLS LAUS and QCEW, IRS SOI, HUD, FHFA, NIBRS, CDC PLACES, CBP. +5. Congressional Record speeches; ethics actions; House/Senate disclosures. +6. Cooperative Election Study; election results; topic taxonomy. +7. GDELT news-quoted statements, then social media if obtainable. + +Each needs a source contract (coverage, licence, size) before acquisition. + +## 5. Decisions needed from the operator +1. Approve dropping `stage.acs_pums_record` (286 GB derived rows only; files retained)? +2. Approve dropping `stage.fec_row` after the typed FEC pilot passes (not now)? +3. Approve the priority order in section 4? +4. Confirm the short list of card sections for the first published card. + +## 6. Operator decision on PUMS (2026-10-10) +Keep all ACS PUMS: every 1-year release (2005-2019, 2021-2024) and every 5-year release +(2005-2009 through 2020-2024), person and housing, as typed tables with the person weight and all 80 +replicate weights, plus a per-year harmonization mapping. No sampled subset. The 286 GB raw-JSON staging +is dropped only after the typed tables are loaded and reconciled per release; retained source ZIPs are +never touched. Size estimate (about 100-200 GB typed) must be measured on one release before the full run. +The 5-year file pools five annual survey years (about 5x the records of a 1-year file); it is not a +change measure, and its respondents overlap with the 1-year files. Reason: policy-impact and +standard-of-living analysis, multilevel models, and small-area estimation all need it.