Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
109 lines (78 loc) · 6.4 KB

ARCHITECTURE.md

File metadata and controls

109 lines (78 loc) · 6.4 KB
Raw

Architecture

SEO Data Console is a local-first analytics application with an optional cloud replica.

Current Choice

  • Frontend: React, TypeScript, and Vite under apps/web/.
  • Backend: FastAPI under apps/api/.
  • Local source of truth: normalized canonical SQLite facts in data/local/seo_dashboard.sqlite.
  • Active ingest evidence: deterministic, replaceable raw responses under data/*/raw/.
  • Cloud database: Supabase Postgres as an optional replica using the same canonical keys.
  • Frozen reference: apps/seo_dashboard/ is not an active development or runtime dependency.

Capture timestamps, request ranges, file hashes, GSC snapshotId, GA4 scopeId, and current/comparison labels are not analytical fact identities.

Data Flow

explicit source action
  -> read-only readiness + canonical coverage preflight
  -> user confirmation
  -> lightweight persisted job/partitions
  -> execution-time server re-plan
  -> connector CLI with complete pagination
  -> temporary raw response
  -> normalize and validate
  -> canonical SQLite upsert
  -> deterministic active raw replacement
  -> optional Supabase canonical-key upsert

cached read
  -> requested date range
  -> canonical SQLite facts + coverage ledger
  -> FastAPI range contract
  -> React analysis workflow

Cached reads never enter connector execution or cloud health checks.

Canonical Identities

  • GSC: property, report family, grain, date, dimensions, filter identity, and search type.
  • GA4: property, report family, grain, date, dimensions, filter/segment identity, and metric schema.
  • CrUX: normalized target, target type, form factor, metric, and collection period.
  • PageSpeed: normalized requested URL and device strategy; only the latest validated success is retained.

A returned canonical key overwrites its previous value. A previously stored key omitted from a later response remains stored. Aggregate rows that cannot prove a stable daily identity are not promoted into canonical daily facts.

Coverage And Incremental Sync

source_coverage records attempted daily partitions per source, owner, report family, grain, and filter/segment identity. A cached query reports complete, partial, missing, failed, or no_dataset from this ledger and the facts actually present.

Explicit GSC and GA4 syncs:

  1. preflight requested canonical coverage without source/cloud calls or local writes;
  2. show gaps, coverage hits, named three-day reconciliation overlap, known minimum versus dynamic calls, and persistence targets;
  3. require explicit confirmation, persist only lightweight operational job/partition metadata, and recompute the plan server-side immediately before execution;
  4. fetch every still-planned interval with full pagination;
  5. validate, normalize, and upsert returned facts while retaining facts not returned by the new response;
  6. stop once on fatal source/configuration errors while allowing unrelated report-local failures to continue;
  7. retry only prior failed/interrupted partitions whose ranges remain canonically missing after another server re-plan.

Search Appearance is intentionally two-stage. Preflight can know discovery calls but cannot know the count of filtered appearance calls. After discovery, each returned appearance is deduplicated and planned against its own canonical coverage before its dynamic partition is added.

CrUX uses a seven-day refresh cadence because its facts are collection-period based. PageSpeed remains an explicit live action with latest-only URL/device retention.

Backend Responsibilities

  • Plan bounded source work from canonical coverage.
  • Keep preflight read-only and ignore client-supplied report/interval execution instructions.
  • Complete connector pagination before promotion.
  • Reject or quarantine unprovable rows without deleting valid existing facts.
  • Redact API errors, credentials, and private paths from UI diagnostics.
  • Serve range-based cached analytics without snapshot selection.
  • Keep local persistence successful even if optional cloud replication fails.
  • Persist bounded restart-safe job/partition progress without analytics payloads, metrics, credentials, commands, stack traces, or paths.
  • Recover orphaned active work as interrupted and prevent automatic loops for fatal auth, permission, quota, network-policy, configuration, or request failures.

Frontend Responsibilities

  • Let users edit requested dates independently of collection events.
  • Keep “read cache” separate from quota-consuming “query source”.
  • Show an inline preflight/confirmation before a source job, then render non-color-only partition progress and selective retry.
  • Display requested range, canonical coverage, gaps, retained facts, freshness, comparison status, and pagination truthfully.
  • Preserve saved views, exports, URL state, annotations, and responsive behavior without storing capture identities.

Database Responsibilities

SQLite holds:

  • gsc_facts, ga4_facts, and crux_facts;
  • source_coverage and source_sync_attempts;
  • source_sync_jobs and source_sync_partitions for lightweight operational progress;
  • latest-only PageSpeed facts and attempts;
  • API run history, saved views, annotations, AI tasks, and unrelated operational state.

Supabase migration 003_canonical_facts.sql mirrors canonical fact and coverage keys. Cloud replication is optional and may fail without invalidating the local commit.

Timestamped GSC/GA4/CrUX analytical raw history and legacy capture tables are retired. Deterministic active raw evidence is replaceable and is not the source queried by analysis endpoints.

Migration And Safety

tools/migrate_canonical_facts.py performs a targeted, repeatable migration. It promotes provable daily facts, compacts source-derived legacy tables and raw payloads, verifies protected row counts and SQLite integrity, and never deletes the whole database. Saved views, annotations, tasks, configuration, credentials, PageSpeed latest results, and unrelated files remain protected.

Cloud-primary storage is not approved. .env remains unchanged, and source/cloud calls require an explicit action and authorization.

Active And Frozen Applications

apps/api/ and apps/web/ are the sole active development path. The dependency-free dashboard under apps/seo_dashboard/ remains a frozen behavioral reference and temporary fallback. It may be removed only after the parity, data, operations, browser, rollback, and explicit-approval gates in .ai/FRONTEND_BACKEND_MIGRATION_GATE.md pass.