Skip to content

[v0.4] Sync completeness, source epochs and recoverable partial imports #38

Closed

Description

@JetSquirrel

Tracks #31. Target: v0.4 — Static graph → Change investigation.

Problem

The current plugin protocol may keep records written before an interrupted sync. Investigations must not confuse partially ingested data with complete source coverage.

Scope

  • Track per-source sync-run IDs, start/end, status (complete/partial/failed/cancelled), cursor and mapper version.
  • Mark evidence and findings with the source epoch/completeness they rely on.
  • Make full-sync removals contingent on a verified complete enumerated source; preserve idempotent stable-ID merge behavior.
  • Define recovery/resume/retry handling for interrupted imports, avoiding silent "all clear" interpretations.
  • Surface coverage warnings in CLI, UI and generated investigation reports.

Acceptance criteria

  • Plugin failure mid-stream is visible as partial/failed with affected source IDs.
  • A partial sync does not produce false confirmed deletions or "no exposure" conclusions.
  • Repeated completed full/incremental syncs remain deterministic and idempotent.
  • Fixtures test cancellation, duplicate IDs, missing edge endpoints, changed cursors and source disappearance.
  • No secrets or raw credentials written to sync logs.

Out of scope: generic distributed exactly-once ingestion.

Activity

  1. JetSquirrel commented on Nov 3, 2026

    @JetSquirrel
    OwnerAuthor

    All acceptance criteria have tests; shipped in #51.

    Parts of the scope that were not built as written:

    • Coverage warnings in generated reports: those reports don't exist yet. [v0.5] Investigation audit trail and evidence-backed report export #41 already requires reports to explain incomplete sources.
    • Resume: a retry syncs again from the same cursor. There is no resuming partway through a sync.
    • Mapper version: runs record the plugin version, with no separate mapper version.

    Real AWS syncs are complete or failed, never partial, because the plugin stops at the first failed call. Nothing has run against a real AWS account.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions