diff --git a/README.md b/README.md index 85df68b..ff61388 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Internal Marmot audit-log explorer. -Goggles accepts sensitive `marmot-forensics-audit` JSONL audit logs from Dark Matter clients (current schema `marmot-forensics-audit/v2`; legacy `v1` is still accepted), preserves the exact uploaded text and raw lines, normalizes common forensic columns into PostgreSQL tables, and gives the team a login-gated dashboard for comparing what multiple account-device engines saw and decided inside each group. +Goggles accepts sensitive `marmot-forensics-audit` JSONL audit logs from MDK clients (current schema `marmot-forensics-audit/v2`; legacy `v1` is still accepted), preserves the exact uploaded text and raw lines, normalizes common forensic columns into PostgreSQL tables, and gives the team a login-gated dashboard for comparing what multiple account-device engines saw and decided inside each group. The audit event schema is committed at `docs/schemas/audit-log-event.v2.schema.json`. See also `docs/api-v1.md` for the authenticated read API, `docs/deployment.md` for VM deployment notes, and `docs/audit-debugging-platform-prd.md` for the platform's product requirements. diff --git a/docs/api-v1.md b/docs/api-v1.md index 45736cf..1682d23 100644 --- a/docs/api-v1.md +++ b/docs/api-v1.md @@ -148,7 +148,7 @@ the convergence-decision evidence refs that produced those child rows. `message_id` filters convergence runs by selected branch id, losing branch id, candidate branch id, or candidate commit ids. `epoch` filters by run-level, candidate-level, and evidence-level convergence epoch fields. -When Dark Matter does not emit a stable convergence `run_id`, Goggles groups +When MDK does not emit a stable convergence `run_id`, Goggles groups contiguous convergence and epoch-state evidence for the same group engine into an inferred run and sets `inferred` to `true`. If multiple engines have the same run id, the detail endpoint returns `409` with diff --git a/docs/audit-debugging-platform-prd.md b/docs/audit-debugging-platform-prd.md index 3eb70bd..ffc4e1c 100644 --- a/docs/audit-debugging-platform-prd.md +++ b/docs/audit-debugging-platform-prd.md @@ -2,7 +2,7 @@ Status: Draft Date: 2026-06-24 -Audience: Dark Matter, Goggles, and client application engineers +Audience: MDK, Goggles, and client application engineers ## Summary @@ -27,12 +27,12 @@ content and transport wire identifiers. ## Background -Dark Matter audit logging currently writes sensitive JSONL records using +MDK audit logging currently writes sensitive JSONL records using `schema_version = "marmot-forensics-audit/v1"`. Goggles ingests those files, preserves exact raw text, normalizes common fields into database columns, and renders group-level dashboards. -The latest Dark Matter audit changes add richer group and epoch breadcrumbs: +The latest MDK audit changes add richer group and epoch breadcrumbs: - `epoch_state_changed` records engine epoch-state transitions such as stable, pending publish, recovering, and unrecoverable. @@ -45,7 +45,7 @@ The latest Dark Matter audit changes add richer group and epoch breadcrumbs: attribution on every row. These changes are necessary but not sufficient. Goggles needs new product -surfaces, derived data models, and programmatic APIs. Dark Matter and client +surfaces, derived data models, and programmatic APIs. MDK and client libraries may also need additional audit events so Goggles can explain branch selection, transport gaps, and client/device behavior with confidence. @@ -90,7 +90,7 @@ selection, transport gaps, and client/device behavior with confidence. ## Audit Data Modes The audit recorder should expose a two-option data-mode setting. The selected -mode must be stored in Dark Matter settings, stamped into recorder or session +mode must be stored in MDK settings, stamped into recorder or session context, visible in Goggles, included in API responses, and auditable when it changes. @@ -189,7 +189,7 @@ Important distinction: Question: What did the convergence engine see, what did it choose, and why? -The view should organize events by convergence run. Once Dark Matter emits a +The view should organize events by convergence run. Once MDK emits a stable `convergence_run_id`, that id should define the run. Before then, Goggles can infer provisional runs from contiguous convergence and epoch-state events with the same engine id and group ref. An inferred run begins when an engine @@ -214,7 +214,7 @@ Each run should show: - resulting epoch-state transition; - raw evidence links for every row. -Current Dark Matter data can show high-level decisions through +Current MDK data can show high-level decisions through `convergence_decision`, `error_kinds`, and `epoch_state_changed`. It likely does not yet expose enough candidate-level detail to fully explain branch selection. @@ -312,10 +312,10 @@ projection model: - Update fixtures and parser tests for the V2 JSON Schema. - Include derived projections in agent export and API output. -## Dark Matter Audit Data Gaps +## MDK Audit Data Gaps The latest audit changes are a strong foundation, but Goggles probably needs -additional Dark Matter events for complete explainability. +additional MDK events for complete explainability. A first draft of the proposed V2 event-line schema is in [`docs/schemas/audit-log-event.v2.schema.json`](schemas/audit-log-event.v2.schema.json). @@ -480,9 +480,9 @@ API behavior: Phase 0: Alignment -- Review this PRD with Dark Matter, Goggles, and client owners. +- Review this PRD with MDK, Goggles, and client owners. - Turn P0 items into issues. -- Decide which Dark Matter schema gaps are required before UI redesign. +- Decide which MDK schema gaps are required before UI redesign. Phase 1: Greenfield Storage And Ingest @@ -510,15 +510,15 @@ Phase 3: UI Redesign Phase 4: Producer Gaps -- Add Dark Matter audit settings and schema support for obfuscated versus full +- Add MDK audit settings and schema support for obfuscated versus full data auditing. -- Add missing Dark Matter audit events for convergence branch explainability. +- Add missing MDK audit events for convergence branch explainability. - Add TypeScript/client audit-log parity. - Validate emitted logs against the committed schema. ## Decisions From Initial Review -- Convergence runs should eventually be keyed by a Dark Matter emitted +- Convergence runs should eventually be keyed by an MDK-emitted `convergence_run_id`. Until then, Goggles may infer provisional runs from contiguous convergence and epoch-state rows with the same engine id and group ref, and must label those runs as inferred. @@ -566,7 +566,7 @@ Phase 4: Producer Gaps ## Remaining Open Questions -- What exact schema names and field shapes should Dark Matter use for +- What exact schema names and field shapes should MDK use for convergence rule outputs, expected-recipient sets, and full data auditing message contents? - Which derived projection tables and indexes are required for the first fast @@ -578,7 +578,7 @@ Phase 4: Producer Gaps - Inferred missing transport events may be mistaken for proven non-delivery. - System-stamped audit rows may overwhelm human-action views if not separated. -- Branch explainability will be incomplete unless Dark Matter emits candidate +- Branch explainability will be incomplete unless MDK emits candidate details, not just final decisions. - Rich APIs may accidentally expose sensitive forensic data too broadly without tight authentication and careful response design. diff --git a/fixtures/sample-audit-log-acme-dana.jsonl b/fixtures/sample-audit-log-acme-dana.jsonl index 2eb336c..56541a5 100644 --- a/fixtures/sample-audit-log-acme-dana.jsonl +++ b/fixtures/sample-audit-log-acme-dana.jsonl @@ -1,4 +1,4 @@ -{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730086400000,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"obfuscated_sensitive_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{"source":{"account_label":"Dana Whitfield","device_name":"MacBook Pro","platform":"macos"}},"kind":{"type":"recorder_started","recorder":"darkmatter"}} +{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730086400000,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"obfuscated_sensitive_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{"source":{"account_label":"Dana Whitfield","device_name":"MacBook Pro","platform":"macos"}},"kind":{"type":"recorder_started","recorder":"mdk"}} {"schema_version":"marmot-forensics-audit/v2","seq":1,"wall_time_ms":1730086400120,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"full_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{},"kind":{"type":"audit_data_mode_changed","previous_mode":"obfuscated_sensitive_data","new_mode":"full_data","reason":"forensic_capture_enabled","recorder_restarted":true}} {"schema_version":"marmot-forensics-audit/v2","seq":2,"wall_time_ms":1730086400240,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"full_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{},"kind":{"type":"source_context","source":{"account_label":"Dana Whitfield","account_pubkey_hex":"a773887e1170c9c8bc47e0c6a574fb0d9cbb7ea8ddc14a0b345674d4fa40bc0f","device_name":"MacBook Pro","platform":"macos"}}} {"schema_version":"marmot-forensics-audit/v2","seq":3,"wall_time_ms":1730086401440,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"full_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{},"kind":{"type":"message_content_decoded","msg_id":"4818733d5badb40bfb46cd744814e96ea21ed78f2a96b851a21458dee8abfd52","artifact_kind":"application_message","author":{"member_ref":"5299b84c07796db0995edb1fe2ea0b7b","account_pubkey_hex":"a773887e1170c9c8bc47e0c6a574fb0d9cbb7ea8ddc14a0b345674d4fa40bc0f"},"decoded_payload":{"content_type":"text/plain","text":"Standup in 5. Drop blockers in the thread."},"decoded_app_event":{"format":"nostr","kind":9,"content":"Standup in 5. Drop blockers in the thread.","pubkey_hex":"a773887e1170c9c8bc47e0c6a574fb0d9cbb7ea8ddc14a0b345674d4fa40bc0f"}}} diff --git a/fixtures/sample-audit-log-trailhead-maya.jsonl b/fixtures/sample-audit-log-trailhead-maya.jsonl index 33b04c7..553f2b4 100644 --- a/fixtures/sample-audit-log-trailhead-maya.jsonl +++ b/fixtures/sample-audit-log-trailhead-maya.jsonl @@ -1,4 +1,4 @@ -{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730000000000,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"obfuscated_sensitive_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{"source":{"account_label":"Maya Okonkwo","device_name":"iPhone 15 Pro","platform":"ios"}},"kind":{"type":"recorder_started","recorder":"darkmatter"}} +{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730000000000,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"obfuscated_sensitive_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{"source":{"account_label":"Maya Okonkwo","device_name":"iPhone 15 Pro","platform":"ios"}},"kind":{"type":"recorder_started","recorder":"mdk"}} {"schema_version":"marmot-forensics-audit/v2","seq":1,"wall_time_ms":1730000000120,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"full_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"audit_data_mode_changed","previous_mode":"obfuscated_sensitive_data","new_mode":"full_data","reason":"forensic_capture_enabled","recorder_restarted":true}} {"schema_version":"marmot-forensics-audit/v2","seq":2,"wall_time_ms":1730000000240,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"full_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"source_context","source":{"account_label":"Maya Okonkwo","account_pubkey_hex":"0096d5e0a3a8479e8a72d04f4020eab2bfc9796a47a07ba938acb76a93b4f246","device_name":"iPhone 15 Pro","platform":"ios"}}} {"schema_version":"marmot-forensics-audit/v2","seq":3,"wall_time_ms":1730000000720,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"full_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"message_content_decoded","msg_id":"30f6d1e85e6710f17dddc4aca3cd06a0e1338f197cd6e364a1140eeecbae6dee","artifact_kind":"application_message","author":{"member_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","account_pubkey_hex":"0096d5e0a3a8479e8a72d04f4020eab2bfc9796a47a07ba938acb76a93b4f246"},"decoded_payload":{"content_type":"text/plain","text":"Trail this Saturday? Thinking Eagle Creek, 8am."},"decoded_app_event":{"format":"nostr","kind":9,"content":"Trail this Saturday? Thinking Eagle Creek, 8am.","pubkey_hex":"0096d5e0a3a8479e8a72d04f4020eab2bfc9796a47a07ba938acb76a93b4f246"}}} diff --git a/fixtures/sample-audit-log-trailhead-theo.jsonl b/fixtures/sample-audit-log-trailhead-theo.jsonl index 54ce47f..56c26bf 100644 --- a/fixtures/sample-audit-log-trailhead-theo.jsonl +++ b/fixtures/sample-audit-log-trailhead-theo.jsonl @@ -1,4 +1,4 @@ -{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730000000360,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"obfuscated_sensitive_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{"source":{"account_label":"Theo Almeida","device_name":"Pixel 8","platform":"android"}},"kind":{"type":"recorder_started","recorder":"darkmatter"}} +{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730000000360,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"obfuscated_sensitive_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{"source":{"account_label":"Theo Almeida","device_name":"Pixel 8","platform":"android"}},"kind":{"type":"recorder_started","recorder":"mdk"}} {"schema_version":"marmot-forensics-audit/v2","seq":1,"wall_time_ms":1730000000480,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"full_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"audit_data_mode_changed","previous_mode":"obfuscated_sensitive_data","new_mode":"full_data","reason":"forensic_capture_enabled","recorder_restarted":true}} {"schema_version":"marmot-forensics-audit/v2","seq":2,"wall_time_ms":1730000000600,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"full_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"source_context","source":{"account_label":"Theo Almeida","account_pubkey_hex":"b20478b2a54600d0b62fc099448bbcc52c474208545bf438c9d6af9aeab689e1","device_name":"Pixel 8","platform":"android"}}} {"schema_version":"marmot-forensics-audit/v2","seq":3,"wall_time_ms":1730000001500,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"full_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"transport_received","msg_id":"30f6d1e85e6710f17dddc4aca3cd06a0e1338f197cd6e364a1140eeecbae6dee","transport":{"transport":"nostr","delivery_plane":"relay","relay_url":"wss://relay.primal.net","nostr_event_id":"b43904c2a866006115391c243f68b9f885cf4447c3be460b5885a447cc9fe45b","nostr_kind":445},"payload_len":95,"payload_digest":"a0775e72269e1b375d19fa35d08ba57d63e776f0d30b368c6ab111f8e4dcea70"}} diff --git a/forensics/seed_data.py b/forensics/seed_data.py index 359a6bb..2a3bec5 100644 --- a/forensics/seed_data.py +++ b/forensics/seed_data.py @@ -148,7 +148,7 @@ def start(self, participant: Participant) -> None: """ self.emit( participant, - {"type": "recorder_started", "recorder": "darkmatter"}, + {"type": "recorder_started", "recorder": "mdk"}, mode=OBFUSCATED, context={ "source": { diff --git a/forensics/tests.py b/forensics/tests.py index 14db5f5..cafd4c7 100644 --- a/forensics/tests.py +++ b/forensics/tests.py @@ -147,7 +147,7 @@ def audit_event_v2( "engine_id": engine_id, "group_ref": group_ref, "context": context if context is not None else {"operation_id": f"op-v2-{seq}"}, - "kind": kind or {"type": "recorder_started", "recorder": "darkmatter"}, + "kind": kind or {"type": "recorder_started", "recorder": "mdk"}, } @@ -1580,7 +1580,7 @@ def test_v2_upload_builds_audit_projections(self): account_ref=ACCOUNT_BOB, recorder_session_id="session-b", context={"source": {"account_label": "Bob", "device_name": "Bob laptop"}}, - kind={"type": "recorder_started", "recorder": "darkmatter"}, + kind={"type": "recorder_started", "recorder": "mdk"}, ) ), content_type="application/x-ndjson", @@ -2216,7 +2216,7 @@ def test_context_subobjects_must_be_objects_when_present(self): audit_event_v2( 0, context={"source": "alice laptop", "convergence": ["run-1"]}, - kind={"type": "recorder_started", "recorder": "darkmatter"}, + kind={"type": "recorder_started", "recorder": "mdk"}, ) ) @@ -5203,7 +5203,7 @@ def test_upload_log_list_shows_successful_and_failed_uploads(self): HTTP_X_GOGGLES_DEVICE_LABEL="MacBook", HTTP_X_GOGGLES_PLATFORM="macOS", HTTP_X_GOGGLES_APP_VERSION="1.2.3", - HTTP_USER_AGENT="DarkMatter/1.2.3", + HTTP_USER_AGENT="MDK/1.2.3", REMOTE_ADDR="203.0.113.10", ) self.assertEqual(valid_response.status_code, 201) @@ -5267,7 +5267,7 @@ def test_upload_log_list_does_not_select_raw_text(self): data=representative_audit_log(source={"account_label": "Alice"}), content_type="application/x-ndjson", HTTP_AUTHORIZATION=f"Bearer {raw_token}", - HTTP_USER_AGENT="DarkMatter/1.2.3", + HTTP_USER_AGENT="MDK/1.2.3", REMOTE_ADDR="203.0.113.10", ) self.assertEqual(upload_response.status_code, 201)