Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Internal Marmot audit-log explorer.

Goggles accepts sensitive `marmot-forensics-audit` JSONL audit logs from Dark Matter clients (current schema `marmot-forensics-audit/v2`; legacy `v1` is still accepted), preserves the exact uploaded text and raw lines, normalizes common forensic columns into PostgreSQL tables, and gives the team a login-gated dashboard for comparing what multiple account-device engines saw and decided inside each group.
Goggles accepts sensitive `marmot-forensics-audit` JSONL audit logs from MDK clients (current schema `marmot-forensics-audit/v2`; legacy `v1` is still accepted), preserves the exact uploaded text and raw lines, normalizes common forensic columns into PostgreSQL tables, and gives the team a login-gated dashboard for comparing what multiple account-device engines saw and decided inside each group.

The audit event schema is committed at `docs/schemas/audit-log-event.v2.schema.json`. See also `docs/api-v1.md` for the authenticated read API, `docs/deployment.md` for VM deployment notes, and `docs/audit-debugging-platform-prd.md` for the platform's product requirements.

Expand Down
2 changes: 1 addition & 1 deletion docs/api-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ the convergence-decision evidence refs that produced those child rows.
`message_id` filters convergence runs by selected branch id, losing branch id,
candidate branch id, or candidate commit ids. `epoch` filters by run-level,
candidate-level, and evidence-level convergence epoch fields.
When Dark Matter does not emit a stable convergence `run_id`, Goggles groups
When MDK does not emit a stable convergence `run_id`, Goggles groups
contiguous convergence and epoch-state evidence for the same group engine into
an inferred run and sets `inferred` to `true`.
If multiple engines have the same run id, the detail endpoint returns `409` with
Expand Down
32 changes: 16 additions & 16 deletions docs/audit-debugging-platform-prd.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Status: Draft
Date: 2026-06-24
Audience: Dark Matter, Goggles, and client application engineers
Audience: MDK, Goggles, and client application engineers

## Summary

Expand All @@ -27,12 +27,12 @@ content and transport wire identifiers.

## Background

Dark Matter audit logging currently writes sensitive JSONL records using
MDK audit logging currently writes sensitive JSONL records using
`schema_version = "marmot-forensics-audit/v1"`. Goggles ingests those files,
preserves exact raw text, normalizes common fields into database columns, and
renders group-level dashboards.

The latest Dark Matter audit changes add richer group and epoch breadcrumbs:
The latest MDK audit changes add richer group and epoch breadcrumbs:

- `epoch_state_changed` records engine epoch-state transitions such as stable,
pending publish, recovering, and unrecoverable.
Expand All @@ -45,7 +45,7 @@ The latest Dark Matter audit changes add richer group and epoch breadcrumbs:
attribution on every row.

These changes are necessary but not sufficient. Goggles needs new product
surfaces, derived data models, and programmatic APIs. Dark Matter and client
surfaces, derived data models, and programmatic APIs. MDK and client
libraries may also need additional audit events so Goggles can explain branch
selection, transport gaps, and client/device behavior with confidence.

Expand Down Expand Up @@ -90,7 +90,7 @@ selection, transport gaps, and client/device behavior with confidence.
## Audit Data Modes

The audit recorder should expose a two-option data-mode setting. The selected
mode must be stored in Dark Matter settings, stamped into recorder or session
mode must be stored in MDK settings, stamped into recorder or session
context, visible in Goggles, included in API responses, and auditable when it
changes.

Expand Down Expand Up @@ -189,7 +189,7 @@ Important distinction:

Question: What did the convergence engine see, what did it choose, and why?

The view should organize events by convergence run. Once Dark Matter emits a
The view should organize events by convergence run. Once MDK emits a
stable `convergence_run_id`, that id should define the run. Before then, Goggles
can infer provisional runs from contiguous convergence and epoch-state events
with the same engine id and group ref. An inferred run begins when an engine
Expand All @@ -214,7 +214,7 @@ Each run should show:
- resulting epoch-state transition;
- raw evidence links for every row.

Current Dark Matter data can show high-level decisions through
Current MDK data can show high-level decisions through
`convergence_decision`, `error_kinds`, and `epoch_state_changed`. It likely does
not yet expose enough candidate-level detail to fully explain branch selection.

Expand Down Expand Up @@ -312,10 +312,10 @@ projection model:
- Update fixtures and parser tests for the V2 JSON Schema.
- Include derived projections in agent export and API output.

## Dark Matter Audit Data Gaps
## MDK Audit Data Gaps

The latest audit changes are a strong foundation, but Goggles probably needs
additional Dark Matter events for complete explainability.
additional MDK events for complete explainability.

A first draft of the proposed V2 event-line schema is in
[`docs/schemas/audit-log-event.v2.schema.json`](schemas/audit-log-event.v2.schema.json).
Expand Down Expand Up @@ -480,9 +480,9 @@ API behavior:

Phase 0: Alignment

- Review this PRD with Dark Matter, Goggles, and client owners.
- Review this PRD with MDK, Goggles, and client owners.
- Turn P0 items into issues.
- Decide which Dark Matter schema gaps are required before UI redesign.
- Decide which MDK schema gaps are required before UI redesign.

Phase 1: Greenfield Storage And Ingest

Expand Down Expand Up @@ -510,15 +510,15 @@ Phase 3: UI Redesign

Phase 4: Producer Gaps

- Add Dark Matter audit settings and schema support for obfuscated versus full
- Add MDK audit settings and schema support for obfuscated versus full
data auditing.
- Add missing Dark Matter audit events for convergence branch explainability.
- Add missing MDK audit events for convergence branch explainability.
- Add TypeScript/client audit-log parity.
- Validate emitted logs against the committed schema.

## Decisions From Initial Review

- Convergence runs should eventually be keyed by a Dark Matter emitted
- Convergence runs should eventually be keyed by an MDK-emitted
`convergence_run_id`. Until then, Goggles may infer provisional runs from
contiguous convergence and epoch-state rows with the same engine id and group
ref, and must label those runs as inferred.
Expand Down Expand Up @@ -566,7 +566,7 @@ Phase 4: Producer Gaps

## Remaining Open Questions

- What exact schema names and field shapes should Dark Matter use for
- What exact schema names and field shapes should MDK use for
convergence rule outputs, expected-recipient sets, and full data auditing
message contents?
- Which derived projection tables and indexes are required for the first fast
Expand All @@ -578,7 +578,7 @@ Phase 4: Producer Gaps

- Inferred missing transport events may be mistaken for proven non-delivery.
- System-stamped audit rows may overwhelm human-action views if not separated.
- Branch explainability will be incomplete unless Dark Matter emits candidate
- Branch explainability will be incomplete unless MDK emits candidate
details, not just final decisions.
- Rich APIs may accidentally expose sensitive forensic data too broadly without
tight authentication and careful response design.
Expand Down
2 changes: 1 addition & 1 deletion fixtures/sample-audit-log-acme-dana.jsonl
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730086400000,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"obfuscated_sensitive_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{"source":{"account_label":"Dana Whitfield","device_name":"MacBook Pro","platform":"macos"}},"kind":{"type":"recorder_started","recorder":"darkmatter"}}
{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730086400000,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"obfuscated_sensitive_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{"source":{"account_label":"Dana Whitfield","device_name":"MacBook Pro","platform":"macos"}},"kind":{"type":"recorder_started","recorder":"mdk"}}
{"schema_version":"marmot-forensics-audit/v2","seq":1,"wall_time_ms":1730086400120,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"full_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{},"kind":{"type":"audit_data_mode_changed","previous_mode":"obfuscated_sensitive_data","new_mode":"full_data","reason":"forensic_capture_enabled","recorder_restarted":true}}
{"schema_version":"marmot-forensics-audit/v2","seq":2,"wall_time_ms":1730086400240,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"full_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{},"kind":{"type":"source_context","source":{"account_label":"Dana Whitfield","account_pubkey_hex":"a773887e1170c9c8bc47e0c6a574fb0d9cbb7ea8ddc14a0b345674d4fa40bc0f","device_name":"MacBook Pro","platform":"macos"}}}
{"schema_version":"marmot-forensics-audit/v2","seq":3,"wall_time_ms":1730086401440,"recorder_session_id":"recorder-8be53cef","audit_data_mode":"full_data","account_ref":"5299b84c07796db0995edb1fe2ea0b7b","engine_id":"8be53cefb9219687e353aa0c07778877","group_ref":"3571276c8a3b50cd550cd1783520cd83038e65fc614462c9a3d7c2f8b2fd804f","context":{},"kind":{"type":"message_content_decoded","msg_id":"4818733d5badb40bfb46cd744814e96ea21ed78f2a96b851a21458dee8abfd52","artifact_kind":"application_message","author":{"member_ref":"5299b84c07796db0995edb1fe2ea0b7b","account_pubkey_hex":"a773887e1170c9c8bc47e0c6a574fb0d9cbb7ea8ddc14a0b345674d4fa40bc0f"},"decoded_payload":{"content_type":"text/plain","text":"Standup in 5. Drop blockers in the thread."},"decoded_app_event":{"format":"nostr","kind":9,"content":"Standup in 5. Drop blockers in the thread.","pubkey_hex":"a773887e1170c9c8bc47e0c6a574fb0d9cbb7ea8ddc14a0b345674d4fa40bc0f"}}}
Expand Down
2 changes: 1 addition & 1 deletion fixtures/sample-audit-log-trailhead-maya.jsonl
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730000000000,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"obfuscated_sensitive_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{"source":{"account_label":"Maya Okonkwo","device_name":"iPhone 15 Pro","platform":"ios"}},"kind":{"type":"recorder_started","recorder":"darkmatter"}}
{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730000000000,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"obfuscated_sensitive_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{"source":{"account_label":"Maya Okonkwo","device_name":"iPhone 15 Pro","platform":"ios"}},"kind":{"type":"recorder_started","recorder":"mdk"}}
{"schema_version":"marmot-forensics-audit/v2","seq":1,"wall_time_ms":1730000000120,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"full_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"audit_data_mode_changed","previous_mode":"obfuscated_sensitive_data","new_mode":"full_data","reason":"forensic_capture_enabled","recorder_restarted":true}}
{"schema_version":"marmot-forensics-audit/v2","seq":2,"wall_time_ms":1730000000240,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"full_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"source_context","source":{"account_label":"Maya Okonkwo","account_pubkey_hex":"0096d5e0a3a8479e8a72d04f4020eab2bfc9796a47a07ba938acb76a93b4f246","device_name":"iPhone 15 Pro","platform":"ios"}}}
{"schema_version":"marmot-forensics-audit/v2","seq":3,"wall_time_ms":1730000000720,"recorder_session_id":"recorder-cc0658eb","audit_data_mode":"full_data","account_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","engine_id":"cc0658ebd36118bbc64873aa5932ca19","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"message_content_decoded","msg_id":"30f6d1e85e6710f17dddc4aca3cd06a0e1338f197cd6e364a1140eeecbae6dee","artifact_kind":"application_message","author":{"member_ref":"e47a8cf4fcf0cf34ce1da2b01f2fbf64","account_pubkey_hex":"0096d5e0a3a8479e8a72d04f4020eab2bfc9796a47a07ba938acb76a93b4f246"},"decoded_payload":{"content_type":"text/plain","text":"Trail this Saturday? Thinking Eagle Creek, 8am."},"decoded_app_event":{"format":"nostr","kind":9,"content":"Trail this Saturday? Thinking Eagle Creek, 8am.","pubkey_hex":"0096d5e0a3a8479e8a72d04f4020eab2bfc9796a47a07ba938acb76a93b4f246"}}}
Expand Down
2 changes: 1 addition & 1 deletion fixtures/sample-audit-log-trailhead-theo.jsonl
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730000000360,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"obfuscated_sensitive_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{"source":{"account_label":"Theo Almeida","device_name":"Pixel 8","platform":"android"}},"kind":{"type":"recorder_started","recorder":"darkmatter"}}
{"schema_version":"marmot-forensics-audit/v2","seq":0,"wall_time_ms":1730000000360,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"obfuscated_sensitive_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{"source":{"account_label":"Theo Almeida","device_name":"Pixel 8","platform":"android"}},"kind":{"type":"recorder_started","recorder":"mdk"}}
{"schema_version":"marmot-forensics-audit/v2","seq":1,"wall_time_ms":1730000000480,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"full_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"audit_data_mode_changed","previous_mode":"obfuscated_sensitive_data","new_mode":"full_data","reason":"forensic_capture_enabled","recorder_restarted":true}}
{"schema_version":"marmot-forensics-audit/v2","seq":2,"wall_time_ms":1730000000600,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"full_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"source_context","source":{"account_label":"Theo Almeida","account_pubkey_hex":"b20478b2a54600d0b62fc099448bbcc52c474208545bf438c9d6af9aeab689e1","device_name":"Pixel 8","platform":"android"}}}
{"schema_version":"marmot-forensics-audit/v2","seq":3,"wall_time_ms":1730000001500,"recorder_session_id":"recorder-12230e6b","audit_data_mode":"full_data","account_ref":"880f262de55f8871e03cdbaa4df0a100","engine_id":"12230e6b0a0dbd3ae48e785c16b18584","group_ref":"c877a7ea4191a38c3354968353f527957254ccd761f4c14bd4e90f9808f5a198","context":{},"kind":{"type":"transport_received","msg_id":"30f6d1e85e6710f17dddc4aca3cd06a0e1338f197cd6e364a1140eeecbae6dee","transport":{"transport":"nostr","delivery_plane":"relay","relay_url":"wss://relay.primal.net","nostr_event_id":"b43904c2a866006115391c243f68b9f885cf4447c3be460b5885a447cc9fe45b","nostr_kind":445},"payload_len":95,"payload_digest":"a0775e72269e1b375d19fa35d08ba57d63e776f0d30b368c6ab111f8e4dcea70"}}
Expand Down
2 changes: 1 addition & 1 deletion forensics/seed_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ def start(self, participant: Participant) -> None:
"""
self.emit(
participant,
{"type": "recorder_started", "recorder": "darkmatter"},
{"type": "recorder_started", "recorder": "mdk"},
mode=OBFUSCATED,
context={
"source": {
Expand Down
10 changes: 5 additions & 5 deletions forensics/tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ def audit_event_v2(
"engine_id": engine_id,
"group_ref": group_ref,
"context": context if context is not None else {"operation_id": f"op-v2-{seq}"},
"kind": kind or {"type": "recorder_started", "recorder": "darkmatter"},
"kind": kind or {"type": "recorder_started", "recorder": "mdk"},
}


Expand Down Expand Up @@ -1580,7 +1580,7 @@ def test_v2_upload_builds_audit_projections(self):
account_ref=ACCOUNT_BOB,
recorder_session_id="session-b",
context={"source": {"account_label": "Bob", "device_name": "Bob laptop"}},
kind={"type": "recorder_started", "recorder": "darkmatter"},
kind={"type": "recorder_started", "recorder": "mdk"},
)
),
content_type="application/x-ndjson",
Expand Down Expand Up @@ -2216,7 +2216,7 @@ def test_context_subobjects_must_be_objects_when_present(self):
audit_event_v2(
0,
context={"source": "alice laptop", "convergence": ["run-1"]},
kind={"type": "recorder_started", "recorder": "darkmatter"},
kind={"type": "recorder_started", "recorder": "mdk"},
)
)

Expand Down Expand Up @@ -5203,7 +5203,7 @@ def test_upload_log_list_shows_successful_and_failed_uploads(self):
HTTP_X_GOGGLES_DEVICE_LABEL="MacBook",
HTTP_X_GOGGLES_PLATFORM="macOS",
HTTP_X_GOGGLES_APP_VERSION="1.2.3",
HTTP_USER_AGENT="DarkMatter/1.2.3",
HTTP_USER_AGENT="MDK/1.2.3",
REMOTE_ADDR="203.0.113.10",
)
self.assertEqual(valid_response.status_code, 201)
Expand Down Expand Up @@ -5267,7 +5267,7 @@ def test_upload_log_list_does_not_select_raw_text(self):
data=representative_audit_log(source={"account_label": "Alice"}),
content_type="application/x-ndjson",
HTTP_AUTHORIZATION=f"Bearer {raw_token}",
HTTP_USER_AGENT="DarkMatter/1.2.3",
HTTP_USER_AGENT="MDK/1.2.3",
REMOTE_ADDR="203.0.113.10",
)
self.assertEqual(upload_response.status_code, 201)
Expand Down