Skip to content

Add a 'How Dayflow figured this out' transparency panel#297

Open
caezium wants to merge 1 commit into
JerryZLiu:mainfrom
caezium:feature/card-transparency
Open

Add a 'How Dayflow figured this out' transparency panel#297
caezium wants to merge 1 commit into
JerryZLiu:mainfrom
caezium:feature/card-transparency

Conversation

@caezium

@caezium caezium commented Jul 3, 2026

Copy link
Copy Markdown

What this adds

A "How Dayflow figured this out" panel, shown inline in each activity card's detail (collapsed by default). It makes the AI categorization transparent — useful for trust in an always-on screen recorder, and for debugging odd cards. For each card it surfaces:

  • The model's reasoning for how it grouped/categorized the session.
  • The on-screen observations it wrote, each timestamped and paired with a downsampled screenshot from that moment — so you can see what the model actually saw.
  • Which model/provider generated the card.
  • An optional raw LLM-call view (operation, status, latency, response).

How it works

  • Reasoning is persisted in the timeline_cards metadata JSON at save time — no schema migration; it rides the existing envelope alongside appSites. For older cards (pre-this-change), it falls back to parsing the logged llm_calls, so the panel is populated retroactively too.
  • Observations, screenshots, and raw calls are read from already-persisted data by batchId. No new tables.
  • Screenshot decoding is downsampled via ImageIO and loaded off the main thread.

Tests

Unit tests for the reasoning extraction (CardInsightParsingTests) covering bare JSON, code-fenced JSON, and the OpenAI / Gemini envelope shapes, plus graceful nils.

Notes

  • Branch is based on current main (v1.14.1); the app builds and the tests pass.
  • Screenshots: the per-observation screenshots are the most compelling part ("see what it saw"), but if you'd prefer to land a text-only version first for privacy reasons, that split is easy — happy to adjust.
  • I'll add a screenshot of the panel to this PR.

Surfaces how each activity card was produced, inline in the card detail
(collapsed by default):
- the model's reasoning for grouping/categorizing the session
- the on-screen observations it wrote, each with a downsampled
  screenshot from that moment, so you can see what it actually saw
- which model/provider generated the card
- an optional raw LLM-call view (operation, status, latency, response)

Reasoning is persisted in the timeline_cards metadata JSON at save time
(no schema migration — it rides the existing envelope alongside
appSites), and falls back to parsing the logged llm_calls for older
cards. Everything else reads already-persisted observations + llm_calls
by batchId.

Includes unit tests for the reasoning extraction (bare/fenced JSON and
OpenAI/Gemini envelope shapes).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant