PathCowork

PathCowork: Shadowing Pathologists to Learn Diagnostic and Research Reasoning from Gaze, Voice, and Interaction

What Is PathCowork

PathCowork is a multimodal platform that lets AI "shadow" expert pathologists during whole-slide image review, so models can learn not only final diagnoses but also the diagnostic process.

Instead of training only on static slide-report pairs, PathCowork synchronizes four behavioral streams during routine work: viewport navigation, eye gaze, voice reasoning, and interaction logs (including AI-assist tool usage). These signals are processed into structured supervision, including spatial grounding, trajectories, revisitation patterns, and aligned text/thinking traces. The goal is to close the analysis-navigation gap: helping AI learn both where to look and how to reason.

Main Manuscript Figure

image-2.png is the main figure used in the manuscript.

Demo Video

Click the GIF preview below to open the MP4 demo:

Repository Structure

• viewer/: whole-slide viewer app and runtime capture entrypoint
• processors/viewport/: viewport denoising, keyframe extraction, stable navigation segments
• processors/gaze/: gaze cleaning, fixation/saccade parsing, attention grounding
• processors/voice/: offline/online ASR pipeline and transcript alignment
• processors/interaction/: interaction/tool-log normalization
• processors/mouse/: mouse trajectory processing and hover/click event extraction
• processors/revisit/: revisit detection from viewport/mouse traces (heuristic module, evolving)
• final_outputs/: converts processed segment signals into human learning material, machine datasets, and QA benchmarks
• schemas/: unified JSON contracts shared across all processors
• history/: lightweight shareable case-history bundle (seg_*_raw.json + seg_*_union.webp)
• streamlit_app.py: Streamlit debugging UI for history/segment browsing and on-demand gaze rendering
• videos/: raw and example recordings, logs, and derived artifacts
• image-2.png: main manuscript figure

Data Product Strategy

PathCowork standardizes multimodal traces into a unified event representation for agent-style training.

• Viewport output: tool-use style events (zoom, pan, inspect, peek) + stable keyframes + revisit candidates
• Gaze output: tool-use style events (highlight) aligned to viewport context
• Interaction output: tool-use style events for AI-assist or user-invoked tools
• Voice output: aligned transcript/thought text (not forced into tool-use)
• Mouse output: movement/hover/click traces supporting revisit and intent inference

See schemas/README.md and schemas/unified_event.schema.json for the common format.

Gaze Rendering Note

For IVT-IDT fixation comparison panels and manuscript-ready gaze/boundary overlays, see:

• processors/gaze/README.md
• processors/gaze/GazeData.py

By default, gaze overlays are rendered without any union.png crop fix. If a dataset has canvas-border artifacts, crop can be enabled manually via canvas_crop_insets=(top, bottom, left, right).

Streamlit Prototype Constraint Policy

For the Streamlit debugging prototype, we explicitly prefer truthful constraints over UI hacks.

• If a requested UI behavior is not supported by Streamlit natively, we state that directly.
• We do not ship fragile CSS/JS hacks as substitutes for unsupported core behavior.
• If needed, we escalate to a proper web stack (for example React + FastAPI) for richer UI controls.

Streamlit Debugger

Run:

cd /Volumes/ExternalHD/PathCowork
streamlit run streamlit_app.py

What it does:

• Auto-discovers case histories from history/
• Uses summary as the first selectable item (montage + PDF export + table)
• Shows per-segment debug view with mode buttons (Eye, Viewports, Mouse, All)
• Computes gaze artifacts on demand from segment files

History Bundle Format

history/ is designed to stay compact and shareable:

• Required per segment:
  • seg_xxxx_raw.json
  • seg_xxxx_union.webp (lossy WebP, currently q=85)
• JSON stays small and portable; image payload is compressed for Git sharing.