carwatch for Python - Tools for processing CARWatch and saliva data

This Python package supports the processing of sampling logs recorded by CARWatch and their integration with laboratory biomarkers. It is designed for ambulatory sampling studies in which researchers need auditable sampling times, protocol deviations, manual diary fallbacks, and biomarker features.

The package reads the app log exports, reconstructs each study day, compares planned and recorded sample times, lets you review unclear records, and combines the result with laboratory data, such as cortisol. The original app exports are never changed. The package creates a separate table of questions and decisions when information is missing or inconsistent. This makes it possible to rerun the same analysis later and understand how each decision was made.

The features include:

• Import CARWatch logs from CSV files, ZIP archives, and participant folders
• Reconstruct expected study days, sampling positions, and timing compliance
• Create and reload a review table for conversion issues
• Review conversion anomalies in a structured, editable two-pass issue report
• Patch missing timestamps from manual measurement diary
• Read and write complete Study Results CSV files
• Merge laboratory saliva measurements by physical sample ID or scheduled sample position
• Correct documented tube swaps
• Compute cortisol response features and static quality-control plots

The examples directory contains end-to-end notebooks and focused gallery workflows. The user guides explain individual parts of the workflow in more depth.

Installation

CARWatch requires Python 3.10 or newer. Install it from PyPI in an existing Python environment:

pip install carwatch

With uv:

uv add carwatch

The beginner setup tutorial explains how to install uv, create an isolated CARWatch analysis environment, install the package, and run Jupyter notebooks on macOS, Linux, and Windows.

Install the optional Jupyter decision editor and interactive plots with:

uv add "carwatch[interactive]"

Install the current development version directly from GitHub with:

uv add "carwatch @ git+https://github.com/carwatch-tools/carwatch-python.git"

The main branch can contain unreleased changes. Development setup is documented under For developers.

Typical workflow

Workflow at a glance

CARWatch keeps data import, researcher decisions, and analysis as separate steps:

CARWatch app exports
        |
        v
Import logs and check which files were used
        |
        v
First conversion: reconstruct the study and create a review report
        |
        v
Review the reported issues and record your decisions
        |
        v
Final conversion: create complete Study Results
        |
        +----> inspect timing and compliance
        +----> save and reload the processed study
        +----> merge laboratory data and compute saliva features (e.g., AUC, max increase, slope)

The two conversion passes are intentional. The first pass shows how CARWatch understands the app logs and identifies records that need review. The second pass applies the submitted decisions to the original logs. The app exports remain unchanged throughout.

Basic concepts

Term	Meaning
Raw logs (raw_logs)	The events exported by the CARWatch app, such as saved study settings, awakening events, and barcode scans. These are the original input data.
File-import log (source_audit)	A table showing which CSV or ZIP files were used or skipped and why.
Registration	A set of study settings saved in the app for a participant: study name, number of days, sample IDs in their planned order, and sampling times. Saving a changed setup creates another registration.
Protocol	The intended order of registrations, study days, and samples across the whole study. CARWatch normally reconstructs it from the registrations found in the app logs.
Protocol manifest (protocol_manifest)	An optional Python list in which you state the intended protocol order explicitly. Most studies do not need one. It is useful when the available app logs do not establish one unambiguous order.
Conversion report	The review table produced by the first conversion. It describes missing or inconsistent information and contains suggested decisions. Suggestions are not applied automatically.
Manual diary	A table of awakening and sampling times that were written down outside the app (e.g., on paper). This can be used as fallback information when the app logs are incomplete. It is used only when a decision explicitly sets the source of an awakening or sampling time to "manual diary" because the app logs are missing or inconsistent. The manual diary is not used automatically.
Scheduled sample	The tube expected at a particular position according to the registration.
Recorded sample	The tube actually scanned in the app. It may differ from the scheduled tube, for example after a tube swap.
Sample position	The first, second, third, and so on sampling position defined by the order saved in the registration. CARWatch does not derive this from the spelling of a tube ID or file name.
Study Results (study_results)	The final processed study data. They contain study days, planned and recorded sampling times, compliance, information about where each time came from, and later any merged laboratory values.

Detailed workflow

1. Load raw CARWatch logs

For one folder per participant, provide the participant IDs explicitly. Folders can contain nested CSV exports and ZIP archives.

from pathlib import Path

import carwatch as cw

participant_folders = {
    "vp01": Path("data/carwatch/vp01"),
    "vp02": Path("data/carwatch/vp02"),
}

raw_logs, source_audit = cw.io.load_raw_logs_from_participant_folders(
    participant_folders,
    create_report=True,
)

source_audit records which files CARWatch used or skipped and why. When the exact sources are already known:

raw_logs = cw.io.load_raw_logs(
    ["data/carwatch/vp01.csv", "data/carwatch/vp02.zip"]
)

2. Create a first review report

The first conversion creates provisional results and a list of records that need review. It also calculates planned sample times and timing compliance.

initial_results, conversion_report = (
    cw.logs.convert_raw_logs_to_study_manager_summary(
        raw_logs,
        errors="warn",
        create_report=True,
    )
)

print(conversion_report["summary"])
conversion_report["issues"].to_csv("conversion_issues.csv")

The suggested decisions are recommendations. Nothing is corrected until an edited report is submitted in the final conversion.

If CARWatch cannot determine the protocol order

Inspect the reconstructed protocol with:

cw.logs.summarize_protocol(raw_logs)
cw.logs.extract_registration_schedule_from_raw_logs(raw_logs)

Most studies do not need a protocol_manifest. Create one only when the logs cannot establish the intended order reliably, for example because participants completed registrations in conflicting orders, one registration is absent, or only incomplete participant records are available.

protocol_manifest = [
    {
        "study_name": "Control",
        "study_days": 2,
        "saliva_ids": ["control-1", "control-2", "control-3", "control-4"],
        "saliva_times": [0, 30, 15, 15],
        "saliva_absolute_times": [],
    },
    {
        "study_name": "Challenge",
        "study_days": 2,
        "saliva_ids": [
            "challenge-1",
            "challenge-2",
            "challenge-3",
            "challenge-4",
        ],
        "saliva_times": [0, 30, 15, 15],
        "saliva_absolute_times": [],
    },
]

initial_results, conversion_report = (
    cw.logs.convert_raw_logs_to_study_manager_summary(
        raw_logs,
        protocol_manifest=protocol_manifest,
        errors="warn",
        create_report=True,
    )
)

The list order defines registration order. study_days defines the number of days in each registration. saliva_ids defines planned tube order, while saliva_times contains relative timing intervals in minutes. Fixed clock times belong in saliva_absolute_times. Pass the same manifest to the first and final conversion and to the interactive editor.

3. Review and apply decisions

Resolve the reported issues in a spreadsheet or in the optional Jupyter editor. Both routes produce the decisions table used by the final conversion.

Load the manual diary when report decisions use it as a fallback. Omit this line and the manual_diary arguments below when the study has no accepted manual-diary decisions.

manual_diary = cw.io.load_manual_diary("manual_diary.csv")

Option A: edit the report in a spreadsheet. Open the CSV, choose a decision for every issue, and save it without changing identifying columns. Then reload it:

decisions = cw.logs.load_conversion_issue_report("conversion_issues.csv")

Option B: resolve issues interactively. This requires the interactive extra.

from IPython.display import display

editor = cw.logs.interactive_conversion_issue_report(
    raw_logs,
    conversion_report["issues"],
    manual_diary=manual_diary,
)
display(editor.widget)

# After completing and refreshing the issue queue:
decisions = editor.decisions
decisions.to_csv("conversion_issues.csv")

Select an issue, choose a valid decision, and apply it. Use Refresh remaining issues to rerun conversion with decisions made so far. Accepted upstream decisions remain in the history while the visible queue is replaced by issues that still need attention. If a diary-backed decision cannot be applied, that issue is reset to Leave unresolved and remains visible while successfully applied decisions are hidden.

The editor offers only decisions that are valid for the selected issue:

Decision	Effect	Decision value
accept	Apply the suggested action shown in proposed_action. Read its description before accepting because the effect depends on the issue.	Not required.
keep	Mark the issue as reviewed without applying the suggested correction. The current reconstruction, including any missing value, is retained.	Not required.
drop_sample	For an issue tied to a sample, clear that sample from the participant's Study Results while retaining the planned sample position.	Not required.
drop_day	For an issue tied to a study day, clear the complete participant-day from Study Results.	Not required.
drop_participant	Remove the participant associated with the issue from Study Results.	Not required.
override_expected_sample	Assign a scan whose expected sample cannot be resolved to another sample in the active registration. This option appears only for the corresponding issue type.	The exact registered sample ID to use.
change	Apply an alternative, issue-specific correction instead of the proposed action. This option appears only when the issue supports it.	Required; the allowed value depends on the issue, as shown below.

When change is available, user_decision_value supports these values:

Issue	Allowed value for change
Multiple collection dates	use_earliest_collection_date, use_latest_collection_date, or a complete JSON mapping such as {"2026-01-05":"D1","2026-01-06":"D2"}.
Possible re-registration	A JSON target such as {"registration":2} or {"registration":"Challenge"}.
Sampling times are not increasing	sort_samples_by_time.
Missing awakening time	use_manual_diary_awakening_time, or an explicit local timestamp such as 2026-01-05 07:10.
Missing scheduled sample	use_manual_diary_sampling_time or use_default. The latter reconstructs the time from the registered schedule or supplied sampling_schedule.

Run the final conversion against the original logs:

study_results, final_report = (
    cw.logs.convert_raw_logs_to_study_manager_summary(
        raw_logs,
        errors="raise",
        create_report=True,
        issue_decisions=decisions,
        manual_diary=manual_diary,
    )
)

Omit manual_diary when no accepted decision uses it. If the first pass used a protocol_manifest or sampling_schedule, pass the same object to the editor and final conversion. Strict conversion stops if a submitted decision is invalid or an issue remains unresolved.

4. Inspect and save complete Study Results

study_results contains the complete information needed for later merging, analysis, and plotting. Helper functions provide focused study-day and sample tables for inspection.

study_days = cw.logs.extract_day_summary_from_summary(study_results)
sample_events = cw.logs.extract_sample_events_from_summary(study_results)
cw.io.save_study_results(study_results, "study_results.csv")

analysis_results = cw.compliance.drop_non_compliant_samples(study_results)
cw.compliance.summarize_compliance(study_results)
cw.compliance.find_non_compliant_samples(study_results)

The study-day table includes collection dates, awakening information, registration context, and day compliance. The sample table includes planned and recorded times, sample positions, tube IDs, timing deviations, and sample compliance. Filtering preserves the complete wide Study Results structure.

5. Restore results or import a Study Manager export

study_results = cw.io.load_study_results("study_results.csv")
display_results = cw.io.load_study_results("study_results.csv", simple=True)
study_results = cw.io.load_study_manager_export("study_manager_export.csv")

Use simple=True only for a compact table to inspect or share. Keep the default complete form for merging, analysis, compliance checks, and plotting.

6. Load laboratory saliva data

For matching by physical tube ID, use a long CSV with participant, sample, and one biomarker column:

participant,sample,cortisol
vp01,tube-a,8.2
vp01,tube-b,12.6

saliva = cw.io.load_saliva("cortisol.csv", saliva_type="cortisol")

For position-based laboratory data, create an index containing participant, canonical day, and sample position. Additional day-level labels such as condition can be retained as named index levels.

import pandas as pd

saliva_by_position = (
    pd.read_csv("cortisol_by_position.csv")
    .set_index(["participant", "day", "sample_position", "condition"])
)

7. Merge sampling and laboratory data

Recorded physical IDs correct documented swaps by default. Set correct_swaps=False to match scheduled IDs instead.

merged_results = cw.merge.merge_saliva(
    study_results,
    saliva,
    match_on="sample",
    correct_swaps=True,
)

merged_results = cw.merge.merge_saliva(
    study_results,
    saliva_by_position,
    match_on="position",
)

The result remains complete Study Results. It records whether a laboratory value was found, which tube was used, and whether a documented swap was corrected.

8. Compute features and quality-control plots

The CARWatch feature adapter groups by participant and day, orders samples by sample_position, and uses actual sampling times.

cortisol_features = cw.saliva.compute_features_from_carwatch(
    merged_results,
    saliva_type="cortisol",
)

The sampling timeline compares protocol targets, app-updated targets, and recorded collection times for one participant-day. Arrows show signed timing deviation; color indicates compliance and marker shape identifies the source of the recorded time.

fig, ax = cw.plotting.plot_sampling_timeline(
    study_results,
    participant="VP_01",
    day="D1",
)

The compliance overview summarizes compliant, non-compliant, and unassessed samples at each sampling position.

fig, ax = cw.plotting.plot_compliance_overview(study_results)

The deviation plot shows how early or late samples were collected at each sampling position. Individual points remain visible behind the boxplots.

fig, ax = cw.plotting.plot_timing_deviation(study_results)

The saliva curve retains individual participant-day trajectories and adds the mean response with its confidence interval.

fig, ax = cw.plotting.plot_saliva_curve(
    merged_results,
    value="cortisol",
    group_by="condition",
)

The figures are generated from deterministic synthetic data with:

uv run python tools/generate_readme_figures.py

The Python and R figure generators use the same study configuration, 40 participants, anomaly ratios, random seed, bootstrap settings, figure sizes, and resolution.

For generic long-format saliva data, use compute_features(), auc(), max_value(), initial_value(), max_increase(), or slope().

Optional interactive review

In Jupyter, launch a participant/day selector around the same static timeline:

from IPython.display import display

timeline_widget = cw.plotting.interactive_sampling_timeline(study_results)
display(timeline_widget)

Synthetic data

Generate deterministic local example data without changing real study files:

study_root = cw.example_data.generate_synthetic_study_data(
    "carwatch-example",
    n_participants=4,
    random_state=42,
    non_compliant_sample_ratio=0.10,
    missing_awakening_time_ratio=0.01,
    missing_sampling_time_ratio=0.02,
    create_cortisol_data=True,
)

The generated directory contains raw logs, manual_diary.csv, a ready-to-submit issue_decisions.csv, and position-indexed cortisol.csv when requested.

For developers

Install uv, clone the repository, and synchronize the project environment:

git clone https://github.com/carwatch-tools/carwatch-python.git
cd carwatch-python
uv sync

Run the relevant checks after changing the package or documentation:

uv run poe format
uv run poe ci_check
uv run poe test
uv run poe docs_clean

Use uv run poe docs_preview to inspect the built documentation. Dependencies are managed through pyproject.toml. Package invariants and task-specific guidance are documented in AGENTS.md, the repository-local skills, and the agent-assisted workflow guide.

Contributing

Bug reports, feature requests, and reproducible examples belong in the GitHub issue tracker. Changes should include tests and documentation for the affected research workflow.

Citation

Report the package version used in an analysis. For research using CARWatch, cite:

Richer, R., Abel, L., Küderle, A., Eskofier, B. M., & Rohleder, N. (2023). CARWatch — A smartphone application for improving the accuracy of cortisol awakening response sampling. Psychoneuroendocrinology, 151, 106073. https://doi.org/10.1016/j.psyneuen.2023.106073

The installed package version is available as:

import carwatch

print(carwatch.__version__)

License

CARWatch for Python is available under the MIT License.