Data Inspection Pipeline

Pipeline for extracting activity intervals, computing HR/HRV metrics, and generating baseline comparisons from ADL logs plus ECG/PPG/HR signals.

Install

python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt

Quick start

1. Copy the template config:

copy config.example.yaml config.yaml

2. Edit config.yaml with your data paths and settings.
3. Run the pipeline:

python run_inspection.py --config config.yaml

Configuration

Key fields in config.yaml:

• data.adl_path - ADL CSV with start/end events
• data.ecg_path - ECG/PPG/HR signal CSV (or folder of CSV.GZ)
• data.imu_paths - optional dict mapping sensor_name -> path (if omitted, pipeline auto-discovers IMU sensors in subject folder)
• data.eda_bioz_path - optional corsano_bioz_bioz file/folder path (if omitted, pipeline auto-discovers in subject folder)
• signal.signal_type - ecg, ppg, or hr
• activities.* - keywords and minimum durations
• activities.extra - custom activities (e.g., washing_hands) to extract and compute metrics

You can also generate a template via:

python run_inspection.py --config config.yaml --create-config

Batch processing (optional)

Batch processing expects a dataset folder with subject subdirectories. Set the base path via environment variable:

set DATA_BASE_PATH=D:\path\to\dataset
python batch_process_subjects.py

See BATCH_PROCESSING_GUIDE.md for details.

Notebooks

• analysis_notebook.ipynb - single-subject exploration
• multi_subject_analysis.ipynb - batch analysis

Outputs

Results are written to output/ (single run) or output_batch/ (batch runs). Each run includes:

• propulsion_activities.csv, resting_activities.csv
• propulsion_hr_metrics.csv, resting_hr_metrics.csv
• baseline_activity_comparisons.csv
• pipeline_summary.csv

Optional diagnostics and window overlap reports are created when enabled and data is available.

(Optional Stage 2) ML report visualisations

Use visualize_ml_run_reports.py to generate figures from one output scope at a time.

Single training run:

python visualize_ml_run_reports.py --mode single --run-name hybrid_icf_20260306_234802

Cross-validation report directory (for example LOSO output):

python visualize_ml_run_reports.py --mode cv --cv-report-dir output_ml/tuning/hybrid_tuning_v1/trial_reports/cv_loso

Tuning study leaderboard:

python visualize_ml_run_reports.py --mode tuning --tuning-dir output_ml/tuning/hybrid_tuning_v1

Notes:

• --mode auto (default) attempts single-run first, then CV, then tuning.
• Add --maximize-objective in tuning mode when your objective is higher-is-better (for example val_qwk).
• Figures are written to output_ml/figures/ under mode-specific folders.

Hyperparameter tuning (random search)

You can run multi-trial tuning on top of ml_train_hybrid.py using:

python ml_tune_hybrid.py \
	--base-config experiment_config.example.yaml \
	--study-name hybrid_tuning_v1 \
	--trials 20 \
	--epochs 12 \
	--objective val_mae

Notes:

• --objective can be val_mae, val_rmse, val_ordinal_mae, or val_qwk.
• Lower is better for MAE/RMSE/ordinal MAE. Higher is better for QWK.
• The tuner samples model/loss/optimizer parameters and launches ml_train_hybrid.py once per trial.

Outputs are written to output_ml/tuning/<study_name>/:

• leaderboard.csv: one row per trial, objective, sampled hyperparameters, and status.
• best_config.yaml: copy of the best trial config.
• best_summary.txt: quick best-trial summary.
• trial_configs/: generated per-trial config files.
• trial_reports/: evaluation outputs generated by each training run.
• trial_checkpoints/: checkpoints generated by each training run.

Feature selection workflow

Use ml_feature_selection.py to run a combined filter + wrapper feature selection pass.

From an existing training table (defaults to output_ml/training_table.csv):

python ml_feature_selection.py --target-cols target_score

Directly from a batch output (builds table first):

python ml_feature_selection.py \
	--build-from-batch \
	--batch-dir output_batch/batch_20260310_145652 \
	--icf-csv C:/Users/Nicla/Documents/ETHZ/Lifelogging/Data/ICF_scores_nursing_home.csv \
	--target-cols "Basic Movements,Walking,Oral Care,Grooming"

Outputs are written to output_ml/feature_selection/ by default:

• feature_target_correlations.csv
• feature_scores_aggregated.csv
• high_collinearity_pairs.csv
• correlation_filter_selected_features.csv
• wrapper_forward_selection_trace.csv
• wrapper_backward_elimination_trace.csv
• training_table_selected_features.csv
• feature_selection_summary.json