Skip to content

DATA_SCIENCE.md

Latest commit

 

History

History
98 lines (79 loc) · 5.37 KB

DATA_SCIENCE.md

File metadata and controls

98 lines (79 loc) · 5.37 KB
Raw

Data Science & ML in MedSift

MedSift ships a self-contained, local-first machine-learning library at medsift/ml/ plus a set of data-science skills that operate directly on the structured family archive. Everything runs on the user's machine; no health data leaves the device (see medsift/templates/AGENTS.md).

This document describes the modules, the skills that wrap them, how to run them, and the design choices behind them.

Design principles

  1. Additive, not invasive. The ML code lives in one package (medsift/ml/) and is surfaced through skills (SKILL.md + scripts/). The agent loop, channels, and providers are untouched.
  2. Graceful degradation. Heavy libraries (SHAP, spaCy/scispaCy, Snorkel, rank_bm25, rapidfuzz, sentence-transformers) are imported lazily. When one is missing, the relevant function falls back to a transparent, dependency-free implementation and reports which backend it used.
  3. Local and private. Indexes, models, and extracted entities stay on disk. External lookups, if ever needed, send only minimal non-identifying terms.
  4. Explainable by default. Predictions come with feature-level reasons in plain language, not just a score.

The medsift/ml library

Module What it does
synth.py Generates a clearly-labeled synthetic cohort and vitals series so models and demos run without any private data.
features.py Feature engineering from cohort frames and from metrics/*.jsonl time series (summaries + trend slope).
models.py RiskModel: L2-regularized logistic regression baseline and gradient boosting, with stratified cross-validation and probability calibration.
explain.py Global and per-prediction explanations via SHAP (if installed) or model coefficients / feature importances, rendered as a plain-language summary.
anomaly.py Robust z-score / IQR / EWMA for univariate series and IsolationForest for multivariate readings.
quality.py Fuzzy entity resolution (drug names), unit/date normalization, validation rules, and fuzzy de-duplication — with a per-change audit trail.
textmining.py Clinical entity extraction (conditions, medications, dosages, vitals, negations) using spaCy/scispaCy with a regex+lexicon fallback.
evaluation.py Metrics (AUROC, PR-AUC, F1, Brier), calibration tables, bootstrap confidence intervals, McNemar's test, and a two-proportion A/B test.
search.py BM25 lexical search over the archive with a TF-IDF cosine fallback.

The data-science skills

Skill Wraps Try it
clinical-entity-extraction textmining scripts/extract.py --demo
data-quality-normalizer quality scripts/normalize.py --demo
metric-anomaly-detection anomaly scripts/detect.py --demo
weak-supervision-labeler labeling functions (+ Snorkel) scripts/label.py --demo
archive-search search scripts/search.py --demo --query "blood pressure"
model-explainability models + explain scripts/explain.py --demo
experiment-eval evaluation scripts/evaluate.py --demo
risk-monitoring (ML mode) models + explain scripts/train_risk_model.py --demo then scripts/score.py --demo

Each skill folder is under medsift/skills/<skill>/ with a SKILL.md and a scripts/ directory. Skills declare their requirements in frontmatter (metadata: {"medsift":{"requires":{"bins":[...],"pips":[...]}}}); the skills loader marks a skill unavailable, with a clear hint, until its requirements are installed.

Installation

pip install -e ".[datascience]"

The lightweight skills (clinical-entity-extraction, data-quality-normalizer, archive-search, weak-supervision-labeler) run without the extra via their fallbacks; the modelling skills need it.

Modelling notes

  • Baseline first. The logistic model is the interpretable baseline; the rule-based risk-monitoring scan is an even simpler control. Gradient boosting is the stronger model. experiment-eval compares them with a McNemar test so an improvement is shown to be real, not noise.
  • Bias / variance. Generalization is checked with stratified 5-fold CV (cv_auc_mean ± cv_auc_std); regularization strength (logistic C) and tree depth control the trade-off.
  • Calibration. Probabilities are sigmoid-calibrated so a "70% risk" means roughly 70% observed frequency — important when the number drives a reminder.
  • Loss. Training optimizes log-loss; evaluation reports Brier score and a reliability table in addition to ranking metrics (AUROC/PR-AUC).

Scaling notes

The data code is written on vectorized pandas/numpy operations, so the same feature and quality transforms map onto Spark DataFrames (or pandas-on-Spark) for large corpora. archive-search mirrors an Elasticsearch index: each document add is an index op and each query a match query, so the local index can be swapped for an ELK backend without changing the skill interface.

Privacy

All processing is local. Do not send archive text, extracted entities, embeddings, model files, or predictions to external services. Network lookups, if unavoidable, must carry only minimal non-identifying terms, per medsift/templates/AGENTS.md.

Note: the synthetic data generators exist purely to make the models runnable and reproducible. Synthetic records must never be presented as real patients.