Systematic data management: versioned datasets, model provenance, and reusable artifacts

[mark-e-deyoung]

Problem

WineBot generates and consumes a growing collection of datasets, model weights, and checkpoints with no systematic management. Currently stored as flat files in models/, data is lost on container restart (until recent fix), has no versioning, no provenance tracking, and no way to trace which dataset trained which model.

Current State (15+ model directories)

Directory	Contents	Versioned?	Provenance?	Reusable?
models/yolo/	YOLO v2/v3 weights (.pt)	❌	❌	⚠️ Filename only
models/wine-dataset/	Synthesized training images + labels	❌	❌	❌ Can not regenerate exactly
models/wine-dataset-real/	Real desktop screenshots + labels	❌	❌	❌ Only 30 pending
models/cross-validation/	5-fold CV results	❌	❌	❌ Lost after restart
models/eval-dataset/	Held-out evaluation split	❌	❌	❌ Lost after restart
models/state_classifier/	Trained ML state classifier (.pkl)	❌	❌	❌ Unreproducible
models/clip/	CLIP ONNX exports	❌	❌	❌ Must re-download
models/frame_index/	CLIP semantic search index	❌	❌	❌ Must rebuild
models/florence2/	Fine-tuned captioning LoRA adapters	❌	❌	❌ Can not regenerate exactly
models/tessdata/	Tesseract language data	❌	❌	✅ Static
models/screenparser/	ScreenParser weights	❌	❌	❌ Can not regenerate
models/huggingface/	HF cached downloads	❌	❌	✅ Auto-downloaded
models/annotations/	Bounding box labels	❌	❌	❌ Lost on container restart

Requirements

Priority	Requirement	Why
P0	Dataset versioning	Reproduce any training run exactly
P0	Provenance/lineage	Know which data + code → which model
P0	Self-hosted	Data stays on TrueNAS, no cloud dependency
P0	Open source (Apache 2/MIT)	No licensing costs
P0	REST or programmatic API	Sidecar, WinBot, and scripts need access
P1	Upload/download API	Push training results, pull for inference
P1	Multiple collections	Separate GT, YOLO, CLIP, classifiers
P1	Metadata + search	Find datasets by description, date, params
P2	Access control	WineBot vs WinBot permissions
P2	Web UI	Browse without API calls

Platforms Evaluated

Platform	License	API	Versioning	Provenance	Self-host
DVC	Apache 2	CLI + Python	✅ Git-like	✅ DAG	✅ S3/minio
LakeFS	Apache 2	REST S3	✅ Git-like	✅ Hooks	✅ Kubernetes
Dolt	Apache 2	SQL+HTTP	✅ SQL Git	✅ Diff	✅ Single binary
MLflow	Apache 2	REST	⚠️ Logged	✅ Runs	✅ Docker
Quilt	Apache 2	REST	✅ Packages	✅ Metadata	✅ Docker

Recommendation: DVC (Primary) + Dolt (Adjunct)

DVC for ML artifacts: version datasets, YOLO weights, checkpoints, CLIP indexes, state classifiers.
Dolt for structured metadata: wine class taxonomy, annotation catalogs, experiment configs.

Phased Bring-Up

Phase 1 (Day 1): Initialize

• pip install dvc && dvc init
• dvc remote add truenas s3://models --endpoint-url https://truenas.fritz.box:9000
• Configure MinIO or use TrueNAS S3-compatible endpoint

Phase 2 (Day 1-2): Version existing artifacts

• dvc add models/yolo/wine-finetuned-v3.pt
• dvc add models/wine-dataset/
• dvc add models/state_classifier/
• git add *.dvc && git commit && dvc push

Phase 3 (Week 1): Pipeline provenance

• Replace raw training scripts with dvc run:

  dvc run -n train_yolo_v3 \
    -d models/wine-dataset/ \
    -d scripts/train_yolo.py \
    -o models/yolo/wine-finetuned-v3.pt \
    python3 scripts/train_yolo.py

• dvc dag now shows lineage: dataset → training → weights
• dvc reproduce train_yolo_v3 re-runs with same data

Phase 4 (Week 1): Structured metadata with Dolt

• dolt init && dolt table import docs/wine_classes.csv
• Version annotation catalogs alongside image data

Phase 5 (Week 2): CI/CD integration

• CI pulls DVC-tracked data before test runs
• CI commits and pushes new weights after training pipeline
• Dataset version = git commit; model version = git tag

Phase 6 (Week 2+): Evaluate MLflow for model serving

• If WinBot needs HTTP model serving, add MLflow on top of DVC storage
• MLflow registers model versions, serves via REST API

What Success Looks Like

# A colleague can reproduce any model at any tag:
git checkout v0.9.8
dvc checkout  # pulls exact dataset version
dvc reproduce train_yolo_v3  # re-runs training

# See full provenance:
dvc dag
#   +---------------------+
#   | wine-dataset.dvc    |
#   +---------------------+
#             |
#             v
#   +---------------------+
#   | train_yolo_v3       |
#   +---------------------+
#             |
#             v
#   +---------------------+
#   | wine-finetuned.dvc  |
#   +---------------------+

# Query model metadata:
dolt sql "SELECT * FROM models WHERE f1_score > 0.95"

Full Research Report

memory/dataset-management-research.md in the project memory.