Senior ML engineer, 10 years of production systems. Independent contractor working on evaluation, training data, and the infrastructure underneath.
The five repos below cover one loop: measure a model, serve it, feed it, and check whether the grader doing the measuring can be trusted. Each runs on its own without an API key, and ships with real numbers committed.
Five broken versions of an order management service, and a harness that drops coding agents in to fix them. Verification runs in three layers: unit tests, health probes under load, and assertions on the telemetry the run emitted. The telemetry layer is the one that earns its place. One task passes its full test suite while still being broken, and only the metrics catch it. Also includes an LLM judge scored against my own labels, and an SFT exporter that filters trajectories by rubric score.
python 路 agent evaluation 路 benchmark design
A gateway in front of any OpenAI-compatible provider: failover, request hedging, circuit breakers, per-key rate limiting, load shedding, streaming passthrough, and cost metering. Standard library plus one YAML parser. In the committed chaos benchmark I killed the primary provider 3 seconds into a 3000-request run. All 3000 still returned 200, and the breaker math is visible in the metrics.
go 路 distributed systems 路 observability
Raw conversation dumps to SFT-ready data. MinHash near-deduplication, PII scrubbing with a Luhn check gating card numbers, quality filters that name their reasons, and LLM judging under a hard dollar cap enforced pessimistically so concurrency cannot overspend it. CI checks the conservation law on every push: kept plus dropped equals parsed.
python 路 data engineering 路 dataset curation
Measures whether your graders and your rubric hold up. Krippendorff's alpha, weighted kappa with bootstrap intervals, and a diagnostic that names which two rubric anchors your annotators keep swapping. Statistics written by hand and fuzz-tested against the reference implementation. The test suite plants a known defect in each simulated grader and asserts the report finds it.
python 路 statistics 路 annotation quality
A CNN for surface defect classification, exported to ONNX behind a numerical parity gate and benchmarked for edge inference. 23k parameters, 95.4% validation accuracy, sub-millisecond CPU inference at batch 1.
pytorch 路 onnx 路 edge inference