AI Engineer — agent reliability, AI control & evaluation
ebinbt.dev · linkedin.com/in/ebinbt · peerlist.io/ebinbt · ebinbabuthomas@gmail.com
I work on AI control and evaluation: agents whose evidence cannot be fabricated, human approval over tool calls, preregistered experiments on open models, and benchmarks for auditing agents. Before that, three and a half years shipping LLM, RAG and agent backends for startup clients in four countries, with merged open-source contributions. I publish failed tests next to passing ones.
| Project | What it is | One number | Status |
|---|---|---|---|
| diffing-agent-bench | Sealed, preregistered benchmark for black-box model-diffing agents | 0 of 13 agent attempts asked a database question; a $0.15 prompt battery found the plant | MATS 12.0 work sample |
| incidentgate | Lab for policy gates, monitors and human approval over an incident agent | 0 and 12 side effects from two covert actions under the same full safeguard chain | closed at baseline, Sep 6 |
| digital-grimace-scale | Preregistered test for involuntary markers of adverse treatment in LMs | 65.8% of distress language trained away, the margin effect stayed | Apart Research sprint, Aug 2026 |
| proofpack | Pre-approval reviewer that cites a hashed screenshot for every finding | $0.02–$0.19 per review vs 20–40 min by hand; 87 offline tests | v0.2.0, pilot |
| odd-number-forensics | Forensic study of a published reward-hacking environment | 0% → 87% gaming across single-line prompt edits | practice take-home |
| whose-voice | Blind attribution of hidden principals in poisoned corpora | 12–44% top-1 of 47 (chance 2.1%) | Apart hackathon, Jul 2026 |
| exactdoc | PDF to editable DOCX, checked by rendering back and diffing | 16/16 corpus, 0.9588 live-text retention | 1.0 |
Each README carries its own limitations and the command that produced every number.
Built at Zackriya Solutions for startup clients in the US, Canada, Europe and Australia, where most of the code is private. Open-source work from those years sits under my work account @ebinzack15.
- Real-estate search. Turned plain-English queries into SQL over Cloud SQL through GPT-3.5, served by FastAPI on GCP Cloud Run and load-tested with Locust.
- DocuAI. Chunked and indexed 1000+ documents in Qdrant and returned the closest parent documents, behind a Next.js frontend. Live demo.
- Speech assessment. Scored one-minute candidate videos with Whisper ASR and the Microsoft Pronunciation API on AWS Fargate, tuned against ground-truth scores.
- FinBot. Fine-tuned Mistral-7B with QLoRA, added a Bytewax news pipeline into Qdrant, and served it with vLLM on a GKE L4 node. Three Kaggle notebooks cover the fine-tune, the adapter merge and the inference run.
- Data and backend. Piped live MQTT sensor streams into MongoDB for a bioreactor startup, built a BPMN to PDF report engine, and wrote healthcare data-cleaning pipelines.
- Meetily. Refactored the backend and added OpenAI provider support and a CLI testing script to a privacy-first meeting-notes tool. PR #75, merged.
- bpmn-io/refactorings. Proposed a cosine-similarity connector-template recommender as a lighter alternative to LLM function calls, and built the implementation fork. Issue #33.
Python, FastAPI, LangGraph, MCP/FastMCP, Claude Agent SDK, Gemini API, PydanticAI, vLLM, Modal, LoRA/QLoRA/DPO, Playwright, PostgreSQL, Qdrant, Docker, Kubernetes, AWS, GCP, OpenTelemetry/Langfuse, pytest.