Case Study · 2026 · Open source
mlfactory
An LLM-orchestrated ML factory. Specialist agents own the judgment: designing the target, hunting leakage, ranking model families. A deterministic, unit-tested Python CLI owns every number. Typed artifacts chain each result back to the data that produced it. It is domain-agnostic, validated end to end on real churn data, and gated by a human at every real decision, so the call you make actually propagates through the pipeline.
- Role
- Solo: design & build
- Stack
- Python 3.11 · scikit-learn · Pydantic v2 · Optuna · Claude
- Evidences
- Rigor: 284 tests · sha256 lineageAI: multi-agent orchestration
- leaked vs. honest AUC
- 0.9999 → 0.66leaked vs. honest AUC
- AUC on real Telco data
- 0.8304AUC on real Telco data
- tests · ruff + mypy clean
- 284tests · ruff + mypy clean
- your file, Postgres or SQLite
- Any tableyour file, Postgres or SQLite
Problem
Two failures ruin models. Both hide in the notebook.
Leakage. The model reads a column that secretly encodes the answer. It scores near-perfectly in testing, then collapses in production.
Un-auditable automation. Letting an LLM just do the ML is fast. But a language model narrating an AUC is not a fact you can re-run. A pinned CLI computing it is.
I built mlfactory after noticing how much careful work disappears. The column you dropped. The split you chose. The bar you set for shipping. Those calls live in a notebook or in someone’s head, and by the time the model matters they are gone.
So mlfactory writes them down. Every number is reproducible, every artifact is lineage-verified, and the AI never substitutes for a computation.
How it works
AI judges. Tested code computes. Artifacts remember.
Agents own judgment: designing the target, hunting leakage, ranking model families. They never compute a number.
A unit-tested CLI owns every number. You can’t unit-test a vibe. You can unit-test that PSI (the standard score for “has this data shifted?”) comes out at exactly zero for two identical distributions. Same input, same output, always.
Typed artifacts carry the lineage. Any number traces back through a parent_sha256 chain to the data that produced it.
Architecture
Narrow specialists. One tested engine. A human between them.
┌──────────────────────────────────────────────┐ │ AGENT LAYER · multi-agent orchestration │ │ • orchestrator playbook (/mlfactory-run) │ │ • specialist subagents: leakage-scanner, │ │ column-profiler, model-recommender, … │ │ • propose & judge, never compute a number │ └──────────────────────────────────────────────┘ │ ▼ HUMAN-IN-THE-LOOP GATE · the DS decides │ the call is written to the decision record ▼ that every downstream stage then reads ┌──────────────────────────────────────────────┐ │ CLI · deterministic, unit-tested Python │ │ • split · engineer · train · evaluate · ship │ │ • same input → same output, seeded │ │ • 284 tests: numbers are covered, not trusted│ └──────────────────────────────────────────────┘ │ typed artifacts · schema + parent_sha256 lineage ▼ validate-artifact walks the chain & re-verifies
A playbook spawns narrow subagents: leakage-scanner, column-profiler, model-recommender, advisor, artifact-validator. Each owns one judgment call and nothing else.
Between the agents and the engine sits a human-in-the-loop gate at every real decision: target, leakage, features, model, ship. The next section shows how that gate works.
Proof · the leakage lesson
It catches the leak instead of shipping it.
The reference domain ships with a planted leak, so you can watch the factory catch a mistake that would sink a real model. Real seeded run, nothing staged:
$ mlfactory leakage-scan --config churn.yaml ⚠ 1 leakage risk(s): [near_perfect] cancel_page_visits_30d strength +0.917 → inspect $ mlfactory train --model logistic --json # features: auto {"train_metrics": {"auc": 0.9999, "ks": 0.9877, "lift": 9.90}} # 0.9999 AUC is not a triumph. It is the fingerprint of a leak. # drop the flagged column, re-run the very same pipeline: $ mlfactory exclude-columns --add cancel_page_visits_30d $ mlfactory evaluate --model model.pkl --test test.parquet AUC 0.6626 | PR-AUC 0.1906 | KS 0.2288 | top-decile lift 2.10x
The scanner flags the trap before training. Train on it anyway and the score is a suspicious 0.9999. Drop the column and the same pipeline reports an honest model: held-out AUC ≈ 0.66, top-decile lift ~2.1× (the riskiest tenth churns at twice the base rate).
Held-out AUC · same pipeline, same data
looks perfect, but it is reading the answer
honest, defensible, shippable
The 0.34 drop is the leak, measured. A model that only ever ran the first configuration would have shipped at 0.9999 and collapsed in production.
Proof · it survives real data
Not just the demo. Real data too.
The same pipeline runs on the public Telco Customer Churn dataset. No planted trap, no synthetic tidiness. A real model, documented by a model card whose numbers are verbatim CLI output:
Model card · logistic
real data · 1,371 held-out rows
| baseline floor AUC | 0.5000 |
| test AUC | 0.8304 |
| test PR-AUC | 0.6343 |
| test KS | 0.5096 |
| top-decile lift | 2.53× |
| log-loss | 0.4443 |
| calibration error (ECE) | 0.0313 |
Limitations (authored honestly)
- Held-out split of the provided data (Telco). Not a synthetic reference domain, and not measured production performance.
- A risk ranking, not a calibrated business decision. Pair it with the policy layer.
- Snapshot horizon: no date column, so
tenureandTotalChargesare cumulative-to-date; the scoring cutoff must be stated before this is used next quarter.
A real 0.83 AUC, well calibrated. An expected calibration error of 0.031 means a predicted 30% really does happen about 30% of the time. The card describes itself honestly as a held-out split of the provided data, and renders every section the run supports: Purpose · Training Data · Features · Performance · Calibration · Limitations · Lineage.
The decision loop
How a decision actually gets made.
“AI proposes, tested code computes, the human decides” isn’t a slogan. It’s the mechanism. Every judgment call runs the same three steps, each backed by a concrete command:
A subagent reads the data and recommends. The suggestion itself comes from the tested CLI: leakage-scan tiers the risks, compare ranks the model menu on held-out performance and stability, advise --json emits the call. The agent explains it in plain language. It never invents a number.
The AUC, the drift score, the leakage strength and the ship verdict are all produced by the CLI. Same input, same output, seeded and covered by tests. The LLM orchestrates and explains. It never does the arithmetic, so no figure is ever reasoned out in a prompt.
At each gate the data scientist accepts or overrides. The decision is written to a typed record (record-decision → config.decisions) that split, train, evaluate and ship all read, so the choice changes what gets computed. Defaults reproduce the prior behaviour, so gates add control without surprise.
The AI makes the careful workflow the fast one. The tests make every number trustworthy. The decision record makes every call auditable. Nothing is taken on faith.
What it is, and what it isn’t
mlfactory takes you from a dataset to a model you can defend. Tabular binary classification. It is not a serving platform: deployment, monitoring at scale and feature stores are out of scope. The contract tier is the bolt-on toward them, not a substitute for them.
Engineering rigor
Rigor you can unit-test.
- A clean-room metric core. Decile-KS, gain/lift, ROC/PR-AUC, log-loss, calibration and drift, all reimplemented from public methods in numpy/pandas and property-unit-tested. Held-out results break out by segment to show where the model is weaker, report a calibration error, and flag drift before a stale model is trusted again. Behind 284 passing tests, ruff + mypy clean.
- A model menu ranked on stability, not luck. Logistic, a pruned tree, random forest, XGBoost and histogram gradient boosting, each trained in the same leakage-safe pipeline. Ranked on held-out performance andstability against a floor, so a model that got lucky on one split can’t top the list. A seeded Optuna search tunes the shortlist when you want it.
- Leakage-safe by construction. A split guard that catches the classic panel-data entity leak. A time-aware split for dated data. A feature stage that fits on the training split and applies outward. No test-set information ever reaches the model.
- Typed artifacts, and decisions that propagate. Every stage writes a schema-validated file carrying a
parent_sha256link, andvalidate-artifactre-walks both the chain and the on-disk contents. Each gate decision goes into a typed record every downstream stage reads, so a choice changes what gets computed instead of living in a chat log. - Layered and genuinely domain-agnostic. A generic compute core (numpy/pandas only) sits under a swappable reference domain, so pointing it at your own file, Postgres or SQLite runs the same pipeline unchanged. That is what makes it a factory rather than one churn project with good tests.
Go deeper