What we look at
Six areas, in the order they tend to hurt.
The assessment covers one system end to end - a model you want to deploy, a
pipeline that keeps breaking, or a prototype that has to become a product. We are
looking for the things that will page someone at 3am, not for style points.
01 · Reproducibility
Can it be rebuilt without you?
Whether the current model can be reproduced from data and code by an engineer who
did not write it. Pinned environments, seeds, data snapshots, and whether the
artifact in production matches anything in your repository.
02 · Pipeline
What runs unattended, and how it fails.
Training and inference paths, orchestration, scheduling and retries. Where state
lives, which steps are idempotent, what happens when one dies halfway, and how
long you would take to notice.
03 · Evaluation
Is the metric the one you care about?
The validation strategy, the split, and the distance between your offline number
and the business outcome. Leakage, temporal ordering, and whether a good score is
measuring generalization or measuring your test set. A leaderboard is not
generalization, and we have paid to learn that.
04 · Serving
Latency, cost, versioning, rollback.
How inferences are served and what they cost per request. Batch versus online,
model versioning, the rollback path, and what happens to traffic while a bad model
is being taken out.
05 · Monitoring
Would you even know?
Concept and performance drift, input distribution checks, and whether your alerts
fire on things that matter. If the model silently degrades over a quarter, we want
to know which graph moves first.
06 · Data
Lineage, freshness and what breaks upstream.
Where features and labels come from, how stale they are by the time the model sees
them, and which upstream change silently poisons the whole thing. Label quality
gets the same scrutiny as model code.