How To Finally Ship Your AI Thing — Without Burning Another 6 Months, Blowing Your Runway, or Admitting to Your Investors That You Still Don't Know What's Wrong

A founder reached out. Building an AI system that needed to understand spoken commands. Hundreds of different commands. Real time. On hardware that couldn't be upgraded.

They'd been at it for three months. The headline number was stuck at 44%. They needed 80%. Twelve different configurations. Nothing moved the number.

I asked them to send me everything. Not the model. Everything else.

In week one I found it. Nothing wrong with the model.

The way they had organized their command categories created a specific problem in how the AI learned to tell things apart. It was like asking someone to sort red and dark-orange poker chips into two piles in dim light. Almost impossible. And that was the sorting job the system was being asked to do because of decisions made months earlier.

Reorganized the categories. Changed the training approach. Fixed the emergency category specifically.

The model was fine. The categories were wrong. Nobody thought to question the categories because they were set before anyone on the current team joined.

Oil and gas. AI system to detect faults across 87 rigs. Downtime costs $50,000 to $150,000 per day. Stuck at 27% accuracy. Target: 85%. Months of iteration.

I found two problems.

Problem one: The training setup was backwards for their situation. The data was 95% “fine” and 5% “broken.” The model was being quietly discouraged from confidently identifying failures — the exact thing it needed to do.

Problem two: Some equipment stored numbers in a slightly different format. The pipeline was reading every sensor value wrong from a third of the fleet. Not obviously wrong. Just wrong. Valid-looking numbers that weren't the actual readings. The AI had been learning from 6 months of bad data and had no way to know it.

A byte-order bug in one vendor's sensors. Invisible unless you specifically looked at raw readings vs. parsed values. Nobody had.

Document classification system. Insurance company. The AI sorted incoming documents into categories — claims, appeals, medical records, correspondence. At launch it was 95%+ accurate. Everybody moved on to other things.

Six weeks later, a customer service manager noticed a spike in mis-routed documents. By then the accuracy had dropped to 71%. Silently. Over six weeks. No alerts. No dashboards. No one watching.

The problem was upstream.

A third-party vendor changed the format of incoming PDFs. Slightly different metadata. Slightly different text encoding. The parsing pipeline handled it “fine” — no errors, no crashes. But the text coming out was subtly different from what the model was trained on. Different enough to confuse it. Not different enough to trigger any error.

The model hadn't changed. The data feeding it had. Nobody was watching the input distribution. That's the monitoring gap that kills most production AI systems.

Manufacturing quality control. Computer vision system inspecting parts on a production line. In the test environment: 92%. On the actual line: barely 60%. The team was convinced something was wrong with the deployment.

They redeployed. Twice. Checked the model export. Checked the inference code. Everything matched. Same model, same weights, same code. Still 60%.

The problem had nothing to do with the model or the code.

The test environment images were captured under controlled lighting with a specific camera at a fixed distance. The production line had different lighting, slight vibration from the machinery, and a camera at a slightly different angle. The preprocessing pipeline normalized image sizes but not exposure or orientation. So the production images looked subtly wrong to the model — every single one.

“Same data” wasn't the same data. It was the same objects, photographed under completely different conditions. The model had never seen what production actually looked like.

Recommendation system. E-commerce. The team would improve click-through rate and conversion would drop. Improve conversion and average order value would tank. Optimize for revenue and customer satisfaction scores cratered.

Four months of this. Every sprint produced a “win” in one metric and a loss in another. The PM was losing confidence. The team was demoralized.

The features were coupled in ways nobody had mapped.

Three of the input features were derived from the same underlying data. When you changed how one was calculated, it indirectly changed the effective meaning of the others. The model had learned to rely on relationships between these features — relationships that broke every time someone “improved” one of them.

This is a known problem in ML called CACE — Changing Anything Changes Everything. But knowing the name doesn't help if you can't see where it's happening in your specific system.

The team wasn't bad at optimization. They were optimizing inside a system where the variables were secretly connected. Once we mapped the dependencies, each change did what they expected.

E-commerce search. The AI powered product search and recommendations. Internal metrics showed 94% accuracy. NPS was dropping. Support tickets about “can't find what I'm looking for” were rising.

The team was confused. The numbers said it was working. The customers said it wasn't.

They were measuring the wrong thing.

94% of the time, the model returned a result that was technically “relevant.” But relevance was measured by category match. A customer searching for “blue running shoes size 10” would get blue shoes — technically relevant — but not running shoes in size 10. The metric said success. The customer said failure.

The metric was designed early, before anyone understood how customers actually used search. It had never been revisited.

When your metric doesn't match what your customer actually cares about, you can optimize forever and still lose. The model was doing exactly what you asked. You were asking the wrong question.

Healthcare analytics. The AI flagged patient risk factors from medical records. Technically accurate. Clinically validated. Published-paper-level performance.

Adoption was 12%. Doctors would look at it, ignore it, and document their own assessment. Six months after deployment, usage was declining, not growing.

The problem wasn't the model. It was the output.

The system produced a risk score: a number between 0 and 1. Technically precise. Clinically meaningless. Doctors don't think in probabilities. They think in “what do I need to do next?”

The score didn't map to any clinical action. A score of 0.73 didn't tell a doctor anything they could act on. And the model didn't explain which factors drove the score. So doctors couldn't evaluate whether to trust it.

The AI worked. The interface between the AI and the human didn't. This isn't a model problem. It's a deployment problem. And it's the most common reason technically excellent AI systems fail in practice.

Computer vision API. The model analyzed images and returned results. Accuracy was great. Latency was 4.2 seconds average. For their use case, anything over 2 seconds meant users abandoned the flow.

The team spent two months trying to make the model smaller. Model compression. Pruning. Distillation. They got it to 3.1 seconds. Still too slow. And accuracy had dropped.

The model was not the bottleneck.

The inference pipeline was: receive image → decode → resize → normalize → run model → post-process → return. The model took 400ms. The other steps took 3,800ms. Image decoding was happening on CPU with a single-threaded library. Resizing was using a high-quality algorithm designed for photo editing, not real-time serving. Post-processing was making a database call for every single request.

They spent two months compressing the thing that took 400ms. The other 3,800ms was sitting right there, unexamined, because everyone assumed the model was the slow part.

NLP startup. The demo they showed investors was compelling — the AI understood complex queries and gave useful answers. The actual product, using the same model, was inconsistent. Sometimes great. Sometimes embarrassingly bad.

The founder was terrified. The next investor meeting was in three weeks. The demo always worked. The product sometimes didn't. “Same model” but different behavior.

The demo and the product were not using the same inputs.

The demo used a curated set of example queries. Clean text, well-formatted, moderate length. The product received real-world user input: typos, abbreviations, incomplete sentences, copy-pasted text with hidden formatting characters, queries in unexpected languages.

The preprocessing pipeline didn't clean any of this. The model received raw user input and the demo received polished input. Same model, completely different experience.

The gap between “it works in the demo” and “it works in the product” is almost never about the model. It's about the distance between curated inputs and real-world inputs.

Fraud detection. Fintech company. The model was catching 43% of fraudulent transactions. Target was 80%. Every two weeks they collected more labeled data, retrained, saw a small improvement, deployed — and within days the number settled back to the low 40s.

Eleven retraining cycles. Four months. The team had started calling it “the rubber band” because the number always snapped back.

The training data was structurally biased.

The fraud labels were assigned by the existing rule-based system plus manual review. But manual review only happened when the rules flagged something. So the model was being trained on “fraud that looks like what we already catch.” New fraud patterns — the ones they actually needed to detect — were almost never in the training data because the old system didn't flag them for review.

More data didn't help because more data was more of the same bias.

You can't train your way out of a data collection problem. The model was learning exactly what you taught it. You were teaching it the wrong thing.

SaaS company. Their ML engineer built the recommendation system, the ranking system, and the anomaly detection pipeline. All three. Single person. Undocumented. Then they left.

For two months, everything kept running. Then the recommendation quality started degrading. Then the ranking started producing weird results. The remaining team could read the code but couldn't understand why specific decisions had been made. Constants in the code with no comments. Threshold values that seemed arbitrary. A preprocessing step that looked unnecessary but broke everything when removed.

The system wasn't broken. It was undocumented and drifting.

Three things were happening: the input data distribution had shifted since launch and nobody was monitoring for drift. A hardcoded threshold was calibrated for the data profile at launch time and was now wrong. And the “unnecessary” preprocessing step was compensating for a known data quality issue that only the departed engineer knew about.

This is the most common story I see. Not a dramatic failure. A slow degradation that nobody can diagnose because the knowledge walked out the door. The system works until it doesn't, and when it doesn't, nobody knows where to look.

Content moderation. The AI sorted user-generated content into categories: safe, borderline, violation. The model was good. 85% agreement with human reviewers.

But the 15% disagreement wasn't random. It was systematic. The model consistently classified certain types of content as “borderline” that human reviewers called “violation.” And it classified other content as “safe” that reviewers called “borderline.”

The category definitions had drifted from reality.

The categories were defined 18 months earlier. The platform and its community had evolved. New types of content didn't fit cleanly into the original categories. The labeling team had gradually developed their own unwritten rules for edge cases. The model was trained on the original definitions. The humans were judging by the evolved ones.

The model wasn't wrong. The labels were wrong. Or rather, the labels were right for 18-months-ago and wrong for now.

If the categories your model learned don't match the categories your business now needs, no amount of model improvement will fix it. The foundation has to match reality.