People
When the model is wrong
Tobiloba Odejinmi · 21 May 2026 · 6 min · 982 words
Direct answer
When the model is wrong, the case moves into an escalated state. A person owns it. The reason and the original payload stay attached. You do not “nudge the prompt” and hope the next run is luckier. You treat the miss the way you would treat a junior who filed the wrong number: stop, correct, write the rule, then let the loop continue.
- Wrong is a workflow state, not a mood in Slack.
- Keep the payload. Screenshots are how context dies.
- There are different kinds of wrong. They need different owners.
- If you cannot explain the miss, you are not ready to stay live.
Wrong is a process event
People talk about hallucinations as if they were weather. I treat a wrong output the way I treat a failed disbursement. It happened to a specific case, with specific inputs, at a specific time. If you cannot point at those three things, you will argue about the model instead of fixing the work.
At 10mg, a row that is wrong is not an interesting ML discussion. Someone does not get treated. The same gravity applies when an AI employee writes a follow-up that invents a promise, or ranks a CV on a field you told it to ignore. Name the case. Then decide what state it is in.
Escalation-as-state
I want three honest endings for a unit of work: done, waiting on a system, or escalated. Escalated means a person has it, the reason is stored, and the payload is still there. You can list every escalated item. You can see how old they are. You can see who is sitting on them.
A Slack mention is not that. A “can you check this?” that disappears by lunch is not that. Anthropic’s 2026 agents work is blunt about bounded agents: tools, a stop condition, a path back to a person. The path back is the product. The fluent paragraph is the easy part.
Three kinds of wrong
The first is a missing fact. The system did not have the document, the policy, or the last ticket. That is a retrieval or access problem. Sending it back to the same prompt will not grow the fact.
The second is a broken shape. It returned a paragraph where you needed a status, an amount, and a next step. That is a structured-output problem. Zod is unglamorous. It is also how you stop a model from inventing a field.
The third is a judgment call dressed up as a fact. The system picked a tone, a priority, or a ranking that a person would not defend. That belongs with the process owner, not with a retry button. If you mix these three together in one “AI error” bucket, you will tune the wrong thing.
What I learned from the review pile
At SmartComply we did not pretend the model never missed. We made the first pass cheap and left the weird cases for a person. Review work fell by about half. The misses we still caught were the ones that did not fit the shape. That is the deal I will still make.
If you cannot explain a miss, you are not ready to stay live. “The model got confused” is not an explanation. “It used last year’s price list because retrieval returned the wrong PDF” is an explanation. One of those you can fix by Tuesday. The other one you will still be saying in six months.
Who gets the miss
The owner of the process gets the miss. Engineering gets it if a tool failed or a schema broke. The vendor gets it if their API started lying. Do not send every miss to the person who wrote the first prompt. That turns one engineer into an unofficial support line and hides the real rate from the business.
Write the routing in the runbook. Missing field → reviewer. Wrong policy document → whoever owns the knowledge store. Tool timeout → engineering. Judgment → the named owner. If routing takes a meeting, the state machine is not done.
What you write down after
After a real miss, add one line to the standard: what we will not accept again. Then decide if the system should have escalated earlier. Most “surprising” errors are cases that should never have been marked done.
Keep a short log. Date, case, kind of wrong, what changed. You do not need a museum. You need enough history that a new reviewer does not relearn the same lie. That log is also what a buyer, a regulator, or a tired Monday standup will ask for. Write it while you still remember.
Questions people ask
What should happen when an AI employee is wrong?
The case is marked escalated, a person takes it, and the original inputs stay with it. After it is fixed, you decide whether the rule, the retrieval, or the scope needs to change. You do not only rerun the model.
Should we turn the system off after one miss?
One miss is a review. A pattern of misses is a stop. Write the pattern in advance: wrong rate, missing required fields, or a tool that started returning garbage. Do not invent the threshold during the incident.
Who tells the customer?
A person. The AI employee can draft the note. It should not be the last voice on a correction that involves money, health, or someone’s job.
Is this the same as human-in-the-loop?
HITL is the broader idea. Escalation-as-state is the implementation. A human is not “in the loop” if they only see a chat summary after the work is already marked done.
What if the model sounds sure?
Confidence in prose is not a score you can use. Structured output and a required field check catch more lies than a paragraph that reads like a brief. If the system cannot fill the shape, it escalates. That is the point of the shape.
Written by
Tobiloba Odejinmi
Head of Engineering at 10mg Health. I have run engineering at Zeeh Africa and sold Insurpass and Shopl. I still write the code. If you have one process that still runs on people copying things, we can look at it in thirty minutes.
