Build
Monitoring an AI employee in production
Tobiloba Odejinmi · 31 Mar 2026 · 6 min · 959 words
Direct answer
Monitor an AI employee the way you monitor a path that moves work: volume in, volume out, miss rate, escalation rate, age of cases waiting on a person, and failed write-backs to the tools you already use. Alert when it goes quiet, when a hard gate is skipped, or when the queue of human decisions piles up. Do not alert on 'AI usage.' That number makes people feel busy.
- Watch the paths a customer or a colleague will shout about.
- Silence is an incident. So is a skipped gate.
- Escalation rate around 10 to 15 percent is a health signal.
- A dashboard without an owner is decoration.
What should you watch after go-live?
Start with the same questions I use on a health-credit path. Can a case start. Does a decision come back. Does the write-back land. Can someone say why it failed. Translate that to your pile. Can a ticket enter. Does the employee act or escalate. Does the help desk show the result. Can the owner open one log and talk.
Add the rates. Items per day. Share that escalated. Share that hit a hard gate. Median time to a human decision. Failed API calls. Empty lookups. If you only watch token spend, you are monitoring the vendor bill, not the job.
Which alerts are worth waking someone?
Zero output with non-zero input. A burst of permission errors. A gate that never fires when it used to fire. A human queue older than the promise you made on day 7. A write-back that started failing. Those wake a person. A slightly longer reply does not.
I keep the list short. Alert fatigue is how teams mute the only channel that mattered. If everything is urgent, nothing is. The handover names who gets the ping and who is backup. An alert to a group of twelve is an alert to nobody.
How do you read a miss without a war room?
You keep a case ID that exists in the tool the team already uses, and you store the working state next to it. When someone says 'this one is wrong,' they paste the ID. You see the fields, the lookup, the decision, and the action. If that requires six tabs, we failed the build, not the review meeting.
Explain the miss in one of three buckets: map, connection, model. Map means the real process was not on the page. Connection means the tool lied or timed out. Model means the fields were there and the output was still wrong. Fix the bucket you have. Do not rewrite the prompt for a timeout.
What does 30 days of support actually mean?
On the $7,500 week it means I am still in the loop while week two and week three teach you the cases you did not sample. We rerun the test set after changes. We tighten a gate. We fix a write-back. We do not use the month to start a second process unless you booked several workflows.
It is not an SLA for your whole company. It is coverage for the employee we shipped. If you want someone owning architecture across more than one process, that is the $15,000+ conversation. I will not pretend 30 days is a department.
How do you tell volume from quality?
Volume is how many items moved. Quality is whether the moved items were right, and whether the ones that should not have moved were stopped. A busy employee that closes the wrong tickets is a cost. I report both numbers. Leadership often wants only the first. The doer cares about the second. I write both on the same line so they cannot be separated in a slide.
Escalation rate belongs with quality. If it sits near 10 to 15 percent and the briefs are usable, the slice is honest. If people start completing the case in a side channel because the brief is junk, your quality number is lying even if the model looks busy.
What do you review every week?
A handful of misses. Any gate that did not fire. Any case that aged. Any tool error that repeated. One question: did the map change in real life. Processes drift. A new product line. A new refund rule. A hiring freeze. If the world changed and the employee did not, you will see it here first if you look.
This review is thirty minutes if the logs are right. It is an afternoon if they are not. I will spend days 4 to 6 making it the thirty-minute version. Pretty dashboards do not replace a person who can explain the database. Same rule. Pretty token charts do not replace a person who can explain the case.
Questions people ask
What is included after the week?
The $7,500 week includes monitoring after it goes live and 30 days of support. One workflow at $5,000 includes a write-up and a call. Several workflows at $15,000+ include someone on the architecture and faster replies when something breaks.
Do we need a new monitoring product?
No. We need logs you can query, a short view of the rates, and a place the alert already lands. Slack you already have is fine.
How often should we review?
A short look every day in week one. A weekly look after that, plus any alert. If you need a war room to understand a miss, the logs are wrong.
What if volume is low?
Then sample every case. Low volume is not an excuse for no monitoring. It is a reason to question whether the process was worth a week.
Can the employee monitor itself?
It can flag its own low confidence. It cannot be the only watcher. Self-report without a counter in the tools is how silent misses hide.
Written by
Tobiloba Odejinmi
Head of Engineering at 10mg Health. I have run engineering at Zeeh Africa and sold Insurpass and Shopl. I still write the code. If you have one process that still runs on people copying things, we can look at it in thirty minutes.
