Build
Testing an AI employee before go-live
Tobiloba Odejinmi · 12 May 2026 · 6 min · 1,005 words
Direct answer
Test an AI employee on a set of real recent cases before it goes live. Score what it got right, what it missed, and whether a person can explain the miss. Include the escalations and the hard gates, not only the happy path. If you cannot explain a miss, you are not ready. Days 4 to 6 of the week are for this, not only for wiring tools.
- The test set is last month's pile, not invented examples.
- A miss you cannot explain is a ship-blocker.
- Gates and escalations are tests, not documentation.
- A person still sits on the cases that do not fit.
What do you test before an AI employee goes live?
You test the job, not the model. Did it start from the right trigger. Did it read the required fields. Did it look up the right record. Did it take the allowed action. Did it stop on a hard gate. Did the person receive a working state and a decision brief. Did the write-back land where the team already looks. Those are yes or no questions. If you cannot ask them, the map was too soft.
I also test the boring failures. Timeouts. Empty search. Duplicate ticket. Attachment the parser cannot read. Permission denied. People skip these because they are not 'AI.' They are why week two breaks. Days 4 to 6 include them because I have already been on the other side of a Monday morning when a clinic could not open a path.
How do you build a test set from real work?
Take the last two to four weeks of the pile. Strip what you must strip for privacy. Keep the mess. Annotate each item with the step it should take and the exit it should hit. This is slower than writing five ideal examples. It is also the only set that predicts production.
If the team is nervous about history, we sit together and pick the set in a room. I do not need your whole archive. I need a fair sample, including the cases they are proud of and the ones they buried. Buried cases are usually the gates.
What counts as a miss?
A wrong fact. A skipped gate. A missing escalation. An action in the wrong tool. A brief that forces the person to re-read everything. A correct-sounding reply that answers a different question. Soft tone with a hard error is still a miss. I score those separately so nobody hides a fact error under 'the customer would have liked the voice.'
At SmartComply we used structured output so the system had to return a shape we could check. Same habit here. If the employee can wander in prose, you will argue about quality forever. If it has to fill fields, you can count.
Who sits in the review?
The doer, the queue owner, and me. The doer says whether the output is usable. The queue owner says whether the exits match how they run the team. I say whether the miss is a map problem, a connection problem, or a model problem. Those three causes need different fixes. Mixing them is how you spend a day rewriting a prompt that cannot see the customer ID.
If the doer is excluded because they are 'too busy,' you will go live on a fantasy. Find an hour. The $5,000 and $7,500 weeks assume that hour exists. If it does not, the company is not buying an employee. It is buying a story.
What do you refuse to ship?
I refuse a go-live when we cannot explain the misses on the test set. I refuse when a hard gate is prompt-only and the tool permission still allows the action. I refuse when there is no named owner for the 10 to 15 percent that should escalate. I refuse when the only 'test' is a happy path we typed ourselves.
This is not purity. It is the same bar I use on systems that move money or keep clinics online. If you cannot say why it failed, you will not know why it fails on Thursday at 4pm, and the team will turn it off. Turning it off after a public launch is more expensive than one more day of testing.
How long should testing take in a seven-day week?
It starts as soon as the first connection writes back, not on the morning of day 7. By day 6 I want the set run, the gates proven, and the ugly cases reviewed. Day 7 is handover of something that already passed, plus the subset you are actually opening to traffic.
If testing blows the week, we shrink the slice or we do not go live. We do not 'ship and watch.' Watching is monitoring, and monitoring is for known behaviour. Unknown behaviour is still a build. The week has a finish line. The finish line is not a date on a slide. It is a test set you can rerun after you change a prompt.
Questions people ask
How many cases do you need?
Enough to cover the happy path, missing fields, duplicates, and every hard gate. For most first processes that is a few dozen real items, not thousands. If you cannot find a few dozen, volume may be too low to bother.
Who reviews the misses?
The person who does the work today. Not only me, and not only a sponsor. They know the unofficial correct answer.
Do you use automated evals?
Yes, for structured fields and for 'did it escalate when the gate fired.' I do not use a single vibe score as a ship decision.
What if it is good enough on tone but wrong on facts?
It is not good enough. Tone is cheap. Wrong facts in a customer reply or a hiring note are the thing we are here to prevent.
Can we go live on a subset?
Yes. A tagged queue, one mailbox, one role family. A subset with a monitor is a go-live. A quiet sandbox with no owner is not.
Written by
Tobiloba Odejinmi
Head of Engineering at 10mg Health. I have run engineering at Zeeh Africa and sold Insurpass and Shopl. I still write the code. If you have one process that still runs on people copying things, we can look at it in thirty minutes.
