How To Tell In Week Two Whether An AI Pilot Will Survive
Vivek Thomas, Founder & CEO of AISensum, building AI teammates that help store managers and frontline teams make better decisions every day.
gettyA few weeks ago, I looked at a retail pilot that was working perfectly. The software scored every conversation, the reports arrived each morning, and the dashboards were green. Then someone checked the logins. For seven days, almost nobody in the stores had opened the app. Nobody had noticed, because every number we were watching described the technology. None of them described the people.
Most AI pilots are judged this way. Vendors report accuracy and uptime. Buyers report “adoption” as one blended number. When the pilot stalls, both sides argue about whose fault it is, and the pilot is either extended out of politeness or killed for the wrong reason.
After years of running pilots in stores, venues and offices, I now judge every pilot and every live project with three plain questions: Do people show up? Does the work score up? Does it pay off?
The first question is whether people follow the routine the technology depends on. This is discipline, and it is almost always the first thing to break.
What discipline means changes with the use case. For a coaching tool, it means the cashier logs in, listens to their own conversations and does the practice. For a sales tool at the counter, it means the cashier actually makes the suggested offer to the guest, every time, including on a busy Saturday. For a decision tool, it means the manager opens the morning recommendations and either acts on them or writes down why not.
Measure discipline as a funnel. Out of everyone who should be using the tool, how many logged in? How many completed each step? Most teams stop at the first line. The real story usually sits two steps further down.
The second question is whether the work itself is getting better. This is performance, and it is the result of the actions people take.
For a coaching tool, performance is whether people’s scores are good and rising. For a sales tool, it is the extra revenue from the offers that were made. For a decision tool, it is what happened after each decision was carried out.
Discipline and performance must be measured separately. A team can follow every step and still score badly, which tells you the content or the model needs work. A team can score well while hardly using the tool, which tells you the habit has not formed and the results will fade.
The third question is whether the business is better off. This is impact, and it takes the longest to prove.
• Consistency: Are improvements holding across people and across months, or do they depend on a few stars?
• Business Result: Did store sales or margin move against similar stores without the tool?
• Payback: Does the result pay for what the tool costs?
Until a client shares sales data, impact stays marked as unknown, never as zero. An honest “not yet proven” protects both sides.
Put discipline on the vertical axis and performance on the horizontal one, each scored from 0 to 100. Draw a line at 60 on both. Each pilot, or each store within a pilot, becomes one bubble, and the size of the bubble is its impact. See the figure here.
• Healthy (high discipline, high performance): People show up and the work is improving. The job now is to prove the payoff and grow the bubble.
• Coach (high discipline, low performance): People are doing everything asked, but scores stay low. This one is on the provider: Improve the content, the model or the training.
• Reengage (low discipline, high performance): The results are good where the tool is used, but the habit is weak. This one is on the client’s managers. A store manager who asks about the routine every morning fixes more than any software update.
• At Risk (low discipline, low performance): Nobody is using it and nothing is improving. Act the same week, at the most senior level on both sides.
The map only works if someone looks at it every week. Review it in the same meeting, at the same time, with one named owner for every bubble outside the healthy zone.
It also gives you a clean rule for ending a pilot. If a pilot sits in the at-risk zone for two weeks and the client’s sponsor does nothing, end it. Weak technology can be fixed by the provider. A routine nobody enforces cannot be fixed from outside the building.
This is also why I no longer accept “the technology works” as a status update. In the pilot I described, the technology did work. Within a week of putting discipline on the screen, the gap was visible to everyone, and the conversation moved from blame to a clear owner and a date.
Technology is the part of a pilot a vendor controls. The habit belongs to the leaders who run the floor. Measure both, separately, every week, and most pilots will tell you how they end long before the final review.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
