Insights

Is your AI still paying off? The check most teams skip

A gain measured at launch is a snapshot. Models, staff and volumes all move afterwards.

  • Ravi R. Singh
  • 4 min read

The result nobody checked again

Most AI projects are measured once. The team reports a result a few weeks after launch, the sponsor is satisfied and attention moves to the next thing. A year later the same figure is still quoted in board papers, and nobody knows whether it is still true.

Often it is not. I would put more weight on a modest gain that has been checked every month than on a large one measured once.

Gartner listed escalating costs among the reasons generative AI projects are abandoned after proof of concept. Costs escalate after launch as well as before it, and gains erode. Both happen quietly.

Four ways a gain erodes

The first is that the tool changes. Vendors update models without asking. An update can improve one task and make another worse, and instructions that worked in March may behave differently in September.

The second is that people change. The person who built the workflow leaves, and their replacement inherits a process without the reasoning behind it. Small workarounds accumulate. Review steps get skipped when the team is busy.

The third is that the work changes. Volumes rise or fall, a new customer sends documents in a different format or a regulation adds a field. The process was designed for last year's inputs.

The fourth is cost. Usage-based pricing grows with adoption, and a vendor's price rise can turn a comfortable margin into a thin one.

What to measure each month

A monthly check does not need a dashboard project. Four numbers on one page are enough.

Track the volume handled, so you know the tool is being used. Track the net time or cost against the baseline in the business case. Track the error or rework rate, taken from a small sample that a person reviews by hand. Track the total running cost, including licences, usage fees and review time.

Add one question for the process owner: what changed this month. A new document format or a staff change explains most movements in the numbers before they become problems.

Keep the sample honest. Twenty items chosen at random and checked properly tell you more than a thousand items scored by the same tool that produced them.

Every review ends in a decision

A review that only reports is a ritual. Each one should end with one of three decisions. Continue as is. Adjust, which might mean a changed instruction, an added review step or a retraining session. Or stop, because the gain has fallen below the line set in the business case.

Stopping is a legitimate result. An organisation that can retire an AI workflow cleanly can be trusted to start the next one.

The review is also where the next use case comes from. The owner of a working process usually knows which neighbouring task should follow, and they have earned the right to be heard.

Who owns the number

Every live AI workflow needs one named owner for its result, and that person should sit in the business. IT can report whether the system is running. Only the process owner can say whether the work is better.

Ownership tends to lapse at handover. The project team disbands, the sponsor moves to the next initiative and the workflow becomes part of the furniture. Put the owner's name on the monthly page, and review that name whenever someone changes role.

I have seen the same lapse with media and analytics tools over the years. A dashboard built with care was still being circulated long after anyone could say what its figures meant. AI workflows decay faster, because the model underneath them keeps changing.

A worked month

Suppose an accounts team uses a tool to match invoices to delivery notes. In the launch month it matched 90 in every 100 without help. Six months later the monthly sample shows 78. The owner's note explains why: two new suppliers send delivery notes as photographs.

The decision is to adjust. The team asks those suppliers for a different format and adds a check for image quality. Without the monthly sample, the first sign would have been a late payment and an unhappy supplier.

Keep the check small and regular

The habit matters more than the method. One page, once a month, with the owner in the room, will catch most erosion within weeks. Put the review in the calendar for the next twelve months on the day the workflow goes live.

In the way I work this is the Grow stage. I offer it as AI Performance Advisory at US$2,000 a month, with an initial commitment of three months. Many teams can run the check themselves once the measures are set up.

If you have an AI workflow that went live more than six months ago, look up the number that justified it and measure it again this week. Whatever you find, you will know more than you do today.

Sources

AI Performance AdvisoryAll articles