Six months ago you bought AI seats for the team. There was a kickoff, some enthusiasm, a couple of people who took to it immediately. Since then, nothing much has been said about it. The invoice arrives, the invoice gets paid, and the renewal date sits on a calendar nobody has opened.
This is how shelfware happens. Not through a bad decision, but through the absence of a second decision. Software nobody evaluates gets renewed by default, and AI tools are especially prone to it because they are easy to feel good about and hard to measure. Here is the review we run with clients at the six month mark. It takes about ninety minutes, it produces a real answer, and its most important feature is that it happens before the renewal date rather than after.
Why Six Months, and Why Before the Renewal Date
Six months is long enough for novelty to wear off and short enough that people still remember what the work looked like before. At three months you are measuring enthusiasm. At twelve you have already renewed.
The timing is not a detail. A review that happens after you renew is a report, not a decision. Put it on the calendar at least thirty days before the renewal date, and put the renewal date somewhere visible while you are at it. Most annual subscriptions renew silently, and your leverage with a vendor is highest before that happens. The NIST AI Risk Management Framework, published by NIST in 2023, describes the same rhythm, calling for the “appropriateness of AI metrics and effectiveness of existing controls” to be “regularly assessed and updated.” Regularly is the operative word. A one time rollout is not a program.
What to Gather Before the Meeting
Do not walk in with opinions. Walk in with five things, and give yourself a week to collect them.
- Seats paid versus seats actually used. The single most revealing number, usually available in the admin console of whatever you bought. Microsoft’s documentation for the usage reports in its admin center puts the purpose bluntly: you can “see who uses a service to the fullest extent and who barely uses it and might not need a Microsoft 365 license.” Those reports sit under the Reports and Usage area and cover trailing windows from about a week up to roughly six months. Most AI vendors offer something comparable. Find it before the meeting.
- The tasks it actually changed. Not the tasks it could theoretically change. Ask each department for two or three concrete jobs done differently now than they were in the spring. If a department cannot name one, that is data.
- Time saved, broken out by role. Averages hide everything here. A tool that saves your proposal writer six hours a week and your bookkeeper zero is not a tool that saves three hours per person. It is a tool for one department that you are buying for everyone.
- Quality complaints, including the quiet ones. Rework counts. If someone spends twenty minutes fixing what the tool produced in two, log that honestly.
- What people quietly stopped using, and why. The most valuable item and the hardest to get, because abandonment is embarrassing. “What did you try that did not stick?” gets better answers than “why aren’t you using it?”
One note on usage data. Some platforms conceal individual names in reports by default as a privacy measure, and Microsoft’s documentation describes exactly that for its admin center reports. That is fine here. You want patterns by department and role, not a case against a person.
The Four Questions That Cut Through
Data gets you to the room. These questions get you to the answer. Ask them out loud, and write down the first response rather than the polished one that follows.
- Would we notice if it vanished tomorrow? The best question in the set because it is nearly impossible to answer politely. If most people would not notice for a week, you have your result.
- Would the team fight to keep it? Enthusiasm in a survey is cheap. Willingness to argue for something in a budget meeting is not. If two people would go to the mat and everyone else shrugs, you are paying for a company-wide deployment of a two person tool.
- What would we buy instead with the same money? This forces a comparison against real alternatives rather than against nothing. Sometimes the answer is a competing tool. Sometimes it is fixing the process the AI was papering over.
- What did we expect at rollout, and did it happen? Go back to whatever you wrote when you bought it. If you wrote nothing down, that is the first process fix, and it costs nothing to correct next time.
Document What Worked So the Next Tool Starts Smarter
Whatever you decide about this tool, the review produces something more durable than a renewal decision: a record of how your business actually adopts software. That record makes every future rollout cheaper.
Capture four things on a single page. Which prompts and workflows genuinely worked, written specifically enough that a new employee could reuse them. Which ones failed, and what the failure looked like. Which roles got real value, which tells you how to size the next purchase. And what your rollout got wrong: training, timing, or nobody owning it.
This is also where you catch what never shows up on an invoice. If half the value came from personal accounts people expensed themselves, you have a shadow IT problem, and the seat count you are reviewing was never the real picture. Ask directly whether people use tools you did not buy, and ask without consequences attached, or you will not get a truthful answer.
What to Do With the Answer
There are four outcomes, and three of them are not “keep everything as it is.”
- Renew as-is. Usage is broad, value is clear, complaints are minor. Set the next review date before you leave the room.
- Renew smaller. The most common honest outcome. Two departments get real value, three do not. Cut the seat count to match reality and put the savings somewhere useful.
- Renew, but fix the rollout. Usage is low because training was thin or nobody owned it, not because the tool is wrong. A legitimate call, but make it only once.
- Cancel. Export your data before the account closes, revoke any connections the tool had into your email or file storage, and tell people what to use instead. Cancelling is a normal outcome, not an admission of failure.
The Bottom Line
Ninety minutes, twice a year, is not a heavy governance program. It is the minimum honest attention any recurring expense deserves, and AI tools currently get less of it than almost anything else on your books because everyone is slightly worried about looking like they do not get it.
Cancelling a tool that did not earn its keep is not a statement about AI. Treat AI like a junior staff member and this becomes obvious: you would review a new hire at six months, say honestly how it is going, and decide. You would not keep paying and hope. AI is the dumbest it will ever be today, so a tool that failed this review may well deserve another look in a year, and having written down why it failed is what lets you tell the difference. Same practical mindset behind preparing your team for AI without the hype.
If you want a second set of eyes on your AI spend, we can pull the real usage numbers, run the review with your team, and give you a straight answer on what to renew and what to cut. We do this for small and mid-sized businesses throughout Denton County. Contact us today.
Sources:
Comments are closed