Open five AI vendor websites and you will read the same page five times. Powered by advanced AI. Automate your workflow. Save hours every week. Enterprise grade security. Every product sounds identical because every product describes the technology instead of what changes in your business on Tuesday morning.
That becomes your problem the moment you sign an annual contract based on a demo built to impress you. We have watched small businesses spend real money on tools that worked beautifully in a controlled walkthrough and got quietly abandoned six weeks later. The fix is not more skepticism about AI. The fix is changing what you shop for. You are not buying software. You are buying a specific outcome, and outcomes can be defined, tested and measured before you commit.
Start From the Task, Not the Technology
Almost every disappointing AI purchase starts the same way: somebody sees a tool, gets excited, then goes looking for a problem it can solve. Reverse that. Start with the work genuinely costing you hours.
Spend a week writing down where time actually goes, not where you assume it goes. Then pick one task that is repetitive, high volume, well defined, and low consequence if it needs correcting. Quoting from a standard price list. Summarizing service tickets. First drafts of routine correspondence. Those are good candidates. Tasks requiring judgment, relationships, or where a mistake is expensive are not where you start.
The NIST AI Risk Management Framework calls this MAP, the function that “establishes the context to frame risks related to an AI system.” In plain terms: define the job before evaluating the candidate. Write one sentence describing the outcome you want, with a number in it. “Cut the time our team spends writing service summaries from six hours a week to two.” That sentence becomes your evaluation criteria, and it is what you show vendors instead of asking what their product does.
Make Them Demo It on Your Data
A vendor demo is a performance. The sample data was chosen because the product handles it well, the prompts were rehearsed, and none of it resembles the messy reality of your files.
So ask for a demonstration on your material. Your invoices with their inconsistent formatting. Your support tickets with the typos and industry shorthand. Your documents with tables that never convert cleanly. If a vendor cannot or will not do this, that is your answer. The NIST framework makes the same point under MEASURE, asking that performance criteria be “measured qualitatively or quantitatively and demonstrated for conditions similar to deployment settings.” A result on clean sample data tells you almost nothing about yours.
Bring your hardest examples, not your easiest. The odd cases are where the hours actually go, and a tool that handles the simple 70 percent while failing the difficult 30 percent may save you nothing once someone has to check the output.
Ask What Happens When It Is Wrong
Not if. When. All of these systems produce incorrect output sometimes, and a vendor who will not discuss that openly is either inexperienced or overselling.
- How does it fail? Does it decline the task, or does it produce a confident and plausible wrong answer? The second is far more dangerous, because nobody catches it.
- Can we see its work? Can a person trace where an answer came from? Verifiability determines how much review time you have to budget.
- What is the review step? Harrison’s rule applies here: treat AI like a junior staff member. Capable, fast, occasionally wrong in ways someone experienced needs to catch. You would not let a new hire send work to a client unreviewed in week one.
- How do we turn it off? The NIST framework specifically calls for mechanisms to supersede, disengage, or deactivate AI systems that perform inconsistently with their intended use. Ask what the shutdown looks like before you need it.
- What happens to our data? Where it is stored, whether it trains the vendor’s models, how it gets deleted. Get it in writing and confirm it currently, because vendor terms and plan tiers change often.
Ask Who Is Accountable for the Output
Here is the answer no vendor volunteers: you are. If an AI-generated quote goes out with the wrong price, your customer holds you to it. If a summary misstates a client instruction, that is yours to fix.
So accountability needs a name on it before rollout, not after the first mistake. The NIST framework’s GOVERN function asks that “roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and clear.” For a 30 person company that means writing down who reviews the output, who decides when review can loosen, and who can shut it off. One or two names on one page.
The framework also recommends contingency processes for handling failures or incidents in third-party AI systems. Translated: what do you do the morning the vendor has an outage, changes the product, or goes out of business? If the work simply stops, you built a dependency you did not price in.
Check How It Fits What You Already Own
Two things get missed here constantly, and both cost real money.
First, check whether you already pay for this. Major business software suites keep adding AI capabilities to existing plans, and we regularly find clients buying a standalone tool that duplicates something already in a subscription they hold. Audit before you shop, and confirm current feature availability and pricing directly with the vendor, since these change constantly.
Second, ask what integration really means. A tool that does not connect to your systems creates copy and paste work, which is exactly where promised time savings disappear. Ask whether it connects to the systems you name, whether that connection is included or costs extra, and who does the setup. Ask about single sign-on too. A tool outside your identity system is another set of credentials to manage and another account somebody forgets to remove.
Run a Pilot With a Number Attached
Never sign an annual agreement off a demo. Run a two week pilot first, structured so it produces an answer rather than an impression.
- Measure the task before you start. Hours per week, errors, time per item. Without a baseline you cannot tell whether anything improved.
- Pick three to five people who do the work daily. Not the enthusiasts. The people whose honest opinion you need.
- Define success as one number in advance. Written down before the pilot begins, so nobody redefines the goal to match the result.
- Track review time as part of the cost. If a task drops from four hours to one but someone spends two hours checking output, you saved one hour, not three.
- Decide in advance what a failed pilot means. Give yourself permission to walk away. That is much harder after you have announced the tool to the whole company.
The Bottom Line
The companies getting real value from AI are not the ones that bought the most impressive product. They picked a specific expensive task, tested a tool against their own messy data, kept a human accountable for the output, and measured the result honestly enough to abandon what did not work.
That approach does not go stale when the next generation arrives, because it is a purchasing discipline rather than a product recommendation. For the companion piece on getting your people ready rather than your software, see how to actually prepare your team for AI without the hype. And if staff are already signing up for tools on their own, that is worth reading alongside our piece on shadow IT.
We help businesses across Denton County figure out which tasks are worth automating, evaluate vendors without the sales theater, and run pilots that produce a clear yes or no. If you are being pitched something and want a straight second opinion before signing, that is a good use of a phone call. Contact us today.
Sources:
Comments are closed