Somebody on your team found an AI tool that saves them two hours a week. It is cheap, it works, and they want to start feeding it customer records on Monday. Saying no is the easy answer and usually the wrong one. Saying yes without asking anything is the fast answer and definitely wrong.
What you want is a short, repeatable review that takes half an hour and gives you enough to decide. Not a hundred question questionnaire built for a bank. Ten questions, asked before company data touches the tool, with what a good answer looks like and what a bad one is telling you.
Why an AI Tool Needs Its Own Review
You may already review vendors. AI tools deserve an extra pass for two reasons that do not apply to accounting software.
First, the data you put in may become part of how the system behaves later. The National Institute of Standards and Technology, in its guidance on generative AI, notes that models can memorize and later reveal sensitive information in their training data, and can sometimes infer personal details never supplied directly. Ordinary software stores your data. An AI system may absorb it.
Second, the output can be confidently wrong. NIST calls this confabulation: systems that generate and present false content with complete assurance. Your invoicing software does not invent an invoice. An AI tool will, and somebody has to catch it.
Questions One Through Five: Where the Data Goes
- Where does our data go, and is it used for training? Ask plainly whether anything you submit is used to improve their models, and whether that is on by default. Vendors serving businesses generally publish this. OpenAI’s enterprise privacy documentation, for instance, states it does not train its models on business and enterprise customer data by default. Get the answer for the exact plan you are buying, because consumer and business tiers often differ.
- Is there a business tier with contractual protections? Free and personal plans usually carry weaker terms than paid business plans from the same company. Ask what changes when you upgrade, whether a data processing addendum is available, and confirm current terms and pricing with the vendor rather than trusting an article or a sales deck.
- Who at your company can see our content? There is almost always some human access, and a straight answer is a good sign. OpenAI’s documentation, for example, describes access limited to authorized employees for engineering support, abuse investigation, and legal compliance, plus specialized contractors reviewing solely for abuse and misuse. A vendor claiming nobody can ever see anything is either using a technical arrangement they should be able to explain, or is not being careful with words.
- How long is our data kept, and can we delete it? You want a stated retention period, a way to shorten it, and deletion that actually removes things. OpenAI’s enterprise documentation describes customer controlled retention with deleted conversations removed within thirty days unless it is legally required to keep them. That is the shape of a real answer: a number and a named exception.
- Is there administrator control and audit logging? Somebody needs to see who is using the tool and what is happening in it. Joint guidance from the National Security Agency, the Cybersecurity and Infrastructure Security Agency, the FBI, and partner agencies in Australia, Canada, New Zealand, and the United Kingdom, published in April 2024, recommends collecting logs covering inputs, outputs, and errors alongside role based or attribute based access controls. Without an admin console, every account is its own island.
Questions Six Through Ten: Control, Exit, and Accountability
- How do users sign in, and does it support our single sign-on? Single sign-on means staff log in through the account system you already run, so when somebody leaves, one action cuts their access everywhere. The same joint guidance recommends phishing resistant multifactor authentication for access to systems and services. A tool with standalone passwords only becomes an orphan account problem the day after a resignation.
- What happens to our data if we cancel? Can we export everything in a usable format, how long do you keep it after we leave, and does deletion include backups. Test the export during your trial. Discovering it is a partial spreadsheet on your last day is a bad way to learn this.
- What is your breach notification commitment? You want a commitment to notify you, a rough timeframe, and a named channel that is not a status page. NIST’s generative AI guidance recommends service level agreements that address incident response, response times, and availability of critical support. What any notification obligation means for your business is a legal question, so have your attorney read the contract language.
- Do you have a security certification or audit report? An independent audit does not guarantee good security, but it proves an outside party looked. OpenAI, for example, states its business and enterprise products have completed a SOC 2 Type 2 audit and makes details available through a trust portal. If a vendor has nothing, ask what they do instead and judge the answer.
- Who is accountable when the output is wrong? This one is not about the vendor. It is about you. NIST recommends clear assignment of liability for incidents in vendor arrangements, and no AI vendor will take responsibility for a bad quote you sent a customer. Decide which outputs get checked by a human before they leave the building, and write that down as part of approving the tool.
How to Run This in Thirty Minutes
Most of these answers are already published. Start on the vendor’s security or trust page, fill in what you can find yourself, then email the rest to sales and give them a week. Save the replies and the pages you relied on in one dated folder, because terms change and you will want to know what you were told at the time.
Two habits make this stick. Recheck at renewal, since AI products change faster than almost any other software. And keep a running list of every AI tool anyone uses, because your reviews only cover the tools you know about. That gap is the subject of our piece on shadow IT and the risks inside your own organization, and with AI tools it is unusually wide.
Judge answers on substance, not enthusiasm. NIST cautions against extrapolating a system’s capabilities from narrow, anecdotal assessments, which is a formal way of saying a great demo is not evidence.
What Bad Answers Actually Mean
- Silence is a result. A vendor who cannot answer basic questions about their own product in a week has shown you how they will handle your support ticket.
- Vague reassurance is a warning. “Your data is completely secure and private” is marketing. Ask again. If specifics never arrive, that is the answer.
- Bad answers do not always mean no. They mean limit what goes in. A tool with weak terms can still be fine for work involving no customer data.
- Good answers do not mean stop thinking. Strong terms cover the vendor’s obligations, not your staff’s judgment.
The Bottom Line
Ten questions, half an hour, before company data touches the tool. That is the whole system. It will not catch everything, and it beats the alternative most businesses run, which is a credit card and a hope.
Treat AI tools the way Harrison describes treating AI itself: like a junior staff member. You would check references before handing a new hire your customer list. Same instinct here, and the same expectation that somebody experienced reviews the work before it leaves the building. If you are rolling tools out across a team, our guide to preparing your team for AI without the hype covers the rest of the groundwork.
If you would rather not run this review every time somebody finds a new tool, we do it for businesses across Denton County: evaluating the vendor, reading what they publish, and telling you plainly whether it is safe for your data. Contact us today.
Sources:
Comments are closed