How to Test an AI Vendor Before You Commit

A smooth demo is not proof

A vendor controls the data, the question and the route through a demo. Your business will not be that tidy.

Before you commit, test the product on work that looks like yours.

1. Bring one real job

Choose a job with a clear owner and result. Use safe but realistic data. Avoid a made-up example that everyone knows the product can handle.

Ask the vendor to show the full path from input to checked result.

2. Ask it to show doubt

What happens when facts are missing or two sources disagree? A useful product should show that it is unsure and ask for help.

If every answer looks polished, inspect the weak cases more closely.

3. Name the human check

Ask who reviews the work, what they see and how they change it. Check whether the source and the reason remain visible.

“A person is in the loop” is not enough. You need to see the loop.

4. Test your way out

Ask how you export your data, rules, prompts and results. Find out what still works if you stop paying.

The right vendor should add value beyond access to a common AI model. That value may be trusted data, strong checks, better tools or hard work already built into the product.

5. Agree the proof before the trial

Write down:

• the job being tested;
• the old time and error rate;
• what a good result looks like;
• who decides whether it worked;
• the date you will stop, change or continue.

Do this before the trial starts. A test with no finish line will always find a reason to carry on.

Questions worth sending in writing

1. Which model does the product use today?
2. Where does our data go?
3. Is our data used to train anything?
4. What can we export?
5. What happens when the model or price changes?
6. Can we speak to a customer using it on similar work?

Buy the result you can test, not the future shown on a slide.