Six questions to ask before you buy an AI agent

Phrases like "runs your follow-ups end to end" are easy to find on AI product pages this year. Far fewer of those pages explain what the agent does when it is unsure, or who finds out when it gets something wrong.

In June 2025 Gartner predicted that over 40% of agentic AI projects would be cancelled by the end of 2027, citing rising costs, unclear business value and weak risk controls. In the same release it said that of the thousands of vendors marketing agentic AI, only about 130 were offering something it considered real. The rest, it said, were relabelling chatbots, assistants and automation. It called this agent washing.

That leaves a business owner sitting through a demo with no easy way to tell which kind they are looking at. These are the questions we would ask.

1. What exactly will it do without asking a person first?

Ask for a list, in writing. Sending emails? Changing records? Issuing invoices? Committing a price? A good vendor can tell you which actions run on their own, which wait for approval, and how you change that per action. A vague answer here is the answer.

2. Where does it keep what it learns about our business, and can we read it?

An agent that is meant to improve has to keep something: your rules, your corrections, your customers' preferences. Ask where that lives. Ask whether you can open it, edit it, see who changed it, and take it with you if you leave. If the honest answer is "inside our product, and no," then every correction your team makes is building value you do not own.

3. What happens when the AI part fails?

Every agent depends on a model call that will sometimes time out, error or return nonsense. Ask what the product does then. Some fall back to a template and carry on without telling anyone, which means you can receive generic output that looks normal. We have been caught by exactly that in our own systems. You want a product that says plainly when it fell back, and alerts someone.

4. Who on our side owns it?

This is a question for you more than the vendor. If nobody in your business is named as the owner, the agent will drift: rules go stale, errors go unreported, and it ends up in the cancelled 40%. Many vendors will be happy to sell to a business with no owner. That does not make it a good idea.

5. What does it need from us that we do not have yet?

Most agents assume clean customer data, a written process and agreed rules. Ask the vendor what a typical customer had to prepare before go-live, and how long it took. If your follow-up process exists only in someone's head, the agent will either invent one or need you to write it first. Better to know which before you sign.

If the answer is a long list, that is useful too. It tells you the first month of the project is preparation work on your side, and you can decide whether to do that work before you pay for the agent rather than during.

6. How do we stop it?

There should be one switch that halts all actions and returns everything to review. Ask to see it. Ask who can use it. Ask what happens to actions already in flight.

What these questions will not tell you

A vendor can answer all six well and the project can still fail, because the task was wrong or the team never used it. Good answers mean the product is honest about its limits. They do not mean it fits your business.

And some of the most useful automation is not an agent at all. A plain rule that flags overdue quotes every morning, plus a person who acts on it, often beats an agent that costs far more. Agent washing works in both directions. Plenty of firms buy an agent because the word is fashionable when a rule would have done.

Before your next demo

Print the six questions and take them in. Write the answers down during the call, in the vendor's words. Afterwards, show those answers to the person who would own the agent, and ask them whether they would be comfortable explaining each one to a customer who complained.

Book 20 minutes at pgco.world/book.