CRM agents scored near zero on knowing what is confidential

Before you let an AI agent loose in your CRM, read one benchmark. Salesforce AI Research published CRMArena-Pro in May 2025. It tests AI agents on the kind of work sales and service teams do every day: looking up accounts, qualifying leads, handling cases, answering questions about pipeline. The results are worth knowing even if you never use Salesforce.

Salesforce sells an agent product, so it has an interest in framing the problem in a particular way. The numbers come from the paper itself, though, and they line up with independent work.

Finding one: agents get worse as the conversation goes on

On single-turn tasks, where everything the agent needs is in one request, the best agents tested completed about 58%. On multi-turn tasks, where information arrives over a back-and-forth like a real conversation with a colleague or customer, the best score fell to about 35%.

This matches what a separate Microsoft and Salesforce paper found the same month across general tasks: an average 39% drop when the same work is spread over several turns.

For a sales team this is the normal case, not the edge case. Nobody hands an agent a perfect brief. A rep asks for a summary, then a follow-up, then a correction, then a draft email. Each turn is a new chance for the agent to lose track of what was said before.

Finding two: agents often fail by not asking

The paper notes that many failures came from agents not asking for information they were missing. Instead of saying "which account do you mean?", they picked one and carried on.

A human assistant who did this would be coached out of it in a week. An agent does it confidently, every time, unless something tells it not to.

Finding three: confidentiality awareness was close to zero

This is the result that should change how you deploy. The benchmark tested whether agents recognised when information was confidential and should not be shared, such as internal pricing, another customer's details or notes not meant for the client. The agents showed almost no built-in awareness of it.

In plain terms: if your CRM holds a note saying a client is behind on payments, or a margin figure, or another customer's contract terms, an agent drafting an email has no instinct telling it to leave that out. It will use whatever looks relevant.

What to do about each one

For drift over long tasks, keep the rules out of the conversation. Store the agent's working rules in a written memory it reads at the start of every task and every step, rather than relying on instructions given earlier in a thread. That is how our own agents at Provecta Group work, and it is the same approach Anthropic's engineers recommend for long-running agents.

For not asking, write it as a rule with a reason. "If the account, contact or deal is ambiguous, ask one question before acting, because a wrong guess sends the wrong client the wrong email." In our own work, a rule that carries its reason gets applied to new cases more sensibly than a bare command. That is our observation, not a measured result.

For confidentiality, a written rule helps and is not enough. List the fields and note types that must never appear in anything sent outside the firm: internal pricing, margins, payment status, notes from other accounts. Put that list in the agent's memory. Then add a check that does not depend on the agent remembering it: the agent's access excludes those fields, or every outbound draft is scanned for them, or a person approves anything going to a client.

That last layer is the one that actually stops a leak. Memory makes the agent less likely to try. Permissions and approval gates make sure that when it does try, nothing leaves.

What we would not do

We would not give an agent write access to a CRM, or the ability to send client email on its own, on the strength of good instructions alone. The benchmark is a reminder that instructions degrade over a conversation and that confidentiality is not something these models bring with them.

We also would not read this as a reason to keep AI out of sales work. Agents in the benchmark did handle a good share of single-step tasks. The question for a sales team is where to draw the lines, and those lines need to be written down before the agent starts.

Three fields to find

Open your CRM and find three fields that would embarrass you if they appeared in a client email. Write them down as a list titled "never send outside the firm". That list is the start of your agent's rules, and of your own team's.

The RevOps audit looks at how your sales data and processes would hold up with AI in the loop. Book 20 minutes at pgco.world/book.