How to Evaluate AI Customer Support Tools: The Resolution-Rate Scorecard That Survives a CFO

You run support for a 60-person SaaS company, or you own CX inside a mid-market retailer, and someone above you has decided this is the year AI handles tier-one tickets. Now you have to pick the platform, sign a contract, and then sit across from a CFO who will ask one question. “What did this actually save us, and how do you know?”

This guide is for that person. The 60-second version: pick on resolution rate, not deflection rate, model the per-resolution fee at your real ticket volume (not the demo’s), and walk into the budget meeting with a cost-per-ticket-before-versus-after number you can defend. Sticker price is the smallest line in the bill.

The reason this matters more than in most software categories is that AI customer support pricing is usage-metered. You are not buying seats. You are buying resolutions, and the vendor decides what counts as one.

The buying problem before the buying

Most teams buy an AI customer support tool to cut cost per ticket. The trap is that they measure success with the vendor’s favorite metric, which is deflection rate, and deflection is not resolution. A bot can “deflect” a customer by ending the chat. Whether the customer’s problem got solved is a different number entirely.

Here is the gap in plain figures. Industry-average AI resolution sits around 44.8% in 2026 , while legacy chatbots resolve only 10 to 25% and agentic platforms with real backend access reach 70 to 85% . A vendor can show you a 90% deflection dashboard while the real resolution rate is 40%, because half those customers just gave up. Gartner found AI deflects 45%+ of queries but only 14% are fully self-service resolved .

That gap is the buying problem. You are evaluating a usage motion: tickets come in, the AI tries them, some resolve, some escalate, and you pay per attempt or per resolution depending on the vendor. If you pick on the wrong metric, you sign a contract that bills you for thousands of “deflections” that quietly became repeat contacts and angry humans.

So define the failure as a number before you start. Failure is a resolution rate below 50% after 90 days of tuning, or a repeat-contact rate inside 72 hours that climbs after launch. If a platform cannot show you those two numbers from real customers, it is selling you a dashboard, not an outcome.

The weighted scorecard for AI customer support platforms

Score every shortlisted platform against the same twelve criteria with the same weights. The weights below add to 100. They are tilted hard toward resolution quality and true cost, because those are the two things that survive contact with a CFO. Demand evidence, not claims.

“We resolve 70%” means nothing without the cohort, the time window, and how they counted.

Run all candidates through this once and the spread is usually obvious. The expensive part is not the scoring. It is forcing yourself to demand the evidence column for every row, because vendors will happily fill the criterion column with marketing.

The true multi-year cost of AI customer support

Sticker price in this category is almost a fiction. The number on the proposal is a per-resolution rate or a base platform fee, and neither is what you pay. Gartner research found 71% of enterprise buyers underestimate AI support costs by at least 40% in year one , and that undercount is structural, not careless.

Start with the per-resolution math, because it scales with you in a way seats never did. Intercom Fin runs about $0.99 per resolution . Zendesk charges $1.50 per automated resolution on committed volume and $2.00 on pay-as-you-go overage , on top of a per-agent base. Salesforce Agentforce is $2.00 per conversation and requires Service Cloud Enterprise starting around $175 per user per month . Ada lands at roughly $1.00 to $3.50 per interaction with annual contracts that start near $30,000 and reach $100,000 to $300,000+ for enterprise .

The overage rate is where it bites. At 100,000 monthly resolutions, the difference between two vendors can be $51,000 a month, or $612,000 a year . That is not a rounding error. That is a headcount.

Then add the layers nobody quotes. Implementation and integration can add 20 to 30% to initial cost , and enterprise implementations for platforms like Ada run $40,000 to $100,000 depending on integration complexity . Budget an additional 30 to 50% beyond subscription in year one , dropping to 20 to 30% later as the build settles. Then there is the human you still pay to maintain the knowledge base, the helpdesk seats the AI needs to sit on, and the overage when a product launch spikes your volume 3x for a month.

The number you bring upstairs is not the per-resolution rate. It is the fully loaded three-year cost divided by resolutions, compared against your current cost per human ticket. That is the only framing a CFO trusts.

The adoption discount the CFO applies

Every CFO has been burned by software that got bought and never used, so they mentally discount your projected savings. They are right to.

Across all SaaS, 30 to 50% of software spend is lost to shelfware , and in customer relationship software specifically, Gartner found 42% of CRM licenses go unused .

The pattern repeats with AI support: the contract gets signed, the bot launches at a low resolution rate, the team loses faith, and tickets quietly route back to humans while the per-resolution meter keeps running on the few it still handles.

The thing that kills adoption in this category specifically is hallucination. Ungrounded chatbots hallucinate 15 to 27% of the time, while grounded LLMs drop to 0.7 to 1.5% . One confidently wrong refund answer in front of an angry customer, and your agents stop trusting the tool.

After that, no amount of license cost gets you adoption. So your rollout plan, not just your tool choice, is what determines whether the savings show up.

Now the conservative ROI anchor, because the vendor’s number is inflated. Vendors love to quote $3.50 returned per $1 and 210% three-year ROI . Do not put those in your board deck. Use the floor instead. IBM measured an average 30% operating cost reduction for tier-one support across 412 enterprises , and a board-credible net figure is 20 to 35% reduction with a 6 to 9 month payback for mid-market deployments with custom integration . The math underneath is simple and defensible: AI-handled tickets cost $0.50 to $1.05 each versus $8 to $12 for a human ticket . Apply that delta only to the share you genuinely resolve, not the share you deflect, and you have a number that survives the CFO’s discount.

The security and procurement gate

In this category, security review is not a checkbox at the end. Customer tickets contain order numbers, account details, sometimes payment context and health information, and all of it flows through a third-party LLM. Procurement will block the deal if the data handling is loose, and they should.

Treat each item below as pass or fail, with evidence, before you spend time on price.

• SOC 2 Type II report, current, with the actual report shared under NDA, not just a badge on the website. SOC 2 Type II is the floor, ISO 27001 is expected, ISO 42001 is increasingly required for AI systems .
• A signed DPA with sub-processor list, covering the LLM provider behind the platform.
• Zero data retention at the model provider, in writing. Some platforms retain conversation data by default and require explicit opt-out ; confirm yours does not train on your tickets.
• Real-time PII redaction that runs before the request hits the LLM, not retrospective scrubbing of logs after the fact.
• Data residency you can pin to your region (US, EU, or both) if you serve regulated customers.
• Encryption at rest and in transit, with key management you can document.
• Role-based access and audit logs for who can see ticket transcripts.
• GDPR alignment and, if you touch health data, a HIPAA BAA the vendor will actually sign.
• A defined incident response and breach notification window in the contract.
• Clear ownership of the conversation data, including export and deletion on contract exit.

If a vendor cannot produce the SOC 2 Type II report and a no-training-on-your-data clause, end the evaluation there. No resolution rate is worth an undisclosed data leak in front of your customers.

The buying committee, mapped

A mid-market AI support purchase touches six or seven roles, and each one can kill the deal for a different reason. Walk in with the evidence each one needs, or you will spend a quarter chasing approvals.

The mistake is treating this as your decision alone. It is the support leader’s tool, but it is the CFO’s budget, the security team’s risk, and the agents’ daily reality. Bring the right proof to each, separately, before the group meeting.

Running the trial like a test

A demo is theater. A proof of concept is the only thing that tells you what your real resolution rate will be, and it has to run on your tickets, not the vendor’s sandbox. Give it four to six weeks and design it like an experiment.

Pull a representative sample of your actual historical tickets, including the messy ones, refunds, account changes, and edge cases, not just password resets.

Connect the AI to a sandbox of your real backend so it can attempt true actions, because agentic platforms hit 70 to 85% resolution only when they have backend access . Then measure three things and write them down weekly.

Measure genuine resolution rate (problem solved, no human, no repeat contact in 72 hours), not deflection. Measure escalation quality by reading transcripts where it handed off, checking whether the human got context. Measure cost per resolution at the trial volume and project it to your full volume.

Run the same hard tickets through every shortlisted vendor so you are comparing on identical inputs.

One more test that catches the worst vendors. Deliberately feed it a question it should not answer, a policy edge case or a made-up product detail, and watch whether it hallucinates a confident wrong answer or escalates. That single test predicts your adoption curve better than any feature grid.

The one-page summary you bring to the C-suite

Strip the evaluation down to one page, because that is all the C-suite will read. Lead with the decision and the money, then the proof, then the risk you have already handled.

Open with the recommended platform and the one-line reason, tied to resolution rate and cost. Then the cost comparison: current cost per human ticket versus projected fully loaded cost per AI resolution, with the three-year total and the 6 to 9 month payback .

Show the POC result, your measured resolution rate on real tickets, not the vendor’s claim. Show the conservative savings, the 20 to 35% net reduction, applied only to resolved volume. Then one line each on security clearance (SOC 2 Type II, no model training on your data), the renewal cap you negotiated, and the rollout plan with the kill criteria.

The CFO wants to see you already thought about how this fails. That is what gets it signed.

Red flags that should end an evaluation

Some signals mean stop, not negotiate. A vendor that quotes deflection rate and dodges genuine resolution rate when you ask directly is hiding a weak product.

A vendor that will not share the SOC 2 Type II report under NDA, will not put a no-training-on-your-data clause in the contract, refuses a renewal-increase cap, or cannot demo a real backend action against your sandbox is telling you what year two looks like. If two or more of those show up, walk.

Questions buyers ask before they sign

How do I tell resolution rate from deflection rate?

Resolution means the customer’s problem was solved end to end with no human and no repeat contact within 72 hours. Deflection just means the AI ended the conversation, which it can do by sending someone to a help article that does not help.

Industry-average resolution is around 44.8% while deflection dashboards routinely show 80 to 90% . Always ask for resolution by cohort, with the definition in writing.

What should I actually budget beyond the per-resolution fee?

Plan for 30 to 50% on top of subscription in year one , covering implementation, integration, helpdesk seats, and knowledge-base upkeep.

Gartner found 71% of buyers undercount by at least 40% precisely because they model only the headline rate. Build the fully loaded three-year cost and divide by resolutions before you compare vendors.

Is per-resolution pricing better than per-conversation?

It depends on your resolution rate. Per-resolution (like Fin at $0.99 ) only bills when the issue is solved, which protects you when the AI is weak. Per-conversation (like Agentforce at $2.00 per conversation ) bills whether or not it worked, so you pay for failures too. Model both at your real volume and overage rate, since the gap can hit $612,000 a year at scale .

What is a realistic resolution rate to promise my CFO?

For a year-one deployment, 55 to 70% is realistic for a strong AI-native platform with backend access , and you should promise the lower end. Anything above 80% on a sales slide is either deflection rebadged or a cherry-picked cohort. Promise conservative, then beat it.

How do I keep ticket data out of model training?

Require zero data retention at the LLM provider and a contract clause stating your conversations are never used for training. Some platforms retain data by default and need an explicit opt-out .

Also require real-time PII redaction that runs before the request reaches the model, plus the SOC 2 Type II report and a signed DPA.

What ROI can I defend to the board without overstating?

Use the floor, not the vendor’s number.

A defensible figure is a 20 to 35% net operating cost reduction with a 6 to 9 month payback , built from the real delta of $0.50 to $1.05 per AI ticket against $8 to $12 per human ticket , applied only to genuinely resolved volume.

Skip the $3.50-per-dollar and 210% claims in the board deck.

How long should the proof of concept run?

Four to six weeks on your real historical tickets, connected to a sandbox of your actual backend. Shorter and you only see the easy tickets. Run identical hard tickets across every shortlisted vendor, measure genuine resolution and escalation quality, and deliberately test hallucination before you sign.

For the tools that actually clear this bar, see our tested ranking , and for how we score and verify every platform, read /about/methodology/ . If you are also weighing the broader stack, our customer success category hub lists the adjacent tools worth evaluating alongside.