How to Evaluate AI Coding Agents (Before the Token Bill Eats the Budget)

You run engineering or platform tooling at a growing company, every developer is already pasting code into some chatbot, and someone above you just asked you to “pick an AI coding agent and tell me what it costs per year.”

You are the person who has to evaluate AI coding agents, choose one, roll it out, and then defend that spend to a CFO who does not care that the tool writes a slick React component. Here is the 60-second version.

The per-seat price is no longer the number that matters, because the whole category just shifted to usage-based token billing, and a single power user can run a $39 seat into hundreds of dollars a month. The biggest risk is not picking the wrong AI coding agent, it is buying it for everyone and watching less than half your developers open it in a given week.

And the code these agents write carries measurably more security holes, which is a real cost line, not a footnote. Evaluate for true usage cost, real adoption, and code security first. The autocomplete demo comes last.

The buying problem before the buying

Most AI coding agent evaluations start in the wrong place. Someone watches three demo videos, gets dazzled by an agent that scaffolds a to-do app in 40 seconds, and ranks the tools by how impressive the autocomplete feels. That demo is how you end up paying for a tool your seniors quietly turn off and your juniors use to ship bugs faster.

Here is the failure as a number. In a controlled trial, experienced developers were actually 19% slower when allowed to use AI tools, even though they believed they were 20% faster . That perception gap is the trap.

A tool can feel like a rocket and still slow your best people down on code they know cold.

The deeper problem is the usage motion, and it changed under everyone’s feet in 2026. An AI coding agent is not bought once at a flat seat price anymore.

GitHub Copilot moved every plan to usage-based token billing on June 1, 2026 , and Cursor already runs a credit pool that depletes by which model you pick. Two motions now.

The seat motion you can forecast, and the token motion that moves with how hard each developer leans on agentic, multi-file work.

That second motion is where the bill goes sideways. After Copilot’s billing change, developers reported costs jumping from $29 to $750 a month and from $50 to $3,000 once agent usage ramped.

The real question is not which AI coding agent writes the nicest code. It is which one your developers still use in 14 months, at a token bill you can predict, producing code your security team will sign off on. Everything below scores for that.

The weighted scorecard for AI coding agents

Score every candidate against the same 12 criteria, with the same weights, before anyone watches a demo. The weights matter more than the criteria. They drag the conversation away from the autocomplete sizzle reel and toward the things that actually decide whether this purchase survives a CFO review. Demand evidence for every line.

A vendor claim is not evidence. A token bill from your own two-week trial, a contract clause, or a result on your own repo is.

The weights are deliberate. Usage cost, adoption, and code security carry 39 points between them because those three are where AI coding agent deals quietly fail.

A tool can win the demo and still be the wrong call if the token bill triples the month your team goes agentic, or your seniors stop using it, or it floods your codebase with subtly insecure code that passes review.

The true multi-year cost of AI coding agents

The pricing page lies by omission. It shows you a tidy per-seat number. It does not show you the part of an AI coding agent budget that now hurts, which is tokens. The seat is the floor, not the bill.

GitHub Copilot Business stays at $19 per user per month and Enterprise at $39 , but those numbers are now just an included AI-credit allowance, and everything past it bills at per-token rates.

Run the real math. Say you buy 100 seats. The license alone looks predictable. Then your engineers start using the agent the way the vendor advertised, and the token motion kicks in.

Cursor itself says daily Agent users typically need $60 to $100 a month in total usage, and power users often need $200 or more , which is why its Ultra plan is priced at $200 a month to cap that anxiety.

Multiply your heaviest 10% of developers by $200 and the seat price stops being the story.

Then there is everything that is not the tool. A DX breakdown puts the real annual cost for 100 developers at $66,000 or more, with teams paying 2 to 3x the advertised monthly rate once training, security review, and admin overhead are counted .

Budget $50 to $100 per developer for enablement , and expect a productivity dip while people learn the tool. None of that is on the demo.

The thing that makes this category nastier than most is that the variable bill is unbounded by design. Copilot’s own change removed the old fallback to a cheaper model when credits run out, replacing it with budget caps an admin has to set .

Without a cap, a single developer running long agentic sessions can spend more in a month than their salary covers in tooling for the quarter. The CFO has seen the sticker-versus-real gap before.

Walking in with the all-in three-year number, with token overage on your heaviest users modeled and a hard budget cap already in place, is what separates an approved request from “come back with real numbers.”

The adoption discount the CFO applies

A CFO who has bought software before mentally discounts every ROI slide you bring. They are right to. The reason is adoption, and for AI coding agents the adoption gap is brutal precisely because the demo is so good.

At a 500-person engineering org that bought Copilot for everyone, only about 45% opened it in a given week , and the rest was expensive shelfware. Access is not habit.

It gets harder. Even teams that hit 100% license coverage routinely stall at 30 to 40% weekly active users , and trust is collapsing alongside usage.

Only 29% of developers say they trust AI tool output, down from over 70% in 2023 , and just 16.3% report it made them significantly more productive while 41.4% say it had little or no effect. An AI coding agent your seniors do not trust is one they quietly stop opening.

Now anchor the ROI conservatively, because a CFO believes a modest number faster than a vendor’s best case. Healthy returns on AI coding tools land around 2.5 to 3.5x on average, with top-quartile teams reaching 4 to 6x , but the honest comparison number is hours, not a multiple.

Developers save roughly 3.6 to 4 hours a week on average when they actually adopt the tool. Build the case on those hours at a realistic adoption rate, not the vendor’s 6x ceiling.

Here is the honest framing for finance. If the AI coding agent pays back when only 45% of your licensed developers use it weekly and you account for the 19% slowdown your seniors may hit on familiar code , you have a number that survives scrutiny.

If it only pays back at full adoption and a 6x return, you do not have a business case, you have a hope. Build for the 45% reality, then treat anything above it as upside.

The security and procurement gate

This is where AI coding agent deals die, and most buyers underestimate it. Reportedly 73% of AI coding tool implementations are terminated by enterprise security reviews , because vendors treat security as an afterthought and buyers bring it up too late.

Walk into security with evidence already collected, or your evaluation stalls for a quarter.

The category has two security problems most software does not.

First, the code itself is riskier: AI-generated code carries 2.74x more vulnerabilities than human-written code , and even AI-generated patches introduce new vulnerabilities in about 9.5% of cases while fixing the original bug.

Second, your proprietary code, API keys, and customer data patterns flow to an external model unless you contractually stop them.

Treat this as a pass or fail checklist, collected before procurement, not during:

• Current SOC 2 Type II report, reviewed under NDA, proving controls operate over time rather than on paper.
• A zero-retention or training opt-out clause in writing, so your code is processed but never stored for model training.
• Contractual IP indemnity covering generated code, so you are not liable if the agent reproduces licensed code.
• Signed Data Processing Agreement covering any customer or EU data that could appear in your repos.
• Data residency with US or EU region pinning, plus a self-hosted or VPC option for regulated teams.
• SSO and SAML on the tier you are actually buying, not gated several tiers up.
• An integrated or compatible security scanner that catches the extra vulnerability rate before code merges.
• Admin budget caps and usage analytics, so an unsupervised agent cannot run up an unbounded bill.
• A sanctioned, governed deployment, because if you do not provide one, developers run shadow AI on their own.
• Audit logging of who prompted what, and the right to export your data and turn the agent off cleanly.

That last point matters more here than in most categories. When organizations fail to provide a sanctioned tool, developers do not stop using AI, they take it underground as shadow AI , pasting proprietary code into unvetted consumer tools.

Choosing and rolling out an approved AI coding agent is itself a security control.

The buying committee, mapped

No AI coding agent gets approved by one person. Map the committee early, learn what each role actually worries about, and bring the specific evidence that closes their objection. Surprise objections at the finish line are how a six-week evaluation slips into next quarter.

The CFO worries about predictable spend and a believable payback. Bring the all-in three-year cost with token overage modeled, a hard budget cap, and a conservative hours-saved ROI at 45% adoption. The engineering VP worries about whether it makes the team faster without wrecking code quality.

Bring trial results from real tickets plus the security-scan diff on AI versus human PRs.

The security or AppSec lead worries about code risk and data leaving the building. Bring the SOC 2 Type II report, the zero-retention clause, the IP indemnity, and the vulnerability scan from the trial. Procurement worries about contract terms and the unbounded token bill.

Bring the written budget cap, the overage rate, and the data-handling and exit clauses.

Your senior developers worry about whether the tool actually helps or just gets in the way. Bring the pilot feedback and their own week-three usage numbers, not a vendor demo. The executive sponsor worries about whether this was a defensible call. Bring the one-page summary that ties cost, adoption risk, code security, and the recommendation together.

Running the trial like a test

A vendor demo is theater. Run your own trial as a controlled test on your own code, because that is the only evidence that survives a committee. Pick one pilot team, five to eight engineers spanning seniority, and run at least three weeks to see past the honeymoon.

Give the agent real work. Have it complete three actual backlog tickets in your real repo, not a greenfield demo app, so you see how it handles your conventions and legacy code. Run a multi-file agentic task and watch where it acts without asking, because that autonomy is where both the value and the risk live.

Then scan the AI-authored PRs against human-authored ones and count the difference in findings.

Watch the token bill the whole time. Turn on the usage dashboard day one, set a budget cap, and project per-developer spend across your heaviest 10% of users, not the average. Track time-to-first-merged-PR and the daily-active rate in week three, because the honeymoon week always looks good.

The trial that tells the truth runs on your code, with your cap on, long enough for the novelty to wear off.

The one-page summary you bring to the C-suite

The deck that gets approved is short. One page, four numbers, one recommendation. Lead with the all-in three-year cost, broken into seats, modeled token overage on your heaviest users, enablement, and security review, with the hard budget cap stated as a line item. That number, with the cap, is what a CFO actually reads.

Then the conservative ROI: hours saved per developer at 45% adoption, not the vendor multiple, with the 19% senior-developer slowdown acknowledged rather than hidden.

Then the security verdict in one line: SOC 2 Type II confirmed, zero-retention and IP indemnity signed, vulnerability-scan delta measured and mitigated. Then the adoption evidence from your own pilot: week-three weekly-active rate and time-to-first-merged-PR.

Close with the recommendation and the one risk you are accepting, named plainly. A committee trusts a buyer who states the downside out loud. For the underlying tool-by-tool data, point them to our tested ranking and to how we test so the numbers have a visible source.

Red flags that should end an evaluation

Some signals mean stop, not negotiate. The first is the unbounded bill. If the vendor will not put a hard budget cap and the per-token overage rate in writing, the variable bill is designed to surprise you, and you have no defense when a power user spends $3,000 in a month. Walk, or do not sign until that cap exists.

The second is the security and trust stack failing together.

If the trial is a canned demo with no access to your real repo, the vendor will not commit to zero-retention or IP indemnity, SOC 2 Type II or SSO is gated above the tier you can afford, and the AI-versus-human scan shows a flood of new findings, you are looking at a tool that security will reject and developers will stop trusting.

That combination ends the evaluation.

Questions buyers ask before they sign

How much does an AI coding agent really cost beyond the per-seat price?

Plan on the all-in number dwarfing the seat. The license is just an included token allowance now, and everything past it bills per token.

A DX breakdown puts the real annual cost for 100 developers at $66,000 or more, with teams paying 2 to 3x the advertised rate once enablement, security review, and admin are counted.

Cursor itself says power users need $200 a month or more in usage . Budget on the three-year all-in figure with token overage on your heaviest users modeled, not the pricing page.

What changed with GitHub Copilot’s billing in 2026?

Copilot moved every plan to usage-based token billing on June 1, 2026 . The base prices held (Business at $19, Enterprise at $39), but those now buy an AI-credit allowance, and usage past it bills at per-token rates.

The old fallback to a cheaper model when credits ran out was removed. Developers reported bills jumping from $29 to $750 and $50 to $3,000 . Set admin budget caps before you roll out, not after.

What ROI number is safe to put in front of finance?

Anchor low and you will be believed. Average returns land around 2.5 to 3.5x with top teams at 4 to 6x , but build your case on hours, not the multiple. Developers who actually adopt save roughly 3.6 to 4 hours a week . Model that at 45% adoption and acknowledge that experienced developers can be 19% slower on familiar code . A case that pays back at half adoption survives scrutiny. One that needs full adoption and a 6x return does not.

Why do AI coding agent rollouts end up as shelfware?

Adoption and trust. At a 500-person org that bought Copilot for everyone, only about 45% opened it weekly , and teams routinely stall at 30 to 40% weekly active even with full coverage.

Trust is falling too: only 29% of developers trust AI output, down from over 70% in 2023 . Buying everyone a seat does not create the habit. Watch week-three usage in the pilot, because if your seniors stop opening it, you are funding shelfware.

Is the code these agents write actually less secure?

Yes, measurably.

AI-generated code carries 2.74x more vulnerabilities than human-written code , and AI-generated patches introduce new vulnerabilities in about 9.5% of cases even while fixing the original bug.

That does not mean do not buy, it means budget for the scanning and review that catches the extra findings. Run a security-scan diff on AI versus human PRs during the trial so you size the review cost honestly.

What security and IP evidence do I actually need to collect?

The non-negotiables are a current SOC 2 Type II report under NDA, a written zero-retention or training opt-out clause, contractual IP indemnity on generated code, and SSO on your buying tier.

Reportedly 73% of AI coding tool implementations die in security review , so collect this before procurement, not during. Add a DPA, data residency pinning, and a self-hosted option if you are regulated.

The IP indemnity matters more here than in most categories because the agent can reproduce licensed code.

Should I worry about lock-in to one model?

Some. The tools that let you pick among several frontier models, and that publish a clear deprecation policy, protect you when a model is retired or its price moves.

Pricing risk is real and recent: the whole category just flipped to token billing , and Microsoft reportedly cancelled most direct Claude Code licenses about six months after granting them.

Demand the supported-model list and the deprecation policy in writing, and confirm your token overage rate is fixed, not floating with model prices.