How to tell if your AI support agent is any good

I once watched a team celebrate their AI support agent hitting 78 percent deflection. Balloons-in-the-Slack-channel level celebration. Three weeks later, their app store reviews started filling up with a new phrase: "impossible to reach support." The agent was not resolving 78 percent of questions. It was absorbing them, answering something adjacent, and customers were giving up.

The dashboard said success. The customers said maze. Both were describing the same system.

Deflection measures whether a conversation ended without a human, not whether the customer got what they came for. A customer who rage-quits after three wrong answers counts exactly the same as a customer who got a perfect answer in four seconds. If your primary quality metric cannot tell those two apart, you are not measuring quality. You are measuring silence, and silence has more than one cause. I have made the longer argument in stop counting deflected tickets and start counting revenue created, so here I want to focus on the practical question: what should you do instead?

The answer is an evaluation kit. Nothing fancy, no data science team required. Four practices, maybe three hours a month.

How do you actually test an AI support agent?

The same way you test software: with known inputs and expected outputs. The tool for this is what I call a golden-question set.

A golden set is 20 to 30 real customer questions where you know exactly what a correct answer looks like. Not questions you invented in a meeting; questions pulled from real tickets, with real customer phrasing, typos and vagueness included. For each one, write down the correct answer and the source it should come from (which page, which policy, which Q&A pair).

Build it to mirror your actual traffic. If a third of your tickets are shipping questions, a third of your golden set should be shipping questions. Then include a handful of each of these deliberately:

• Edge cases: the return question about a final-sale item, not the standard return question.
• Ambiguous phrasings: "it's not working" and other underspecified openers.
• Questions the agent should refuse: things outside your product's scope, where the right behavior is a graceful "I don't know" or a handoff, not an improvised answer.

That last category surprises people. An agent that never says "I don't know" is not a good agent; it is an unsupervised one. You need test cases that check the brakes, not just the engine.

Run the whole set against the agent and record what comes back. Now you need a way to score it.

Score correctness, not fluency

Here is where most informal evaluations go wrong. A fluent, confident, well-structured answer feels right, and modern models produce fluent, confident, well-structured answers even when they are wrong. Fluency is the one property you can safely ignore, because it is never the problem.

Score each answer on three questions instead:

• Correct? Is every factual claim in the answer true? One wrong detail (an old price, a wrong timeframe) fails the answer, no matter how good the rest is.
• Grounded? Does the answer come from your actual content, or did the model improvise? An answer can be accidentally correct but ungrounded, and ungrounded answers are the ones that mutate into confident nonsense after your next content change.
• Actionable? Does it resolve the question or restate it? "You can manage billing in settings" is true and useless; "Settings, then Billing, then Update Card" is an answer.

A simple sheet is all the tooling you need:

| # | Question             | Correct | Grounded | Actionable | Notes                    |
|---|----------------------|---------|----------|------------|--------------------------|
| 1 | Return w/o receipt?  | PASS    | PASS     | PASS       |                          |
| 2 | Ship to Canada cost? | FAIL    | PASS     | PASS       | Quotes old $12 rate      |
| 3 | Cancel anytime?      | PASS    | FAIL     | PASS       | Right, but not in docs   |
| 4 | API rate limits?     | PASS    | PASS     | FAIL       | Says "see docs", no nums |

Pass/fail per column, notes for anything interesting, done. An answer passes overall only when all three columns pass. Your golden-set score is simply the percentage of questions that fully pass, and that single number is worth more than any deflection dashboard, because you know precisely what it means.

Read twenty transcripts a week

Now the counterintuitive part. The highest-value evaluation practice I know involves no metrics at all: read twenty full transcripts every week, chosen at random. Not a dashboard summary. Not flagged conversations. The actual back-and-forth, top to bottom, twenty of them, randomly sampled.

This feels too manual to matter, which is exactly why almost nobody does it, which is exactly why the teams that do it have such an unfair advantage. Aggregate metrics answer questions you already thought to ask. Transcripts show you the failure modes you did not know existed: the question category you never anticipated, the polite customer quietly giving up on message three, the answer that is technically correct but tone-deaf, the loop your tripwires miss.

Twenty transcripts takes about thirty minutes once you get fluent at skimming. Keep a running note of patterns, and turn each pattern into either a content fix, a new Q&A pair, a handoff rule, or a new golden question. That last one matters: your golden set should grow out of your transcript reading, so your regression tests keep tracking the failures you actually see in the wild.

Watch resolution and handoff quality, not containment

When you do look at metrics, look at these:

• Resolution rate: of conversations the agent handled alone, how many actually ended with the problem solved? Post-conversation "did this solve it?" responses, no-repeat-contact within a few days, and your transcript reading triangulate this.
• Handoff quality: of conversations that went to a human, how many arrived cleanly, with full context, at the right moment? A well-timed handoff is a success, not a failure. Punishing handoffs in your metrics teaches everyone to build a bot that never surrenders, and that path ends in bot jail.
• Repeat contact rate: customers who come back about the same issue within 72 hours were not resolved the first time, whatever the first conversation's metrics claimed.

Notice what this reframing does to your incentives. Containment-thinking asks "how do we keep customers away from humans?" Resolution-thinking asks "how do we solve problems fastest, with whichever path fits?" Those produce very different systems, for the same reason that fast responses are not the same thing as good support: the customer's definition of success is "my problem is gone," and every metric you steer by should approximate that definition or be treated with suspicion.

The monthly evaluation ritual

Here is the whole kit assembled into a repeatable process. First run takes half a day; monthly runs take two to three hours.

1. Build (or refresh) your golden set: 20 to 30 real questions mirroring your ticket distribution, each with a documented correct answer and source, including edge cases and should-refuse questions.
2. Run every golden question against the agent in a fresh conversation and paste each answer into your scoring sheet.
3. Score each answer pass/fail on correct, grounded, and actionable. Compute the full-pass percentage and log it next to last month's.
4. For every failure, trace the cause: missing content, stale content, a bad Q&A pair, or a retrieval miss. Fix the source, retrain, and re-run the failed questions until they pass.
5. Read your twenty random transcripts (do this weekly, but fold the month's patterns in now). Convert recurring failures into content fixes and new golden questions.
6. Review resolution rate, handoff quality, and repeat-contact rate. Investigate any drop by reading the relevant transcripts, not by staring at the chart.
7. Retire golden questions that no longer reflect real traffic and add new ones, keeping the set between 20 and 30 so the ritual stays light enough to actually happen.

Step 7 is quiet but load-bearing. Evaluation rituals die from bloat, not neglect. A 200-question set you skip is worth less than a 25-question set you run every month.

If your platform makes retraining cheap (in Fetchply, for example, editing content or Q&A pairs and retraining is a couple of clicks), the fix-retest loop in step 4 tightens from days to minutes, which is what makes this ritual sustainable for a small team.

The uncomfortable truth underneath all of this: you cannot fully outsource judgment. The golden set, the scoring, the transcript reading, all of it works because a human who knows the business looks at real outputs regularly. Three hours a month of that beats any dashboard you will ever build.

• Deflection measures the absence of humans, not the presence of resolution; a rage-quit and a perfect answer score identically.
• Build a golden set of 20 to 30 real questions with known correct answers, including edge cases and questions the agent should refuse.
• Score answers on correctness, groundedness, and actionability; ignore fluency entirely.
• Reading twenty random transcripts weekly is the most underrated evaluation practice available.
• Track resolution, handoff quality, and repeat contacts instead of containment.
• Re-run the golden set after every retrain; treat retraining like a deploy that needs a test suite.

What is a good resolution rate for an AI support agent?

It varies too much by industry and question mix for a universal number. What matters is your trend: establish your baseline with the golden set and transcript reading, then improve month over month. A rising resolution rate with clean handoffs beats any borrowed benchmark.

How is deflection rate different from resolution rate?

Deflection counts conversations that ended without a human, whatever the outcome, including customers who gave up. Resolution counts conversations where the problem was actually solved. Only the second one correlates with customer satisfaction.

How big should a golden-question set be?

Twenty to thirty questions. Large enough to cover your main categories, edge cases, and should-refuse questions, small enough that you will genuinely re-run it monthly and after every retrain. A small set you use beats a large one you skip.

Can I automate the scoring with another AI model?

Partially. A model can pre-screen for obvious groundedness failures and save time, but a human who knows the business should make final pass/fail calls, especially on correctness. The judgment is the point; automate around it, not through it.