How to write Q&A pairs your AI agent can actually use

A friend of mine runs support for a small ecommerce brand. When she set up her AI agent, she did what most people do: she exported the internal wiki, pasted a hundred entries into the Q&A section, and hit train. The agent got worse. Not slightly worse. Noticeably, embarrassingly worse.

The wiki entries were written for her teammates. They said things like "Escalate per the RMA matrix" and "See fulfillment SLA doc." Customers do not ask about RMA matrices. They ask "my order came broken, what do I do?" The agent had a pile of answers, and almost none of them matched a question a real human would ever type.

Here is the thing I want you to take away before anything else: Q&A pairs are the single most direct way to shape what your AI agent says, and also the easiest input to get wrong. A crawled website gives the agent broad context. A Q&A pair gives it a precise, pre-shaped answer to a specific intent. That precision is a superpower when the pair is good, and a liability when it is not.

Why Q&A pairs punch above their weight

Most AI support agents learn from three kinds of material: crawled web pages, uploaded files, and Q&A pairs. The first two are bulk knowledge. The agent reads them, chops them into chunks, and pulls relevant pieces when a question arrives. That works, but it is indirect. The agent has to find the right chunk, interpret it, and compose an answer.

A Q&A pair skips most of that. You are telling the agent, in effect: "When someone asks this, say this." It is the closest thing you have to putting words directly in the agent's mouth.

That is exactly why quality matters so much. A bad web page dilutes the knowledge pool a little. A bad Q&A pair actively injects a wrong or confusing answer at the exact moment a customer asks the matching question. The blast radius of a bad pair is bigger because the match confidence is higher.

So let us talk about what makes a pair good.

What does a good Q&A pair actually look like?

After watching a lot of teams do this well and badly, the pattern comes down to four properties.

One intent per pair. A pair should answer exactly one question. If your answer contains the word "also" doing heavy lifting ("You can return items within 30 days. Also, exchanges work differently..."), you have two intents fused together. Split them. When intents are fused, the agent retrieves the whole blob for both questions and buries the relevant half.

Phrase the question the way customers phrase it. This is the mistake my friend made. Your internal vocabulary is not your customers' vocabulary. You say "RMA," they say "send it back." You say "provisioning delay," they say "why isn't my account working yet." Go read ten real tickets and steal the exact wording. The question side of the pair is a matching target; the closer it sits to real customer language, the more reliably it fires.

Make the answer self-contained. The answer should stand alone without the customer needing to know anything else. "See our returns page" is not an answer. "Yes, you can return any unworn item within 30 days for a full refund. Start at fetchply.example/returns and you'll get a prepaid label by email" is an answer. Include the number, the timeframe, the link, the exception. Assume the reader has zero context, because the model composing the reply effectively has zero context beyond what you gave it.

Cover the variants that matter. Customers ask the same question five different ways: "do you ship to Canada," "can I order from Canada," "international shipping?" You do not need a separate pair for every phrasing (the model generalizes reasonably well), but you should cover genuinely different angles of the same topic. "Do you ship to Canada" and "how much is shipping to Canada" and "how long does Canadian shipping take" are three different intents wearing similar clothes.

Now here is the counterintuitive part: the best Q&A writers spend more time on the questions than the answers. The answer is usually sitting in your head or your docs already. The question is the retrieval key, and a beautiful answer behind a question nobody asks is a beautiful answer nobody ever sees.

Bad pairs versus good pairs

Concrete examples beat theory, so here are three before-and-after pairs pulled from patterns I see constantly:

BAD
Q: What is the RMA procedure?
A: Refer to the returns policy for RMA eligibility and process.

GOOD
Q: How do I return something that doesn't fit?
A: You can return any unworn item within 30 days for a full
refund. Go to yourstore.com/returns, enter your order number,
and we'll email you a prepaid shipping label within a few
minutes. Refunds land 3 to 5 business days after we receive
the item.

BAD
Q: Billing information
A: We support monthly and annual billing. Invoices are
available in the dashboard. Also, you can update your card
under Settings, and annual plans get a discount, and for
enterprise billing contact sales.

GOOD
Q: How do I update my credit card?
A: Go to Settings, then Billing, then Payment Method, and
click Update Card. The change applies to your next invoice
immediately. If a payment already failed this month, we'll
retry it automatically within 24 hours of the update.

BAD
Q: Do you integrate?
A: Yes, we have many integrations.

GOOD
Q: Does this work with Shopify?
A: Yes. Install the app from the Shopify App Store, connect
your store, and the agent can look up order status and
recommend products from your catalog automatically. Setup
takes about five minutes and doesn't require any code.

Notice what changed each time: one intent, customer phrasing, and an answer that resolves the question completely instead of pointing somewhere else.

When to write a pair, and when to fix the page instead

This is the judgment call people skip. Not every gap deserves a Q&A pair.

A Q&A pair is the right tool when the answer is short, stable, and frequently asked; when customers phrase the question very differently from how your docs phrase it; or when you need a precisely worded answer for something sensitive (refund exceptions, cancellation policy) where you cannot afford paraphrase drift.

Fixing the source page is the right tool when the underlying documentation is wrong, outdated, or missing. If your shipping page says 5 to 7 days and reality is 10, a Q&A pair saying 10 days just creates a contradiction the agent has to arbitrate, and it will not always arbitrate in your favor. Never use Q&A pairs as patches over broken documentation. Fix the page, retrain, and reserve pairs for the intent-matching work pages cannot do. If you have not audited your content yet, do that first; I wrote up the full process in how to prepare your content before training an AI support agent.

And some questions deserve neither a pair nor a page. Anything requiring account lookups, judgment, or empathy belongs on a different path entirely; the triage framework in not every customer question deserves AI covers how to sort those.

Mining pairs from real conversations

The best Q&A pairs are not invented in a brainstorming session. They are excavated from support history. Here is the process I recommend, and it takes about two hours the first time:

1. Export your last 100 to 200 support conversations from your helpdesk or inbox. Real tickets, not your FAQ page.
2. Tag each conversation with the customer's underlying intent in a few words ("return without receipt," "change delivery address"). Ignore your agent's replies for now; you only care about what was asked.
3. Sort by frequency and keep intents that appear three or more times. These repeats are your candidate pairs, and the long tail of one-offs is mostly noise.
4. For each candidate, copy the best real phrasing of the question verbatim, typos cleaned up but vocabulary intact.
5. Write a self-contained answer using the checklist above: one intent, complete resolution, concrete numbers and links, no "see our docs."
6. Check each new pair against your existing trained content for contradictions, then retrain the agent.
7. Two weeks later, review conversations where the agent handled these intents. Tighten any pair that produced a wobbly answer, and add newly surfaced variants.

The frequency filter in step 3 is doing quiet, important work. Twenty pairs that cover your real top twenty intents will outperform two hundred speculative pairs every single time, because every pair you add is another document competing for retrieval.

A note on maintenance

Q&A pairs rot quietly. The pair about your pricing is accurate until the day you change pricing, and then it is a confident, precise, wrong answer. Put a recurring reminder on your calendar (monthly is plenty for most teams) to skim your pairs for anything that references prices, timeframes, policies, or product names that may have shifted. Delete ruthlessly. A missing pair degrades gracefully into a generic answer or a human handoff; a stale pair degrades into misinformation delivered with total confidence.

If you use a platform like Fetchply, pairs live alongside your crawled site and uploaded files, and retraining after an edit takes a click, so there is no excuse to let a known-stale pair sit for weeks.

• Q&A pairs put words directly in your agent's mouth, so a bad pair does more damage than a bad page.
• One intent per pair; split anything held together by "also."
• Write questions in customer vocabulary, stolen from real tickets, not internal jargon.
• Answers must be self-contained: numbers, timeframes, links, exceptions, no pointing elsewhere.
• Fix broken source pages instead of patching over them with pairs.
• Mine pairs from your last 100 to 200 real conversations and keep the intents that repeat.

How many Q&A pairs should I start with?

Twenty to forty well-mined pairs covering your most frequent intents beats hundreds of speculative ones. Start small, watch real conversations for two weeks, then add pairs for the gaps you actually observe.

Should I write a separate pair for every phrasing of a question?

No. Modern agents generalize across phrasings well. Write separate pairs only for genuinely different intents, like shipping cost versus shipping time, even when they mention the same topic.

What if a Q&A pair contradicts my website content?

The agent may answer from either source, so treat contradictions as urgent. Fix whichever source is wrong, keep both aligned, and retrain. Reserve pairs for intent matching, not for overriding pages you know are outdated.

Can I just import my internal knowledge base as Q&A pairs?

Usually not directly. Internal articles use staff vocabulary and assume context customers lack. Translate each one: rewrite the question the way a customer would ask it and make the answer self-contained before importing.