Training an AI Agent: The Material Decides the Quality, Not the Model

When an AI agent gives a customer a wrong answer, the instinct is to blame the model. In practice, the overwhelming majority of wrong answers are a knowledge problem: the material did not contain the answer, or contained two contradictory versions of it, or contained a version that stopped being true eight months ago.

Fixing the model is not available to you. Fixing the material is, and it is the work that actually moves quality.

What should go into the knowledge base?

Things that are true, specific and already in use. In rough order of value:

1. Your price list, with the conditions attached. Not “prices from €40” but the actual structure, including what makes it more.
2. Policies: returns, delivery, cancellation, guarantee. The exact wording, including the exceptions.
3. Service or product descriptions, with the details customers actually ask about — materials, sizes, durations, what is included.
4. The answers you have already typed a hundred times. Export or copy your real replies. They are written in your voice and they are already known to work.
5. What you do not do. This is the most underrated entry. “We do not ship outside the EU” prevents a whole class of invented answers.

What should stay out?

• Marketing copy. “We are passionate about quality” contains no answer to any question and dilutes retrieval.
• Anything internal. Margins, supplier names, staff notes. An agent will use what you give it, and it cannot reliably keep a secret you put in its source material.
• Out-of-date documents kept “for reference”. The agent cannot tell which of two price lists is current. If both are in there, it will sometimes pick the wrong one, and that is your fault rather than the model’s.
• Long documents whose answer is one line. A forty-page brochure to establish the delivery time is worse than a sentence.

How should the material be written?

The format that retrieves well is close to how a careful colleague would write an internal FAQ:

• One topic per section, with a heading that names it. “Delivery times and costs”, not “Everything you need to know”.
• Self-contained paragraphs. A paragraph that says “as mentioned above, this also applies here” is meaningless when retrieved on its own.
• Explicit numbers and names. “2–3 working days”, not “quickly”. “Within 14 days of delivery”, not “a reasonable period”.
• Plain phrasing of the question somewhere in the text. If customers say “how long does postage take”, having that phrasing in the document helps it be found.

How do you tell the agent how to behave?

Separately from the knowledge. The knowledge is what is true; the instructions are how to act. Four instructions do most of the work:

1. Answer only from the supplied material. If it is not there, say so and offer a person.
2. Match the customer’s language. Someone writing in Spanish gets Spanish.
3. Keep it short. Direct is read on a phone. Two or three lines, answer first.
4. Never commit to something you cannot verify. Stock, a delivery date, an appointment slot, an exception to a policy.

How do you find out what is missing?

Not by imagining. By reading what the agent could not answer.

Any serious setup keeps a list of conversations where the agent said it did not know or handed over. That list is the single most valuable document in the whole system: it is your customers telling you, in their own words, what your material does not cover.

The routine that works is monthly and takes an hour: read the list, group it, and write the two or three paragraphs that would have answered the most common gap. Quality improves in steps, and each step is traceable to something a real person asked.

What do you do when it answers wrongly?

In this order, because the order saves time:

1. Find the source it used. If your tool can show which passage produced the answer, start there. Usually the passage is genuinely ambiguous or genuinely out of date.
2. Check for a contradiction. Two documents saying different things about the same policy is the most common cause of an answer that is right half the time.
3. Check whether the answer exists at all. If it does not, the agent was guessing, and the instruction not to guess needs to be stronger — or the material needs the answer.
4. Only then consider the instructions. Tone, length, over-eagerness.

What not to do: add a special-case rule for that one question. Three months of that and you have rebuilt a keyword tree behind the agent, with all of its brittleness and none of its predictability.

How much material is enough?

Less than people fear. A small business with a proper price list, three policy pages and twenty real answers usually has enough to handle most routine questions.

What matters more than volume is coverage of the actual distribution. Fifty pages about your philosophy and nothing about delivery is a worse knowledge base than one page that answers the four questions you are actually asked. The method for finding those four is in the FAQ bot article.

How often should it be updated?

Whenever the truth changes, and that is the discipline that decays first. A price change that reaches the website and not the agent produces a confidently wrong quote to a customer, and nobody finds out until somebody complains.

The practical safeguard is to keep the agent’s material as close to the source as possible — pointing at pages you already maintain rather than copies of them — so that updating one thing updates both.

What the agent should do when it is out of its depth is covered in when an AI should hand off to a human, and the distinction between an agent and a rule-based bot is in AI agent versus chatbot. Chatonaut’s AI agent is trained on material you supply; plans are on the pricing page.