What to put in an AI knowledge base (and what to leave out)

The instinct is to gather everything — the folder of policies, the old handbook, the two hundred help articles — and upload the lot. It feels thorough. It reliably produces a worse assistant than a careful hour would have.

Work from the other end. Open your support inbox, your chat log, your phone, and write down the thirty questions people actually ask, in the words they actually use. That list is your specification. Everything else is a guess about what people want to know.

Now go and find which documents answer those thirty questions. It is usually a surprisingly small pile, and some of the answers turn out not to be written down anywhere — which is the most valuable thing this exercise produces, because those are the gaps that cost you support time every week regardless of whether you ever build an assistant.

Material that is current, specific, and written to be read by an outsider:

• Policies people are held to — returns, refunds, cancellation, shipping, warranty, admissions. High-stakes and constantly asked.
• Product and service detail — what’s included, what isn’t, compatibility, specifications, setup instructions.
• Process explanations — what happens after someone books, orders, applies, or enrols. These generate enormous question volume and are rarely documented properly.
• Troubleshooting — the ten things that go wrong and what to do about each.
• Your existing help articles, assuming they’re current. They were written for exactly this purpose.
• Recorded explanations — training sessions, walkthroughs, briefings. Video works alongside documents, and answers cite the timestamp.

One quiet advantage worth planning around: material that was never a file counts too. The explanation that only ever lived in an email or in one person’s head can be typed straight in as text. That is often where the best answers are.

Leaving things out is the skill. Four categories to keep firmly outside:

• Anything superseded. Last year’s price list, the previous policy, the old handbook “for reference”. This is the single biggest cause of bad answers, and it’s self-inflicted.
• Anything you wouldn’t publish. Internal margins, staff notes, supplier terms, anything with personal data in it. Assume everything you upload is quotable to whoever can reach the assistant, and decide on that basis.
• Drafts and proposals. A document describing what you might do reads exactly like a document describing what you do.
• Marketing copy where facts belong. A brochure saying you offer “industry-leading turnaround” gives the assistant nothing. The document that says “three working days” does.

There is a security-shaped version of this too. Prompts and permissions are not a substitute for not uploading the file — the reliable way to keep something out of an answer is to keep it out of the knowledge base.

If two documents answer the same question differently, you have not given the assistant more information. You have given it a coin to flip, and it has no way to know which document you consider authoritative — nothing in a PDF says “this one supersedes the other”.

Worse, the contradiction is invisible until a customer surfaces it. Everything looks fine in testing because you happened to ask in the words of the current document.

So before uploading: for each of your thirty questions, decide which single document is the authority, and remove or update the others. It’s tedious, it’s an hour, and it does more for answer quality than any amount of configuration. If your answers are wrong, this is the first place to look — not the settings.

A knowledge base decays. Policies change, prices move, processes get rewritten, and an assistant that was accurate in March is quietly wrong by September if nobody touches it.

The fix is not a review schedule that nobody keeps. It’s a habit: whoever edits a policy re-uploads it the same day. With a retrieval-based assistant that’s a file upload and the answers follow immediately — there’s no retraining step and no wait, which is precisely why retrieval rather than fine-tuning is the right architecture for this job.

Two habits worth adding. Re-run your thirty questions once a quarter and read the answers — it takes fifteen minutes and catches drift. And keep a list of what the assistant couldn’t answer: every honest refusal is a documentation gap, and that list is the best content roadmap you will ever get for free.

If you want to see the shape of this as a product, our knowledge base assistant does the version described here — your help material, your branding, a citation to the exact page on every answer.