The Knowledge Base Problem

Hey Wranglers,

Deflecting your tickets to AI is one of those things that sounds simple until you try it.

The pitch is straightforward: your AI reads your knowledge base, an employee asks a question, the AI finds the answer and sends it. No agent required. Ticket deflected.

What actually happens: the AI confidently answers questions with information that was accurate in 2022, pulls from a document nobody has looked at in fourteen months, and occasionally sends an employee to a Confluence page that no longer exists.

Then someone gets a wrong answer, leadership hears about it, and the project stalls.

The technology worked fine. The knowledge base wasn't ready.

Here is the sequence I recommend before turning on AI deflection. None of it requires a technical background - you just need someone with ownership and a few hours a week for about a month.

Not the most complex tickets. The most frequent ones. The questions that come in over and over: how do I reset my password, what's the PTO policy, how do I submit an expense report, who do I contact about X.

These are the questions AI deflection will handle first, and they're almost never the ones that are best documented. Write them down in plain language, exactly the way employees ask them, not the way your documentation is organized.

For each of the 20 question types, find the document or policy that answers it. Then check three things:

If the answer to any of those is no, that document needs work before your AI touches it.

Most internal documentation was written by someone who knew the topic deeply and wanted to capture everything. That produces documents that are technically complete and practically unreadable. Even when the AI finds the right document, the answer may be buried inside a long, technical explanation or surrounded by caveats that make a concise response difficult.

For each document, ask: if an employee asked this question out loud, what's the two-sentence answer? Make sure that answer exists somewhere in the document, clearly, near the top.

If the answer to "how do I request time off" lives in three different places - the HR handbook, a Notion page, and a pinned Slack message that's two years old - your AI will find all three and may pull from the wrong one.

Pick one canonical source for each question type and make it clear which version is authoritative. Archive or delete the competing versions wherever possible. This is the part most teams skip because it feels tedious. It's also the part that makes the biggest difference.

Knowledge bases go stale because nobody owns them. Before you turn on deflection, assign a document owner for each major category and set a regular review cadence - quarterly for stable policies, and immediately after any process or policy change.

This doesn't need to be a big process - the goal is to catch drift before your AI starts confidently sending employees wrong information.

The playbook above focuses on improving your source material. But one of the most important trust-building controls is deciding upfront what the AI should not answer on its own.

Before launch, decide:

When the system is uncertain, the sources conflict, or the question involves a sensitive issue, it should escalate rather than improvise. This is what separates a deflection system employees trust from one they learn to work around.

Knowledge bases aren't AI-ready because they were built for a different job.

Internal documentation has historically been written for humans who can read between the lines, follow a link, skip the parts that don't apply to them, and ask a follow-up question if something doesn't make sense. AI is not a forgiving reader. It reads what's there and synthesizes an answer from it. If what's there is ambiguous, the answer will be ambiguous. If what's there is outdated, the answer will be outdated.

The other thing that's worth saying directly: most teams dramatically underestimate how scattered their documentation actually is until they go looking.

I've talked to IT leads who were confident their knowledge base was in reasonable shape, then spent an afternoon auditing it and found three different versions of the same policy in four different tools, two of which contradicted each other. That's not an edge case. That's most organizations.

The audit is uncomfortable. Do it before your AI does it for you.

The teams that get AI deflection right almost always did one thing the others didn't: they ran a soft launch with a small scope before opening it up broadly.

Instead of turning on deflection for all ticket types across all departments, they picked one department and one category of questions - usually password resets, software instructions, or other high-volume, low-risk questions.

They watched the outputs closely for two weeks, they fixed the documents that produced bad answers, and then they expanded.

This approach is slower on paper. In practice it's faster, because you're not spending three months rebuilding trust after a visible failure.

You're finding the gaps in a controlled environment before they become visible to the whole company.

Thanks for reading. As always, reply if something resonated or if there's a question worth digging into in a future issue.

Adam Smith
CEO & Founder, Wrangle

PS — If you've looked at your knowledge base recently, what's the oldest document you found that was still being actively used? Curious how far back the drift actually goes.