Managing Knowledge in the Age of AI: Myths and Truths

AI is changing so many aspects of post-sales support, including customer success, that I thought it would be useful to review some of the more common claims I’m hearing about knowledge management and put them to the test.

Knowledge management no longer matters

False

Seeing how deftly AI tools can pull information from unstructured data, it’s tempting to think that knowledge management is a quaint practice of the past. Just point your favorite LLM at your case-tracking tool, bug-tracking system, Teams, Slack, other collaboration tool, etc., and they will magically retrieve, organize, and summarize the information you need. You can certainly adopt this strategy with your internal team while you build a knowledge base. But the risks of exposing all these systems and the data they contain to customers are too great. Plus, without a defined “source of truth,” older, no longer accurate information may pop up.

Knowledge management is more important than ever as it allows you to define the source of truth for all users, and also for LLMs.

We must gate all content

Absolutely false

At a time when LLMs can consume all ungated content and regurgitate it using their special twists, it may seem prudent to hide all the good stuff behind the safety of a customer login. The problem, of course, is that a lot of support content is incredibly useful for prospects, including “invisible” prospects, not yet known to your sales team. Hiding content can backfire for your brand. Also, LLMs will use whatever alternate content is available if you gate yours, and that content may be outdated, misleading, or plain wrong. (See our recent post about DIY support.)

It is foolish to keep all the content behind gates, even if you decide to withhold some, as we discuss next.

We must make all content available to all

Maybe true

While I’m a great partisan of removing gates, there are good reasons to keep some content locked up for customer use. For instance, security or other major bugs can be catnip to competitors, or hackers. And you may want to keep content protected as a demonstration of value for paying customers. In general, vendors that cater to enterprise customers gate more content, while vendors that cater to consumers or SMBs gate less.

We can automate content management

True, to a very large extent

The main argument I hear against investing in formal knowledge management is that it requires a lot of resources. Actively managing knowledge is certainly not free, but AI tools make it very easy to create well-written, technically solid, perfectly formatted content from a case, a requirements document, a best-practice webinar, and other documents and recordings. Train everyone on the team on how to harness your AI tools and you will build your knowledge base in no time.

We should structure content differently

True

LLMs consumes information differently than humans. As of now, we need to make sure that both audiences are satisfied and, fortunately, what works well for humans can also work well for LLMs. Consider these principles:

Question/answer structure. LLMs love being able to map user questions to the exact query they get.
User voice. Your users, whether or not they employ an LLM, may not use your terminology so your knowledge base must meet them where they are, in their words.
Journalistic answers. Use topic sentences, like your 8th-grade English teacher told you.
High signal to noise content. Stay away from fluff (this is where the support knowledge base excels, or should excel, compared to marketing documents). Short and dense is best.
Contextual data. Make it clear if the answer applies only to a certain version or edition.
Definitive tone. LLMs don’t like hedged or imprecise answers. Humans don’t either.

Our users will tell us if things go wrong

True, but can you afford to wait?

Of course you want to collect user feedback, both for internal and external users, but most users won’t bother to speak up–and the consequences of a wrong answer can be embarrassing or even catastrophic. Regularly and systematically test your knowledge base, chatbots, and the LLMs that your customers use. Ask me how if you’d like an automated, no-hands-on solution.

What do you think? Please add your own myths in the comments and I will opine!