When humans and LLMs want the same things out of the KB… and when they don’t

Are you proud of your knowledge base (KB)? If so, are you proud of how it works for human users (your customers, your team members), for LLMs, or (nirvana) for both? Let’s explore the many KB features that work well for both humans and bots, as well as the features humans may not need, but LLMs certainly do. Fortunately it’s not too hard to retrofit KB articles that already work well for users so they can also serve LLMs well.

Features that work well for both humans and bots

Many knowledge features that work well for humans also feed RAG models very well. So if your knowledge base is already working well for customers and team members, it will likely perform properly for your chatbot or third-party LLMs.

Collectively exhaustive and mutually exclusive

Fancy terms to mean that you should aim to cover all topics, and avoid overlaps. No overlaps is an absolute rule. Covering all topics is your job guarantee: you may never get there, but that’s the goal.

Precise titles

Don’t keep them guessing: write titles that describe the contents. Titles with questions perform well. Think (” How do I use X feature?” as opposed to “X Feature” or even “10 Things You Should Know About X Feature”)

One document, one topic

Cramming multiple topics into a single document (as we often do with legacy documentation, by the way) makes the document longer, hence forbidding, and harder to navigate. Humans are particularly allergic to multi-page documents so make everyone happy by sticking to one well-contained topic per document.

Short paragraphs

You’re not writing In Search of Lost Time: sentences and paragraphs need to be short and (for paragraphs) focused on a single topic. Proust was not writing KB articles.

Front-loaded answers

Don’t bury the lede. An information-full first sentence helps humans and LLMs both.

Consistent terminology

We often want to vary our writing, so will use different phrasing throughout an article. That can be confusing, both for newbies and also for LLMs. In support, we have the added constraint that customers may use terms that do not match the terminology on the marketing side. Create a list of product names, features, and concepts and stick to it. It’s very useful to include customer terminology on the list, with guidance on whether it can be used by itself, or always accompanied by the official terminology.

Up-to-date content

Old, no longer valid content gums up the work. And newer articles, including newly-updated articles, have more weight. If you are using a KCS model, or you have an effective process to review articles regularly, keep at it.

Features that LLMs need (and humans may not)

As we saw, many features that work well for humans also make LLMs perform well. Still, there are a few areas where LLMs have needs that humans do not have. Let’s take a look at them and how you can adapt an existing knowledge base to work better for LLMs.

Structured documents

Humans do very well, thank you, with smart formatting such as bolding text for emphasis, using white space, or using callouts. LLMs need HTML headers. Humans can easily see whether an article is an FAQ, or contains instructions steps. LLMs really benefit from adding an FAQ schema or a HowTo schema.

It’s a lot easier to maintain a knowledge base that has structured templates so start there. The extra work to add schemas is definitely worth it (and can be automated). Eventually, you can retrofit existing documents (start with with the most popular ones).

Alternatives to visuals

Humans love visuals. I’m not talking about pretty pictures, but rather artifacts like workflows, block diagrams, and the like. Provide captions for any visuals, and alternate descriptions for any visually-presented knowledge so LLMs can play, too.

Educate the KB authors on why they need to provide both visuals and annotations. If you are using KCS, make sure the coaches are helping authors stay on the right path.

Thorough explanations

Some humans do love thorough, detailed explanations, but most of us prefer more direct, shorter language. We are also able to make connections between content elements. For instance, we can instantly connect a question in the title of a document with the answer in the body. LLMs do better when questions and answers are “obviously” linked and when explanations are a little more thorough.

It’s somewhat challenging to strike the right balance, especially since testing the output of third-party LLMs is time-consuming. Tools such as isalo.ai can help.

Your to-do list

Is it worth constructing your KB so it can be used by LLMs? Absolutely. For one thing, you may offer your own chatbot, which will need the extra care (and if you do not currently use a chatbot, chances are very high you will, soon!) And your customers are already using their own LLMs to find answers about your products and services. If what they find is incorrect, your reputation will suffer.

Adapt your knowledge management strategy so the outcome serves LLMs better. You need better templates (perhaps), some education for the authors, and a touch of technology to add schemas. It’s doable, and so worthwhile.

Need help getting your KB ready for LLMs? Just ask. And please share your experience training your team to add LLM-friendly features to the KB.