Measuring Case Deflection, and Other Tales of Knowledge Management Effectiveness

How do I know that my knowledge management program is working? Knowledge management is not free, so it would be good to know whether it’s bringing tangible benefits. Here are two approaches and eight techniques, with descriptions and pros and cons.

Approach #1: Are we deflecting cases?

Many knowledge management ROI analyses rely on replacing cases with searches. It’s a very narrow view (unacceptably narrow, if you ask me!), but a very common one.

Deflection during case creation

Defined as the ratio of website users who embark on a case creation process (click the Create Case button) but do not complete the process (click the Submit Case button) after being presented with search results.
This assumes that there is a case creation process that includes an automated presentation of solutions.
It is the most restrictive way to look at case deflection, since users may well do the right thing (perform a search) before they start along the path of creating a case.

Deflection through website search

Defined as the ratio of website users who perform a search but leave before they create a case.
You can use this whether or not your case creation process includes automated suggestions.
Note that users may well do a search for something they would not log a case for. Or users who search may not be authorized to log cases. (Both situations unduly inflate the deflection number.)
It only considers users who perform the search on the website itself. Check your website analytics! In many circumstances, a third or more of the visits to the KB start with an internet search, not on your lovingly-curated marvel of a website.

Deflection through website usage

Defined as the ratio of website users who use the website but do not create a case.
If users often visit the website for reasons other than case creation (e.g. to download software, check on case progress, or other), the ratio will be mistakenly large.

Deflection through KB usage

Defined as the ratio of users who look up knowledge on the website and leave before they create a case.
This includes users who start their searches on the internet rather than the website, so is more inclusive than the deflection through website usage.

Incident rate decrease

Defined as the decrease in the number of cases created per user after implementing a knowledge base program or making a significant improvement.
It’s not an ideal metric since the decrease may be attributable to all kinds of other factors, and it’s not very helpful with an ongoing knowledge management program.

Approach #2: Look at internal benefits

Knowledge management has many internal benefits so stopping at case deflection is shortsighted.

Resolution time gains for cases that reuse knowledge

Defined as the ratio of resolution time (or effort time, if you measure that) for cases linked to knowledge articles compared to average.
This will always be a happy metric since cases linked to knowledge are also less complex, so look for ongoing positive trends, proving that the knowledge management program has long-term effects.

Overall resolution time decrease

Defined as the decrease in resolution time (or effort time, if you measure that) for all cases, before and after the introduction of knowledge management, or ongoingly.
If you are experiencing a big increase in deflection, resolution time may actually go up for a while as the easy cases are deflected and the ones that are left are more complex! But it’s a good measurement for ongoing knowledge management efforts.

Case productivity increase

Defined as the increase in cases closed per staff member
Case productivity is determined by many factors, and if case complexity increases (because of high deflection rate), this number may dip rather than increase, but it’s not a bad number to monitor for the long run.

Customer productivity increase

Defined as the increase in the number of customers supported per head
This metric encapsulates all your efforts–and it is problematic for the same reason: what is the part of knowledge management versus everything else.

Finally, I specifically ignored increases in customer and employee satisfaction. They matter, too!

How do you measure the success of your knowledge management program? Please share in a comment.