The FT Word
The FT Word is a free monthly newsletter with support management tips. To subscribe, send us email with your email address. The subscription list is absolutely confidential; we never sell, rent, or give information about our subscribers. Here’s a sample.
Welcome to the July 2010 edition of the FT Word. Please forward it to your colleagues. (They can get their own subscription here.)
Topics for this month:
- Community metrics that make sense – why page views just don’t cut it
- Support’s involvement in product development – a special article by FT Word subscriber Jennifer Walker
- And, as always, an invitation to attend the upcoming Third Tuesday Forum breakfast, which will welcome Marty Messer of VMware to discuss what we in support can learn from Open Source. Sign up now.
Community Metrics that Make Sense
Now that many support organizations are diving, mostly happily, into communities, the popular question I’ve been getting is, “How do we measure success?” I’m still seeing a lot of page view counts as the one and only community metric (see rant at the end of this article), so clearly we have a ways to go as an industry. Here are the community metrics I like:
Adoption, aka registered users, as a percentage of the customer base. This number will give you a good idea of the penetration of the idea of community. If only a tiny fraction of potential users is taking advantage of the community, surely you have a ways to go. I much prefer working with the percentage of registered users rather than the raw number of users because the percentage automatically scales as the user base grows. (Naturally this metric only makes sense if you mandate registration for community users.)
A good target for the percentage of registered users would be 10%, and the more the better.
Activity, aka active users, as a percentage of registered users. While a registered user has taken the initial step to participate in the community (at least once!), an active user is someone who actually visits the community (in a given month, typically). So this number gives a good idea of how interesting and relevant the community is to potential users.
A good target for active users would be 10% again – the more the merrier. Alternatively you can measure visits rather than visitors. It’s a little different than unique users, but provides a similar feel for the community being good enough and interesting enough to return to it.
Participation, aka contributors, as a percentage of active users. Going one step further, a contributor is a user who not only visits the community but actually posts to it (either questions or answers). This number is interesting in that it gives you insight into how alive the community is (as opposed to depending on the support organization).
I like to see participation percentages of 10%, but I also notice that, with very large and successful communities, the number of contributors can be very tiny and still sustain the community very well, thank you.
Answers, aka the percentage of answers marked or rated correct or helpful. I’m less worried about this metric since few users bother to mark answers as correct (even though they are!) and if you have a good effectiveness rating (see below), it’s probably more meaningful than the answer rate anyway, but it’s still nice to know that you are delivering answers through the community.
The target here will depend on whether you expect users to rate answers or you have a rating system internally. Relying on user rating will mean a much lower percentage since users may not bother. But either method is just fine.
Effectiveness, aka the users’ rating of their experience with the community. This metric is more challenging to gather since it assumes (requires!) that you have a way of gathering user feedback. If your users are registered you can send them a quick follow-up email after a visit. All you need to ask is, “Did you find what you wanted?”
Harvesting, aka the percentage of threads transformed into knowledge base articles. I like to measure harvesting because, even if you have a fantastic search engine (lucky you!), it’s still nice as an end-user to be able to find nicely summarized questions and answers rather than plough through posts. Perhaps a matter of taste?
Here are some other metrics I don’t like very much:
- User growth. Great to know what usage is growing, but I much prefer to stick with the percentage of registered users since your overall pool of users may be growing as well.
- Content rating. I like using rating mechanisms as flags to attend to problems but let’s face it, very few users bother to rate content so relying on ratings is not a good idea.
- Page views. First there’s the bot problem, and since “automatic” traffic is probably not that meaningful you should leave out bot traffic. Second, even for real users page views are rather meaningless. The best community visit is a short one, so at least for support sites the notion of trying to increase page views is silly. (Using web analytics to figure out traffic flows can be very interesting, however. In particular I always like to know how customers get into the community pages.)
- Case deflection, when measured blindly. Sorry, community enthusiasts, community visits do not equal deflected cases! There are many situations where customers may visit the community (or other self-service resources) but would not log a case. Think of your online banking activity. In the olden days, before online banking, would you call your bank to see whether a check had cleared? Probably not, or very rarely. Today, you can do a quick check every few days if you’d like. This means satisfied customers, but not deflected cases. If you want to measure deflection do it seriously, by comparing incident rates (cases per customer per time period) over time.
Interested in best practices for communities? Contact me.
Support’s Involvement in Product Development
Many thanks to Jennifer Walker Principal Supportability Engineer at EMC for not only suggesting this topic, but actually writing the article!
All too frequently during the development cycle, Support is thought of as the team that only needs to be involved after the product goes out the door. Here are some tips on how to convince Development and Product Management executives to involve Support in all phases of software development – and how doing so can help drastically lower support and maintenance costs.
Product Inception. In the beginning stages of a product lifecycle, being able to provide support for that product once it’s released is often taken for granted. Having Support involved during a product’s inception will help avoid the risk of a project derailing down the road. During this stage, Support can determine, based on a product’s business case, whether it can provide support for the product based on their existing headcount or if additional headcount is required. If additional headcount is required, Support can determine if the revenue projections support the additional resources needed. Any Support resource risks can be raised and mitigated early on.
Requirements Definition. During this phase Product Managers can be primarily focused on the cool new features that are required for their product. The fact that some things might go wrong with this product once it’s deployed at customer’s sites can be easily overlooked. Having Support involved during the gathering of a product’s requirements can help ensure the necessary diagnostics tools; error messages, etc are included as requirements.
Design. Every software developer thinks their code is going to work perfectly but we in Support know that regardless of how talented a development team is, problems will occur once the product reaches the customer base. Support can use its knowledge from existing cases to validate/tweak design proposals in order to prevent future cases from occurring. Some things Support should look out for when reviewing design documents are things that fail silently with no errors returned, features that can’t be disabled, manual steps that could be automated, unsupported items that are being documented instead of enforced in the code, lack of tracing, logging, and diagnostics tools etc. If design document templates are used, ask Development to add a Diagnostics section to the template.
Testing. During this phase, Support can review testing plans and ensure realistic customer scenarios are included. Support can also ensure any tracing, logging; diagnostics tools etc are being tested.
Documentation. Having Support involved in reviewing customer product documentation can help prevent cases by ensuring the documentation makes sense from a customer perspective. Reviewing documentation will also ensure that Support agrees to provide assistance with anything documented or if needed, that the necessary Support disclaimers are included.
Maintenance. In addition to Support assisting customers during this phase, Support can perform root cause analysis on incoming cases to identify problem trends. This will enable Support to feed information back into Product Management, Development, QA, and Technical Publications in order to prevent these cases from occurring in the future.
To help streamline the collaboration between Support and R&D it’s very helpful to have a dedicated team within Support assigned to work on new product development. Good luck!
FT Works in the News
Customer Management Insight published an article I wrote entitled Selecting a Knowledge Management Tool for your Support Center in their June issue. You can read is at http://www.icmi.com/Resources/Articles/2010/June/Selecting-a-Knowledge-Management-Tool-for-Your-Contact-Support-Center.aspx?utm_source=Email&utm_medium=email&utm_content=062410%20Call%20Center%20Insider&utm_campaign=Newsletter#comments
Third Tuesday Forum
Are you based in the San Francisco area (or will you be there on Tuesday July 20th)? That morning, David Kay and I will be hosting The Third Tuesday Forum, a roundtable for support executives to discuss the topics we embrace and wrestle with every day. Our presenter will be Marty Messer from VMware, who will speak about what Open Source can teach us about better support. Turning transparency and accountability into competitive assets instead of feared liabilities should be a challenging topic– and as always, the Forum is PowerPoint-free! To register or for more details, click here. Space is strictly limited to ensure an interactive session.
If you cannot make it this time but would like to be on the mailing list, sign up. You will be the first to know about new events. You can also join the Third Tuesday Forum groups on LinkedIn and Facebook. And check out our speaker for July.
Curious about something? Send me your suggestions for topics and your name will appear in future newsletters. I’m thinking of doing a compilation of “tips and tricks about support metrics” in the coming months so if you have favorites, horror stories, or questions about metrics, please don’t be shy.
650 559 9826