Working with Engineering — or why won’t they fix any bugs?

Many thanks to Kevin Williams for suggesting this topic (the subtitle is all mine!)
Most support organizations manage customer issues, troubleshoot them, suggest workarounds, but stop short of fixing any bugs discovered as a result of the investigation — that’s the province of the Engineering team. (More on support organizations that tackle the bug fixing challenge themselves in point #10, below). And the Engineering team may not exactly be falling over itself to fix the bugs. So what’s a support executive to do?
1. Understand that engineering teams are rewarded for delivering features, on time — not to fix bugs
It may come as a great surprise that almost all Engineering teams are not rewarded at all for fixing bugs. Instead, their incentives rely on the timely delivery of features, as embodied in new products and new releases of existing products. So when Support folks come, hat in hand, asking for a fix, the Engineering team is doing them a favor by creating a fix, if it interprets its formal objectives in a literal sense.
If the Engineering team also has quality goals, the goals are likely to focus on bugs found per line of code, which means that the incentive is now to declare as many problems as possible as “not bugs”. Sounds familiar?
So one of the strategic changes you can work on, as a support executive, is to influence the design and targets of Engineering goals so they match more closely with the goals of Support — and more importantly with the goals of customers.
2. Understand that bug fixing is difficult and un-glamorous

Even if Engineering has incentives to fix bugs, it’s still the case that fixing bugs is less interesting than other work. Wouldn’t you rather develop a new, “cool” feature than dredge through old code, which you probably did not write and is not properly commented, looking for an obscure misfunction? Wouldn’t you rather participate in the scheduled, sometimes biting, but collegial code reviews for new features than be subjected to the hourly “are we there yet?” questions from pressured support engineers, panicked salespeople, and the occasional customer on the warpath?
This means that many engineering organizations start new engineers in the bug-fixing business, with the idea that they will gain an understanding of the code as they explore it to find and fix errors. And it works pretty well, as a training tool — but it’s not a great feeling for our support bugs, is it? Not all organizations do this. Some organizations ask their star developers to fix bugs — but it’s rarely their favorite activity.

3. Develop a personal relationship with the Engineering team

There are practical ways to improve the bug-fixing process, as we shall see below, but nothing replaces a positive personal relationship with your Engineering counterparts, at all levels, from executives on down. I find that many Engineering organizations are doubtful of the technical skills within Support, so the goals are to (1) position the support team as experts in the customer-management domain, which most engineers know they are not so good at so they like the fact that someone else will take care of it for them and (2) establish at least a few individuals within the support team are technically competent and trustworthy when it comes to technical issues (so above and beyond our skills as customer managers).
To accomplish (1), give the Engineering team real-life demonstrations of saying no to customers. There’s nothing so powerful as when we can turn down a customer request firmly but politely and still retain the trust of the customer. Easily done during most escalation calls.
For (2), hire well and allow the most technically-gifted support engineers to show their stuff with their counterparts.
4. Craft a service-level agreement (SLA)

It’s a great idea to create a formal service-level agreement with the Engineering team to include:
  • What issues should be handed off and what type of troubleshooting should happen before the handoff. It’s completely reasonable to expect to perform a number of tests or obtain certain logs before handing off an issue. Resist the pressure to have to fill out oodles of forms to cover all kinds of potential issues, just in case. I call that the “Go get a rock” approach, designed to limit as much as possible the number of issues that bubble up to Engineering — and provoke very unhappy reactions from support engineers and customers alike. Use your technical clout to design a process that’s complete, but not unnecessarily complex.
  • The process for handoffs. Usually this will be through a tool, often the bug-tracking tool. If that’s the case, how do you obtain troubleshooting help, outside an identified bug?
  • The timing for response and resolution. Typically this involves some kind of severity matrix, whereby critical issues get near-instant response and less important issues wait a bit. Keep it simple (the Engineering team may want to make it more complicated). In most cases using three levels is fine: critical, or “customer on fire” level (this one is easy!), important for issues that are not on fire but create a real issue for the customer (usually with a workaround, but perhaps an inconvenient workaround), and minor. It’s smart to add a crowd-sourcing element as well. If a minor issue is inconveniencing dozens of customers, it should be promoted to an important one. Typical response time would be a few hours for critical to perhaps a couple a days for important. Critical issues get a fix as soon as practical, while important issues are fixed in the next release (according to the release schedule, which may mean the one after next if the next one is already in QA).
  • A communication process. This is probably the most important component of an Engineering SLA but it’s also the one that’s most likely to be missing! I like to see a weekly (or, better, twice-weekly) meeting with Engineering and Support represented to review the performance against SLAs and discuss the top bugs. This is a great forum to discuss those high-impact, low-severity bugs, in addition to the current list of troubles, I mean escalations. It’s hard to schedule if the Engineering team is half-way around the world, but it’s a critical piece.
5. Don’t mess with the internal organization of the Engineering team

Press hard to get a favorable SLA but resist the temptation to micro-manage the bug-fixing process. If you believe that the only way to get bugs fixed is to have a dedicated Sustaining Engineering team but the Engineering VP believes otherwise, who do you think will win? Save your head-banging for a more likely-to-succeed cause and focus on monitoring the SLA. Are we meeting targets? Is the throughput matching the incoming load? (This last question is very important. If you get 20 new bugs per day and 5 are getting fixed. you and your customers will soon be in a world of hurt.)
6. Be the funnel

Engineering typically gets requests for fixes from many sources beside Support including the Sales team, Professional Services, and assorted other executives. One great service you can supply is to serve as the single point of contact for all engineering issues, organize the prioritization, and handle all communications. It usually makes sense for the support organization to play this role because of the volume of issues that come from Support. It’s a difficult task and Engineering will be delighted to see someone stepping up to the bar. And you will have an unmatched court-side seat to the action (in exchange for all your hard work!)
7. Don’t jerk around with priorities

Bug fixing demands long periods of concentrated work. If you change priorities all the time, you are wasting valuable effort as the engineers need to switch context and refocus on a new issue. Try not to do that outside dire emergencies.
8. Accept that many (most?) bugs won’t be fixed

Remember those “minor” bugs described above? The reality is that many of them may never be fixed. They are bugs, but the volume of other issues is such that they may never move away from a workaround — expect for those crowd effects we discussed. This is the reality for most software vendors today.
9. Learn to have the tough talks with customers – early on

Once we say to a customer that the problem is due to a bug, the logical next step is to fix it, right? That will be the normal expectation, unless you take action. So equip all support engineers with an assortment of scripts that they can use to set proper expectations. For minor bugs, the normal discussion should go something like this: “This behavior has been reproduced and it is a bug. You have a workaround, . Considering the type of issue, my expectation is that a fix may or may not be forthcoming so I’m committed to helping you build a more robust workaround if needed”. Tough words to say, perhaps, but much, much easier than dealing with an exasperated customer four months later, when it dawns on him or her that a fix is not coming anytime soon.
10. Consider DIY very carefully

OK, you say. It’s hopeless. The Engineering group does not care and will never care, so I will create my very own Sustaining Engineering team and I will fix my own bugs, to my customers’ satisfaction. I have a couple of clients (no more!) who have taken this route and have been successful with it. Many more have tried and failed, because, surprise, it’s pretty hard to fix bugs! Moreover, if Sustaining is a separate team the rest of the Engineering team is less disposed to help out if additional assistance is needed.

One last thought: when working with the Engineering team it’s useful to always come back to the reason why support needs assistance: customers! So it’s not a matter of Engineering helping Support, or Support being dependent on Engineering — it’s the entire vendor at the service of the customer. This simple focus change should go a long way towards aligning the two organizations.

Tagged under:

1 Comment

  • David Kay (@dbkayanda)

    This is a great list!

    Another point I’d add, perhaps as a sub-bullet to “build relationships,” is to expose engineers to the voice of the customer. Most engineers really want their products to be used, and they want to be admired–perhaps even revered–by their customers.

    If you can show real data, whether it’s tweets, knowledge article reuse rates, knowledge article feedback, or online reviews, that shows how bugs get in the way of customer enjoyment of the product, cause frustration, and engender the exact opposite of admiration, this can kick in developer pride in a big, very helpful, way.

Comments are closed.