Dismantling the help desk

Partnerships and automated tools bring effective system administration close to our users.

We've said it many times before, and we're not shy about repeating ourselves: System administration is the most important support function for Unix distributed-computing environments. The two most important aspects of how we approach system administration at Sun are the partnerships we develop with business units and users and the automated systems we use for fast and efficient delivery of services.

In previous columns, we described those partnerships and how, by formalizing the process with our Unix Production Acceptance ("Our #1 priority," June 1994) and Service Level Support Agreement ("SysAdmin: The heart," December 1994), we make sure Sun's distributed-computing systems best serve the company's needs. This month, we take a look at the protocols and automated processes we use to respond to requests for support.

Are you being served?
The help desk -- the centralized system that responds to ad hoc user requests for system administration services -- has become a nearly permanent fixture in most medium to large organizations' computing infrastructures. Like most help desks, ours at Sun was originally based on the most traditional and ubiquitous way users could and would call for help: the telephone. A help desk administrator would either resolve a user's problem then and there over the phone, or take down and relay the necessary information for later system administration response. Telephone-based help desks do work, but they're extremely labor intensive, inefficient, and expensive. Under pressure to reduce our budget (five percent per year!), we did what any red-blooded technocrat would do: automate. We created ServiceDesk, a graphical-user-interface-based on-line application that lets users report their systems problems and request help from system administration electronically over Sun's network (see screen shot of ServiceDesk). ServiceDesk's suite of tools also tracks user problems and resolution.

With ServiceDesk, users prepare their own "trouble tickets" and work request forms, which are then automatically submitted over Sun's network to the appropriate system administration group. That cuts the time help desk administrators spend taking user requests for assistance and directing the call to the appropriate person or group for resolution. Moreover, ServiceDesk tracks trouble tickets and work requests all the way through to resolution, including notifying users of their progress, so few, if any, problems "fall through the cracks." In fact, ServiceDesk will automatically escalate user requests to management's attentions if they haven't been resolved within a specified period of time. Now that's responsiveness!

Help, don't hinder
All of our rightsizing efforts to "Unixfy" Sun's systems have a single motto: SELL the concept; DO NOT shove it down users' throats. Work with your user/customer or a rebellion will surely happen. Regardless of the benefits and efficiencies from ServiceDesk, we still had to sell it to users. Think about it -- picking up a phone is easy, and personalized service is usually preferred by customers. With ServiceDesk, requesting help means filling out a form and sending it through a machine to "who knows where."

As expected, there was some yelping when we first went live with ServiceDesk. True to our "Unixfied" approach to innovation (collaboration versus confrontation), we continued to run the traditional phone-in help desk. Whenever a user would phone in a request, however, we would remind them (politely, of course) to use ServiceDesk. Over time, we were able to show users that our response times to resolve problems improved dramatically through ServiceDesk over the traditional phone-based help desk: On average, the help desk took nearly four hours to resolve problems. Not long after we installed ServiceDesk, our average turnaround time for help was reduced by half. We now promise to respond to user trouble tickets within 30 minutes (the help desk used to take two hours) and guarantee resolution within four hours or else the problem is brought to management's attention. (Just a note: In the nearly five years we've been with Sun, no systems problem ever escalated to management's attention. Whew!) System administration also promises to resolve work requests -- submitted by users through ServiceDesk when they want new systems installed, existing systems upgraded, or their operations moved -- within five business days.

There's one other important aspect to ServiceDesk that truly makes it work: user feedback. ServiceDesk automatically keeps users informed of the status of their requests, including notifying them by e-mail when the task is completed. And ServiceDesk automatically sends users a Feedback Tool (see inset to ServiceDesk screen shot) that lets them quickly and easily rate the quality and timeliness of the service we've performed. We assure them that all their smiley-, neutral-, and frowny-face feedback is given serious consideration -- our system administration performance evaluations by management depend on it. We're allowed no more than two percent frowny faces, and every unsatisfied user gets a personal phone call from someone in system administration. Fortunately, the grand majority of our respondents are all smiles.

Our goal is to eliminate the help desk altogether. We haven't done that yet, but we have made significant progress by streamlining as many of those old MIS help desk functions as possible. For instance, at one time there was a lot of "passing the hot potato" from the Assistance Center (nŽe help desk) to a centralized "focal" queue of system administrators, and perhaps further along to an on-site or near-site system administrator to resolve tough trouble tickets.

Today, ServiceDesk trouble tickets and work requests go directly to a focal queue of system administrators assigned to one of Sun's campuses or business-unit territories for resolution. And our focal queue system administrators and the on-site or near-site administrators, who each have a specialty such as database or systems management, work as a team to solve all problems.

A recipe for support
Here are the steps Sun's system administration staff now takes to resolve trouble tickets:

• All trouble tickets must be electronically submitted and tracked through the ServiceDesk.
• Trouble tickets, prepared by users, are automatically transmitted from the desktop over Sun's network to the appropriate system administration focal queue, the centralized systems and personnel assigned to oversee a specific campus at Sun. Remote business units' requests go to the Assistance Center, our centralized, automated help desk.
• The on-call focal queue system administrator is automatically paged by all Priority-1 trouble tickets (user designated), such as a system failure.
• All requests are acknowledged by either the focal queue administrators or by the Assistance Center within 30 minutes.
• Nearly 80 percent of all trouble tickets are resolved ("closed") within two hours of receipt.
• For trouble tickets initially handled by the Assistance Center, if they cannot resolve the problem within two hours, it is escalated to the regional system administration focal queue.
• If the focal queue cannot resolve the problem, the trouble ticket is then escalated to on-site or near-site sys admins.
• Once a trouble ticket is closed (virtually all within 24 hours of the time they were submitted), ServiceDesk automatically e-mails notification and the Feedback Tool to the customer.
• The user selects one of three face icons -- smiley, neutral, or frowny -- to represent their level of satisfaction with the service, and, with a click of the mouse, returns the Feedback Tool response to system administration.
• System administration reviews all feedback reports, compiling the results for management review, and makes personal contact with users who have reported unfavorable service.

Work requests for systems installation, updates, or relocation (work requests) are handled a bit differently than trouble tickets:

• Like trouble tickets, users submit work requests through ServiceDesk.
• The work request is automatically routed to the on-site or near-site system administrator or his or her backup assigned to the subnet of Sun's organization where the request was made.
• All work requests are completed with five business days.
• Once the work request is closed, ServiceDesk automatically notifies the users and activates their Feedback Tool.
• Feedback is reviewed daily by the system administrator and compiled for periodic management review.
• Unhappy users are personally contacted by system administration.

Harris Kern (harris.kern@sunworld.com) is Sun's Open Systems Migration Consultant for NAAFO Market Development. Randy Johnson (randy.johnson@sunworld.com) owns R&H Associates, a full-time rightsizing consultancy in Boulder Creek, CA. R&H Associates helps people worldwide in implementing and supporting client/server infrastructures based on their proven methodologies. © 1995 Harris Kern and Randy Johnson. All rights reserved.

Pick up a copy of their book Rightsizing The New Enterprise: The Proof Not the Hype, SunSoft Press/PTR Prentice Hall, ISBN 0-13-132184-6, or their new book Managing The New Enterprise: The Proof Not the Hype by Kern, Johnson, Hawkins, Law, and Kennedy, SunSoft Press/PTR Prentice Hall, ISBN 0-13-231184-4. Browse SunSoft Press offerings at: http://www.sun.com/smi/ssoftpress

You can buy Managing The New Enterprise and Rightsizing The New Enterprise at Amazon.com Books.

If you have problems with this magazine, contact webmaster@sunworld.com
URL: http://www.sunworld.com/asm-01-1995/asm-01-unix.html.
Last updated: 1 January 1995.