Originally published in the June 1994 issue of Advanced Systems.

Our #1 priority

Sun's official process for supporting production apps the UPA revealed.

We had little to go on when we started our rightsizing adventures at Sun back in 1989. As far as we knew, mission-critical production applications had never before been supported in a distributed-computing environment. There were no formal guidelines or processes to guide us.

Scores of Unix distributed applications were developing rapidly in Sun's business units all over the world, and we didn't have a process in place to manage them. How were we to make sure they were reliable, available, and secure in the new environment? Our number-one priority was to develop those procedures and guidelines.

Unixfying the UPA
First, we huddled. Actually, we (including Unix systems programmers Andrew Law and Becky McNulty) camped out -- literally -- for weeks in one of Sun's conference rooms, all the while jotting down every process, tool, guideline, and reason we could possibly think of that would be required to support a distributed-computing environment. When we emerged, we had a document of about 200 pages, which we proudly entitled the Unix Production Acceptance (UPA) process.

Thump! wasn't just the sound it made when we dropped our new UPA on Sun's applications developers and users, but also the resounding whack! of doors slamming in our faces.

Could we really blame them? Besides being much too long, it was filled with mainframe disciplines. And that was heresy in the minds of the developers who were "Unixfied" toward open, freely shared computing practices. In no way did they want to deal with centralized controls and restrictions. They'd rather support their own applications. Go it alone, that is, until they realized they were not chartered to support and could not maintain availability of their applications. Rather, we in Sun's Data Center could and just needed to convince them of that reality.

We spent the next three years working closely with increasing numbers of supporters within Sun who not only helped us streamline the UPA, but also helped us champion the methodology throughout the company. Today, the UPA, is about 25 pages long. And applications developers have only seven pages in the UPA to fill out in order to register their work with the Data Center.

The UPA process
The UPA process is synergistic -- much in keeping with the Unix philosophy of shared work and resources: Sun's business units dictate what technologies and applications they want and need to use, and they entrust the Data Center to reliably and securely support those applications. All we ask is that the business units follow the UPA process.

It takes between two weeks to more than six months for an application to move through the UPA process, depending on its size and how it is to be distributed. We begin the UPA process by assigning a Data Center UPA Committee to work closely with the developers and users of a new application, preferably starting with its design stage.

The team collaborates to foster the application through alpha, beta, and preproduction stages. Once an application is finally ready for production, the Data Center infrastructure takes full charge of its implementation and security. Root access to the application is controlled by Data Center systems programmers, regardless of where the application's server is located.

Preparing an application for production happens in four phases. Phase I is the information gathering phase. If you are a user or developer of an application that requires Data Center staff support, your first contact is Production Control. We assign a Data Center operations analyst to oversee the UPA process and manage the UPA committee, consisting of Systems Programming, Technical Support, Database Administration, Production Control, and Computer Operations personnel from the Data Center along with the project leader.

Phase I takes about a week to complete. During that time, the operations analyst completes a preliminary copy of the Production Implementation Process (PIP), our way of tracking an application in the UPA process. This questionnaire contains general information about the project, including its name and the names of its development group, owner, and leader, a description of the application, if it needs a database, what type of hardware it will use, if it requires continuous (24-hour, 7 days; 24 by 7) support, where users are located, whether users need front-end or rlogin access and Assistance Center support, and target dates for server installation, alpha and beta tests, production freeze, software distribution, UPA sign-off, and a production-implementation date.

Phase II is resource planning. The Data Center operations analyst assigned to the project reviews the PIP, formulates an appropriate UPC committee based on the application's needs, and works with Tech Support to define Data Center space, equipment, and personnel allocations and costs to support the project. The operations analyst also works on the UPA template with the application's project leader, who orders needed equipment during Phase II.

Phase III is the implementation period. It usually takes one month to complete depending on the size and complexity of the application. The operations analyst adds the application to the Data Center's Auto-Paging System (APS), a tool we developed that automatically alerts via beeper an on-call technical support person when a server goes down. If that technical support person doesn't respond in a reasonable time, the APS alerts another until the alert is answered and the problem addressed.

During Phase III, Tech Support personnel install the necessary hardware, software, and all supporting utilities on the server. The tape librarian is instructed to create tapes with labels and to install the appropriate Unix backup procedures. The Data Center's Database Administration people work with the application's developers to prepare the supporting database (if needed), and then relay disk-partition information and database-creation scripts for installation and execution by Tech Support.

Finally in Phase IV, the application and all Data Center support systems are brought on line and tested for as long as it takes to ensure that the application can run reliably in a production environment. Also during this final phase, the operations analyst adds the application to our Morning Report, which automatically tracks server and applications availability worldwide (see "Remaking mainframers," March 1994). Tech Support adds the UPA and all related documents to the production server.

Once fully completed, an application's UPA is maintained by the Data Center: Database Administration maintains and upgrades the database and software, making any needed system changes, such as adding dump devices, increasing database sizes, and analyzing and reconciling maintenance errors. Production Control manages job scheduling, restarts applications, and makes crontab changes. Tech Support maintains the operating system software and hardware, and formats and repartitions disk, installs unbundled software, and maintains and configures system security and network services.

The UPA revealed
Sun's Information Resources Data Center's UPA guide contains the following sections. Sun's Integration Services will make the UPA available for purchase sometime this summer, including versions customized to individual customer needs.

• System configuration: hostname location, arch, model, power, swap, memory, and other system functions.
• Disk configuration: the number and type of controllers, quantity, type, and partitioning for disks.
• Peripherals: modems, printers, tape drives, CD-ROMs, etc.
• Database: design, initial size, and placement; special loggings used by applications, batch jobs, support staff, initial sp_configures, Sybase devices, sys_usages, dataserver name, version of Sybase, a copy of /etc/fstab with marked Sybase partition, and the run file.
• Software tools: requirements for Sun-OS, OpenWindows, DBE, compilers, On-line Disksuite, and MVS/NFS.
• Storage management: incremental and full-system backup schedule and requirements.
• Error messages: messages and protocols for operator intervention.
• Interdependencies: environment, applications, tools, I/O-information flow, data-flow and entity-relation diagrams, and the transition document.
• Capacity planning: estimated disk space for database growth, performance characteristics for key transactions, and controls to manage the plan.
• System availability: uptime hours for on-line users and aliases for announcing machine downtime.
• Test plan: alpha through preproduction phase.
• Training: if required.
• Database configuration: size, raw devices, SQL process, and version.
• Administration: job versus batch processing, script monitoring, and restart procedure.
• Disaster Recovery Procedures: if applicable.

Harris Kern (harris.kern@sunworld.com) is Sun's Open Systems Migration Consultant for NAAFO Market Development. Randy Johnson (randy.johnson@sunworld.com) owns R&H Associates, a full-time rightsizing consultancy in Boulder Creek, CA. R&H Associates helps people worldwide in implementing and supporting client/server infrastructures based on their proven methodologies. © 1994 Harris Kern and Randy Johnson. All rights reserved.

Pick up a copy of their book Rightsizing The New Enterprise: The Proof Not the Hype, SunSoft Press/PTR Prentice Hall, ISBN 0-13-132184-6, or their new book Managing The New Enterprise: The Proof Not the Hype by Kern, Johnson, Hawkins, Law, and Kennedy, SunSoft Press/PTR Prentice Hall, ISBN 0-13-231184-4. Browse SunSoft Press offerings at: http://www.sun.com/smi/ssoftpress

You can buy Managing The New Enterprise and Rightsizing The New Enterprise at Amazon.com Books.

If you have problems with this site, contact webmaster@sunworld.com