Bell letters How to manage system files (and anything else) with SCCS

July  1995

Are people stepping on your file's toes? Text editors aren't databases -- they don't impose transactional consistency or concurrency control for multiple updates. This doesn't affect you one bit if you're the sole system manager at your site but as soon as two or more people are chartered to maintain the environment, you need some sort of control system to serialize and document configuration changes. Enter SCCS, a staple of most Unix systems.

Mail this article to a friend

How often does this happen to you? You add a new Web server to the network, inserting its IP address in /etc/hosts with plenty of time to spare before the Demo For Big People. At T-minus one hour to demo, your browser can't resolve the hostname. Neither can anyone else's.

Frantic, you check everything before finally coming back to /etc/hosts. Your change is gone, probably because someone else edited the file around the same time and overwrote or removed your edits. You either need some strong configuration control, or a truly loud warning bell that signals anyone's attempt to modify a critical file.

Text editors aren't databases -- they don't impose transactional consistency or concurrency control for multiple updates. This doesn't affect you one bit if you're the sole system manager at your site, but as soon as two or more people are chartered to maintain the environment, you need some sort of control system to serialize and document configuration changes. The downside is that you'll spend a non-trivial amount of time deciphering changes made by your peers or un-doing valid work that conflicts with items on your own task list.

This month, we'll look at the source code control system, or SCCS, bundled into nearly every Unix operating system and a staple of simple configuration control. (See our comparative review of two configuration management packages elsewhere in this issue. )

After explaining the basics of SCCS file administration, we'll look at the more difficult issues of merging changes and dealing with files owned by root. Our goal is to reduce the mystery and annoyance factor of SCCS, and make it a viable tool for producing an electronic version of your "site book" documenting the who, what, and why of system-configuration changes.

Rewriting history
SCCS is really a collection of tools that control updates to ASCII files. You can use SCCS with binary data, which will be converted into ASCII form using uuencode, but we'll limit this discussion to ASCII data since that's the source for most configuration files. SCCS lets you put files under configuration control, check out read-only copies, acquire write locks for updates, check in and document changes, print histories, and identify and combine specific updates. Any text file can be put under SCCS's control, making it useful for managing plain text documentation and meeting notes.

Before going into the functional details, here's a bit of terminology:

• History files contain the source for the file under control, as well as a log of all changes made to the file, information about revision numbers, and access controls. History files are prefixed with an "s.", and generally live in a subdirectory called SCCS.
• Deltas are specific changes made to a file. Changing a few characters, adding a line, or removing a line constitute deltas to a file. Deltas are numbered as minor release numbers from the main or major release. A particular version of a file, reflecting the cumulative effect of many deltas, is referred to as an SCCS delta ID, or SID. Most SCCS commands take an SID as an argument when a specific version of the file history is needed.
• Branches are subdivisions of deltas. While deltas are used to track the main changes to a file, branches let you create special-purpose minor variations in a file. Branches may or may not be merged back together at some point; a typical branch file might be created when you update your network configuration to host some demo or loaner machines, and plan to remove those edits in a few days or weeks. Branches are a convenient form of short-term memory.

When you place a file under SCCS control, SCCS creates the history file. To change the file, you check it out for editing, and then each subsequent change to the file is annotated in the history file when you check the modified version back in. SCCS locks the history file while one user is editing it to prevent concurrent updates.

Bones of contention
Let's walk through some basic SCCS operations to see how the components fit together, and then get into the grittier problems that make SCCS more of a benefit than an added burden. First, you'll need to have /usr/ccs/bin in your path, since that's where the SCCS commands live (in SunOS, they're part of /usr/bin).

You can call the individual SCCS commands, or use the sccs front-end tool to simplify life. We'll use the front-end for illustrative purposes, but you can also call the SCCS subcommands directly. Make sure you have an obvious place to store history files, such as a subdirectory called SCCS. SCCS commands look for this subdirectory if you don't give an explicit history file location.

Take a vanilla ASCII file and put it under SCCS control, using the admin command:

huey% sccs admin -ihosts hosts

This creates an SCCS history file called hosts initialized with the content of the file named hosts. You want the history file and the actual file to be namesakes unless you're particularly good at associating strange path names with your /etc files. You can choose any file you want for the initialization; if you've just sorted your hosts file into /tmp/hosts.sorted, the above command line might be:

huey% sccs admin -i/tmp/hosts.sorted hosts

If all goes well, sccs admin returns quietly to the shell prompt. The most common complaint is that the initial file doesn't contain any ID keywords, which are magic strings filled in by SCCS with the file name, delta numbers, and date and time stamps. We'll talk about the keywords and how to maximize your enjoyment of them shortly. Successful submission of a file to SCCS creates a new s-file in the SCCS directory. The file is primarily ASCII text, with SCCS records marked with an ASCII SOH (start of header) character, showing up as control-A in most editors. All revisions, delta histories and access control information goes into the s-file.

When you're ready to use the file, check out a read-only copy:

huey% sccs get hosts
1.2
10 lines

SCCS tells us the current SID of the file and its size. The get operation produces a read-only file in the current directory, and it will complain if there's a writeable version of the file already present. After you initialize a history file, be sure to rename or remove the initial file to prevent problems on your first check-out operation.

Edit the file by checking out a writeable version, using sccs get -e or the shorthand sccs edit:

huey% sccs edit hosts
1.2
new delta 1.3
10 lines

This time, we're told the new delta number to be created by our editing session. If someone else is editing the file at the time, SCCS produces an error:

huey% sccs edit hosts
1.2
ERROR [SCCS/s.hosts]: being edited: `1.2 1.3 stern 95/06/16 17:41:22' (ge17)

Our first contention point is removed: any request to edit a file that is already being consumed by another system administrator is met with a cryptic yet gentle slap on the keyboard. If you want to find out who is currently editing SCCS-controlled files, use the info subcommand:

huey% sccs info
     hosts: being edited: 1.2 1.3 stern 95/06/16 17:41:22
     aliases: being edited: 1.45 1.46 wendyt 95/06/17 14:50:33

Make your changes a part of the file's permanent record using sccs delta:

huey% sccs delta hosts
comments? added two new host entries
1.3
2 inserted
0 deleted
10 unchanged

Your writeable source file is removed when you file the deltas, so you have to do another sccs get to fetch the latest, read-only copy, or merge the delta and get operations together with sccs delget hosts.

At this point, you can feed the read-only file into whatever system management step comes next: running an NIS make, executing newaliases, or restarting a daemon with its new configuration file.

Letters of intent
How can you determine the version number of a file, or if it's even SCCS controlled? When you check a file out, the get subcommand fills in SCCS keywords with values such as the SID, pathname of the history file, date, and time. The SCCS magic cookie indicating a keyword is a single, capital letter between percent signs, such as %Z%. Put the SCCS keywords in a comment header in your file, and you have a built-in identification scheme. Here's a sample header for a configuration file that uses the pound sign (#) as a comment character:

#	%M%	%I%	%H% %T%

This set of keywords gives you the filename (M), the file revision or SID (I), the current date (H), and the time of checkout (T). You may also choose to insert the pathname to the s-file (P). (Here is a partial list of SCCS magic cookies.) The %W% keyword generates the filename and SID prefixed with the string @(#), which is assumed to be unique to the SCCS system. The what utility searches for the SCCS prefix and prints any information after it, allowing you to quickly identify any number of files.

To include other information to be picked up by what, use the %Z% keyword to insert an SCCS cookie and then build your own identification string. A more verbose version of the example above is easily found by what:

#  %Z% common hosts file revision  %I% of %H% at %T%

what is smart enough to look in the string tables of executables and libraries, so it will identify the SCCS versions of each object component. Bundle an SCCS string into a C program with a global definition like this:

char *sccs_id = "%Z% %I% %H% %T%";

While peeking at the SID and file origins is useful for quick sanity checks, reviewing the delta history of a file is more likely to tell you who changed something and why. When you create the delta, SCCS asks for a comment which is then recorded with your login in the history file. Dump the delta history using sccs prs:

huey% sccs prs hosts
SCCS/s.hosts:

D 1.2 95/06/16 16:49:32 stern 2 1	00002/00002/00008
COMMENTS:
added alias for wind, new host shower

D 1.1 95/06/16 16:43:30 stern 1 0	00010/00000/00000
COMMENTS:
date and time created 95/06/16 16:43:30 by stern

The line introducing each delta shows you the SID, date and time of change, and the login of the person making the change. The slash-separated numbers are the line counts of new, deleted and unchanged lines. The manual pages for the prs subcommand also list all of the possible SCCS keywords and their expanded values.

Merge ahead
We still haven't tackled two of the hardest problems in change management: how do you get multiple users to access SCCS files, particularly when the files are owned by root, and how do you merge changes together? The first problem doesn't have an easy solution. You can keep all of your SCCS history files in /etc/SCCS, and insist that system administrators include their user names when making changes as root. Since this is fairly unlikely, the next step is to make the SCCS history files group-writeable by members of your system management group (creating a new user group if you need to). Create private SCCS work areas for each system manager using symbolic links to the actual history file location:

huey% mkdir ~stern/etc
huey% ln -s /etc/SCCS ~stern/etc/SCCS
huey% cd ~stern/SCCS
huey% sccs edit hosts

Within ~stern/SCCS, an sccs edit hosts picks up the s-file /etc/SCCS/s.hosts, giving me a private copy of the hosts file to work on.

When I check it back in, the single host-specific copy is returned where other managers (and the system) can find it, but it has my user name attached to changes instead of root. To publicize the changes, I need to su to root, cd into /etc, and then do an sccs get hosts to fetch my latest changes and install the file. Note that the symbolic link points to a machine-specific location, which means I have to be logged on to the machine on which I want to make the edits before doing the checkout. I can always move SCCS files around, as long as files get installed on the appropriate machines.

If you're worried about giving up some measure of security regarding permissions on /etc/hosts, remember that only root can install the file in /etc and rebuild NIS maps or restart daemons. For an added layer of safety, using the SCCS access control feature, explicitly name allowed users with sccs admin -a:

huey% sccs admin -astern
huey% sccs admin -awendyt

But the opening question still lingers: how do I find out what happened to my hosts file at 3:30 on Friday afternoon June 16, and who did it? The easiest way is to look at the delta history since that time:

huey% sccs prs -l -c95-06-16-15-30 hosts

The -l flag says I'm interested in things that occurred after the time specified with the -c flag. The time and date are given in YYMMDDHHMM format, with any non-white space character separating the items. This example shows me the revision history comments and the user names responsible for making changes.

If I want to see the actual line by line edits, it's sccs diffs to the rescue:

huey% sccs diffs -c95-06-16-15-30 hosts

Like the diff command, this compares the current working copy of a file to any older delta, identified by SID or by a timestamp. In this example, I'll see the list of changes between the current hosts file and the one that existed at 3:30 PM on June 16. Want to regenerate the hosts file, minus a few changes? get lets you include or exclude any SID, providing a simple mechanism to drop changes from the current copy of a file:

huey% sccs get -x1.6,1.7 hosts 

The current hosts file is retrieved without the changes applied in SIDs 1.6 and 1.7. If you want to extract the changes made in those deltas, generate the differences with context in a form that can be later fed to sed, just like the output of the standard Unix diff command:

huey% sccs get -r1.6 hosts
huey% sccs diffs -r1.5 hosts > hosts.sed.6

If you plan on applying the patches at a later time, when the hosts file may have undergone some additional minor edits, you'll need to generate context differences that can be fed through patch:

huey% sccs diffs -C -r1.5 hosts > hosts.sed.6

diff takes the -c flag for generating context differences, but sccs diffs takes -C to avoid conflict with the timestamp flag.

Control freaks
Like all powerful system administration tools, SCCS has a number of poorly documented but interesting features and subtle caveats:

• If you damage the list of users allowed to make deltas badly enough, it may be worth hand-patching the history file:

  huey% chmod +w SCCS/s.hosts
  huey% vi SCCS/s.hosts
  huey% sccs admin -z hosts
  

  Edit the list of users that is underneath the ^Au line. Regenerate the SCCS file checksum using admin -z, or you'll get notice of a corrupted history file on your next attempt to edit or check out a copy of the file. If you see corrupted file warnings at other times, you can fix the checksum, but be very certain that the file wasn't actually damaged by someone editing in the wrong place. Once the checksum is fixed, the SCCS history file is assumed to be valid, and any errors introduced will be propagated into future deltas of the file.

• To find out when a particular line showed up in a file, use sccs get -m to preface each line with its SID number.
• You may run into contention when multiple system administrators attempt to create history files at the same time, if the SCCS directory is NFS-mounted. SCCS uses an exclusive file-create operation when opening the s-file for the first time, but exclusive creates aren't obeyed by the NFS Version 2 protocol (NFS Version 3 supports exclusive creates, see "NFS Version 3 Design and Implementation" by Brian Pawlowski, et al.).

  If you find that you can't access a history file you think you just created, make sure you're the owner of the file and that someone else didn't beat you to the sccs admin punch.

• SCCS doesn't preserve the modification timestamps on files. When you check a file out using sccs get, the modification time is set to the current time. Furthermore, if you're accessing the SCCS directory and work area via NFS, the modification time is set to the time on the NFS server, which might have drifted a few minutes ahead or behind the time on other machines.

  Changing the modification time has less than pleasant impacts on make. If you decide to get a fresh copy of all of your NIS source files, figuring that the NIS Makefile will only rebuild those that you've changed, you'll be in for a surprise, because SCCS will change the timestamps on all of them, so make will assume they're all new.

• Hooks for integrating a trouble ticket or request system into SCCS are minimal but present. SCCS calls these modification records, or MRs in short hand. By default, MRs are not collected, but you can force sccs delta to prompt for an MR by enabling them:

  huey% sccs admin -fv hosts
  

  This command turns on the validation flag in the s-file, which is used to signal that MRs should be accepted for each delta. A script or executable specified after the v flag will be invoked on each delta, and given the name of the file and the modification record string entered during the delta check-in. Here's how to update your SCCS history file for hosts so that it calls /usr/local/bin/host-update after each check-in:

  huey% sccs admin -fv /usr/local/bin/host-update
  

  One application of MRs is tying the configuration file edit cycle to trouble-ticket management, so that the validation executable removes the trouble tickets or requests from your work flow system as soon as the change is made.

• These tidbits and dozens more are found in the SCCS manual pages. Locate a particular subcommand's man page using the form man sccs-subcommand, where subcommand is the subcommand in question.
• An alternative to SCCS is the Revision Control System, or RCS, written by Walter Tichy. While SCCS stores your original file and all of the changes needed to bring it forward, RCS stores the latest version and the changes needed to send it backward to an older SID. As a result, RCS checkouts tend to be much faster for heavily edited files. RCS also lets you assign symbolic names to SIDs, for example, naming a configuration file snapshot that was used for a demo or before a major network overhaul.

There's certainly much more that can be done with SCCS. In the last issue of Advanced Systems, Chuck Musciano suggested using a web browser front end for checking files in and out, and viewing the history. A bit of creative perl or awk programming lets you generate HTML out of the sccs prt output. Send us your marriage proposals for HTML and SCCS, and we'll attach the interesting submissions to this page.

The hidden agenda of using SCCS is accountability. You want to know who inflicted a change, and why, and under whose authority. A rigorous policy for attributing changes and accepting responsibility for their implementation and effects is fundamental to any robust, mission-critical environment.

Dan Geer, noted security expert and frequent speaker, tells the story of an investment bank executive who demanded a systems change to circumvent normal reporting and control code. The hole was later exploited to execute trades that violated various internal and external regulations. Who was responsible?

• The developer who changed the source code?
• The executive requesting the change?
• The system administrator who allowed this code to be fielded?

Tracing the changes from idea to deployment gives you the first measure of accountability. It's a good thing to have when you hear those warning bells.

About the author
Hal Stern is a Distinguished Systems Engineer at Sun Microsystems, where he specializes in information systems architecture, technology evangelism, and answering e-mail. Reach him at hal.stern@sunworld.com.

If you have technical problems with this magazine, contact webmaster@sunworld.com

URL: http://www.sunworld.com/swol-07-1995/swol-07-sysadmin.html
Last modified: