Serious programming doesn't have to be difficult
Scripting languages make enterprise programming easier
July 1, 1999: Serious programming doesn't have to be difficult
Cameron and Kathryn look at what makes enterprise software so complicated, and our real-world script of the month zeroes in on Evolutionary Commercial Systems Inc. and its custom supervisory control and data acquisition applications. (1,100 words)
July 15, 1999: Safety in numbers
Put different computer languages together for better results -- How are several languages being combined to produce the most ideal solution? (1,600 words)
In the second half of 1999 we'll probe more deeply into several topics upon which we've already touched lightly: deployment, cross-language cooperation, commercial news of advances in scripting, and the race to embrace such hot technologies as LDAP, SSL, sales force automation, Jini, and multiprocessing. We'll also make space for several stories on enterprise-class scripts.
What is enterprise software?
The label enterprise software designates applications that pervade the computing environment of a large organization and shape the fundamental operations or transactions of a business. When industry pundits bandy about such phrases as "electronic nervous system" or "information-driven enterprise," though, they have something far more elaborate in mind. The conventional promise made of enterprise software is that it will allow executives at a distant central office to know instantly about all sales throughout the organization. Detailed, realtime measurements make it realistic to adjust inventories, tune marketing, negotiate new contracts, and perhaps even change hiring and firing, all in a much more timely fashion than was previously possible.
The architecture of enterprise software is very complex. It needs to connect to historical data kept in mainframe databases; it might generate different kinds of reports for a whole series of departments, so that cases of a given product can be ordered, received, shipped, stacked, and shelved; it probably has interfaces to various physical devices (universal product code scanners, at least) and other organizations' computers; and it's likely to implement a security scheme, to ensure that sensitive data isn't seen by the wrong eyes. The source code for an enterprise application might be smaller than that of a contemporary word processor, but it ranges over more domains and styles of data operation.
Writing enterprise software and putting it into use is hard. At least, that's its reputation, and that's the current reality in the field. Consultants who specialize in enterprise resource planning (ERP) projects typically bill hundreds of dollars per hour -- and it's notorious how long ERP projects last!
Many of the challenges of enterprise programming are inherent. The feudal rivalries between finance and operations departments inevitably complicate technical calculations. Even in isolation, though, engineering enterprise software is usually difficult.
But it doesn't have to be so hard -- at least, not as hard as it is now.
Successes in enterprise software
During the next nine months or so, we'll look at several specific examples of successful serious projects, and the roles of scripting in each. When people talk about serious software in a business environment, they usually mean something that involves big Oracle or DB2 databases, Lotus Notes messaging applications, SAP or PeopleSoft ERP integrations, and so on. As we'll see, each of these has a scripting story to tell.
Oracle, for example, is the second largest software company in the world. Customers pay a lot for its database management systems (DBMSs), then pay a lot more to develop, deploy, administer, and maintain them.
Part of the reason for this is because the user interface is so cumbersome. The fundamental development API for DBMSs is a set of C bindings. This is the case for DBMSs in general -- Sybase, Informix, and all the rest are as guilty as Oracle. Talus Technologies President Tom Poindexter recalls that, during the early days of DBMS vendors in the 1980s, "Emphasis was usually placed on fast, efficient code, probably custom-coded programs in C." There's certainly a place for those, but we've seen in Regular Expressions how often a safer and more expressive language can yield results in a fraction of the calendar time as handcrafted C. Margo Seltzer of Harvard University and Michael Olson of Sleepycat Software aptly articulate the situation in the related domain of database programming: "Application developers need small, fast, versatile systems that can be tailored to a specific environment."
Still, DBMS vendors regard scripting as "slow, ugly, and a hack," in Poindexter's words. Several vendors have responded to their ease-of-use problems, but with the wrong answer. They offer so-called "fourth-generation (access) languages," generally proprietary variations on BASIC. Almost without exception, these are inflexible, syntactically confused, and, perhaps worst, useless for the database administration that always complements real-world DBMS developments.
So, from the start, savvy programmers have abstracted their own higher-order methods. Ten years ago, we often worked in isolation, coding in homegrown combinations of awk, DCL or sh, and sed. Despite the lack of vendor support, scripting bindings have now become commonplace. Most scripting languages support the most popular DBMSs well.
Real-world script of the month: ECSI takes Perl to industrial plants
Scripting often starts with a limited role in a larger project. Evolutionary Commercial Systems Inc. (ECSI) exemplifies this pattern of adoption. ECSI specializes in custom supervisory control and data acquisition (SCADA) applications: controlling operations in municipal water systems and electronic chip fabrication plants, monitoring emissions from chemical factories, and so on. ECSI provides these services for such clients as ARCO and Tosco. Contracts are for turnkey installations -- combinations of hardware and software. As ECSI President A. H. Ongun explains, "In most cases the client is not even aware of the individual pieces."
Several of these pieces are built with the Perl scripting language. A typical ECSI project acquires data from a variety of programmable logic controllers and similar devices, and dumps them into a relational DBMS (RDBMS). These components are generally coded in C. Separate processes digest the data from the RDBMS and issue reports to operators and managers. Over the last few years, ECSI has become adept at using RDBMS-extended Perl to script reporting functions far more quickly than the native C bindings allow.
High-performance, complicated, specialized, serious software often finds its most effective marriage with fun, expressive, general-purpose scripting languages. In the coming months, we'll detail many examples of this. We'll also examine why leading vendors aren't telling their own customers of these possibilities, and the industry trends which hope to change the situation.
July 15: Safety in numbers
It's a good time to be a polyglot.
More precisely, we're in the midst of changes that are making it far more inviting to speak more than one computer language. An application built with two different languages may have felt a bit academic or exotic in the past, but it soon will seem natural.
Cooperation between languages has been a reality for a long time, of course. Digital Equipment Corporation carefully and proudly documented all its VMS compilers as compatible, with interoperable calling conventions. At the time, it was an exciting leap forward to tie together C-coded networking and a BASIC user interface, with engineering libraries written in FORTRAN, and have it all work together comfortably.
More recently, C and Java have become the reference languages for application programming interfaces (APIs); any development system with commercial aspirations needs to be able to bind to libraries coded in the former, and often the latter. New languages invariably advertise their capacity to embrace legacy work. REBOL and Lua, for example, are two niche languages that we've noted here before and that have had new versions released this month; both make interesting connections to other languages, as we'll discuss below.
Such technologies as Inter-Language Unification (ILU), the Common Object Model (COM) (which fuzzily edges into DDE, OLE, ActiveX, COM+, and DCOM), OpenDoc, and the Common Object Request Broker Architecture (CORBA) have abstracted communications one step further. All of these promise language-neutral interface definitions where different objects or modules safely meet. While all of these have largely met their technical goals, OpenDoc foundered on business crosscurrents. Thousands of installed applications built to the other three protocols, though, happily pass data back and forth every day.
There's plenty of work still to do in these directions, in fact. GNOME, the GNU network object model environment, is a project whose mindshare as a desirable desktop will explode over the next year; one consequence will be the increasing acceptance of its CORBA-related object request broker (ORB), called ORBit. The importance of COM will only grow as Unix hosts become more hospitable to it. We'll cover the story of COM's migration to Unix and other server-class operating systems for SunWorld early this fall.
Another current of cross-language work is in some ways more modest. It's now becoming commonplace to tie a few languages together just for one project, without resorting to a big, formal standard like CORBA. Think how FORTRAN and YACC made scientific computing and LALR(1) parsing, respectively, safe for earlier generations of programmers; with these technologies, nonspecialists could achieve useful results in fluid flow simulation and compiler design. In much the same way, new techniques and practices are making it acceptable and even inviting to speak several computer tongues. Let's look at a few of this summer's announcements.
Mike Durian's Tclmidi, "a language designed for creating and editing standard MIDI [musical instrument digital interface] files," is a good place to start. While an original design goal of Tcl was an easy interface with other components, those other components were generally coded in C, at least during Tcl's early history. Preliminary experiments leading to Tclmidi were done with a combination of C and Tcl. Then Tclmidi scripted a collection of C++ objects. Now, with the 4.0 release, the C++ has been entirely swapped out in favor of highly efficient and maintainable Eiffel. The moral to draw from this is that good analysis and design are the most important attributes of a project. Decisions about implementation are pluggable details that can be tuned and adjusted fairly late in the process -- even if those decisions are as far-reaching as a language switch. The decision-making process should be adjustable, anyway; too many programmers fail to take the time to make their work elegant in this way. It's a virtue of Tcl that it's so broadminded about the partners it takes.
Lua, version 3.2 of which was just released, is perhaps the simplest of all useful languages to embed and extend. Lua is also intriguing for its support of a metaprogramming style; the metaprogramming method, roughly speaking, is to design a small, problem-specific language that expresses the essence of a particular domain. It's common to implement these domain-specific languages in base languages like Lua, LISP, Forth, or Tcl. As we described in our November 1998 column, an explicit ambition of the Guile project (related to LISP) is to implement such other languages as Python, Perl, and Rexx.
The high-level architecture of all the approaches is single-process. They create an executable with some pieces from one language, and others from another. Multi-image applications are most often built in terms of object frameworks like COM and CORBA. It's not necessary to make such a big leap, though. Many teams are working in a middle ground where they enjoy the modularization and mobility of network programming, without the weight of COM and CORBA.
Scripting little servers
Mark Harrison and Michael McLennan detail a useful template for this in their book, Effective Tcl/Tk Programming. The principle of simple computing servers applies to any socket-savvy language, though, not just Tcl, as we'll see.
Chris Nelson, author of Tcl/Tk Programmer's Reference, scheduled for bookshelves this summer, relates one success he had with text-based socket communications. Cadence Design Systems (CDS) is a leading developer of software tools for the electronic design automation (EDA) industry. Its automation language is SKILL, a derivative of LISP. Nelson's employer, Pinebush Technologies Inc., develops HyperCDS as an add-on to supply plotting and printing services to CDS products.
Early versions of HyperCDS were coded in around 11,000 lines of SKILL. Nelson rebuilt HyperCDS on 1,000 lines of networking and data-access SKILL; this chunk of code was connected to a separate process with about 100 lines of Tcl he embedded in existing applications Pinebush had written. At a stroke, he cut the maintenance load for the product by a factor of ten, and reunified the functionalities of the company's product line.
The point is not that Tcl is a more succinct or better language than SKILL. Pinebush achieved dramatic results by exploiting the strengths of two different languages in cooperation, rather than being stuck with the limitations of either in isolation.
Enthusiastic plugging leads to chains of control that sound superficially silly. Robin Becker's clever Python server is a nicely portable way to manage ActiveX objects on Windows desktops. There are several administrative applications in use now, programmed in Perl or Tcl, that reach through Python to ActiveX modules coded in Visual C or Visual Basic. This isn't just an exercise in networking calisthenics, though. The techniques are now so dependable that it can be more desirable to reuse than to rewrite.
Message-passing between languages is at the heart of two other technologies in the news recently. REBOL, which we mentioned in our February column, was recently upgraded to version 2.1. Sassenrath Research has trademarked it as "the Internet messaging language," and explicitly bases its ability to integrate legacy components on this model.
Also winning attention this year is XML-RPC, "simple cross-platform distributed computing, based on the standards of the Internet." This is an initiative of Dave Winer's UserLand Software Inc., aimed originally at problems of managing Web content. In addition to Frontier, Java, Perl, PHP, Python, and Tcl coming together on XML-RPC territory, there are also hints that leading Web development tool vendors are deciding whether to support it. Growing use of XML and comfort with multilingualism can only expand interest in XML-RPC.
All this success with network communication hasn't squelched the ambitions of in-process language smiths. University of Chicago Assistant Professor David Beazley's SWIG (simplified wrapper and interface generator) has done yeoman service for a couple of years now, flanging together programs written in C, C++, and Objective-C with scripting languages including Perl, Python, and Tcl/Tk. SWIG's 2.0 release is scheduled to have extensible language interfaces, so that it will be possible to bind together a far wider variety of languages.
Most provocative, though, are Jean-Claude Wippler's experiments. As we reported in our May column, his MetaKit is a very portable and high-performance datastore already in use in a number of cutting-edge projects. His research into ways to bind MetaKit into different development environments has led him to Minotaur, a massively cross-language technology just hatched this summer. Minotaur makes it possible to do chain processing through arbitrary dynamic combinations of Forth, Java, COM, Lua, ICI, Perl, Python, Tcl, and PHP. Minotaur is only a proof-of-concept so far, but it's already demonstrating surprisingly good performance and portability.
Not every problem is a nail waiting to be pounded in, and we can see finally that not every computer language has to look like a hammer. Wise use of multiple languages can simplify development and reduce maintenance costs. The current flowering of cross-language technologies makes this an excellent time to practice picking the right tool(s) for the job(s) you face.
Next month, we return to the theme of scripting the enterprise, with three tales of mission-critical datastores.
Thanks to Geoff Eldridge, Alexandre Ferrieux, Andreas Kupries, and Larry Virden for helping to keep us current on developments in computer languages.
If you have technical problems with this magazine, contact firstname.lastname@example.org