XML and the IT architect
What is this new technology all about and how does it affect enterprise and application architectures?
XML is an interesting and relevant topic for system architects, but there isn't an abundance of practical information available that describes how to leverage this new technology. This month's IT Architect, Jonathan Rich, attempts to remedy that lack of information with a hands-on introduction to reading XML documents and a survey of XML standards and tools -- with a particular look at XML middleware. He also offers his assessment of how XML will affect enterprise and application architectures. (5,700 words)
As system architects, we are always searching for new technologies like XML to leverage in our environments. While most people understand that XML is related to data exchange, there is a lack of truly relevant information available on this emerging technology. In our continuing effort to handle issues of direct relevance to systems architects, we decided to devote this month's IT Architect to XML in order to provide a perspective on when it might make sense to adopt this technology into an environment. We explore not only what XML is all about, but how it can be leveraged. XML appears to be one of the emerging technologies that addresses a key business/technology need, especially as it pertains to the Internet, and this column discusses why we believe this technology is here to stay.
ast year XML (Extensible Markup Language) was quite popular as a buzzword; this year it has almost taken over as the dominant feature needed by every new Internet application. Beyond the hype, XML is a pervasive new technology: publishers use it in both traditional and online publications, and software vendors include support for XML in everything from browsers (both Microsoft Internet Explorer 4 and 5 employ XML) and databases to middleware, ERP, and procurement software.
But what is XML really all about, and how will it affect IT architectures? That's what I'm here to discuss. In this article I will describe the basics of XML (and its related proposed standards) with a few definitions and a few simple examples. Then I'll briefly cover the tools available for building XML applications, with a specific look at the XML middleware called WIDL. Finally, I'll describe the various places where XML can play a role in enterprise and application architectures.
XML: A better ASCII
Back in the early days of computers, character sets for encoding letters and control characters varied among vendors. Moving information, even pure text, could be hazardous and time-consuming. Gradually the adoption of standardized encoding schemes, starting with EBCDIC and ASCII and continuing up to Unicode, made the exchange of text relatively easy and transparent.
But the same cannot be said for data structures. Relational data and object data both have an inherent structure, even if they are implemented differently. In contrast, a text stream does not automatically have any structure unless everyone agrees on a way to encode that structure. With ASCII, you could take a text stream and read the letters in the stream, but still not know how to read the structure and, therefore, not be able to read the message. The power that XML brings is in its ability to define how to encode structured data into a text stream in a standard way. Because the encoding is standard, everyone can read the structure and understand the message. Just as ASCII turned a stream of bits into characters, XML can turn a text byte stream into a structured data object ready for use in applications.
There is another comparison that is useful when first approaching XML as a beginner. XML is a markup language like HTML, which requires special markers called tags to be added to text documents to give some added meaning to the document. For instance, in HTML the <p> </p> tag pair indicates the beginning and end of a paragraph. Naturally, the purpose of each markup language is different, so no one language will suit all tasks.
Now here's the comparison: if the ultimate purpose of HTML is to convert a text stream into the presentation of a page, then XML's ultimate goal is to convert a text stream into a data object, one that can have a highly complex inner structure.
But enough of comparisons -- let's look at an actual XML document.
Basic syntax: Well-formed documents
One of the goals put forth by the authors of the XML standard has been to make XML documents easy to read both by humans and machines. For the most part, they have succeeded, as you can see from the following example, which shows a well-formed XML document for a purchase order:
<?xml version="1.0" ?> <PurchaseOrder> <Customer> <Name>Acme Airline and Stormdoor Company</Name> <Address>34 Poplar St., Watertown, MA</Address> </Customer> <Supplier> <Name>Computer And Printer Paradise</Name> <Address>45 Seabreeze Dr., Miami, FL</Address> </Supplier> <Order> <Item>A brown hat</Item> </Order> </PurchaseOrder>
The simplest observation is that the document is all text, which makes
it easy to transfer between processes, across the enterprise, and between companies
via the Internet. Next, you'll notice that like
HTML, XML tags are delimited by < and > and come in pairs (for
example, the begin tag
<PurchaseOrder> is matched by the end tag
</PurchaseOrder>). The XML tags in this document (apart from the
<?xml> tag) are not part of the XML standard -- they are user
defined. XML is extensible because it allows users to extend it by
defining their own types.
Taken together, all of the defined types in this XML document define a
purchase order: a
PurchaseOrder is composed of a
Customer is composed of a
Name and an
composed of a list of
Items, etc. You get the idea. It is a hierarchy
of tags that provide the structure of the purchase order. This
hierarchical structure is shown graphically in Figure 1.
Figure 1. A purchase order as a parse tree
It looks a bit like an object model (yes, it can easily represent relational and object data). It looks like a parse tree (yes, it is easy to parse). Yet it is still easy for humans to read.
Easy to read, yes -- but how do we know if it is complete? For one thing, the tags match and none overlap. For example, if
<Customer> was followed by
</Order>, the end tag for
Order would overlap the end tag
Customer and would be incorrect as a result. Begin and end tags
for a structure must be nested inside of its parent begin/end tags. So, first we know that the document forms a coherent set of tagged
elements that are properly nested and constructed. At this point, the
document is well-formed in that it satisfies the rules of XML, although
it has not formally defined its own data structure.
Basic syntax: Valid documents
At this point, we can still question whether there is missing data or data of an incorrect type. We solve this problem by supplying a declaration of the exact structure of the document. In this case, the document in question is a purchase order document, using a document type declaration (DTD), which is shown in first half of the document below, following the
<?xml> tag and preceding the
<PurchaseOrder> tag. By
providing a structural definition of what a
PurchaseOrder ought to be,
an XML parser can examine any XML document and tell if it conforms to
PurchaseOrder DTD. If an XML document conforms to its own DTD,
then it is called a valid document.
<?xml version="1.0" ?> <!DOCTYPE PurchaseOrder [ <!ELEMENT PurchaseOrder (Customer,Supplier,Order)> <!ELEMENT Customer Supplier (Name,Address*)> <!ELEMENT Order (Item+)> <!ELEMENT Name Item Address (#PCDATA)> <!ENTITY ctp "Cambridge Technology Partners"> <!ATTLIST Customer uid ID #REQUIRED > <!ATTLIST Order cid IDREF #REQUIRED > ]> <PurchaseOrder> <Customer uid="C0023416"> <Name>&ctp;</Name> <Address>304 Vassar St.,Cambridge,MA</Address> </Customer> <Supplier> <Name>Computer And Printer Paradise</Name> <Address>45 Seabreeze Dr., Miami, FL</Address> </Supplier > <Order cid="C0023416"> <Item>A brown hat</Item> </Order> </PurchaseOrder>
Reading from top to bottom, the
<!DOCTYPE> tag defines the name
of the document structure to be defined. The element named as the
document type is also known as the root element. It is
followed by a list of
<!ELEMENT> tags (enclosed within the square
brackets). Their meaning is generally intuitive:
<!ELEMENT PurchaseOrder (Customer,Supplier,Order)>
This tag indicates that a
PurchaseOrder is composed of the elements
mentioned between the parentheses -- a
Supplier and an
Order. Likewise, both a
Customer and a
Supplier are composed of a
Address. And so on.
The only obscure part of the
ELEMENT declarations are the cryptic use
+, which are a carry-over from
XML's ancestor, SGML, and which indicate how many times a sub-element must
appear. In order to ensure compatibility with SGML, the XML DTD syntax is the same as that used for SGML, although alternate methods of defining
DTD are currently under active discussion (more on that later in the
Here is a quick list of the most commonly used signs for how many times a sub-element must appear:
?-- Zero or one time
*-- Zero or more times
+-- One or more times
That takes care of the main structure of a purchase order, but what
happens when we reach the next level down, as with
These elements are declared as being of type
PCDATA (parsed character
data), which indicates that they are simple text that the parser can read
without special characters like
>. So, for example, the first
<Address>34 Poplar St., Watertown, MA</Address>
Here the content "34 Poplar St., Watertown, MA" is
PCDATA and is placed
between the begin and end tags for
Address. Not too complicated.
The second example is a bit more complex. It has two more commonly used
ENTITY tag is
easy to understand. It is generally equivalent to an inline macro of the sort found in many programming languages. In the
ctp is declared as an
ENTITY with the definition being the
"Cambridge Technology Partners". It is used in the
body as follows:
ENTITY is preceded by
& and followed by a
;. The application reading this XML document would then
know to replace
&ctp; with the associated string.
ATTLIST tag on the following line is the mechanism by which XML
tags can have declared attributes, just as in HTML. In this case,
Customer has an attribute called
uid of type
ID and it is
required. Without going into too much detail, the following types of
attributes are supported in XML:
CDATA: Non-parsed character data
NMTOKENS: A name or a list of names
ID: A value that is unique within the entire XML document
IDREFS: Contains the value of an
ID, or is a list of
ENTITIES: Contains the value of an entity or a list of entities
enumerated: A list enclosed in parentheses using
|(pipe character) to separate elements on the enumerated list; for example,
The DTD allows an attribute to either be required (
IMPLIED), or have a value assigned in the DTD (
ID type attributes are interesting because they function like primary
keys in a data model. An attribute of type
ID must be unique within
the entire XML document. Therefore, if we search the document for an
ID "C0023416", we will find the associated
ELEMENT -- very handy for finding data. This
ID attribute can also be
linked to other
ELEMENTS within the document using an attribute type
IDREF (ID reference). For example, we have declared
<!ATTLIST Order cid IDREF #REQUIRED>
and have used it as follows:
<Order cid="C0023416"> <Item>A brown hat</Item> </Order>
ID reference is not a new
ID; instead, it refers to an
elsewhere. Therefore, for
cid="C0023416" to be valid,
there must exist an
ID with the value
"C0023416" in some
other part of the XML document. In this way, we have linked an
Customer. Now, we have mechanisms for defining elements similar
to both a primary key and a secondary key (or foreign key) -- all
within a text stream.
ELEMENT has only attributes and no content. Rather than
be declared as
PCDATA, it is declared as
EMPTY as follows:
<!ELEMENT SKU EMPTY> <!ATTLIST SKU uid ID #REQUIRED>
EMPTY ELEMENT is used normally in two forms, either with an end tag
or with an extra
/ in the begin tag in lieu of an end tag, as
<SKU uid="3495945-43954-12123"> </SKU>
Finally, there are at least two more types of declarations commonly used: comments, which are read by humans, and processing instructions, which are read by computer applications:
<!-- this is a comment --> <?realaudio version="5.0" frequency="5.5kHz" bitrate="16Kbps"?>
As you can see, even with the simple subset of syntax we've covered, XML provides enough syntax and mechanisms to declare the structure and contents of most data objects, given that the content and data values must also be represented as text strings.
XML and its related standards
XML is the descendent of SGML, the parent of the current generation of markup languages. SGML came out of the publishing and natural-language-processing communities, and had support from a number of small tools vendors, including ArborText and OmniMark. In 1986, SGML was accepted as an ISO standard (ISO 8879) and found its niche at large corporations where it was used to encode and structure technical documents, though it could be and was applied to a vast range of data structuring and storage problems as well. SGML is highly complex, as it has been forced to adapt to the needs of these sophisticated uses.
With the emergence of the Internet and HTML, many felt that SGML needed to be lighter weight for use on the Internet. HTML was not suitable for defining data structures because it focused on a fixed set of tags that determined how to display a document in a browser. HTML cannot be extended with new tags and so does not support user-defined typing. What it boiled down to was that SGML was too powerful and HTML not powerful enough. As a result, the community experts at large pursued other approaches that focused more on data structures and less on presentation.
In the summer of 1996, Jon Bosak of Sun Microsystems recruited a group of SGML experts and formed the XML Working Group under the auspices of World Wide Web Consortium (W3C). According to the formal design goals of the XML Working Group, XML should be:
There were informal design goals also. Rumors have it that one informal goal was to make XML so easy to use that a typical grad student should be able to write a parser for XML in a week. Another important goal was that the standard should be completed in a year's time -- not the traditional decade it often takes to finalize a standard for a programming language like C++.
The XML working group took fifteen months to complete the XML 1.0 standard -- not on target, but well within reason. By this time XML tools were already appearing in the form of parsers, and several auxiliary standards (needed to round out the functionality of XML) were already being proposed. XSL, XLink, and XPointer are among the proposed standards that provide XML support for style sheets, hyperlinks, and other features. Just as SGML existed prior to XML, there were several other existing standards, such as HyTime and DSSL, that were rewritten and reduced to support XML.
Here is a quick summary of the current proposed standards; Figure 2 shows a graphic view:
|Seeks to provide scoping for element and attribute names, avoiding name collisions between documents. A namespace is a collection of names identified by a URI (Uniform Resource Identifier) where the names are used as XML elements and attributes; approved (working draft) on January 19, 1999.|
|Style-sheet language for XML data, meant for use in Web browsers; goes beyond CSS for HTML.|
|Improves on HTML anchors by supporting bi-directional, one-to-many, typed links (a footnote vs. glossary reference, for example); based on HyTime.|
|Allows you to point at anything you want in a target document without requiring a predefined HTML link anchor, among other features; based on the Text Encoding Initiative (TEI).|
|Metadata for XML documents, like HTML Metadata tags (author, copyright, publication date); approved on February 24, 1999.|
|Allows interactions with Web servers to be defined as functional interfaces that can be accessed by remote systems over standard Web protocols; an XML-based middleware.|
|Replaces DTD for defining XML data types; based on earlier proposals including XData, SOX, DCD, DDML; working draft released May 6, 1999.|
|A natural extension to the XSL pattern syntax, XQL is notation for addressing and filtering the elements and text of XML documents; it provides a concise, understandable notation for pointing to specific elements and for searching for nodes with particular characteristics; the proposal was provided in September 1998.|
There is also some effort being made to standardize how an application accesses an XML document once it is in a parser. Two APIs are of particular interest:
|A tree-based API that compiles an XML document into an internal tree structure, and then allows an application to navigate that tree; it is limited by available memory.|
|An event-based API that reports parsing events (such as the start and end of elements) directly to the application through callbacks, but does not usually build an internal tree. As an event-based API, it provides simpler, lower-level access to an XML document. More importantly, you can parse documents much larger than your available system memory, and you can construct your own data structures using the callback event handlers.|
Figure 2. XML and its related standards
Tools for using XML
The first and most important tool to have when developing XML applications is the XML parser, which is needed to parse XML documents. A variety of academic and free parsers are currently available; most are coded in Java. In addition, several commercial companies have started offering updated versions of these parsers, or have built their own. It is not clear how consistent these implementations are across parsers. However, there are many parsers to chose from, as almost every tool vendor includes some kind of parser along with its product. Benchmarks comparing the various options are just now beginning to emerge.
Turning to the tools needed to build standard IT architectures, things are even more in flux. XML support for basic enterprise components, such as databases and middleware, hardly existed in 1998 and have only arrived this year. Many vendors are still at the vaporware stage.
Here is a list of the categories of tools that are needed to successfully develop XML apps:
XML middleware: A WIDL example
As I described earlier, WIDL is an XML-based middleware that encodes an RPC request as an XML document (that is, as an encoded text stream). The company that submitted the WIDL standard to the XML working group, webMethods, has implemented a WIDL server that exchanges WIDL requests with another WIDL server over HTTP. This approach provides a native WAN middleware that can pass requests across Internet firewalls, which is in sharp contrast to the LAN-based middlewares such as CORBA and DCOM that have great difficulty running on the Internet without resorting to HTTP tunneling or other such techniques.
Figure 3 shows the WIDL architecture in a graphic form.
Figure 3. WIDL architecture
We've already walked through a simple XML document, so let's briefly
examine what an XML middleware request looks like. The example below
is taken from the proposed WIDL standard. It defines a package
tracking service for an interface called
"genericShipping", as you can see by looking at the
<WIDL> ELEMENT and its attributes in the first line. The
<Service> ELEMENT includes only attributes and uses no end tag.
Its attributes define two bindings --
input variables and
"TrackOutput" for output variables.
In this example, the values defined in the
binding get passed via HTTP as name-value pairs to a service residing
REFERENCE="doc.title.text") are used in the
"TrackOutput" binding to check for successful completion
of the service, and to extract data elements from the document returned by
the HTTP request.
<WIDL NAME="genericShipping" TEMPLATE="Shipping" BASEURL="http://www.shipping.com" VERSION="2.0"> <SERVICE NAME="TrackPackage" METHOD="Get" URL="/cgi-bin/track_package" INPUT="TrackInput" OUTPUT="TrackOutput" /> <BINDING NAME="TrackInput" TYPE="INPUT"> <VARIABLE NAME="TrackingNum" TYPE="String" FORMNAME="trk_num" /> <VARIABLE NAME="DestCountry" TYPE="String" FORMNAME="dest_cntry" /> <VARIABLE NAME="ShipDate" TYPE="String" FORMNAME="ship_date" /> </BINDING> <BINDING NAME="TrackOutput" TYPE="OUTPUT"> <CONDITION TYPE="Failure" REFERENCE="doc.title.text" MATCH="Warning Form" REASONREF="doc.p.text" /> <CONDITION TYPE="Success" REFERENCE="doc.title.text" MATCH="Foobar Airbill:*" REASONREF="doc.p.value" /> <VARIABLE NAME="disposition" TYPE="String" REFERENCE="doc.h.value" /> <VARIABLE NAME="deliveredOn" TYPE="String" REFERENCE="doc.h.value" /> <VARIABLE NAME="deliveredTo" TYPE="String" REFERENCE="doc.h.value" /> </BINDING> </WIDL>
The full DTD for WIDL appears below. As you read through the WIDL DTD, notice that virtually all of the WIDL DTD was already covered in our earlier discussion of a simple XML document for purchase orders.
<!ELEMENT WIDL ( SERVICE | BINDING )* > <!ATTLIST WIDL NAME CDATA #IMPLIED VERSION (1.0 | 2.0 | ...) "2.0" TEMPLATE CDATA #IMPLIED BASEURL CDATA #IMPLIED OBJMODEL (wmdom | ...) "wmdom" > <!ELEMENT SERVICE EMPTY> <!ATTLIST SERVICE NAME CDATA #REQUIRED URL CDATA #REQUIRED METHOD (Get | Post) "Get" INPUT CDATA #IMPLIED OUTPUT CDATA #IMPLIED AUTHUSER CDATA #IMPLIED AUTHPASS CDATA #IMPLIED TIMEOUT CDATA #IMPLIED RETRIES CDATA #IMPLIED > <!ELEMENT BINDING ( VARIABLE | CONDITION | REGION )* > <!ATTLIST BINDING NAME CDATA #REQUIRED TYPE (Input | Output) "Output" > <!ELEMENT VARIABLE EMPTY> <!ATTLIST VARIABLE NAME CDATA #REQUIRED FORMNAME CDATA #IMPLIED TYPE (String | String | String) "String" USAGE (Default | Header | Internal) "Function" REFERENCE CDATA #IMPLIED VALUE CDATA #IMPLIED MASK CDATA #IMPLIED NULLOK #BOOLEAN > <!ELEMENT CONDITION EMPTY> <!ATTLIST CONDITION TYPE (Success | Failure | Retry) "Success" REF CDATA #REQUIRED MATCH CDATA #REQUIRED REBIND CDATA #IMPLIED SERVICE CDATA #IMPLIED REASONREF CDATA #IMPLIED REASONTEXT CDATA #IMPLIED WAIT CDATA #IMPLIED RETRIES CDATA #IMPLIED > <!ELEMENT REGION EMPTY> <!ATTLIST REGION NAME CDATA #REQUIRED START CDATA #REQUIRED END CDATA #REQUIRED >
How does XML affect IT architectures?
For IT architects, the first question has to be: "So what does XML do for us?" Actually, it offers quite a bit; the technology:
Question number two is usually: "What is XML good at doing?" Again, the answer is pretty complete; XML:
A number of application categories are currently being built using XML. Here is a brief list of the more common ones:
How might XML change IT architectures?
The changes ushered in by XML over the next few years will be pervasive, and will focus on promoting greater sharing between application domains in an enterprise, between tiers of multitiered application architectures, and between corporations over the Internet. Let's conclude by sketching out some of the possible changes.
Enterprise sharing is improved when systems loosely coupled by messages can use a common message format that is independent of any one vendor or domain. In fact, such sharing of XML business objects may be the key to implementing business processes and transactions across enterprise domains using workflow technology.
Application architectures may change in even more dramatic ways.
First, the ability to push XML data to a browser along with presentation instructions in the form of XSL commands may reduce and simplify the information that a Web site has to send to a browser. It may also increase the amount of processing done by client browsers. An HTML page can be displayed in only one way, but when the data driving the page is available, the data can be displayed in many ways. For example, downloading a set of products to a client as XML data is feasible if the client's browser can sort and present the data in different ways, as requested by the client, without having to return to the HTTP server for more information.
Let me suggest an analogy: pushing XML data to a browser is the three-tier equivalent of the old two-tier clients, which received only data from a database server. In the two-tier model, clients did all their own processing -- business logic as well as presentation. This approach would suggest that XML data would enable moving more data-driven processing to the client.
Next, it may be possible to rethink how we use the client tier in an n-tier architecture. As I mentioned above, we normally think of pushing HTML pages and XML data to the client in response to an explicit request. With the ability to move XML data between the client and a Web server, we now have two-way channel for exchanging data. This ability would make it possible to reverse the normal flow of data by having the client initiate a transfer of data to the Web server. In fact, by adding an onboard, personal Web server, clients could have a mechanism for handling incoming requests made by Web servers, an approach that would make Web clients and Web servers roughly peer-to-peer. The protocol for the communication would be HTTP and the data and requests exchanged would be in XML. The requests could be synchronous RPCs (as per WIDL) or asynchronous messages (using an XML-based MOM running over HTTP). Using this two-way communication with a standard XML message/request/data formats is likely to promote the Web client to a full-function tier in an n-tier architecture.
Following this line of thinking, clients could also send XML data and requests out to more than one Web server to drive multiple business processes. The client browser could easily become the integration point between multiple Web sites, just as the application server for a Web site may be the integration point for many company-side systems.
In any case, the main point is that XML data, requests, and messages running over HTTP may be the best way to avoid problems with firewalls and make the Web clients a full-functioning member of an N-tier architecture.
The most exciting new architectural approach maybe the ability to use XML-based middleware and the exchange of XML business objects to integrate business processes across corporate firewalls. Communication between Web applications is far easier to build when a WAN protocol like HTTP is available for the transport layer, and one is not restricted to LAN-based protocols/middleware like IIOP/CORBA and DCOM, which are unable to pass through most Internet firewalls.
Integrating Web applications across the Internet will not be easy, even when all the technical transport problems have been solved. Our experience with EDI has shown that integrating business processes and coordinating documents is hardly trivial. Security and legally binding document exchanges are issues with which the XML community has far less experience than the EDI vendors. Nonetheless, at a time when corporations are trying to integrate their business processes with those of key suppliers and customers and trying to outsource many parts of their supply chain, the rewards for successful inter-corporate implementations may be great.
New B2B applications are already beginning to appear. Some Web sites are already offering to host consolidated up-to-date product catalog services where suppliers can provide XML-based product specs and catalog pages for display to buyers. Even an online auction site has begun providing a similar service to host specialty auctions. The items to be auctioned are loaded as XML documents with individual look and feel style sheets provided to each specialty auction client to maintain a separate look and feel.
XML promises to bring architecture to a new level, and we, as IT architects, have the opportunity to exploit this technology in our future environments.
About the author
Jonathan Rich is a national technology architect for Cambridge Technology Partners's North America Technology Organization, CTP's most senior technology team. As such, Jonathan brings 15+ years of IT industry experience in architecting solutions for a variety of companies. Jonathan specializes in object orientation and database technologies and has worked for product vendors in this technology area, as well as for systems integration firms.
Cambridge Technology Partners, a consulting and systems
integration firm, offers management consulting, process
innovation, custom and package software deployment (including ERP
applications), networking, and training.
If you have technical problems with this magazine, contact firstname.lastname@example.org