Enterprise application integration -- message broker style Pointers on evaluation and implementation By Alex Pan

August  1999

You're familiar with various approaches to application integration, such as point-to-point application integration with remote procedural calls (RPCs) and message-oriented middleware, or database-to-database integration with replication. But what about application integration using a message broker as the middleware? Alex Pan fills in the details of what this involves and the factors you should consider prior to implementation. (3,100 words)

Mail this article to a friend

Foreword
Middleware vendors are now jockeying for a piece of the new enterprise integration architecture market, commonly called EAI. This month, we discuss what EAI is, what it means for the business, and how, as technologists, we can leverage this type of technology. Integration isn't easy -- figuring out what to integrate, what technologies need to communicate, and what they're sharing is a time-consuming effort. And blindly loading EAI products won't help at all; EAI requires in-depth knowledge of the environment and the business model in the organization. Only then will the IT professional be able to provide a seamless infrastructure that is able to link disparate technologies, applications, and package solutions together in a single enterprise technology architecture.

--Kara Kapczynski

istorically, corporations organized themselves around business functions and built or purchased the multiple standalone mission-critical applications needed to support the core business. Such applications were designed to handle a specific business function properly and many have been in use for many years. In the past five years, however, corporations have implemented many new back-office packages (such as SAP, PeopleSoft, and Oracle), and front-office software packages (such as Vantive and Clarify) to support and automate an increasingly complicated business environment.

The information gathered and stored in each of these systems is isolated and not shared with other systems. As time has passed, the need to share information between systems has increased. Industry analysts report that companies spent almost $40 billion on integration problems in 1997, and project that $60 billion will be spent in 2001. Individually, companies are using up to 40 percent of their annual information technology budgets on building and maintaining application integration solutions.

Exchanging information between systems in realtime or near realtime in order to make better, faster decisions is a necessity, not a luxury. Furthermore, corporations are finding out that they need to exchange and/or share information with business partners in a realtime or near-realtime manner. Companies know that they must now move toward managing and automating business processes, such as order processing, rather than individual business functions, such as order entry, shipping, inventory, or billing. This requires companies to seamlessly integrate the disparate applications that implement these business processes. Enterprise application integration (EAI) software is a new kind of software that aids this transformation.

What is EAI?
There are many definitions of EAI depending on whom you ask. For the purposes of this article, we'll say that enterprise application integration is the process of tying together multiple applications to support the flow of information across multiple business units and IT systems.

EAI is all about interoperability and information synchronization across multiple applications -- mainframe, packaged or purchased systems, and custom application systems. EAI enables sharing of information, not just within an enterprise or organization but within a business environment that includes a company, its suppliers, and its customers.

EAI products offer a framework that provides a significant portion of the integration solution right out of the box. An EAI product must provide the following services:

• Messaging
• Connectivity
• Security

Messaging
A messaging backbone is the foundation of an EAI framework. This backbone transports messages between resources, reconciling network and protocol differences. It allows applications to share information with the outside world by sending and receiving messages. It adds quality-of-service options to message delivery, such as security and queuing. The messaging services should ideally support both synchronous and asynchronous messaging. Asynchronous messaging is particularly useful when applications must integrate in broadly distributed environments where the availability of applications and network segments is unpredictable.

Connectivity
Having established a common messaging backbone for interapplication communication, the next step in deploying an EAI framework is to plug the applications into the messaging backbone and allow them to send and receive messages. Because each application has its own interface, communication protocols, and data formats, the connectivity setup must include services for orchestrating the flow of data between integrated resources. It has to be able to resolve data coding differences, translate to and from the common protocol used by the messaging service, and transform data content to and from a standard format.

Security
Typically, each application provides security through the user ID and password entered by the user. Integration technology must provide similar services for the entire business process, including user authentication, resource access control, and information encryption. This technology must also integrate with existing security implementations.

Traditional solutions
Traditional application integration has been done using point-to-point programming with or without a messaging infrastructure and database-to-database integration. Point-to-point programming integrates applications in a pair-wise fashion through low-level communications programming. With this approach, the process and messaging infrastructures are highly fragmented. Messaging infrastructure facilitates communication between applications and shields developers from low-level system and networking details. Database-to-database integration is simple when you have detailed knowledge of both databases. (The June 1998 IT Architect column, "Application integration without band-aids," offers detailed information about these two traditional solutions -- see Resources for the URL.)

Standalone stovepipe applications increasingly must be connected via automated processes. In the beginning, each application that must talk to another application will build its own conduit (see Figure 1). Application A will build a conduit to get information Bb from application B. Application C will also build its own conduit to get information Bb from application B.

The resulting architecture ends up with point-to-point solutions that get complicated when the number of applications and/or systems involved increases. As you can see, if information Bb in application B changed, all the applications that built a conduit to access information Bb would have to change. Modifying programs takes time and demands scarce and costly programmer resources. This type of solution requires writing code on the source system as well as the target system. This makes sharing information with business partners and coordinating changes over time difficult, at best. Maintenance is a big issue and is very expensive and time-consuming. Most types of point-to-point application integration use message-oriented middleware (MOM) to facilitate the communication between systems.

Figure 1. Point-to-point integration

Another type of application integration that has been traditionally used is database-to-database integration. Database-to-database integration means that we simply share information between two applications at the database level. You can use the replication features built into many databases to move information to the target database as long as they both use the same logical schema. Or you can use database replication and transformation software to move information between different schemata on different types of databases.

Simplicity is the main advantage of using database-to-database integration. The downside is that the logic of manipulating the data resides in the database. When dealing with packaged applications such as SAP R/3 or PeopleSoft, you'll have to deal with the application programming interface (API) provided by the vendor. This requires additional expertise and makes the integration work more expensive.

Message broker: A better alternative?
The most flexible type of application integration utilizes the message broker. Built on existing message-oriented middleware, it enables you to integrate systems at either the application level or the database level.

A typical scenario involves "events." Once an event is defined in the source system, those systems interested in that event can "subscribe" to it. A message broker takes care of delivering event messages to subscribers. For example, an order-entry system publishes an order event, and a shipping system, a billing system, and an inventory system subscribe to the event. When a customer orders a book, an order event is published, and the shipping, billing, and inventory subscriber systems receive the event message and initiate whatever processes are required to update their respective systems.

This type of middleware can split, reformat, and combine messages based on predefined rules, forwarding the reformatted messages to appropriate target systems. It utilizes rule engines, transformation engines, and intelligent routing mechanisms. When using message broker software, you typically don't need to change the source or target system.

Most message brokers offer "adapters" or "connectors" to link to common packaged applications such as SAP, PeopleSoft, Baan, Vantive, and Clarify, as well as most relational databases and legacy systems like CICS. Most of them also offer software development kits (SDKs) to build custom adapters, should you need to develop them.

As a developer, when you use MOM for traditional program-to-program communication, you need to take care to transform messages from their incoming format to a different format so that the participating application systems can communicate in a form that each understands. If workflow is involved in the application, you must program that part yourself. Message broker software provides message transformation services and a workflow control engine to reduce the developer's programming load.

Figure 2. Message broker integration

Evaluating EAI packages
Here are a number of things to consider when choosing your EAI package:

• Administration tools
• Development tools
• Scalability and redundancy
• Application integration tools
• Extensibility
• Time-to-implement

Administration tools
An EAI package should provide good administration tools to monitor the messaging backbone and components and to deploy the final solutions.

Development tools
These tools are important to the developers who put the pieces together. The more comprehensive and integrated the tools are with the other parts of the package, the less time it will take to use the package.

Scalability and redundancy
The fundamental infrastructure should be designed to scale up in order to support current message volume and future growth. The package should also provide redundancy to support fault-tolerant configurations in order to be used as part of the mission-critical application solution.

Application integration tools
What are the systems you need to integrate currently and in the future? Pick an EAI solution that supplies an adapter to the packaged applications you intend to integrate and one that makes it easy to build your own adapters for custom-built application systems.

Extensibility
A good integration solution must be customizable and extensible. A company should be able to add to and change business processes without affecting the underlying applications, and IT should be able to change applications without affecting business processes.

The application systems of an organization may support common environment and standards, but every organization will have some applications that are just a little bit different. In addition, common environments and standards are constantly evolving. Integration technology must be adaptable to those changes. For example, although integration technology may support the key packages and languages, it should be extensible to support, through custom development, all the other packages and languages that an installation may use to implement its business processes.

Time-to-implementation
One critical factor in application integration is the time it takes to complete the work. The faster application integration can be accomplished, the faster a business can begin to enjoy its benefits. One of the key selling points of EAI products is that they are able to produce results faster than custom solutions. As a result, they must deliver on this point, even in the first project, in order to maintain credibility with business and IT management.

Implementing EAI solutions
As you start implementing a message broker-based EAI solution, there are a number of things to keep in mind:

• How mature is the message broker software package? The less mature the package, the more prototyping and testing it will require.

• Does the package have the proper infrastructure to support large-scale, mission-critical applications? If not, you'll have to provide some of the infrastructure services through custom programming.

• If you need to integrate into enterprise resource planning (ERP) software packages, call-center packages, or sales-force-automation packages, do you have the expertise required?

• Do you need to integrate with custom-built applications?

• Can you establish an enterprisewide view of data?

Here are a few major obstacles that might prevent you from achieving success in your first project:

• Not understanding the issues surrounding the business integration of different applications, such as the communication, coordination, and ownership issues for translation between different applications that are managed by different groups.

• Not understanding the semantic difference between data elements.

• Not understanding the semantics of different data elements in the context of different applications. For example, the notion of sales price in one application may differ from the notion of sales price in another. The sales price may actually be the same as the list price in the second application.

• Not identifying responses that have been delayed beyond "a reasonable length of time."

• Underestimating the complexity of business rules involved (especially with routing and context-based behavior).

• Data quality and integrity problems in one application clashing with the constraints and rules in other applications.

• A requirement to do reverse-translation may be complicated when unique IDs aren't provided in one application.

As you move toward implementing an EAI solution, you should be aware that the benefits of EAI aren't automatically realized by the use of EAI tools. Effective use of EAI requires the appropriate approach and business infrastructure:

• Understand the applications to be integrated along with the number of interfaces involved.

• Know which departments own the applications and seek executive support for the integration.

• Understand the information to be exchanged and how this information is used in each application.

• Evaluate the effort and risks when finalizing enterprise definitions and identify key executives as owners and mediators of the process.

Architectural guidelines
Given the above description of EAI and how to implement it, the next question might be: When should you use it in an enterprise architecture? The answer is: Not always.

When the integration requirements can be solved using only data movement and/or replication, the database-to-database integration approach makes sense. It's easy to implement and doesn't require additional technical expertise.

The main characteristics that would justify an EAI solution are:

• The integration involves third-party package applications.

• There is the need to transform data between different platforms.

• Business processes change quite often.

• Some kind of workflow is involved.

In such cases, message broker-based solutions should be considered.

Counterindicators would include the following:

• The source code of both source and target application is accessible.

• Integration doesn't involve third-party package applications.

• You need to integrate very few applications.

• You do not see the need to integrate more applications in the future.

You can bring in different message broker packages for evaluation, but you should decide on one package to implement as your enterprise EAI backbone. If you allow one department to implement its own EAI backbone you'll have to integrate the various EAI packages at a later date -- which should be avoided. Keep in mind that point-to-point integration and database-to-database integration are acceptable short-term solutions; when you architect these solutions, do so with the understanding that they'll probably be replaced by a more complete package later on.

Future trend
As more companies move to exchange information between business partners, the need for a common definition of information will increase. Business processes must be more integrated, both within and between companies, to achieve a competitive advantage. Companies that do this first will gain marketshare. A major limit to a business-process focus is the business-function focus of existing applications. Business-function-focused applications are turned into business-process-focused applications by integrating them along business-process lines, both within and across companies. So, high volumes of integration become a strategic business necessity. EAI packages make it possible to achieve this integration much more quickly than you could with "old style" integration. EAI solutions that include XML, in particular, are likely to enable cross-business integration. This is why numerous vendors have made public commitments to support XML in future product releases.

According to the Gartner Group's Gartner Research Note, by year-end 2000, 75% of Fortune 500 companies will be using XML in at least one prototype application integration project and 25% will be using XML in a production application integration project (0.7 probability). Support of XML in products will be an important factor in deciding which middleware you choose.

Conclusion
There are real business benefits to be gained from using brokered EAI software packages, as follows:

• Faster time-to-market for new products through an automated information flow.

• Increased efficiency due to consistent, accurate information available in realtime.

• Improved customer responsiveness due to automated self-service applications.

• Lower cost of operations due to streamlined infrastructure of e-business channels.

• Reduced cost and complexity compared to custom point-to-point integration.

There are also a number of architectural benefits to be gained by using message brokers to do large-scale application integration. The method replaces many point-to-point connections with a reliable, scalable solution that can more easily adapt to changes over time. The flexibility of the publish-and-subscribe paradigm used in many EAI systems decouples applications, allowing them to change independently.

EAI emphasizes the need for a common definition of information between applications. You should always keep in mind that the greatest problems in an EAI implementation arise from difficulties in ownership and meaning of business rules and data, not from the technology.

Resources

• ActiveWorks message broker middleware:
  http://www.activesw.com/
• BusinessWare message broker middleware:
  http://www.vitria.com
• CrossWorlds software suite:
  http://www.crossworlds.com/home.html
• MQSeries middleware:
  http://www.software.ibm.com/ts/mqseries/
• MQSeries Integrator:
  http://www.software.ibm.com/ts/mqseries/integrator/
• "Application integration without band-aids," Sam Wong (SunWorld, June 1998):
  http://www.sunworld.com/swol-06-1998/swol-06-itarchitect.html
• Full listing of previous IT Architect columns:
  http://www.sunworld.com/common/swol-backissues-columns.html#itarchitect

• The SunWorld Topical Index -- a comprehensive listing of all SunWorld articles by subject:
  http://www.sunworld.com/common/swol-siteindex.html
• Visit sunWHERE -- launchpad to hundreds of online resources for Sun users:
  http://www.sunworld.com/sunwhere.html
• Explore SunWorld's back issues:
  http://www.sunworld.com/common/swol-backissues.html
• IDG.net, your one-stop IT resource:
  http://www.idg.net

About the author
Alex Pan is a senior technical architect in the Cambridge Technology Partners's North America Technology Organization (NATO). Alex is NATO's competency leader for middleware technologies. As such, he works on overall architecture designs and helps organizations to leverage strategic technologies.

Cambridge Technology Partners, a consulting and systems integration firm, offers management consulting, process innovation, custom and package software deployment (including ERP applications), networking, and training.

If you have technical problems with this magazine, contact webmaster@sunworld.com

URL: http://www.sunworld.com/swol-08-1999/swol-08-itarchitect.html
Last modified: