Subscribe, It's Free
Letters to the Editor
Link To Us!
Mike Conger, this month's IT Architect columnist, explains the role and value of workflow management in the context of large-scale, enterprise-class systems. An evolving technology, workflow management is likely to provide some compelling solutions to enterprise information systems, approaches, and strategies in the near future. (4,300 words)
|By Mike Conger|
When we talk about adding workflow capabilities to an organization's technology architecture, many people wrongly assume that we are simply implementing a workflow product like Lotus Notes and little else. The reality is that workflow capabilities constitute more of a common service available at the enterprise level. That is, workflow is an enabling technology that can be used throughout the organization, and beyond the organization, to support various business functions. Orchestrating the introduction of an enterprise technology, as opposed to an single application technology, is challenging. This month, we discuss the business justification, potential usage, architectural considerations, and potential pitfalls of taking on workflow technologies. In addition, we explore how workflow can affect an organization's extended enterprise via the Internet, and how workflow relates to XML.
n today's marketplace, businesses are continuously driven to gain a competitive advantage, improving not only the delivery of their products and services, but also their nature and makeup as businesses. Some of the drivers behind this revolution include:
To support this new global marketplace, added demands have been placed on information technology (IT) organizations. In response, the IT industry has increasingly accepted advanced technologies and solutions to fulfill these business demands. These technologies, integrated with a common vision and represented by a strategic enterprise architecture, provide an environment in which business and technology solutions can be applied to address critical needs. The application of workflow management approaches, architectures, and technologies can provide significant value in meeting these demands.
As this model has evolved, there has been a migration from function-focused applications to process-focused applications. A side effect of this change, long after many function-focused applications have been delivered, has been a host technical difficulties. The need for improved process efficiencies, which has already resulted in BPR (business process reengineering) for many companies, is now working its way through the technology arena. A shift in focus toward achieving technical integration and flexibility is now being realized in the real-world usability of process-oriented and component-based architectural solutions. This associated integration flexibility is an absolute IT imperative for the next decade.
However, there is a second IT imperative: the need to coordinate IT and business goals. While Web availability, business-to-business and business-to-consumer (B2B and B2C, respectively) technologies, XML (Extensible Markup Language), EAI (enterprise application integration), middleware, and additional related technologies all contribute to technical connectivity and interoperability, they are for the most part foreign to business. Leveraging formal business process descriptions and corresponding workflow technology, we find common ground between business and IT. Regardless of the implementation technology of workflow management, business process concepts will be a requirement for architects in the next decade.
The fact that there is a technology capable of mapping almost one-to-one onto a business-process language implies that workflow will become a powerful tool as enabling support technologies improve. These technologies are, however, preconditions to the widespread adoption of workflow management across extended enterprise environments (in contrast to successful but limited introduction within large-scale applications). Additional improvements in workflow management tools are also required for the expanded use of workflow in enterprise environments (some of which will be discussed later in this article). On the upside, the long history of such products, and the foresight that was shown in creating workflow management standards means that when these technologies are ready, they will integrate both within and across enterprises.
Overview of workflow management
Workflow management is defined by the Workflow Management Coalition (WfMC) as follows:
The automation of a business process, in whole or in part, during which documents, information, or tasks are passed from one participant to another for action, according to a set of procedural rules.
Workflow management has existed, as a set of either manual or automated activities, for quite some time. Most efforts -- such as the production of an automobile, the processing of customer orders, the creation of information products and services, or the purchasing of raw materials in a manufacturing environment -- often require a complex set of activities, as well as overall coordination and orchestration. Historically, workflow has been supported in organizations via manual, time-intensive processing, which leads to longer processing cycles and costly solutions.
Workflow management has also been closely associated with BPR, and BPR has, unfortunately, obtained a rather dubious reputation. This is due in part to naïve attempts at massive business process change in corporate environments, which are rarely conducted in a phased, controlled, and segmented manner. Moreover, the rapid pace of change, significant impacts across organizational boundaries, and occasional lack of fundamental market forces necessary for large-scale reengineering efforts have also led to questionable results. This is certainly a shame, as workflow technologies and reengineering are very much complementary.
Workflow technologies are often utilized for automation and associated cost-reduction purposes, yet BPR and supporting workflow technologies also offer significant potential in revenue generation. While cost-reduction will likely continue as a catalyst for corporate workflow initiatives, the ability to open new markets, improve revenue potential, and aid in the production of new products and services will also serve to expand workflow opportunities. The opportunity, then, is to leverage workflow management capabilities in order to gain a competitive advantage.
Workflow management also provides a medium by which business and technology solutions can be unified in a disciplined manner. All aspects of delivering solutions -- organizational alignment, corporate culture, the manner in which technology and business solutions are approached, the manner in which information systems are designed, and the formalism presented in describing and agreeing upon standard business processes -- are significantly improved by the proper use of workflow. The focus on cross-organization business process definitions, the implicit and necessary support for the free flow of information across organizational boundaries, the automation of core business processes, and the orchestration of business process activities provided by workflow management all serve as primary catalysts for change.
Traditionally, workflow has played a fundamental role in image processing and document management. Applications in this arena typically utilize forms-based processing, whereby electronic forms are captured, updated, and routed among participants. This form of workflow has long been used by the insurance industry, where document management and claims processing are naturally suited to automation. These document- and image-centric workflow solutions form the roots of workflow technology. Workflow technology has more recently been embedded in business applications and custom software, incorporated into intelligent messaging and groupware engines, and made available as pure workflow products.
Figure 1. Workflow architecture layers
Workflow management architecture
Workflow architecture can be represented by a number of layers:
Workflow routing can vary in complexity and follow three patterns: sequential routing, parallel routing, and conditional routing. Parallel routing permits some degree of concurrency by allowing multiple branches of workflow execution to occur, while conditional routing uses simple to complex conditional logic to control the flow of workflow execution. For example, a loan applicant's credit rating may be based on his or her credit history, past delinquent balances, average outstanding balance, income level, etc. In these cases, factors can be defined, then associated with the workflow process specification and other related activities to assign the applicant a credit rating.
Workflow management standards
The WfMC, which was founded in August 1993 and currently has 200 members, is an international coalition of workflow vendors, users, and industry groups. The coalition's goal is to promote the use of workflow and establish workflow management standards by establishing common terminology, reference models, and interoperability between workflow management products. The following is a brief summation of workflow management standards and architectural components, as defined by the WfMC.
Figure 2. WfMC workflow reference model
The completed process specifications serve as a template to create specific instances of a business process. Process definitions also define the specifications for activities that make up the business process, along with relevant activity sequencing, routing, and dispatching instructions. An activity is defined as a work unit that forms one step in an overall workflow process. An activity could require manual intervention and processing, or it could be a fully automated software process. Workflow vendors provide graphical user interface tools to aid process specification. Workflow engines use these specifications, along with associated metadata, to control processing.
Enactment service and workflow API
Workflow enactment services provide a runtime execution environment in which business process instances are created, managed, controlled, and executed. Standards offer flexibility to workflow management vendors by supporting the enactment service in a centralized or distributed environment. In the distributed approach, numerous workflow engines operate in concert to achieve benefits in functionality and scalability, or support the interaction between execution environments or platform-specific applications and tools. In these cases, the synchronization of process and related process control information across workflow engines is necessary.
Business applications interface with an enactment service via workflow management APIs (WAPIs) and interchange formats. The WAPI enables interoperability between business applications and components of workflow management systems. Interchange formats provide support for the automated interchange of workflow information between different workflow management products. A significant improvement in workflow technology can be gained by using XML.
Workflow engines provide a runtime execution environment and manage the overall processing and execution of workflow process instances. By referencing the business process specifications and rules that govern workflow processing, the workflow engine is able to orchestrate the necessary processing. Operations processing functions provided by the workflow engine include:
Workflow management and the extended enterprise
The use of workflow in the extended enterprise offers a particularly powerful capability. Much progress has been made, particularly in the manufacturing arena, by linking the players of this extended enterprise (for example, manufacturers, suppliers, distributors) in support of just-in-time manufacturing initiatives. Workflow techniques and automation mechanisms between trading partners -- whether supported via electronic data interchange (EDI) or custom interface approaches -- have provided a primary vehicle to more effectively manage processes, manufacture products, and reduce delivery times. The net effect: improved productivity levels and customer satisfaction, faster deliveries, and increased profitability. Applicability of the extended enterprise notion beyond manufacturing (for example, sales, marketing, customer management, etc.) is high.
While EDI techniques are successfully employed in this extended enterprise, they are also quite complex and unwieldy to manage. XML technologies will become a powerful and compelling solution in reducing this complexity. The benefit of XML integration -- a common language and set of rules of operation across workflow engines -- will also administer improved interoperability of workflow solutions across organizations and business partners. XML will facilitate the use of a common and dynamic business language and increase shared business process specifications. (See Resources for more on XML and EDI integration.)
The extended enterprise and associated expansion of the value chain model will be the basis of future growth in workflow technology. The role of workflow in supporting these cross-enterprise ecommerce initiatives will be profound, and will offer tremendous opportunities in allowing businesses to achieve the next level of efficiency, cost reduction, and revenue generation. This extended enterprise offers many benefits, including the capacity to:
Figure 3 illustrates a network model depicting business partners interacting in the fulfillment of a customer order. This model assumes the use of workflow management product(s) by each business entity.
Figure 3. Manufacturing scenario
This scenario relies on intercompany workflow processes that represent an exchange of data and process information between the consumer, manufacturer, and suppliers of raw materials. It also assumes various internal workflow processes utilized by the manufacturer to control internal and raw materials acquisition and production activities. Based on business policy agreements reached between consuming and manufacturing entities, it's feasible that consumers could receive a detailed, real-time view into the activities being undertaken to fulfill their order. The benefits include:
The use of workflow technology to support ecommerce in the extended enterprise model presents a number of challenges, as well:
Scalability and performance
While technologies are in place today that support the enterprise-level usage of workflow management, a key to future success lies in ensuring that high performance and scalable workflow solutions are developed.
Oddly, a number of workflow vendors continue to rely on standalone and nonscalable architectures that are sometimes based on single-threaded models without support for replication or partitioning capabilities. This seems particularly strange in light of the fact that extensions to existing workflow management architectures that support dynamic load balancing, intelligent partitioning of workflow execution, and monitoring facilities to effectively manage and control distributed workflow engine execution are certainly achievable. In fact, workflow is well suited to these highly scalable environments, particularly when coupled with a network built to industry standards, good system management, and middleware tools and technologies.
Figure 4 illustrates a high-level conceptual architecture that leverages distributed workflow services, dynamic load balancing, statistical process metrics and data collection, and distributed remote management and monitoring facilities. The statistical data collection capability collects statistics for process level information, activity execution, aggregate and event statistics, maximum and average duration of processes and activities, etc. These statistics are not only critical in measuring and validating improvements made to business processes, but are also crucial in identifying and resolving performance problems. Statistical information is also used to support rules-based dynamic load balancing and partitioning schemes, and to adjust workflow engine capacity levels to meet performance and throughput requirements.
Figure 4. Workflow platform scalability
Many vendors have made progress in designing high-performance platforms, but few offer the scalability necessary to support the complex enterprise-class platforms needed for large-scale workflow deployments. The enhanced integration of the Internet and commercial middleware solutions (for example, CORBA, MOM, and DCOM) will no doubt help facilitate these improvements. Message-oriented middleware, which provides highly scalable synchronous and asynchronous communications and advanced queuing capabilities, is particularly well suited for the task.
Business transactions used to be managed on mainframe platforms to accommodate transactions that ranged from the very simple to the complex and large grained. The advent of client/server and distributed systems made business transactions manageable across multiple operating and database management systems.
As distributed client/server systems increased in usage, end users relied more on transaction monitors to ensure the overall transactional integrity of business requests. As business application complexity and the use of distributed resources increased, so too did the complexity of these large-grained transactions, sometimes to the point of degraded performance and reduced transactional integrity and application reliability; there was also the damaging side-effect of artificially constrained software reuse.
In contrast, the workflow model relies on a defined business process consisting of discrete activities subject to rules defined for the business process. Each activity is assured transactional integrity, while the set is entirely controlled by the workflow engine. This system provides added support and flexibility, as well as reuse of software assets as software components. It also reduces the need for overly large and complex transactions dependent on increasing resource needs, while providing recovery capabilities in the event of a processing failure. Moreover, formally specifying activities as a workflow-based business processes provides additional management and control benefits. At any point, an interested party can determine the exact status of an execution, including which activities have been processed and which are still pending.
Workflow management, now and in the future
In recent years, the general perception in some circles had been that productivity increases and cost gains afforded via the application of process automation, particularly in the manufacturing sector, had reached their maximum potential. However, in the past two years the explosion of the Internet, wide-scale deployment of intranets and extranets, and the expansion of ecommerce (B2B and B2C) have put these mistaken assumptions to rest. In fact, we have only just begun to reach the potential benefits available. A number of complementary technologies have also converged that apply workflow management to achieve further degrees of process automation. These complementary technologies include, but are not limited to, the following:
Due to greater understanding and use of object-oriented approaches and technologies, as well as improved software development environments, scalable and high-performance middleware, and expanded use of the Internet as a foundation to support business solutions, we will continue to see further progress in automating business processes and reducing inefficiencies.
BPR and the use of workflow management (both principles and technologies) will also provide value in reducing the overall cost of supporting these business solutions. Furthermore, increasing demand and expanding marketspace for enterprise-level solutions, namely EAI, and the expanding role of Web-based transactional systems, will foster a further need for intelligent workflow management solutions.
Workflow market penetration has not achieved its anticipated degree of success. In fact, only recently have signs indicated serious inroads to corporate systems. Reasons for limited market penetration are complex, but they include:
The last item has probably constrained adoption rates more than anything else. Often, corporations are forced to adopt workflow solutions and/or specific application development platforms with embedded workflow, thus limiting their ability to integrate these solutions with core enterprise architecture and deployment platforms. Further complicating the situation is a need to develop additional supporting infrastructure and integration with middleware solutions. Given that the significant area for future workflow management growth is likely to be the use of workflow with core business applications, these issues must be resolved.
The emergence of service-based architectural solutions, combined with distributed object and component-based platforms, will prove to be a significant opportunity for convergence and utilization of workflow technologies. Distributed objects, utilized in Web-based applications, and combined with highly scalable middleware platforms, provide an ideal framework upon which workflow benefits can be achieved. The excellent efforts being undertaken by the Object Management Group (OMG) and WfMC in establishing standards and frameworks in these areas is most promising.
Additional complementary technologies -- namely component-based software development (Enterprise JavaBeans, for example), EAI, and XML -- are all likely to prove beneficial in expanding the capabilities of workflow technology. These technologies will also provide broader appeal to companies requiring enterprise solutions and are likely to play a role in similar future workflow architectures. In order to achieve desired business ends, architects must be aware of these emerging technologies and how to integrate them.
In order for enterprise-level workflow to fulfill its potential, needs must be addressed in the areas of:
Fortunately, market demands and dynamics will continue to be the catalyst for these and other changes.
Ovum has published estimates that state that the growth rates of general purpose workflow
products will reach over $1 billion by 2000, with an
estimated annual growth rate of approximately 38 percent. Embedded
workflow product expenditures are expected to grow to $319 million by the same year, with an estimated annual growth rate of 130 percent. Augmented by
further improvements to core workflow technology, improved
understanding and application of workflow technology, and better
adherence to standards, we could see even more significant increases in
the usage of workflow technology.
About the author
Mike Conger is a senior technical architect with Cambridge Technology Partners, providing enterprise-class information and architecture services and supporting organizations to leverage advanced Internet, object-oriented, and middleware technologies in achieving business solutions.
Cambridge Technology Partners, a consulting and systems integration firm, offers management consulting, process
innovation, custom and package software deployment (including ERP
applications), networking, and training.
Advertisement: Support SunWorld, click here!
|Resources and Related Links|
Workflow integration initiatives
Workflow technology and practices
Workflow management business benefits, strategies, and standards
Other SunWorld resources
|Tell Us What You Thought of This Story|
If you have technical problems with this magazine, contact firstname.lastname@example.org