THE DATA ADMINISTRATION NEWSLETTER – TDAN.com
ROBERT S. SEINER – PUBLISHER

Enterprise Integration Today

The integration environment is volatile with many tools, vendors, approaches and concepts bombarding us daily. The emerging ‘alphabet soup’ of concepts and methods relating to integration in the enterprise is causing considerable confusion. EAI, EBI, EII, EDI ELI and others are bandied about as solutions to all our problems. In reality they represent different areas of focus for the integration effort that often overlap. Further, they tend to be focused on a particular tool vendor solution, service provider, business issue or direction of interest. How can we bring order to the proliferation of tools, ideas and concepts of integration in this type of environment?

Enterprise business integration is much more than any one approach or collection of approaches to integration concerns. Applying the most appropriate approach or solution demands a systematic way of looking at integration opportunities and responses.

There are many integration issues an enterprise faces today. Most of us can relate to the few that are listed below:

• Data is scattered among a number of databases, data warehouses and datamarts.
• Rewrite of applications is costly and time consuming.
• Changes in strategy and direction of the enterprise come at an increasing pace.
• Changes are enterprise wide and not limited to a single product, operating unit or location.
• Management is faced with multiple choices of investment in integration.
• Integration options are increasingly expensive with large investments needed in infrastructure, processes, applications, and data management.

Bi–Directional Integration

There is an orderly way we can look at the integration landscape to help sort out the approaches and solutions. Integration has been addressed from two directions in the enterprise often each unaware of the other. One direction is driven by the business strategy and strategy execution such as a merger, acquisition, consolidation, expansion etc. The other is driven by operational needs and is technology driven such as data access, application interfaces, process renovation and others. Not coordinating the two directions can be costly for an enterprise in terms of time and money. It is like a rail track starting at each end of a country and not meeting in the middle.

We call the business driven/technology driven pairing a ‘bi-directional’ view of business integration. It takes into account the two major directions of integration and attempts to organize them into a simple framework with a set of layers to help determine the situation and approach most appropriate for your enterprise. The diagram below shows the layers of integration and the two directions. We have observed five basic layers going from purely technology oriented to purely business oriented. Both of these are going on at the same time in the enterprise. The motives for each direction are different:

1. Business Driven – Outward focused as a response to the business environment. This is usually a business need looking for an enablement technology.
2. Technology Driven – Inward focused as a response to operational pressures. Often a solution looking for a business justification.

The business oriented top layer may have very little to do with technology. This level is featured by strategic business actions such as merger, acquisition, consolidation etc. It is the result of the response an enterprise makes to the forces in the economy, market, industry and government. This approach is effectiveness driven, that is, it answers the question ‘are we delivering the right products to the marketplace?’ Integrating a business is not the same as integrating technologies but some of the principles are common to all layers of integration.

The Layers of Integration

Business Integration – The key effort here is the restructuring of the enterprise from a business perspective. The use of technology is of secondary consideration. The concern is about integrating processes, locations, product lines, infrastructure, intellectual property, markets, production and delivery operations, catalogues and so on. These are all ‘pure’ business things and only impact IT when there is a need for integration of infrastructure such as used in the delivery of financial data. The techniques used involve enterprise analysis and do not involve IT based methodologies.

Partner Integration – This layer addresses linkage to enterprise partners including suppliers, customers, investors, analysts, interested parties, potential customers and employees. While it may seem strange that an employee would be considered a partner it is reality that employees have both an internal and external view of their enterprise particularly when it involves remote access such as for sales or technical support. Architecture and requirements are critical to this layer with an understanding of the goals and objectives of the stakeholders. IT enablement begins to take on importance at this layer as the means of delivering data, knowledge and information. At this layer, IT methodologies are blended with enterprise analysis approaches to get a linkage between the business view and the technology enablement.

Content Integration – Content integration pulls together intellectual property from many sources and displays the results to interested parties. In some cases they are power users inside the enterprise and in other cases they are external users that are looking for support information such as catalogues, product sheets, maintenance data, parts and replacement information, service manuals, operations manuals and other bounded knowledge domains. New types of requirements articulation and tools are needed here. Methods to identify the knowledge requirements (e.g. essential knowledge identification) are needed with the capability to convert them to delivered functionality for users.

Application Integration – The intent of application integration is to provide access to functionality across multiple applications or processing data across several applications. There are a number of integration strategies for applications depending on the scope and reach of the requirements. The concepts of scope and reach are important to integration. If the scope includes only two applications then pair-wise integration is used. If the scope is three or more applications then a strategy of logic, data and functionality encapsulation come into play. If the scope of the integration is within a small area of the enterprise such as a single operating unit or location, then there is high likelihood of success. If the scope is across many operating units then other factors will impact the effort such as company politics, conflicting business objectives, competitive technology strategies, executive incentives and many more that have little to do with integration. Reach is a very different idea. It relates to the architecture of the enterprise, both business and IT architectures. If the impact of the integration effort changes the business or IT architecture dramatically then the reach risk is very high and there is a likelihood of failure. If the reach risk is low, that is, the architecture will not change much, then the likelihood of success is high. Clearly an alignment approach is needed to determine if the scope and reach will place the enterprise at risk in its integration efforts.

Data Integration – This is the core integration approach in many enterprises today. Managing data well and achieving data integration go together. The need to manage data carefully varies among types of enterprises. Data purveyors (firms that sell data about consumer purchases, how many people watching television programs, who sees which kind of movies and so on) have a very high need for quality metadata, efficient data management, and consistent quality of data. Hence they do considerable data cleanup, completion and management. Manufacturing firms on the other hand have a mixed and moderate need for data management. Banks and insurance companies have a low need though they submit that the need is high. There is considerable pressure today due to integration in the layers above the data layer that is forcing better data management.

Often companies use application integration as the justification for data integration and as the source of requirements for the data integration. This is a risky strategy as the application may be subject to considerable duress when the enterprise acquires a package or a business integration need overrides the application impact. This could remove the driver for data integration. The data integration need should be driven by a business requirement not an application requirement.

While data integration is the core of most integration efforts and strategies, there are several key issues that need to be considered before investing in a large data integration effort:

• Metadata is not well managed and has varying degree of importance depending on the particular industry.
• Business need may vary from industry to industry, market to market and strategy to strategy.
• Logical data integration may be sufficient and a viable alternative instead of physical integration.
• Data quality may not be consistent with the integration need.
• Action is not always warranted by business need.

The table on the right summarizes the forces in the enterprise that drive the need for integration at each layer. For each layer there is a suite of technology and IT enablement architectures that must be taken into account. This often causes confusion when enterprises look at their integration needs and ultimately realize that these layers are nested and that the ability to do application integration will depend to a large extent on the ability to do data integration. Content integration requires skills in application and data integration as well as many new skills related to management, use and deployment of knowledge.

Layers and Integration Approaches

Each of the layers has a primary or predominant approach with several secondary or supporting methods of integration that may be called into play.

• Business integration – the primary integration approach is Enterprise Business Integration. Secondary and supporting approaches are Knowledge, Information and Application Integration.
• Partner Integration – the primary integration approach is Enterprise Information Integration for linking to partners, investors and others external to the business. A secondary form is Knowledge and Data integration particularly the use of data hub concepts and portals for delivery of knowledge, information, and data interchange.
• Content Integration – the primary integration approach is Enterprise Knowledge Integration with secondary supporting approaches of data integration and application integration.
• Application Integration – the primary integration approach is the interoperability of applications. Secondary supporting integration approaches deal with data integration. Encapsulation tools provide access to functionality and data via standardized services that can also be used as web services.
• Data Integration – This is the core integration approach in many enterprises today. There is no primary or secondary approach in this case since it is basic. What we have instead is technology enablement and a variety of data integration architectures depending on the enterprise need. The key feature of data integration is the use of ETL tools, (extract transform and load) coupled with brokers for routing the result of the extraction. Further enablement may come from conversion to XML formats for greater interoperability.

Integration Alignment

The layers of integration need at least some coarse alignment so they do not conflict with each other for resources. In fact the alignment should amplify the benefits derived from the integration efforts at each layer. Earlier we mentioned and described the two components of analyzing risk of integration efforts, scope and reach. Just to refresh:

Scope – The degree to which integration is needed on or across layers to achieve the goals of the enterprise. Is it the whole enterprise, all applications every piece of data or is it a small subset of these?

Reach – The degree to which the structure/architecture must be changed to achieve integration. Will the architecture change radically or very little?

These allow for a range of problems that an enterprise can encounter:

• Over Ambitious Projects – If the integration effort in the enterprise has a large scope and reach then integration may be beyond the ability of the enterprise to accomplish.
• Unrealized Potential - If the integration is tightly controlled by cost or is managed by another limiting factor then the scope may be too small or the reach too tight and the integration goal may not be achieved.

So, if the risk were plotted on a grid you would have a data point for each type of integration. The scope and reach data points for each layer define a pattern of enterprise integration. If the pattern shows conflicting strategies across the layers then one layer will act as an impediment to another.

The goal is to achieve a balanced solution across layers. If the enterprise has considerable funds to expend on the effort then all the data points should be aligned and moved as a group for success. If the enterprise has some cost containment issues then the path should go from business integration to the data integration to maximize the business value.

The direction and number of layers of integration interest is important. If the direction is from technology to business then the focus starts with efficiency and the alignment with the layer above should be close. If the direction starts from the business layers then the alignment to the next layer down is close. If there are multiple layers of integration of interest then the need for alignment is more important since the integration architecture and strategy will determine if the layers work together.

Here is an additional note on integration layers and their alignment. When mapping the data points to a grid keep in mind that the cluster shape will help determine the integration strategies. Alignment does not always mean the assessment grid has matching data points. The points may suggest complementary strategies for integration to be successful. The strategy for the whole set of integration layers requires balancing factors such as funding, skill, time and commitment for success.

Conclusions

Reducing the volatility of the integration environment leads to predictable results of the integration efforts. Understanding where the enterprise is positioned with respect to its integration situation contributes to removing the uncertainty that it faces in attaining the structure needed for success. To accomplish this reduction in uncertainty, several key steps are needed:

• Identify the layer of integration of most interest. Particularly note whether the integration is business, technology or both driven.
• Assess the scope and reach of your effort at each layer. Each layer forms a data point on an integration grid. An excessive scope on several layers requires considerable investment in money and management energy.
• Make sure the efforts align and do not conflict. Align in the direction of the drivers. If the integration is technology driven the least likely alignment is with the highest layer.
• Approach each layer and the suite of layers in a systematic manner. A simple and basic integration methodology is required that identifies the integration ‘As Is’ situation, articulates the patterns, strategies, principles and finally states the ‘To Be’ result of the integration effort.
• Select the discipline and technology enablement appropriate for each layer. Generally one for each layer. Avoid complicated solutions where approaches cover more than one layer. These require careful thought to prevent large amounts of maintenance.
• Integration governance requires careful attention to collaboration and political relationships to prevent ‘push back’ from executives.

Organizations must strive to develop an aligned integration strategy that makes sense across each of the layers, taking into account risk and project funding. It is only then that the enterprise can be assured of the best likelihood of success in applying technology based approaches and solutions.

References

Integration grid adapted from “The Power of Strategic Integration”, by Burgelman and Doz, Sloan Management Review, Spring 2001.