THE DATA ADMINISTRATION NEWSLETTER – TDAN.com
ROBERT S. SEINER – PUBLISHER

What Are Patterns?

The term "pattern" is usually attributed to Christopher Alexander who, in his co-authored book A Pattern Language: Towns, Buildings, Construction,¹ described the concept of having collections of "patterns," forming a pattern language, which can then be combined according to certain rules to provide solutions to larger and more complex problems.

In information technology, the seminal book on patterns is generally regarded as Design Patterns: Elements of Reusable Object-Oriented Software,² which in turn spawned many patterns books covering other areas of IT, not to mention many thousands of websites on the subject.

However, as that book testifies, pattern-based analysis have been around for thousands of years, and defining patterns is the predominant approach taken to capturing essential knowledge in every area of intellectual activity. All knowledge and activity has an underlying pattern. Understanding “Everything” (big “E”, as in “Life, the Universe and Everything”) is about discovering and formulating patterns!

The intent of a pattern is simply to capture the essence of a solution in a form that can be easily communicated to those who need the knowledge and present the information in a coherent and accessible form reduces the potential for misunderstanding.

A pattern is any type of model, template, algorithm, equation or artefact that describes a problem and how it may be solved. It can take many different forms ranging from very precise algorithms, such as mathematical equations, that exactly describe a comprehensive solution to a well-defined problem through to very abstract patterns describing nebulous concepts such as macro-economic behaviour.

For example, this is a pattern (an Equation expressed using Algebraic Notation) for calculating the hypotenuse of a right-angled triangle:

This is a partial pattern (part of a template expressed in the EBNF notation) for constructing a Book:

Book ::= Title [ Contents ] Chapter { Chapter } [Appendices]
Chapter ::= [ Number Title | Number | Title ] Paragraph { Paragraph }
Paragraph ::= ...and so on

This is also pattern (a data domain expressed as a concept map) showing the organisation of information present in a particular knowledge domain:

This is also a partial pattern (part of a model expressed in the Martin Entity-Relationship Notation) showing the construction of a powered vehicle:

Finally, this is also a pattern (an algorithm expressed using a UML Collaboration Diagram) describing the interactions in a message-based authentication process:

All these patterns appear to be radically different, which illustrates the scope that patterns can take, but when analysed they all have the same basic characteristics, namely:

• The patterns are both abstract and concrete at the same time. They are abstract in the sense that they describe a generic solution to a generic problem in generic terms but concrete in that they can be directly applied to a particular instance of a problem to construct a definite solution.
  
  
• They are expressed in a formal pattern language tailored to the domain that the pattern applies to.
  
  
• Once created they are reusable and can be applied many times to other problems of a similar nature. Even if not directly applicable, a pattern can still indicate the general shape of any derived pattern (e.g., the “powered vehicle” indicates the general structure of a car, lorry, aeroplane, train, etc. but may need extending or modifying to deal with the specifics of each).

A couple of the examples may even be considered to be models in the traditional IT sense of the word. They are!

The main difference between a model and a pattern is the purpose for which it was created. A pattern purports to be a general solution to all problems of a particular type, whereas a model only purports to be the solution to an individual instance of the problem.

It could even be argued, though it would be an entire article in its own right, that a pattern is actually the meta-model for a model. Consequently, from the perspective of an individual problem, both pattern and model can be used interchangeably to label the artefacts required to explain the solution to its sponsors and implementers.

Another important point to make is that patterns are not principles!

A pattern suggests a general approach to a solution but, unlike a principle, is not an absolute constraint – there may be other patterns that are equally applicable that may produce the same solution with different techniques.

Architectural patterns are the beginning of the actual architecture where the business requirements and the architectural principles and, if we’re being realistic, the existing architecture are brought together and potential solutions identified and incorporated into the framework.

For any given architectural principle, there may be many different architectural patterns that could be used. For example, we may decide upon a request / response service-oriented environment. Then there are nine distinct interaction patterns that could be applied to the problem, each with different capabilities, benefits and disadvantages.

Equally we may decide to have a “single version of the truth” with a single “database of record” for each “data-item of record,” but does this mean that we have the “single centralised database” or the “distributed federated database” pattern? Either could apply depending on the overall business requirements and future business plans.

In this article, we are going to discuss what architectural patterns are and how they fit into the architectural framework.

A Note on Selecting a Pattern Language

There’s no such thing as a perfect pattern or modelling language that meets all the requirements that an architect may have to address or that can express all the concepts they may want to communicate to someone else.

In IT, perhaps more than any other area of activity, there are literally thousands of possible pattern and modelling languages that could be used to define and communicate any aspect of the architecture to its audience.

There are dozens of different notations available for producing a data model – e.g., UML, Chen E-R, Martin E-R, Express-G, ORM, OMT and just as many for describing Data Flows e.g. UML, Gane-Sarson, Warnier, DSSD, DFD; and there are literally thousands of languages available for describing business processes, though most of them we call programming languages (many a person has observed that the program code is just a model of what the computer is supposed to do – the generated machine instructions are the real programs).

All of these modelling languages have built-in assumptions about how something will be implemented, which in turn limits how something can be expressed used that pattern language.

For example, UML assumes that the deployed system will be an object-oriented software environment and that everything will be identified by an “Object ID” (usually a memory pointer). However, in a business information model where “identity” is externally defined by the business stakeholders’ data intensive environment, assuming that everything has a system-generated “object ID” just doesn’t work when talking to people about the real-world data. Equally, the UML notation suffers some significant limitations when using the business language to label the terms used to identify data items.
If we look at the examples in the opening section, we can hopefully see that expressing the equation for calculating the hypotenuse would be almost impossible to communicate clearly using an entity-relationship diagram or a concept map, and equally the construction of a powered vehicle couldn’t be easily communicated using the EBNF or algebraic notations.

Consequently, it should be easy to conclude that choosing the appropriate pattern language to fit the target architecture is an important choice and, similarly, once a pattern language has been chosen, then it must be consistently applied across the architecture to ensure there is no confusion over what is being communicated.

In a complex environment, it might even be the case that more than one pattern language will be required to express all of the viewpoints required to communicate fully.

Even if a widely acceptable pattern language is selected, it may still be necessary to further constrain the selected language to fit the specific business domain.

Characteristics of Appropriate Architectural Patterns

Architectural patterns are the “engineering” part of architecture. They allow the problem domain that has been decomposed into a discrete set of characteristics (the business requirements) to be solved using previously defined and tested generic solutions.

Choosing architectural patterns is one of the areas where the potential benefits of the activity can save a disproportionate amount of time and provide a return on investment that far exceeds the initial cost spent identifying the appropriate patterns from the almost unlimited number available.

Unfortunately, identifying and choosing architectural patterns can be a bit problematic because there are so many of them with new ones emerging every day; and, just as important, we have to be sure that what is selected is genuinely an architectural pattern and not one of the lower-level design or implementation patterns.

Hence, we need to consider the characteristics that a good architectural pattern should have and how to differentiate an architectural pattern from a design or implementation pattern. The important characteristics are:

• Solution Focused
  
  
• Platform Independent
  
  
• Flexible
  
  
• Diagrammatic

The following sections discuss these characteristics in more detail.

Solution Focused

Enterprise architecture is nearly always seen as an “ivory-tower” activity, and there is always resistance from development teams to taking advice from people who are not directly involved with the implementation of the application.

This is not just a downwards problem; sometimes people up the management chain have the same problem in that they need to see how a particular pattern directly applies to their area of interest. Otherwise it is seen as just an intellectual exercise.

If either party doesn't "get it," then the pattern will probably be ignored and fail in its intent.

The architectural principles and business requirements define the restrictions that everyone will be working under, and the architectural patterns should focus on enabling something to be done and shaping the boundaries of the solution.

A good pattern is one that clearly shows its applicability to meet one or more of the business requirements and also clearly describes to the implementers how the pattern assists in deployment of the solution but does not dictate how it will be deployed.

Flexible

The best patterns solve more than one of the current problems and can be adapted to solve other, as yet unforeseen, problems as they arise.

Unfortunately, flexibility has the risk of introducing complexity. If we return to the equation for calculating the hypotenuse, which was:

This equation covers all of the stated cases but can be considered inflexible because it only applies to right-angled triangles. A more generic solution to calculate the length of the line opposite a known angle in any triangle is:

However, it may be more flexible, but the equation is also more complicated because we’ve introduced the concept of cosine, which now needs defining and the need to know one of the internal angles because we’re no longer sure that it’s a right angle.

In the long-term, the second formula is a much more useful pattern to have available. Whether the additional flexibility and complexity are useful or not will depend purely on the business requirements and the number of required scenarios that the simpler pattern solves compared to the number of scenarios that the more complicated pattern solves.

The same principle applies to all patterns – sometimes the introduction of a little extra complexity produces a far more useful pattern than what was originally conceived.

Platform Independent

Architectural patterns should be about “What the solution is” not “How the solution is implemented”.

Two maxims that I believe should always be borne in mind when establishing platform independence are:

“Your data will always outlive your applications.”³

“The business stakeholder couldn’t give a hoot about the implementation. ”

The business of business is to provide a service to its customers, make profits (even charities should be increasing their working capital year-on-year to allow for future activities) and meet its regulatory obligations. In nearly all business cases, IT supports this by moving information around the company and delivering it to people (or computers operating as proxy people) to act upon, then storing it away somewhere safe so that it can be examined again later. IT is simply an enabler of the business activities but is never the reason for an activity to exist.

Consequently, any architectural pattern that forces a particular operating environment to be used is not platform independent so should not form part of the architecture. If platform-specific features appear to be essential, then either the business requirements are incorrectly stated or the architectural pattern is not correctly defined to an appropriate level of abstraction.

Platform independence isn’t always easy to identify because often the pattern language is the same as the design language, and sometimes the same implementation language, so the differences aren’t always easy to spot. For example, the following pattern (created in Altova XML-Spy) describes the general structure of a service response message:

We could interpret this as being a specific XML message (in which case, it’s an implementation model), but it’s actually establishing the general characteristics of a service response message but doesn’t provide any detail about how each development team must implement it (i.e., it’s defined using XML schema but could be implemented using DTD) and, in particular, the content is left completely undefined.

In most cases, platform independence can be achieved just by removing platform-specific terms and replacing them with generic terms to identify a concept. If the pattern still makes sense and can be migrated into other problem areas, then there is a high probability that it is genuinely an architectural pattern rather than a design or implementation pattern.

Diagrammatic

This may appear to be a subjective observation; but there are many, many research papers published in the area of human learning that conclude that the human mind is predominantly a visual processor and that people gather and process more information by sight than through all the other senses put together.

The adage that “a picture is worth a thousand words” is very true and should be leveraged wherever possible.

Many parts of the architectural framework are generally better represented using diagrams rather than being described in words. For example, expressing the following quite simple diagram on coherence boundaries solely in words would be tedious and difficult:

Words, as shown in the diagram, should be reserved for highlighting interesting points and providing definition of terms.

Obviously there is more explanation required and there are a number of terms that need to be defined (so would be included in the glossary discussed in Part 2); but as part of an architectural pattern, a diagram is much better than a purely textual explanation.

However, a diagram is only meaningful if it communicates successfully. So, because at the architectural level we are generally dealing with high-level concepts, any diagrammatic notation should utilise the minimum number of representative symbols to describe different things of interest and the minimum of detail required to convey the message.

The Key Architectural Views

Most of the time when we talk about patterns, we are usually discussing individual components and features of the overall architecture (micro-patterns); but there are also patterns that can apply to the entire enterprise architecture as well (macro-patterns⁴).

All of the recognised architectural frameworks (Zachman, TOGAF, IE, etc.) are macro-patterns in the most generic sense because they describe an approach to solving the problem of managing the full life cycle of an enterprise’s IT.

To make the architecture-description content effective and well-accepted, we must identify the scope of our architecture coverage and the intended audience so that we can provide the right material to meet peoples' interests and backgrounds.

The standard architectural frameworks suggest doing this by presenting different “views” of the architecture. All of them define a different set of views, or maybe the same view with a different label; and there are four core views that are consistently described which are:

• The Functional Business or Domain View representing all the business processes and activities that must be supported.
  
  
• The Information or Data View representing the data that is required by the business to support its activities.
  
  
• The Integration or Data-Flow View representing the flow of information through the business, where it comes from and where it needs to go.
  
  
• The Deployment or Technology View representing the physical configuration and technology components used to deploy the architecture in the operating environment.

Although the final “to-be” architecture would provide a business specific description of each of these views, the architectural framework should satisfy itself with providing patterns underpinning the final architecture once it is formulated.

In the following sections, we look at what forms an architectural pattern might take in each of these views.

Functional Business Area or Domain View

This is an area where few patterns already exist or have been documented, and most business analysts generally start all of their analysis from scratch with a blank sheet of paper and no pre-conceived ideas of what a given process should look like within an organisation.

However, patterns do exist and could be reused if we took the time to recognise them.

Back in my early days, I cut my teeth designing and developing accounting systems (we didn’t do architecture in those days) covering all the major business administration functions  – sales, purchasing, payroll, asset management, financial reporting, etc.

Last year as part of an engagement with a financial investment company, I was asked to analyse the investment trading and portfolio management processes. Nearly every activity in the process had a direct analogy with the activities identified during my accounting days. In many cases, the activities even had the same labels.

From an architecture perspective, it is the similarities rather than the differences in these apparently disparate processes that are important. All organisations undertaking a similar business activity will probably have similar high-level business processes that will only differ at the detailed level.

Any manufacturing company will create a production to systematically build the product, and at various points in that process particular components will need to be available. A publishing house will have its own processes that are different than the manufacturing processes but similar to all other publishing houses.

This is why an organisation tends to hire people from the same business sector or with a given professional background because they recognise at the corporate level that experience is transportable between organisations.

It is therefore logical to deduce that there are patterns of activity that can be applied to the functional business area, and these should be identified.

Information or Data View

Many people, including myself, would argue that information or data architecture is the only true form of enterprise architecture simply because it is the use and distribution of information that makes a business run smoothly. It's also the case that most business users understand their data far better than they understand the software that will process it.

It is for this reason that some sort of high-level model of the information within the enterprise is important. Unfortunately most data-models dive straight in at the granular level and provide far too much detail to be able to communicate concepts effectively.

A common misconception when producing a business information model is that it has to be detailed in order to be useful. This isn’t true! At the architectural level, the information model just has to be explicit enough to explain what types of information is available and how it is organised.

Instead of a detailed data model, an information view is required that represents the higher level business concepts and how they are associated with each other.

Returning to the diagram presented as an example in the opening section, which was:

From an information view perspective, this provides all the information necessary to understand the content of this particular data domain. It describes all the main groupings of data (the data domains), the major entities within those groupings and the main associations between those entities.

It does not contain so much detail that the basic concepts are lost and provides just enough information to show the integration points where new data sets would probably be incorporated if and when they are introduced.

The Data Integration or Data-Flow View

Data integration is an area that has an extensive number of patterns already defined probably because it is the area that most technology vendors focus on to sell their products. Unfortunately, many of those patterns are published as deployment models to accentuate the product being marketed but with a bit of thought could easily be adapted into macro-patterns.

At the micro level, a data-flow simply involves two parties, either of which can be man or machine, and the transfer of information between them. There is an almost infinite number of ways that a collection of parties or data-flows could be combined together (theoretically the limit is a factorial of a Cartesian join of the number of parties involved); but, in reality, for any given business activity, there are a small number of optimal, or nearly optimal, patterns for minimising the data-flows to efficiently complete the activity.

For example, the following diagram represents the data-flow for a large business intelligence and marketing data warehouse for which I recently drafted the business proposal:

This is a pretty generic data integration pattern that could be utilised for a marketing company or a research company or be used by a government department responsible for formulating and monitoring policy. For a specific industry, decisions will need to be made to identify who constitutes each category of user, how each of the identified data-sets will actually be deployed and exactly what data they contain, but the pattern itself is reusable.

The following diagram (first seen in Part 1) is also a pattern for a data integration view that represents a significantly different environment:

Like the previous diagram, it covers all the main locations that data may be present and how it flows between those locations. The main difference is that this is a highly distributed environment involving a large number of different organisations. This isn’t apparent from the diagram (nor should it be) because once the interfaces between each component are established and well-defined, the actual physical deployment should not be a concern at this level.

With a bit of squinting and generalisation of the terms, the same approach could be applied to any report production, registration and storage environment. In fact, in the last 2 years exactly the same overview has been used on four separate projects⁵ and I’m about to use it for at least one more project this year.

So, to conclude, macro-patterns exist for the data integration view and are reusable once they are identified.

The Deployment or Technology View

There are also a significant number of existing patterns in this area; but with the vast array of technology available, I don’t believe that there is much benefit in trying to document characteristics of an individual piece of technology.
Instead, the other three architecture views and the business requirements should identify the characteristics of each technological component and then we find some technology that fills the gap.

In many cases, much of the technology will already be established in an organisation and in-house expertise will be available to ensure it fits the architecture. Where the technology isn’t already on site, then most solution vendors will happily try to show you how their products will meet your requirements.

Documenting “New” Patterns

Information technology in all its forms is a continually evolving activity; and patterns, processes and solutions are constantly being discovered – mostly at a faster rate than any of us can possibly absorb.

On the other hand, there’s rarely anything new, and most “discoveries” are just refinements of what came before. The only reason we keep discovering new ways of doing things in IT is usually because we haven’t bothered to fully document the underlying patterns for the previous way we did things and hence don’t realise that the “new” way was the “old” way all along.

Unlike many other academic areas, there is not a single repository available to IT architects that we can go to as a resource for researching architectural patterns and proposing new ones that we think we have discovered. IT is an innovative and creative industry; and over the years, I have seen some absolutely stunning solutions to business problems that could have been applied to many other business areas in addition to the initial deployment but never made it into the patterns arena.

We could, of course, just leave it to the “industry heavyweights” to do all the thinking and identify the patterns and document their findings, but that means that the rest of us miss out on all the fun parts of creative thinking in IT.

Personally, I frequently find “new” patterns and join dots between subject areas that I previously thought were isolated. It may not always be original thought (I’m not arrogant enough to think I’m really the first person to think of something) and someone else may have gotten there first many years before me; but if I’ve never seen the pattern before or heard any mention of it, then it should be considered “new” and dealt with accordingly.

There are many online resources that describe the sort of things that a well-defined pattern should include. My personal choice, because it focuses on the pedagogical aspect of patterns, is the approach described in "Documenting Good Education & Training Practices Through Design Patterns" (Mitchell Weisburgh – see link) in which is proposed nine potential aspects to building a pattern.

The approach is aimed at defining a scenario for a particular pattern that looks something like this:

If I find myself in some [Scenario] like [Examples], and I face this [Problem], with these [Constraints], but my situation is different from these [Related Patterns]. Then I should think like this [Rationale] and if I want this [Resulting Context], then I should follow this [Pattern]. And here is a [Name] to help me remember this scenario.

Each of the elements in brackets represents a characteristic of the pattern that may need to be described, although not all characteristics need to be described for all patterns.

“New” patterns must be treated with more care than “existing” patterns because of the absence of a body of work to build a consensus against. Applying an approach like the one described above should provide a systematic analysis of a “new” pattern to decide whether it is robust enough to be used as part of an architectural framework.

Conclusion

Finally after four gripping instalments, we've covered the main artefacts for creating an enterprise architecture framework, namely:

• Creating a glossary of terms so that so that each significant term and concept can be clearly defined in a single location for all to read.
  
  
• Defining the architectural principles so that we have a consistent set of overarching "rules" to govern the detailed design and deployment of the architecture.
  
  
• Identifying the underlying architectural patterns that will be used to shape the overall solution, its components and how they join together.
  From this point on, we will be looking at activities that might be undertaken to produce a specific architecture for a particular problem domain and discussing the analysis necessary to objectively produce a "fit for purpose" and defensible architecture.

In the next column, we will look at purpose and benefits of producing the integrated business information model.

References:

1. A Pattern Language: Towns, Buildings, Construction. C. Alexander; S. Ishakawa & M. Silverstein, Oxford Press 1977.
2. Design Patterns: Elements of Reusable Object-Oriented Software. E. Gamma, R. Helm, R. Johnson, J. Vlissides; Addison Wesley 1994
3. This isn’t strictly true – they do care about cost and time to delivery, but the details of how it is done are normally irrelevant.
4. I nicked these terms from the world of economics. Macro-economics deals with very large domains and micro-economics deals with very small domains, but the core economic theory applies equally to both domains. There is a change of emphasis with scale, but not a change of purpose.
5. I probably shouldn’t have said that because the customer or business sponsor like to think that something has been created especially for them; but I see patterns everywhere in everything I’ve ever done and I remember them, so why re-invent the wheel or re-draw a diagram just to make it look bespoke. It’s much more productive if we’re honest and don’t waste time trying to fool each other.