THE DATA ADMINISTRATION NEWSLETTER – TDAN.com
ROBERT S. SEINER – PUBLISHER

Why Is Defining Terms and Principles So Important?

Producing an architectural framework is primarily about describing a proposed set of components needed to meet some defined business requirements and establishing a set of design patterns, rules and principles that need to be applied across the architecture to ensure consistency and interoperability.

Over the last few years I've been involved in a number of distributed data projects; and on each of those projects, the same principles, such as “service-oriented,” were used, but they each had a different project specific definition.

On one project, part of the build and deployment was put out to tender on a "Best And Final Offer" basis, and all of the potential bidders didn’t have time to read all the specifications in detail so made assumptions (i.e., provided their own definitions) to underpin the tenders. This wasn't recognised until after the procurement had been completed with the consequence that the delivery was significantly delayed whilst the gaps were sorted out.

On another project, a requirement was for the long-term storage of a large number of reports and the data, in an XML format, used to produce them. Being a distributed environment, the users of the report register were regarded as "low trust" so the warehouse operator should assume responsibility for ensuring all data received conformed to the published data specification.  The supplier missed this principle with the result that they didn't plan for the necessary development and testing time and failed the initial user acceptance testing criteria.

A third project involved integrating e-business processes in an open marketplace where there were many companies aiming to supply the services for each process step. Due to the diverse IT platforms, each company was making assumptions based on the capabilities and limitations of their own platform, leading to differences in behaviour in the interfaces. This, in turn, then caused project delays whilst the integration issues where resolved.

In all these cases, the problem wasn't with the functional specifications themselves but was due to a misunderstanding in the principles that underpinned the architecture. Many of the problems could have been avoided if the underlying terms and principles had been explicitly and objectively defined.

Unfortunately, objectively defining want we want is more difficult than it first appears because many concepts used in IT have multiple, even contradictory, definitions that all need to be resolved.

This article discusses an objective approach to defining architectural terms and principles.

What is a Meaningful Definition?

The characteristics of a good definition are that it should:

• Be objective and restrict itself to explaining the desired outcome of applying the principle and any constraints that have to be met.
  
  
• Be concise – this doesn't mean a definition has to be short, but rather that it shouldn't waste words explaining things that are not of functional benefit in solving the specific domain problem.
  
  
• Contain reasons for why each definition needs to apply in the target architecture.
  
  
• Provide sufficient information to allow the reader to understand exactly what is required, including references to any detailed external technical specifications and criteria for judging successful implementation.
  
  
• Be reasonable and achievable given any business constraints that have been imposed on the delivery of the architecture.
  
  
• Explain why any unwanted characteristics are considered irrelevant in this particular domain. Only including the characteristics that are essential to meeting the business requirements makes our target architecture technically simpler and, consequently, easier to build and maintain.
  
  
• Be cross-referenced to ensure that the definitions are consistent with each other and that a statement taken in isolation does not have an entirely different, and possibly even contradictory, meaning when cross-referenced with another statement.
  
  
• Be light on jargon. This is easier said than done; but with the exception of system names and globally accepted acronyms, the use of jargon as terms assumes that the reader knows what the intended translation is. Given that most acronyms have multiple translations, this is not a good assumption.

To illustrate this, let's consider this definition of a service (it’s a real example from a widely available public specification):

Service: A specific activity that is carried out by one party on behalf of another party.

It qualifies as a “definition” in the general sense but doesn't really tell us much at all and certainly wouldn't tell a software engineer anything about what he is expected to produce as a result of having “services” as part of the architecture.

Taking a service-oriented approach implies a whole set of characteristics about the deployed services that could be considered as architectural principles including: Location Independent, Dynamic Service Discovery, Stateless Transactions, Low Trust and probably a dozen or so other characteristics depending on which “experts” we listen to and which software paradigms we want to adhere to.

However, not all of these characteristics will be appropriate in every environment. We may, for example, simply need peer-to-peer messaging which doesn't require "Dynamic Service Discovery" or have a single service provider for each service so “Location Independence” isn’t required.

Consequently, a domain-specific definition of a service would identify and describe only the characteristics that are important, such as:

Service: A specific activity that is carried out by one party (the Service Provider) on behalf of another party (the Service Requester). The Service Provider and Service Requestor interact by passing Service Messages between each other that meet a mutually agreed specification (the Service Definition).

All Services within the [domain name] are based on a Request / Response Service pattern where the client send a Service Message to the Service Provider requesting a Service to be carried out and the Service Provider, after carrying out the service, returns a Service Message describing the outcome of the activity.

The key characteristics of a Service are:

• Stateless – each invocation of a service is assumed to be in isolation of any other service invocation; and except for data stored in a database, there is no persistent of information between service calls. All client defined information either must be passed as part of the Request Message or retrieved from persistent storage.
  
  
• Location Independence – each Service is identified by a URL (uniform resource locator) that can be used to remotely access the Service, but the actual physical location of the Service Provider is not significant to the Service itself.
  
  
• …and so on.

As expected, the definition contains other terms that provide a more precise definition and possibly even a full specification of what is required depending on the type of term being defined.

So, extending the highlighted terms in the above definition, we might have the following:

Service Message: A message that is either sent to or received from a Service Provider. All Service Messages are XML Documents and follow a uniform format consisting of two parts: (1) A Message Header containing [stuff] plus (2) a Message Body containing [stuff].

The Service Message may be either (a) a Request Message inbound to the Service Provider to initiate the service operation or (b) a Response Message from the service Provider returning the results of the requested operation to the Service Requestor.

The full specification of the Service Message patterns can be found at [link]

Message Header: The part of the Service Message containing metadata about the message such as the originator, intended recipient, routing instructions, access control information and so on.

To enable transparent interoperability all Message Headers follow a standard format and must conform to the XML Schema specification defined in [link]...

Message Body: …and so on

This is where the definitions starts being useful because we have started to introduce cross-references to the technical specifications describing exactly how something will look and behave within this domain.

In other words the definitions are becoming objectives-based and much more applicable to the specific implementation.

How Much Is Enough?

As previously stated, developing a robust definition of a term is similar to developing a business requirement. It also has the same potential problems in that each definition raises questions that may give rise to other terms that, in turn, also need definition.

So given the potential number of things that could require a definition then a key question is:

How much is enough to allow us to move forward?

If we tried to fully explain everything in advance of starting to build something, then we'd probably never get to building anything. Instead, providing definition of terms should be an ongoing activity of elaborating principles, rules and definitions over time as further clarification needs to be provided.

Consider this case in point from a recent project defining the message exchange format:

We decided that:

"All messages between functional components would be exchanged using XML."

This didn't seem to be a problem until someone asked whether the interfaces would be specified in XML-DTD or XML Schema. So the definition was extended to:

"All messages between functional components would be exchanged using XML that conforms to the definition specified in XML Schema v1.0 (as published at [link])"

A while later in order to enforce consistency of data usage. it was decided to embed cross-field and conditional validation rules into the XML specifications. Unfortunately, XML Schema has no facility for complex rule definitions so we needed to look at extensions to XML Schema to meet this requirement and settled on Schematron. So our definition of XML was extended to:

"All messages between functional components would be exchanged using XML that conforms to the definition specified in XML Schema v1.0 (as published at [link]) with validation rules defined using Schematron v.1.1 (as published at [link]). The Schematron rules will be treated as an integral part of the XML Schema definition."

And so it evolved…

This illustrates an essential principle when evolving definitions over time, which is that each modification of the definition encompasses the previous definition because avoiding “breaking changes” in architectural principles is critical to avoid unnecessary re-engineering of any components that have already been deployed.

How to Make the Glossary Accessible

This isn’t really anything to do with the activity of defining terms and principles, but it is still an important point to consider.

"Providing a definition" also means publishing the definition because, to the outside world, not making a definition easily accessible is really the same as not defining it; and if the readers cannot easily access a definition, then they will supply their own definition.

Generally this means publishing a glossary of terms; but, as is often the case with large scale IT architecture, the full specification will be a large set of interrelated documents, and maintaining a glossary of terms in each document would almost certainly result in inconsistencies appearing over time.

So a global glossary is required.

There are many approaches to publishing a global glossary, and one of the most flexible approaches is an online WIKI-style glossary that can be easily edited online. By publishing it online, the glossary can be referenced by each document (and individual terms can be directly referenced) through embedded URL.

An even more powerful approach is publishing a topic map presenting a graphical road map of all the terms and how they relate to each other. For example, if we look at our definition of a Service, we can see a topic map would look something like this:

Figure 1: Topic Map Example

Topic maps are particularly useful in complex environments because they allow multiple access routes to be defined into the glossary, and the terms and definitions can be clustered in multiple different ways. In addition, the graphical representation of a topic map gives an impression of overall structure to the glossary from the topics that are listed and how they are linked together.