9.1 Architectural Mining

Architectural mining is rapid intelligence gathering for making better decisions. Its benefit is intelligence amplification; it makes the architect appear smarter and more experienced.

Architectural mining should be fast and effective, or it should not be pursued. The industry changes too quickly for any procedure that takes more than a few days, weeks, or months-depending upon the scope of the decision-to be effective. Architecture, in its role as a planning discipline, should reduce timelines and make software development more effective and efficient. Note this point because architectural mining has the potential for becoming a career-length activity, instead of a short-turnaround intelligence amplification.

Top Down and Bottom Up

In a top-down design approach, abstract concepts are progressively transformed to concrete designs and implementations. The highest level of abstract design might be the system vision or its initial requirements document. In a bottom-up approach, a new design would be created from fundamental programs or parts. Bottom-up design can be very productive when it involves incremental change or reuse from existing designs.

Top down and bottom up can be compared to up front and after the fact. An up-front approach would generate plans for designs before implementation commences. In an after-the-fact approach, the project would document designs based upon the as-built configuration.

In general, software architecture is considered to be initiated as an up-front approach. Architecture embodies a system plan that enables estimation and efficient system construction. Ideally, architecture is configured through a bottom-up approach called architectural mining.

Interestingly, object-oriented approaches usually define architecture after the fact, as an outcome of the detailed design process, whereas the recommended approach would create architecture as an input to detailed design.

Architectural Farming

Most software design approaches assume that design information is invented as the process proceeds. In a top-down process, design information is generated from requirements, which may be represented as software analysis models. Requirements-driven architecture design is called architectural farming. In a spiral process, design information is invented during each iteration. As the spiral process proceeds, architects invent new design information as they learn more about the application problem. It is fair to say that these approaches reinvent much of their design information.

Precursor designs exist for most information systems applications and problems. These designs are in the form of legacy systems, commercial products, standards, prototypes, and design patterns. In my experience, it is not difficult to identify a half-dozen or more precursor designs for any given application problem. Valuable information is buried in preexisting designs-information that allowed earlier architects to build useful systems. Extracting this information for use in software architectures is called architectural mining.

Architectural Mining Process

Architectural mining is a bottom-up design approach. It exploits historical design and implementation experience to create new software architectures. Because software architects rely on successful previous designs, architectural mining represents substantial risk reduction. The challenge of software architectural mining is to discover, extract, and refine the nuggets of design knowledge. Because there is often a great deal of implementation detail to review, the process is analogous to mining the earth for precious metals.

Mining is a bottom-up design approach, incorporating design knowledge from working implementations. Mining can incorporate design input from top-down design processes, too, so that there can be both top-down traceability and bottom-up realism.

Before mining starts, it is necessary to identify a set of representative technologies that are relevant to the design problem. Technology identification can be done by various means, such as searching literature, interviewing experts, attending technical conferences, and surfing the net. All available resources should be pursued.

The first mining step is to model each representative technology. Technology modeling produces specifications of relevant software interfaces. Using OMG UML as the modeling notation is recommended because it is the industry standard for modeling. OMG UML is effective in describing software processes and entities because it is concise and free from implementation detail. OMG UML is also a good design notation for the target architecture because it is language independent, platform neutral, and distribution transparent. By modeling everything in the same notation, a good basis for design comparison and tradeoff can be created.

In the modeling step, it is important to describe the as-built system, not the intended or desired design. Relevant design information includes specifications, modeling diagrams, and user manuals. Frequently, the software interfaces may not be adequately documented. For example, some of the sought-after functionality may be accessible only through the user interface. Other key design lessons may be undocumented. It is useful to capture this design information, too.

In the second step, the mined designs are generalized to create a common interface specification. This step entails more art than science, more architectural intuition than meticulous engineering. The goal is to create an initial strawman specification for the target architectural interfaces. It is usually not sufficient to generate a lowest-common-denominator design from the representative technology. The generalized interfaces should resemble a best-of-breed solution that captures the common functionality as well as some unique aspects inspired by particular systems. Unique aspects should be included when they create valuable features in the target architecture or represent areas of known system evolution. A robust assortment of representative technologies will contain indicators of likely areas of target system evolution.

At this point, it is appropriate to factor in the top-down design information as one of the inputs. Top-down information is usually at a much higher level of abstraction than bottom-up information. Reconciliation of these differences involves some important architectural tradeoffs.

The final step in the mining process is to refine the design. Refinements can be driven by the architect's judgment, informal walkthroughs, review processes, new requirements, or additional mining studies.

Applicability of Mining

Mining can be a fast, inexpensive process that yields significant benefits in risk reduction and architectural quality. The real product of mining is the edification of the software architect. With a mature understanding of the problem and previous solutions, the software architect is well prepared to make good architectural decisions.

Other mining artifacts include the OMG IDL interface models, but they should not be treated as formal deliverables. These artifacts are simply design notes used in the creative process that produce good architecture.

Mining should be done for most high-quality reusable designs. It is not necessary to do mining for all designs in a system, especially the ones that have an impact on only a small number of developers or subsystems. It is appropriate to consider mining for high-risk or widely used interfaces that have an impact on significant aspects of the system or enterprise.

Given the right documentation and access to expertise, architectural mining can be done very rapidly. In fact, most mining studies can be completed within a few days for each representative technology. After several mining studies, it is possible to undertake significant designs with confidence.

Mining for Success

How does a software architect gain sufficient knowledge to design and defend a good architecture? Knowledge can come from years of experience of designing similar architectures. Alternatively, the learning process can be greatly accelerated by explicit mining of design knowledge from existing technologies and experts.

Most software architectures are designed in a vacuum. It is easy to ignore or reject preexisting designs when confronted by a new design problem, but there are serious consequences. "Design-in-a-vacuum" invariably produces immature, custom designs with minimal potential for reuse, interoperability, and adaptability. Because technology transfer between multiple systems rarely occurs in practice, the positive effects of software architectural mining can be quite dramatic.

Horizontal Versus Vertical

Understanding the subtle differences between horizontal and vertical design elements is essential, especially with respect to aspects of a software interface design at a system level. At this scale of design, software architects are interested in managing complexity and change effectively. Their designs must be flexible, but simple and reusable, if at all possible.

An important goal of architectural design is to have a well-thought-out balance between horizontal and vertical elements. Horizontal and vertical design elements often confuse people because these essential design extremes are unfamiliar concepts to most programmers. Generally, these extremes refer to a continuum of design choices that vary in flexibility and reusability (on the horizontal extreme) with its ability to solve the point-design solution (on the vertical extreme).

Vertical design refers to designs that are unique to one software implementation, unique to one system, or unique to one set of application requirements. Vertical designs are the norm. It is likely that most designs (with the exception of vendor APIs) are decidedly vertical. And people wonder why software reuse is so difficult to achieve in practice?

What makes a design vertical is the presence of specialized details that are hard-coded into the solution. It is well known that attribute names and schemas are application specific and vary over time in an application-dependent way. When these attributes are hard-coded into the system APIs, a vertical API exists. Other examples include APIs that specify very specialized functions or contain uniquely constrained sets of parameters.

Programmers favor vertical designs because they resolve the current design problem in an obvious way. There are certain attributes and operations to implement, and they are coded in a straightforward and direct way. Why should it be done any differently?

Architects are concerned with additional design forces that have an impact over a longer term. Experience teaches that requirements change frequently and that certain ways of designing can support change more flexibly than other ways of designing. A rational approach to change management that lists the likely sources of change and their design impacts could be undertaken. One such approach is called the Software Architecture Assessment Method [Bass 1998]. These methods address the coarse-grained issues of change. But architects can make hundreds of fine-grained decisions that accommodate for change, as they proceed with design. The rationale for these fine-grained decisions could be called architectural intuition (or the art of building architecture). However, the quest for flexibility must be balanced with practicality.

Can a Design Be Too Flexible? In a nutshell, YES. It's easy to make designs that are too flexible for their own good. Proper architecture building is all about common sense and balanced design. When designing system-level interfaces, software architects certainly don't want to hard-code vertical design details that are expected to change overnight. On the other hand, the design should not be so flexible that it has flexibility disease. The potential consequences of making a design too flexible include: Inefficiency. Highly flexible designs require extra runtime processing on both sides of an interface. For example, parameter encodings for flexibility may require application-programmed translations from native types to dynamic self-identifying types. In a distributed system, additional marshaling time may be required for dynamic parameters. These inefficiencies can lengthen interface processing latencies by two orders of magnitude or more. The most frequent complaint that an architect will encounter when attempting to insert some architectural qualities into a design will be inefficiency. In most cases, this argument should be resisted. Overoptimization is a source of many unnecessary architectural compromises. Lack of understandability. When the designs are too flexible, developers won't understand the flexible features. Some developers will mention this bias directly; others will ask questions that are difficult to answer. But the most dangerous response is when developers don't understand, yet go ahead and make assumptions and use the features in unintended ways. Lack of understandability is a primary limiting force on design flexibility. Designs should be made flexible, but only up to the practical limit of how easy to understand the designs will be for the team's developers. That includes the likelihood of developers misusing the designs because they do not understand them (which is also a failure to communicate on the part of the architect). Extra coding. Flexible designs require extra software on both sides of the flexible interface. This extra code often hard-codes exactly what was supposed to be flexible in the design. For example, passing a set of dynamic attributes could be hard-coded as a fixed set of attributes for each usage of the interface. This situation is normal because some part of the implementation must do data-handling operations; hard-coding is the most direct way to do it. The benefit of flexibility is that the attribute set is not hard-coded into the architecture, and attribute sets may vary without architectural modifications. In addition, the extra code from hard-coding may introduce additional errors, require additional testing, and incur increased maintenance costs. Extra documented conventions. A key price of flexibility is the need to constrain usage through conventions. Without usage constraints, achieving interoperability between implementations is often impossible, no matter how carefully specified the interface is. A design may be too flexible when the usage conventions become cumbersome or may even outweigh the original specification. There is an important design balance in trading off implementation details between hard-coded architecture and usage conventions.

A vertical design often has many potentially negative consequences. Vertical elements are tied to application-specific requirements, which are always subject to change or reinterpretation. Vertical designs are unique to one implementation and embody the antithesis of reusability. Vertical designs are often complex, containing many application-specific details. In that sense, it is difficult to manage complexity effectively in a vertical design.

Horizontal Design Elements

Horizontal design represents the common requirements from more than one application. One way to describe horizontal design is that it is the design that remains after the vertical elements are removed or refactored. The elimination of the vertical design elements is an explicit intellectual exercise practiced by software architects. Start with a design and remove redundant vertical elements or refactor them into horizontal elements. The remaining design contains elements that are flexible, used internally in different configurations and often addressing the needs of multiple applications; hence, it is horizontal by definition.

One formalized approach to vertical design elimination is called domain engineering. Domain engineering is the systematic management of reusability. It starts with domain analysis. Typically, domain analysis begins with a set of application requirements [Rogers 1997]. The requirements are sorted into groups representing various application functions. Functions that are judged more horizontal (e.g., common to multiple applications) are selected for rewriting. These requirements are rewritten iteratively to remove application specificity, while still retaining their domain functionality. The rewritten requirements are checked with domain experts to ensure that they retain their value to the domain.

Software design begins after the requirements are domain analyzed. The horizontal requirements are used to define software interfaces in an application-independent manner. Several software authorities claim that domain analysis helps programmers to design much higher quality interfaces [Rogers 1997; Coplien 1999]-in particular, interfaces that are much more adaptable and reusable, and subjectively better structured.

This observation can be attributed much more to the experiential process than to the artifacts. The process of domain analysis not only yields the deliverable of horizontal requirements, it also gives the software designer new insight and perspective about the design problem. This new knowledge is an invaluable resource for creating high-quality designs.

It is important to note that domain analysis is an exercise that helps designers differentiate between horizontal and vertical design elements in a given problem domain. Another such exercise is architectural mining. In both cases, the knowledge gained by the analyst is more important to the design process than the artifacts (e.g., documents) generated by the domain analysis or architectural mining process.

Good horizontal designs meet the requirements of multiple applications. Logically that should be very hard to achieve in practice. However, in a perverse sense, horizontal design is easier to do than vertical design. In a famous result from a mathematics educator, George Polya claims that solving more difficult, generalized problems is often easier than solving specific problems (i.e., solving vertical specializations) [Polya 1971]. In other words, solving a more general (harder) problem is often easier in practice. This result is called Polya's Paradox. This result applies to software design, as well.

One reason why Polya's Paradox works is that specialized problems are often overwhelmed by details. Because a specialized problem refers to one concrete instance, as much detail as desired can be extracted from the real-world situation. This excess information itself becomes a problem for the designer. In the more generalized problem, we are freed from addressing the details of any single situation, except as an example from which we can easily discriminate the relevant and irrelevant details.

In the abstract world of the generalized problem, we can define the solution structure in a most advantageous way. This new solution structure is relatively easy to formulate at the abstract level, unburdened by details. When this solution is applied to specific cases, the generalized solution defines the underlying principles for the specialized solution. Any similar specific problem can be resolved with these same principles. In effect, this solution is reusable.

In software, the situation is often not so clear-cut. It is possible to have mixtures of horizontal and vertical solutions. Whether there is a good or bad structure depends upon how these elements are intertwined. For example, the vertical qualities of an API are not diminished if it contains intermixed horizontal design elements. In fact, intermixing of vertical and horizontal design elements is the norm, and the design problem for the architect or programmer is to separate these elements from each other. If horizontal design elements are properly separated (e.g., in a separate interface), then it is possible to intermix these design elements (e.g., through inheritance) in a controlled way that does not compromise the inherent advantages of separation.

In many instances, the differentiation between what is vertical and what is horizontal is an intuitive, subjective judgment. We believe that the ability to distinguish between these extremes is an important and essential ability of software architects. An important design choice (e.g., between horizontal and vertical) for an architect may be invisible or unimportant to programmers on the project. And that's okay. That is one of the reasons why software architects are different from programmers. The ability to see these distinctions clearly and to know why they are important is an important indicator of the architect's instinct.

What About Traceability?

Internal and external designs are different views of the system. Traceability for vertical designs is relatively easy to prove because the details correlate closely with external requirements. As a design is refactored into its horizontal form, the traceability becomes less obvious. A horizontal design (of an interface architecture) represents the internal structure of a software system. The internal structure is different and separate from the external requirements. The relationship between the two can be shown only indirectly, at best. When a design comprises horizontal elements, the dichotomy between internal and external views can be extreme.

One approach to show how the internal design supports the external requirements is through scenarios. Each scenario is the performance of an externally significant function, expressed as a thread of execution through multiple layers of the system. In effect, the internal design is exercised in direct response to the execution of an externally meaningful function. Typically, each horizontal design element will be involved in multiple scenarios, indicating that the design is traceable to many external requirements, but not any individual requirement that solely justifies its design.

Designing for Future Applications

Horizontal designs resolve problems across a range of applications. For a horizontal design to be effective, it must meet the potentially conflicting requirements of several independent applications. The horizontal design must satisfy not only current application needs but also future application needs whose requirements have yet to be specified. But how can we address future, unknown requirements?

The apparent ability to predict future features of systems is a strength of the architectural mining approach. This phenomenon is easiest to understand in terms of commercial software, given that in any given commercial software market there are a number of competing products with differentiated features. For example, the software market includes word processing and geographic information systems (GIS).

There is a common functionality across all products in the market, but there are also key differences that make each competitive. For example, one word processor has great layout abilities, and another has great graphics extensions. As the market evolves, each product will tend to be extended in ways that have proven successful in competing product lines.

This same phenomenon of product differentiation occurs much more dramatically in other situations. For example, in geographic information systems, the product differentiation is so successful that most large companies need multiple vendors' GIS systems to meet their needs.

Because information about commercial products is readily available, it is difficult for competitors to hide anything significant about their products. Competitors monitor each other's products and customers to keep in touch with current conditions and future market directions. With custom in-house software systems, information is less readily available.

In our experience with architectural mining, every legacy system implemented some unique and advanced capability that surprised us in comparison with other known systems. Legacy systems within the same functional area have unique capabilities (that was probably why they were developed in the first place), and all these capabilities are quite different. It was clear that if the legacy owners were aware of all these capabilities, then they would want a new system that embodied most of them. In effect, a study of legacy capabilities pointed to the future best-of-breed for custom systems.

In a limited sense, a future software system is a best-of-breed display of successful features in current software systems. Of course, untested features supporting novel requirements that may or may not become successful can be factored in. One can predict, with some success, how future system features will evolve, given a study of system differences today and how well these product differences are thriving in today's market.