By Richard P. Gabriel

Publication Date: February 3, 1997 (Vol. 19, Issue 5)

Most executives I know have no idea how software is made: They labor under a delusion fueled by abstractions and wishful thinking, fanned by well-meaning but clueless academics. The delusion: Architecture precedes design, design precedes coding, coding precedes testing, and testing precedes maintenance, and it all marches or could march in a well-planned and perfectly budgeted order from conception through delivery on the day mandated by CEOs, boards, investors, or marketing vice presidents. I'm sorry, but the truth is that today the production of software is barely controlled chaos bred from extraordinary complexity, human emotions and frailties, sociological miracles and disasters, and forces acting upon forces and semimovable objects. Software physics more resembles the subatomic, quantum variety than the macroatomic, Newtonian one.

Creating a complex software system or application is more like planning a picnic-heavy vacation in the White Mountains of New Hampshire (where forecasters cannot predict the weather an hour in advance) than one in the coastal range of California (where the weather is boringly predictable).

Similarly, in software development, assumptions and requirements change when technology is not up to the task and when competitors and marketing goals change; and designs and even architectures change when obstacles impose, human creativity fails, and current performance and results are disappointing. As with any complex, chaotic system, insignificant effects in one place can billow into insurmountable forces later on and in other places.

Among the most important factors that influence how well a system is developed are the talents and work habits of the developers, how well the members of the software group come to work together as a team, the faith of the developers in the competence of upper management, the health and well-being of individuals, the size and privacy of offices, and how much natural light is available. One could ask: What about the rigorousness of the software processes, the number of hours the developers are willing to work, the precision of the requirements, and skill of the managers? These have some bearing but not that much. Software development is chaotic because the context of requirements changes as more is learned about the system being built, as the competitive environment changes, and as the dynamics of individual psychology and group sociology play out day to day in a hotbed of development. While observing a project in progress, we are able to recognize general structures -- such as design, implementation, and coding -- just as we observe high- and low-pressure areas, clouds, and squall lines. But, as with the weather systems, recognizing is not the same as predicting, nor is it the same as generating.

WHERE SOFTWARE MEETS ARCHITECTURE. One of the people whose work has begun to help us understand software development is the architect Christopher Alexander. Alexander has strived to capture knowledge about how to design and build houses, offices, neighborhoods, towns, and cities -- that is, anything in the so-called built environment. His emphasis has been on how to make such places alive and beautiful the way they can be and once were, but Alexander's philosophy of how things should be built fits the software world particularly well. In fact, during the past decade, software engineers have attempted to incorporate Alexander's ideas about architecture into software design.

Alexander espouses the idea that the ultimate goal of design and building is to create habitable places and that all building is ultimately piecemeal in nature. Habitability in buildings is the characteristic of serving well and transparently how people live. For example, a room with alcoves provides a way for family groups to have individual space for projects and activities while maintaining familiar closeness. Habitability in software can mean two things: A piece of software, such as a Web browser, is a place where individuals work for long periods, or it can be a place where software developers themselves -- the people who most truly live in the software -- work and alter and maintain the code.

Piecemeal growth in the built world is the process of evolving, completing, and changing rooms, buildings, towns, and cities as the needs of the inhabitants change over time. An easy to recognize example is the New England farmhouse. It starts as a small home with a barn out back. As the family grows and the needs of the farm grow, a back room is added to the house, then a canning room, then a room for Grandma; stables are added to the barn, then a wing for milking more cows. Finally, the house and barn are connected because it is too difficult to get from the house to the barn in a blizzard.

The result is rambling, but each part is well-suited to the family's needs and fits well with the other parts. The result is beautiful because it is a living structure with living people inside. The inhabitants are able to modify their environment because each part is built according to familiar patterns of design, use, and construction and because those patterns contain the seeds for piecemeal growth.

When software is developed, it goes through an inevitable process of piecemeal growth: A typical software system is in use for decades throughout numerous changes and improvements.

Microsoft Word is currently in its sixth major release, each one noticeably different and improved over the previous one; Unix has been around for more than 20 years; and any software project worth more than one or two developers has a hoped-for life span of a number of releases. Further, the developers who work on a software system will inevitably change over time as they move on to other projects or organizations and new developers join.

Because software systems must evolve through piecemeal growth, they must be habitable to those who will change them -- who will inhabit them. Further, because software development is inherently chaotic, it cannot be thoroughly planned. That is, it is not possible to come up with a master plan.

In building homes and office buildings, though, it is possible to come up with a master plan, but Alexander recommends against it. He points out that there are several problems with master plans. First, all building is eventually piecemeal, and it is not possible to foresee enough of the future and future uses of a building to be able to predict how it should evolve. Second, the inhabitants of the place constructed with a master plan are demoralized because they cannot have much, if any, impact on the nature of their space over time. Finally, it simply is impossible to visualize the results of the master plan.

In software, the closest approximation to Alexander's notion of a master plan is the development plan for the life cycle of a software system.

However, because software systems are so complex, the concept applies to the stages in the development of a single release. Of the three problems with master plans, the one that most applies to single releases is the loss of involvement that developers feel if they are required to follow instructions and merely code. Because programming largely requires creative and talented people, making them cogs wastes resources and results in generally poorer systems in the end.

NOT ENGINEERING. Executives find the chaotic nature of the development process discouraging and evidence of the incompetence of development organizations. At worst, it is evidence for the immaturity of software engineering.

Executives wish that software development were like bridge building, where there is a well-defined process of design and construction. But there are crucial differences between bridge building and software development. Let's consider just one aspect of the differences between true engineering disciplines and so-called software engineering. If someone were to ask you about building a bridge, you would not only have some knowledge of successful bridge designs from your experience, but you could drive around and examine bridges, some older than two millennia. On the other hand, it is extremely rare to run into a software developer who has even glanced at the source code for a system that I have seen the source for. In fact, I have yet to find two people who have never worked together who agree that a particular system is beautifully designed and constructed.

In schools there is no such thing as studying the successful designs and implementations of large systems. And there is familiarity only with the outermost user interfaces of systems and their general behavioral characteristics. How can an engineering discipline have been built up from such isolated and generally unknown examples?

We have not begun the work of understanding the forces that go into the construction of good software systems. We have not even begun to build our set of good examples.

PATTERNS. But all is not lost. Software is successfully completed, and companies can make money by writing software. Patterns and pattern languages can form a means to help with the problems in the software process we have been examining.

A pattern is a description of the solution to a problem in a particular context of forces that needs to be resolved. It is easier to explain this by using a pattern from architecture described earlier.

Here is how Alexander describes what to do: "Make small places at the edge of any common room, usually no more than 6 feet wide and 3 to 6 feet deep and possibly much smaller. These alcoves should be large enough for two people to sit, chat, or play and sometimes large enough to contain a desk or table. ... Give the alcove a ceiling which is markedly lower than the ceiling height in the main room."

This is a pattern from Alexander's landmark work, A Pattern Language (Oxford University Press). Notice that it does not specify exactly how to build an alcove. And the point is to describe the forces involved in the problem itself; generally, any problem in design requires balancing opposing or contradictory forces, and the same is true in software.

In software, patterns are more to the point in some ways than patterns such as alcoves. Many software patterns talk about common design problems such as how to design a constellation of objects to cooperate to achieve some goal in the presence of other considerations such as performance or size.

In architecture and in software, groups of patterns together form a pattern language which can help a designer move from the grossest level of design (architecture, for instance) down to actual construction or coding details.

Because patterns and pattern languages can help a developer and development group move freely between design and implementation and reduce the degree of creativity required to complete a task, they are a means to tame some of the confusion of dealing with a chaotic system. Patterns and pattern languages do not represent someone's new ideas of how to solve problems but are descriptions of solutions that have proven successful in a number of systems.

Patterns do not represent laws or principles but practical advice about how to balance forces so as to work one's way out of difficult design and implementation situations. In architecture, patterns represent thousands of years of experience. In software, as noted, we have much less experience, and those experiences are isolated. Only through the test publication of patterns and pattern languages will the experiences of many development teams be captured.

On the other hand, until the substance of software development rises above the level of today's programming languages, there will never be a leap in software development productivity. (Programming teams vary in productivity by several orders of magnitude depending on the expertise of the team members and the familiarity of the software being developed; I am referring to increases in the baseline productivity.)

Therefore, regardless of how thoroughly planned the project may be or how stringent the processes in place, you can be certain that the actual development proceeds through a series of essentially unpredictable forward and backward steps and that the primary activities would be more interesting to a sociologist than to a computer scientist. Just as our modern understanding of some of the behavior of weather enables us to cope with the chaos that dominates our environment, so patterns are a means to at least make our way less serpentinely through the development jungle.

Richard P. Gabriel is a software pioneer, computer scientist, consulting professor of computer science at Stanford University, and the author of Patterns of Software, Tales from the Software Community (Oxford University Press).

Christopher Alexander is a practicing architect and builder and a professor of architecture at the University of California, Berkeley. He has written a series of books that offer a theory of architecture, building, and planning.

Software engineers, many involved in object-oriented programming, have been following Alexander's work for over a decade, applying his architectural theories to their own field.

To learn more about his theories, you can turn to his three-book series: The Timeless Way of Building, A Pattern Language, and The Oregon Experiment (all from Oxford University Press).

Pattern study and resources

Pattern languages in software are proving to be a mechanism for improving productivity and communication, for understanding and improving today's inherently chaotic software development process, and for educating developers for the future.

There are patterns and pattern languages for organizing software development teams so that even the chaos of development can be at least understood, if not partially tamed. The best of these languages is presented in the chapter "A Developmental Process Generative Pattern Language" from a book by James O. Coplien, Pattern Languages of Program Design (Addison-Wesley).

Patterns and pattern languages in software are in their infancy, and only a handful of small pattern languages are available.

Yet some commentators have remarked that using patterns provides the first reliable increase in productivity they have seen, and groups that use pattern languages are able to communicate much more efficiently, thus improving productivity.

Today's pattern languages complement the most recent great hope for software: object-oriented programming. Most of the patterns written today are configurations of communicating objects to solve particular problems, though there is nothing about the concept of patterns that requires the linkage.

Patterns and pattern languages are growing in popularity. The Hillside Group is a nonprofit organization that sponsors two annual conferences -- one in the United States and one in Europe -- where pattern writers exchange and critique patterns and pattern languages. These conferences each year publish a book called Pattern Languages of Program Design with the best patterns and pattern languages; three have been published so far by Addison-Wesley.

The Hillside Group sponsors a number of conferences and workshops, typically in specialized domain areas such as telephony.

In addition, there are a number of books published that present patterns or patterns languages. The most popular one is Design Patterns: Elements of Reusable Object-Oriented Software, by Gamma et al. (Addison-Wesley).