Some people say that a software system architecture can be an emergent property of an appropriate development process.
The term 'emergent' has a specific meaning with respect to chaos theory. A structure, in the past, was always assumed to be imposed by some direct structuring force. This is known as skeletal or imposed structure. This would be an architecture that started perfect and stayed that way.
In this context, emergent means emerging from the system rather than being imposed on the system. Take the example of a river's course. On the one hand the course of the river structures the flow of water. Yet the flow of water is also responsible for defining the course of the river.
Or, to take another example, Prigogyne described a number of chemical reactions where the structures seen were emergent, and related to macro-patterns (order 1cm) described by non-linear micro-interactions (order molecular). The structure was an emergent property because it emerged out of chaos, it was a structure made of interacting dynamics of varying stability.
Now I like the sound of that. I don't think a single skeleton allows you to evolve and thereby adapt to changing requirements. An emergent architecture though, that sounds like refactoring to me. Xp Is Anarchy .
For some of us, the word "architecture" on this page means macro-scale structure, and the question we're asking is whether these structures can be predicted. So the question becomes "Can the resulting architecture of a software product arise without our having been able to predict its form from the start?" And perhaps this is how all new patterns -- including architectural ones -- appear.
You could frame the question in yet another way: "Is it possible to end up with a clear high level design from concentrating on resolving tensions at a detailed level?". Yes, it can, as long as tension resolution is carried on up. When it works, the smaller elements have exerted their collective will over the big picture. That's what we mean by "emergent".
This is not an argument against a reasonable amount of forethought; you always try to plan the obvious. And emergent effects and planned effects can happen comfortably within the same process, and often do.
The question of this page matters because for any planning exercise, there is always an incomplete but reasonable scope. Unless you're God, you can't plan in the scope that includes all scopes, including itself.
Is there an Architect for the Barn in an Amish Barn Raising?
The architect for an Amish Barn Raising (whatever that is!) has long since died and been buried. That they can raise the barn so efficiently is a great testament to his success! You keep confusing team-work with architecture. Who told you these were the same thing? The real question should be who was the architect of the original pattern that allowed the Amish to create barns in very predictable, collaborative, and communal ways. If anything, this is another argument for an architect. Can you see why this is such a successful architecture? -- Robert Di Falco
again I would need evidence for that, I think its much more likely that someone once built something not entirely unlike a modern Amish barn and it has been changed incrementally to its new form. -- ta
I don't think anything I've said asserts architectures don't evolve. In fact, I think I assert that it does. -- rd
About the Amish barn - interesting question! I believe the architecture of these barns (the basic structure used to defy gravity) is a work of folk art, but at a given barn raising, you had better have at least one person who has done this before and knows how to do it. That's your chief builder, that's your 'architect'! Bing bang boom and a booga-rooga, Houston we have lift-off. Man it feels so good to use the word right. -- Walden Mathews
I think the fallacy is to believe that because there is an architecture there must have been one architect. Most things with an architecture have an architect, it just isn't an absolute. -- Tom Ayerst
I think you are missing a temporal property. An architecture persists for all time while an architect is unimportant after they have served their initial purpose. For something that is very old, there is no reason to assume that the current process doesn't use an architect that one did not exist during its first couple of transitions. You keep on leaning towards examples that would be more like integration in software development - i.e. an already created system that we simply erect like leggo blocks. However, that is not the situation we are in. So, to refactor your assertion I would say that just because there is an architecture does not mean that the architect is still around. You also seem to be implying that in our model there can only be one architect. However, if you go to Fred Brooks statements on Conceptual Integrity, you can see that there is nothing that prevents multiple architects, maybe up to 2 or 3. The goal, however, is just not to have everyone be the architect. -- Robert Di Falco
On the basis of my casual observations I currently believe that Architectures generally 'emerge' over time; seldom do they "spring fully armed from the head of Zeus". At least that is the impression I have gained from a general awareness of the Art. I do acknowledge, however, that as in all forms of Art, there appear to be inspired revolutionaries who seem to instantaneously and single-handedly, take the Art to new levels. Since I believe that we all stand on the shoulders of those that preceded us, I am inclined to support Robert's observation that, even in the case of an apparent single architect, there is probably a team in the background. One tangent that interests me is an old observation that a significant difference that could be explored in a page such as Architects Compared To Systems Architects, is that Architects, as a group, seem to have a much better understanding of their Strength of Materials, than do systems architects. -- Hans Wobbe
I think a more correct statement than Can An Architecture Emerge? is how much will the architecture change? Every piece of software has some type of architecture and any piece of software will have its architecture change over its lifetime. The question comes down to how do we best guide the changes in architecture, with some overall prediction of what will be needed or through a series of just in time changes? -- Wayne Mack
Once you've produced code to solve a problem, the structure of that code is just information. There are only a few sources that information could have come from:
Up-Front Design (from BDUF to Metaphor)
The Problem Itself
Noise (everything else)
Small refactoring moves aren't adding any information: They're just rearranging existing information to be more compact, or occasionally throwing some information away. The only way for information to get added to your system is when you add a feature: "When this part of the problem is like this, that part of the solution should be like that."
When is it possible to do a good Big Design Up Front and stick to it? When we know all the information about the problem itself that we need to structure the solution -- i.e., the final code.
Is it possible to do no up-front design and still reach a good solution? We're often asked to start coding without knowing everything about the problem, after all.
We might have something to worry about if there were a trap that your small-scale refactoring moves fell into. But do we know of any refactoring traps?
If the noise in your system -- differences in code formatting, disagreements over duplication, etc. -- are big enough, they could drown out the information from the problem itself. So then you might never reach a working solution, or might progress too slowly to make it worthwhile.
In summary, you could worry about a development process that depends on refactoring getting stuck, or letting the noise take over. But you don't have to worry about anything else.
Do we even have a clear-cut "problem" to solve? What we are doing is automating human processes, and we provide progressively better approximations of those processes. This implies you will have emergent design whether you start out with an up-front design or not.
Also, you're missing a critically important source of information--the solution. Emergent architecture relies on looking at a solution with a poor architecture and making a better architecture. By breaking this cycle into minutes and hours and days instead years and decades and never as in Code And Fix, you can quickly converge on a good architecture without the risks of speculative design. When you get really good at it, you go so fast that people accuse you of thinking ahead even when you don't. -- Kent Beck
Many of the people who believe that an architecture can emerge also happened to be Extreme Programming proponents, and they believe XP is a Programming Value System that allows this emergence to happen.
The party line is that you have to start with a System Metaphor, which constitutes at least a logical architecture, then Model First, then explore concerns and tool choices with one or more Spike Solutions, and make contact between developers and users via a Spartan User Interface. All of these are explicitly architectural activities. They can be conducted iteratively, but enough must be done up front to get to an iterative mode in the first place. It's Big Design Up Front, not Adequate Architecture Up Front that gets the flick. -- Peter Merel
"All of these are explicitly architectural activities." "architectural" in the process sense, certainly, but not in the macro-structural sense, right? -- Francis Hwang
Good architectures do not usually emerge. This is obvious by looking at legacy systems that are doing most of the work in enterprises today. No matter how much work has been done on some of these systems, they are still virtually unmaintainable, and not extendable. I think a better way to frame it would be that good architectures evolve (through careful attention to points of elegant simplicity in the design space) and that architectures tend to converge as in biological systems, i.e. given the same environmental constraints for organisms evolving through many different pathways, you tend to get a similar result. This evolution and convergent was one of the goals of the early SCRUM process. Extreme Programming has, as a principle, careful attention to points of elegant simplicity in the design space, so XP may make the case that constraints implicit in the XP process tend to evolve good architecture. In this sense, XP architecture is emergent through constrained evolution, a concept that the naive reader will certainly miss on first reading. -- Jeff Sutherland
But evolution is an emergent phenomenon, isn't it?
I can see how refactoring can decompose a system into smaller and smaller pieces, what I don't understand is what forces are at work to combine these pieces into meaningful groups. That is what I expect an architecture to convey, a set of generalized, higher level constructs. Any suggestions on what I may be missing or misunderstanding?
Macro-scale structures can form through emergence as small code changes and refactorings aggregate related functionalities.
What you may be missing is that although called 'factoring' (i.e., literally breaking down into smaller pieces), the term is commonly used to mean both decomposition and composition back into dense but loosely coupled units. The force that usually prevails in judgments of what's 'dense' and what's 'loosely coupled' is the force of anticipated change (no, Existing Solutions Drive Architectural Emergence. So, if you stand back and watch, you will find emergence happening during refactoring, not at some later time. The discovery that this thing and that thing are alike, and likely to always be alike, is emergence in progress. Hope this helps. -- Walden Mathews
I do know little about XP, but I think that the refactoring is similar to a process of Simulated Annealing. This works like a recrystallization process that centers around thinking & words & objects & language. If you repeat this process, certain structures of order emerge because they represent states of lower energy. IIRC such phenomena are quite common in thermodynamics. -- Helmut Leitner
Is "architecture" that which remains when the implementation is done? Discussion moved to Architecture Definitions.
I've been working on a small program ( which measures the abstractness and instability of Java packages. see: www.objectmentor.com 
), using Test First and Refactor Mercilessly, and I am watching an architecture emerge. The program has been passing it's test for a few days now, and I have been refactoring since. Very soon now, I will have spent more time refactoring the program than I did writing it. -- Erik Meade
The idea of an emerging architecture that is not known before hand may work okay on small business-based systems, but it does not scale. It particularly does not scale for frameworks, foundations, and pluggable architectures. This doesn't mean it isn't great for smaller systems, but, just look at the inconsistencies, Pattern Chaos, and the Amorphous Blob Of Human Insensitivity that the various Java API's (and their classes) have emerged into. A little planning, design overview, Conceptual Integrity, and consistency rules (for patterns and naming) would have gone a long way. The larger a system grows, the more dangerous it is to simply think an architecture will emerge all nice, mature, and happy like. -- Robert Di Falco
I'll assert the opposite. The larger the scale, the more you must rely on emergence. The Java APIs are an excellent example of very smart people designing their butts off and achieving crap. Nobody said emergence was the same as the path of least resistance. It takes short cycles, concrete feedback, and social networks (basically what you said). -- Kent Beck
Okay, to clarify, I think you are asserting that they didn't emerge but that they were, indeed, designed upfront and quantified where as I understood that they were pretty much just programmed, tested, found to be useful, and so polished up, tested some more and released -- i.e. no overview constraints. You could be right, I wont dispute that but on another point, I wonder if (at least sometimes [is that a compromise!?]) that using only the (definitely great) techniques of refactoring and emergent design can fail for large resusable systems with a wide audience. Especially, where independent systems (parts) can be taken as a whole. The problem is that once one library (i.e. part) is released it can no longer emerge without impacting other libraries (i.e. sibling parts) or their clients. Worse yet, and I'm pretty sure we agree on this, if the various library teams don't augment intra-team bandwidth with 'interteam bandwidth, with no team overlooking the Conceptual Integrity of the overall vision, we end up with each library taking it's own path. This has the usual result of chaos and inconsistency throughout the overall system no matter how absolutely cool each individual part may be. -- Robert Di Falco
As far as when you "rely" on emergence, it seems to matter more what your sphere of influence is than how big the final product will be. When systems that weren't designed to be integrated with each other suddenly get integrated, the patterns of interaction are probably emergent. And the work to finish integrating them is based on those patterns, and sort of quasi emergent because there is both the unexpected part and our planned response. I think this may really be what this page is about, unexpected juxtapositions that we have to make work, and the architectures that result.
If you're talking about architecture as Conceptual Integrity, emergence may not be so easy. If the major System Metaphor's change, then you either chose an inappropriate metaphor or chose too early.
Most effective development efforts are initiated by a small group (often one person) who has the vision, and the ability to instigate that vision with an architecture. Perhaps at some point the group gets too big, and an architect (or architects) is needed to maintain Conceptual Integrity. As to where that line might be, perhaps it varies from project to project.
However, even in small teams, the members will not share all work equally. Each will organically fall into roles they perform a little more than there team mates. Someone will organically rise to the task of dealing with Conceptual Integrity issues more than the others. Another member will rise to the role of dealing with coding standards and project management more than the others. The only difference from a large team is that they may not explicitly recognize, name, or assign these roles on a small team -- the roles happen spontaneously.
Chaos is easy, no question about that. The hard part is maintaining wholeness at larger scales. As you increase the scale, centralized control of Conceptual Integrity definitely falls apart, and you must either rely on some other method or accept that chaos is the best we can do. Open Source is a community that creates wholeness at very large scales. XP strives to create wholeness under high stress situations, which is a different kind of scaling.
Collective Code Ownership isn't done just for the sake of integration and refactoring, but as a design strategy as well. For a non-trivial project, no one person can hold all the design decisions in his head. At some point the team will be making decision decisions, like it or not.
Architectures are always emergent unless the system is a toy. Specifically, a designed architecture would need to provide for all possible constraints for it to remain stable throughout the system lifecycle. But we know requirements always change, right?
Consider termites building a termite colony. My guess is that this works because all termites use the same heuristics to decide what to do. Assuming that's true, the implication is that for a decent architecture to emerge, programmers would similarly have to operate to the same algorithm. That makes the XP metaphor, and the Once And Only Once rule, and coding stands and other heuristics all that more important.
If the people working on the program have different ideas of what is good, then the resulting program will not evolve to a program with a good structure. For a good structure to show up, the programmers need to be making changes to the program that follow a common base algorithm, or they are following a common theory in development. Then all their changes might contribute to an emerging structural pattern. Conceivably, the algorithm they follow is different in different parts of the program. Even with heavy thinking along the way as programmers make local changes, if their operating principles / theories / algorithms / heuristic are different, perhaps the structure won't show up.
Suppose Pat thinks copy/paste is a great idea, and Chris thinks OAOO is a great idea (I worked with a senior tech lead like Pat). Good structure will not show up. Suppose Pat likes to hide business logic away from GUI classes, but Chris disagrees. Good structure will not show up, whether they refactor each other's code or not. They must share a common heuristic of what "better" means, and probably a lot more than that. -- Alistair Cockburn
I don't believe you can posit emergence on a requirement of common thought patterns, especially since conscious intent is at odds with emergence, but to the detached observer watching a pattern emerge from chaos, common behaviors would likely be evident (in addition to common structure in artifacts), although they (the behaviors) might be part of what emerged, not something the system started with. Think "bootstrapping". -- Walden Mathews
Associations between uniformity of workforce and structure of product might be too obvious for their own good, Alistair, meaning that such correlations don't actually exist. I wonder about questions such as:
Is uniformity of workforce a requirement for emergent structure (architecture or whatever)?
Does uniformity of workforce imply the emergence of structure?
Does emergence of structure depend on there being a workforce as opposed to a single worker?
Specific to termites, can we predict what would happen if we introduced a small number of rogue termites into the mix? Can we say whether that would sharpen or blur the colony design?
Still specific to termites, is the workforce really uniform, or are the males a kind of rogue, and is the queen perhaps another kind of rogue?
My intuition is that workforce uniformity and resulting structures are mysteriously, almost magically decoupled, so that the expected results would not be the actual results. Isn't the nature of emergence that it defies predictions such as these?
When structure happens predictably, it's not emergence. When you design for structure, like attacking a problem using only a uniform workforce, and arming that workforce with conscious design tools and models, you are making structure happen, not letting it emerge. By definition, emergence is slippery: defining its causation makes it go away.
Hmmm. Are we talking about working structures? As in the grouping of programmers? Or are we talking about the structures they produce as code? Or both? I think the former is interesting in that the slightest non-uniformity of people results in very heavy patterning. The butterfly effect I guess, but with nasty black-hole attractors when the system gets out of whack. This implies to me that you need a queen-bee to guide the direction and cohesion of effort, effectively damping the feedback to keep the system on track. More extreme intervention may involve imposing a fixed structure, but that sucks for all the reasons that fixed structures suck.
The latter obeys the river simile. What does not vary becomes part of the framework.
Another possible interpretation of the title question: Once a system has become a Big Ball Of Mud, is there any way for it to become otherwise, or is that another word for scrapping the current system and starting over (with the knowledge gained) to build again, but this time with structure?
Where does the "big boned" structure of a product come from? Can it percolate up from the tensions and resolutions found in detailed experience of building and using the system? Or must experience always yield only mess, requiring an intelligent human to analyze and plan a structured substitute for that mess? How does an idea like pipe and filer come about? Can a system say pipe and filter to its developers?
My take on all of the above: Problems have essential structure. Although there may be a few different and valid views of the problem, allowing somewhat different structures, the number of valid structures for a given problem is not infinite, nor is it a matter of analyst style. Plausible solution structures are functions of problem structures, more or less. Second, there is noise around everything, obscuring clarity and structure. This noise is essential for emergence, because when structure emerges, it emerges from a sea of noise (what else?). The critical questions do not include "can an architecture emerge". An architecture "can" always emerge, given the right conditions. The real questions have to do with the minimal conditions under which structure will emerge from apparent noise. There are also questions about for whom structure will emerge, since structure it largely embedded in the processes of observation and interpretation.
I especially like the tension resolution part here. It is about feedback. You need to feel the tension. Where you feel it, your instinct should be to resolve it. A tree for example can kind of "feel" badly balanced branches and puts more wood in the areas of highest pressure (it is more a mechanical effect, of course). He thus achieves statics that are better than those of most cathedrals and can hardly be rivaled by an engineering task force. A tree has an architecture. It emerges by tension resolution.
The tree also achieves better dynamics than a cathedral. It is the constat motion of the flexible parts that tells the tree where to grow stronger. If the tree grows too high and too quickly, it will break in high winds; this is what tells the tree to grow out and not up. If the tree grows out too far and too quickly, it will snap under the weight of a heavy snowfall; this is what tells the tree to grow up and not out. Between the two it achieves balance, without the least bit of planning.
You need to be willing, here and there, to let a few branches break. It is how we learn.
BTW: How about comparing Software Development not with building houses but with gardening? -- Dierk Koenig
Can a system say pipe and filter to its developers?
I like this question, and my answer is "Yes". And a developer who's listening, and who's had some experience with Unix, or Java's I/O Stream Filters, will hear the system loud and clear. But a developer who's never seen pipe and filter probably won't get what the system is saying.
This is now the best explanation I can think of for why patterns are important, even if you're doing emergent (as opposed to up-front) design: you need to see multiple instances of something before you can recognize a new instance of it. To someone who's never seen a hammer, does anything look like a nail?
Books like the Gang Of Four's are phrase books that let you translate what your code is telling you. The Forces section is what you hear from your code (in addition to any obvious code duplication) -- Ha bla piepandfiltr?. The pattern itself is what your code is trying to say (or suggest) - Hey, pipe-and-filter might work nicely here. -- George Paci
Few software developers are building 'cathedrals', they are mostly building 'cottages' and 'small shops'. Craftsmanship and skill are important and they are accessable to everyday folk for this level of construction. It may be that a lot of software projects can get by with Folk Architecture. Just a thought. -- Tom Ayerst
What could a company do to try and create a fertile environment for architectures to emerge? Would that be Agile Architecture?
Be careful not to mix up Evolving Architectures and Emerging Architectures. Architectures absolutely must evolve. They don't particularly need to emmerge (but this depends on your definition of 'emerge'). The larger a system is, the more API's it needs to provide for ISV's, the more it needs to integrate with different enterprise systems, the less possible it is for an architecture to simply emerge. I would say custom apps would probably have a better chance with complete architectural emergence, but it's pretty difficult for a commercial product that needs to support an ISV aftermarket, system integrators, sales channels, technical marketing and so on. If we are using evolution and emergence as synonyms, then I would agree that all architectures emerges. -- Robert Di Falco
Let's say there are 100 workers constructing your building. On room 2097 of your building you discover this new really cool idea. How easy is it to change the other 2096 rooms? How long will it take to retrain the 100 workers?
Difficult, and much time. Luckily software doesn't work like that. Adding a cool idea to the Room abstraction doesn't necessarily make it any harder to instantiate multiple times.
Of course software does work exactly like that. The cool new idea isn't to make the paint color setable. It's more complex than that.
You'll have to demonstrate how writing software is more like building than writing instructions for building. Real buildings are made of material that has weight and (generall) rigid structure. Changing 3000 rooms is non-trivial if they are made of wood and concrete. Changing 300 Room instances can be trivial if they are made of bits. No matter how complex your cool new idea, you don't have to tear down any load bearing beams to implement it (or test it).
Sure, no problem. Let's say you have a commercial software product with an API for integration. You have 1000 ISV's that are using this API and now you decide that you have a much better way to express some integration points and this new way is not going to be backwards compatable. Hmmmm...too bad you didn't spend a little more time in up-front design. You might have caught this before publishing the API. -- Robert Di Falco
Standard procedure is to release a new version of the API and deprecate the current one.
Which is not necessarily a good architectural strategy. In building new things, one has no problem with this, but in maintaining old things or remodeling them, the reliance on stability, endurance and backwards compatibility are critical. Emergent Architecture should not deprecate currently used components on too short a time scale. The time scale should be based on the life-cycle of the dependent software and associated hardware.
The deprecated API lingers until the benefit sinks beneath the cost (usually well below it). That's why the software I'm using right now won't run on an 8088 CPU or MS-DOS 2.0 operating system. No amount of up-front design would have avoided this. -- Eric Hodges
I found this buried in an economic paper about the dynamics of open source development (web.mit.edu ):
... empirical studies have shown that minor innovations are in fact responsible for much or most technical progress. Thus Hollander (1965), found that about 80% of unit cost reductions in Rayon manufacture were the cumulative result of minor technical changes. Similarly, Knight (1963) studied the sources of improvement in general purpose digital computers during the period 1944 to 1962. He found that: "Many of the most economically valuable improvements in computer technology occur as the sum of numerous common engineering and production innovations." ... "These advances occur as the result of equipment designers using their knowledge of electronics technology to produce a multitude of small improvements that together produce significant performance advances." (pp. VII-2 and 3.)
This seems as good an argument for an emergent architecture as any: Good software doesn't come in one inspired bolt from the blue. It comes from the patient accumulation of innovation over time. The best architecture is probably that lets those myriad innovations work on the same problem without stepping on each other's toes. -- francis
Saw something like this about how the human brain works - "at a critical point on the edge of chaos" - phys.org
Buried and success on the same sentence? Being successful means making yourself irrelevant. I don't like that. You are much better off introducing flaws.
-- Dilbert's Pointy Haired Boss
If things are done properly, all architecture is emergent. This is true even for those who follow Big Design Up Front (BDUF). Where do you think that design comes from? A responsible project manager would never just take an idea from thin air and jump straight into production. Only an idiot would commit to something that has yet to be proved. This is not to say the world isn't full of idiots. The point is, all reliable designs emerge from a process of evaluating ideas, mitigating risks, and answering the unknowns. This process is called research & development (R&D). It's the phase that preceeds production and helps to ward off any big gotchas. Both XP and BDUF have an R&D phase. The only difference is that BDUF draws boundaries and segregates it from production, whereas XP does not.
Case in point, take the prior example of building architecture. Large structures, especially those with novel features or locations, are always put through an R&D process before any significant construction begins. The building is modeled and loads are tested. New truss designs are assessed for reliability. The ground is evaluated to ensure it'll hold the weight. An architect doesn't just whip out a bunch of drawings and people start to work. They try things out first. If they run into a problem, they back up and try something else. Thus, the plan they eventually commit to has emerged from a process of repeated testing and evaluation.
In short, BDUF is only BDUF if you ignore the R&D that happens beforehand.
- Milo Hyson
See: Architecture Definitions, What Will This Become, Xp Needs No Architecture, Perfect Architecture, Architect The Negative Space, Can Math Emerge
Contrast: Architects Dont Create Architectures, which is possibly the exact same idea approached from the other direction
See original on c2.com