Re: the evolution of systems:
My vision can shape initial direction, and my attention to the desires of the code can affect how quickly and how well the system finds its desired shape, but the system is riding me much more than I am riding the system. -- Kent Beck (in a mystical mood - don't tell my clients)
There is a funny danger in getting experienced enough to see the universals in engineering work. With the least conscious part of the body doing the engineering work, you start to notice the flow, the universals, the mystery, simplicity and harmonies at work. It increases the pleasure. But a little dangerous for us consultants. The first time I slipped up I was in St. Louis. Everybody stared at me really strangely, so I said, "Oops, a little California sneaked into me there." I tend to try it on my customers perhaps once on an assignment, just to see if anyone is going to be able to handle it. But it seems mostly to work with people who already know it, in which case it is nothing more than a mutual recognition.
I have started teaching it directly to new students. So far it works best with the younger, complete novices, who are able to hear and follow their sensations. They learn good habits before they learn the names for them. But for most engineering people, "follow you sensations" kind of talk is not explicit enough to be of use right away. It needs the long lead in and followup. See also Hear What You Say -- Alistair Cockburn
I find these sorts of things fascinating (see also Egoless Creativity and Systems As Living Things). People don't tend to talk about them much. Considering such talk mysticism is okay, but I think that it is a recognition of abstract form that we just don't have a language for. Witness my attempts in Liskov Substitution Principle at getting across the idea that LSP is part of some universal engineering principle that we do not have a name for yet. As a predominantly visual thinker this happens to me a lot. I can "see" it, but it is tough to put into words. The fact that people who "see" it do "see" it, seems to point to information that is not easily put into words.
Is it mystical? Maybe. I just hope that we can find a good way of talking about such things. The anthropmorphic role-playing that we do with our systems just seems like something that can be presented as a technique for jumping out of traditional ways of thinking to enhance creativity. It seems very valuable. I suppose that it could be taught in an exercise where the traditional linear probing for a solution reaches a dead end, but some perspective switch helps solves the problem. Then we whip out a bag of perspective changing things, like listening to code and anthropomorphism and say "try these when you are stuck." -- Michael Feathers
"Then we whip out a bag of perspective changing things" - have you read de Bono's work on lateral thinking and teaching thought? -- Dave Harris
No. It's good? -- Michael Feathers
See Edward De Bono for a list of his books. Perhaps Dave Harris could write some short reviews?
It's not even just cultural. Different people's intuition runs through different senses. "Listen to what Smalltalk is telling you" is hard for me because my intuition is visual. Apparently Kent's is olfactory. ;-> I've learned that even with people who have highly trained intuition, sometimes my visions confuse them. (I never mention the voices.) There's a whole set of communication techniques that focus on figuring out the other person's mode of thought and using it. "Smell this", you're supposed to say to Kent. "Watch it", he replies. -- Ron Jeffries
At least some of this is to do with aesthetics, and as a student I realized early on that the gurus had a sense of aesthetics which they were trying to impart. This was more in mathematics than programming, by the way. They would present a proof and say, "Isn't this beautiful?" When I've come across bad programmers, I've felt that it was because their sense of aesthetics was poorly formed. They didn't know they were producing ugly code. I'd hesitate to tell novices to "follow their sensations" unless I felt their sensations were likely to be right, because I think this may have to be learnt. -- Dave Harris
When I was studying math, what made me successful (in the sense of becoming teacher's pet and getting grades and such) was that somehow I did develop that sense of aesthetics. Lots of folks could crank out harder integrals than I ever could. In math, aesthetics in a proof is as much about leaving out as it is about putting in. In programming perhaps it's the same. -- Ron Jeffries
Perfection Is Achieved... -- Eirik Mangseth [See also: Wiki Master Original]
I saw someone write in Communications Of The Acm a while back that we need to develop a science of form. It would sure help design across all disciplines. The aesthetics that you mention would probably be part of it.
One example: Alistair Cockburn mentions on another Wiki page that he was told that there are only three numbers in computer science: 0, 1, and N. I also heard once that a programming language is elegant if the answer to all questions "how many ways can you do X?" are 0, 1, or infinity. Mathematicians said this first (with regard to ways that a theorem could be proven). Isaac Asimov quotes it in "The Gods Themselves...". -- Daniel Knapp
On the other hand, there is an aesthetic to imperfection as well that is more appropriate for some engineering and messy organic-like systems. Regardless, there are common aspects to form across disciplines. -- Michael Feathers
My reading of the Nature Of Order is that it is just such an attempt to create a universal, objective aesthetics. -- Kent Beck
Some half-baked thoughts inspired by what Dave Harris wrote above:
Beginners (like me) have to strictly adhere to the rules because they do not have the right sensations to guide them. Rules are a necessary stepping stone (see Wittgensteins Ladder) to develop a feeling for what is and isn't beautiful.
Gurus may seemingly "break" the rules, because their understanding transcends the letter of the rules and follows the underlying aesthetics, which cannot be put into hard and fast words and thus should better stay without a name. [See Shu Ha Ri for more on this.]
BTW, I get the strongest "Isn't this beautiful"-feelings not out of Maths or programming, but out of going through some vintage grand-master chess games. -- Falk Bruegmann
Yes! And this is explicit at the end of The Timeless Way Of Building; that patterns should eventually be discarded.
Patterns, and examples, are merely a way to attempt to instill a sense of aesthetics. -- Dave Harris
I remember reading somewhere that chess players improve to about master strength by learning the rules (and how to calculate etc) but beyond master strength, they improve by learning when to violate the rules. Falk's comment above about following the underlying aesthetics fits to a T. When the best-ranked grandmasters play, the other grandmasters analysing the game occasionally don't even understand why the top grandmaster made a certain move (and the move is seldom a mistake). With computer programs, this would most likely occur because one program could see more ply into the future than the others. But the top grandmasters can't really calculate farther ahead than his peers - he just extracts more value out of what he can see somehow, using this finely-honed aesthetic sense.
You may be interested in Human Perfection In Checkers.
I wonder about this tendency to see form in terms of aesthetics. I prefer to be non-judgemental about form. I want to understand it. If aesthetics is seen as the recognition of internal order, then judgements can be made about the degree of beauty of a form, and those judgements are very contextual, based not only upon the internal order but also the goals and intentions of the observer. I tend to think that forms just "are." And everything else is dependent upon criteria and context. No absolute aesthetics.
My favorite architect is Le Corbusier, and I can sense an incredible aesthetic there, although I probably wouldn't want to live in any of his housing. Does that make his work less aesthetic in the Alexanderian sense? It depends on what one chooses to see. -- Michael Feathers
re: 'I'd hesitate to tell novices to "follow their sensations"'... my current favorite book is Pelle Ehn's Work Oriented Design Of Computer Artifacts, the book from his Ph.D. thesis and already out of print, but he wrote articles around it also, and in it he excerpts someone who studied learning and use of rules in stages of learning, and this is from memory since I am traveling. The first stage is to learn to follow the rules. The second stages is to link the rules to sensations, the third stages is to learn you can break the rules, the fourth is to break the rules casually, the fifth is continuous subvocal performance. I try to teach directly to the second stage these days, and like the result. I try to get the students to notice their feelings, words, facial expressions, whatever, while they are hearing feedback on the designs. They can link their subvocal state to the designs instead of memorizing rules that are wrong anyway.
It is not that I tell untutored people to follow their untutored sensations, it is that I encourage and tutor them to build upon an already mature evaluation system, one that works faster than vocalization and will last longer. Most experienced programmers still can't say what they do or why. -- Alistair Cockburn
Re: Pictures, smells, feelings & 'most experienced programmers still can't say what they do or why'. I seem to remember Wittgenstein also saying that what we cannot explain we can only point at and Camus saying that what we cannot explain, we do not understand. I tend to agree with both positions which means that I have to accept that I have no complete understanding of anything to do with programming (seeing as I too see and try to communicate complex systems in pictures). I sense this is a common predicament when dealing with the animated manifestations of abstractions we call computer programs. Perhaps this means we should strive to understand as well as we can, but not fixate on apparent regularities and extend them as prescriptions. Patterns give us this flexibility, rules do not. Goedel's theorem proved that any formal system is either incomplete or inconsistent. It has always amused me that mathematicians insist that only the incomplete half applies to maths.
Re: Universal aesthetic If there were some universal aesthetic underlying our creations, we would probably have a global monoculture (as we did before Cro-Magnon humans), i.e. a preprogrammed way of making things. An archaeologist friend of mine said he groaned every time he found another pre-Cro-Magnon flint knife - they are all the same all over the world for hundreds of thousands of years. And this is homo sapiens we are talking about here. One of the defining characteristics of modern humans is not that they are able to transcend rules and 'follow the underlying aesthetics' but that they are able to create new aesthetics and derive new rules from them.
Or perhaps there's a single and cohesive aesthetics structure underneath it all but we're just too limited to perceive it or even put it into words? -- Rodrigo Oliveira
For example, in the early 80s in Britain a small Z80 based computer with 48K of RAM, the Sinclair ZX Spectrum, was very popular. Space was tight but some games programmers managed to cram an incredible amount of function into this machine. By reverse engineering, I discovered the technique of self-modifying code. This was a very elegant solution to the problem of balancing the forces of restricted space versus extra function though it would be frowned upon in the context of multitasking re-entrant routines - different forces at work. The point is that different contexts call for different aesthetics (cf. Noh drama and Shakespeare). (BTW: I found reverse engineering 48K of assembly code the closest thing I've ever had to a mystical experience in programming.) -- Sean Ohalpin
Just got two hits on this topic, in a western rational explanation style. One was given to me by my friend Ken Mulligan, who said performance passes through 4 stages,
[...given in Four Levels Of Competence. �- jtg] Second hit was this afternoon I saw in a book on learning organizations essentially the same thing, but in a graph showing performance along the x axis, conscious attention on the y axis, and as performance increased, the attention went up from zero to a max and back down again. -- Alistair Cockburn
[Discussion of the first point above moved to Four Levels Of Competence. -- Jeff Grigg]
When you are designing mystically, how much do you sway away from (or toward) the avant garde? See also: www.geocities.com 
Personally, I use the Force. Well, okay, not really.
I have found that my ability to know transcends my ability to think such that occasionally I have to leave the problem of the moment, deliberately wonder to myself what the best approach is, and allow myself to arrive at an answer without "driving" the thought.
I was being interviewed for a job (which I did not get - they wanted young, energetic guys) and I was shown a piece of C code and asked, "what does this do?" My answer, without thinking, was, "it counts bits." I then "thought" about it and added, "or it might determine parity." I traced the code through a couple of iterations. It counted bits. I'd never seen this code fragment before, but I know C, I don't think C.
I'd never thought to classify it as "mystical" but that's as good a handle as any. -- Garry Hamilton
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -- Buckminster Fuller
The true computational power of the brain is not the mechanistic part that we use to calculate solutions using known algorithms, that part (the 'conscious' part) can cope with only a small number of simultaneous variables in its evaluation of possible answers. Our real intelligence is in the subconscious, an area of mind that can correlate an enormous number of pieces of data and that produces answers that are *probably* *approximately* right, but that are usually more accurate than we would ever expect.
The unconscious (or subconscious) works in a linear (as in analogue) fashion, understands quantities rather than numbers, and expresses itself to our conscious via whichever mechanism is natural to us. That mechanism could be kinaesthetic - meaning a feeling or a shape, 'that fits' or 'that feels right', it could be visual, 'that doesn't look good', and I'm sure it could be olfactory - 'that smells wrong'. This interface is its only weak point, a rather blurry output channel to our conscious mind. Trust the computational power behind it, it's stupendous; the data included is everything you know, instead of just that which you consciously remember at the time. The difficult bit is remembering to listen (or look or feel or smell)
The other side of this though is the (IMO relatively rare) cases where the intuition gets it wrong and the only route to a correct answer is algorithmic calculation. The Monty Hall Problem is an ideal example.
Evidence? Numerous optical illusions suggest the mind gets it wrong quite easily.
You just provided your own evidence. The brain is extremely good at visual perception, vastly better than we can recreate algorithmically. The "numerous optical illusions" you cite were very carefully crafted to fool visual subsystems that are ordinarily quite reliable, and are vanishingly rare in the natural world.
The truism is that basically everything that computers are good at, the brain is not, and everything the brain is good at, computers are not (this could be nitpicked, but don't bother; 90% of the obvious nitpicks are incorrect upon closer examination). That, in a nutshell, is a summary of why AI is so difficult.
So, for a phenomenal amount of evidence of the power of intuition over logic in the general case, look at cognitive science and AI. Essentially every word written on those topics is another piece of evidence for the point.
I can't imagine any claim in any field that has more evidence supporting it. Even "2 + 2 = 4" is sharply less well supported than what Michael said. -- Doug Merritt
The page Refactor Low Hanging Fruit starts with a facetious statement of an important principle. Let's call it the "RubySlippersPrinciple". (And it doesn't deserve its own page!)
The principle implies you consciously know where you want to go and subconsciously know how to get there ("there's no place like home"). You must learn what you already know, so the learning process can make the knowledge relevant.
Kent can't tell if he is riding the system or if it's riding him. He views a resulting design and can't remember if he thought of its shape before or after coding it. -- Phl Ip
See original on c2.com