Refactor Low Hanging Fruit

I said that in Test Driven Development the right design emerges only after you have thrown away a lot of code.

It "emerges" as a combination of you discovering it and you already knowing what it is. Kind of like the end of The Wizard Of Oz, when the Wikid Witch of the North told Dorothy that she had the answer with her all along, but she had to learn it for herself. You just tap the heels of your Ruby Language slippers together three times, and hit the One Test Button.

Of course, with any approach, you will always encounter situations where you have to rewrite major portions of the code, or, worse, change the entire architecture. This is likely to happen due to requirement changes, or performance issues, but can happen because your initial visualization was faulty.

In real life, what typically happens is, after the twister, your house lands on top of the previous lead architect, and you peer tremulously out the door at a twisted landscape of big balls of crufty mud, written by short programmers, their growth stunted by long working hours and junk food, under the spell of some lesser process than XP.

This leaves you wondering where to start. Then Michael Feathers, wearing a Wikid Witch of the North costume, appears with a copy of The Joy of Legacy Code, and smacks you across the forehead with it.

But MIKE floats away inside a SOAP bubble, leaving you all alone. With the short programmers crowding around you.

So, by finding the simplest possible fixes to the lowest level code within that ball of mud, and by isolating the effects of your changes with characterization tests, you can begin to tease apart its design.

... And ... you ... are ...

-- Phlip2004

[NOTE: Refer to the idiotic dialog in Refactor Low Hanging Fruit Original to see what the following discussion is aboout.]

Black Hat: The quote, "But the other architects will laugh at me if they see my code doesn't have a Big Picture in it," smells like whining to me, used primarily to reduce the reader's opinion of the Super Ego.

Yellow Hat: Worrying about not having a Big Picture is a common concern among developers who Refactor Low Hanging Fruit. The quote is merely used to help show developers with this worry about how silly they are being.

Black Hat: I'd like to see some support for that statement. And anyway, what is this ridiculous "discussion" between the non-identities Super Ego and Id? Are these names supposed to convey something? They only serve as a distraction from the supposed "point" of this page.

White Hat: Several different kinds of Dramatic Identities are used throughout Wiki to convey ideas.

Red Hat: But endless repetition of the same statement is not a valid cognitive argument. It is simply naysaying.

Black Hat: Super Ego, as architect, points out that the project is in a state of mess and needs to be resolved into a reasonable architecture. Id, ever the idiot, responds with the only line he has in the whole "conversation."

Yellow Hat: Id is repeating the same mantra as a device to show how all the complex problems can be iteratively solved with a simple solution. The purposeful use of repetition is akin to a Zen Slap.

Red Hat: The pat answer does, however, put me off a little.

Blue Hat: It might be valuable to further explore some of Super Ego's concerns...could we be clearer with a less Zen Master And Student approach? Maybe a simpler explanation before the dialog, or even just a simple explanation?

Black Hat: Refactoring works great when you have a bunch of tightly constructed thingys that all do their own work efficiently and effectively. It's not so great when these thingys don't work together because nobody knows what the state of the system is or what unit has priority over what else and when those priorities need to shift. Refactoring can't address that at all.

Yellow Hat: The point is that although there will be a transitory state, possibly as confusing as you fear, constant and cohesive applications of Refactor Low Hanging Fruit, will avoid such a state in the end.

Black Hat: No, it won't. The architectural issues Super Ego brings up can not possibly be resolved through the simple minded approach that the Id 'bot is spouting. Some of us have experienced projects that desperately needed an architect to figure out how to put things all together. Refactoring doesn't solve all the architectural problems, in my experience.

White Hat: If you envision an advanced design, don't start with the most complex refactor. Try the simplest ones you can do first. Strategically, the design may temporarily get worse. Keep going until the refactor that will install that advanced design is now the simplest refactor available. If you don't envision any advanced refactor, hit the LHF anyway.

Black Hat: Why should I bother with refactoring a bunch of code that may not address the true concerns of the overall system? If I go back to the drawing board and come up with a plan then I can see if any of the objects, procedural functionality, or data constructs look anything like what I envision the system as needing. I'll refactor what I can use and toss what I can't. The use of refactoring needs to be applied judiciously, not as a sledgehammer solution to all scenarios. It is not the Universal Tool.

Green Hat: The Refactor Low Hanging Fruit Original dialog gave me the "Ah Ha" that refactoring low hanging fruit applies to many domains beside programming, such as Refactoring Wiki Pages. The dialog & ensuing discussion on this page don't give me the same insight. The changes made to the dialog above are a perfect example of Disagree By Distorting.

Red Hat: Really? This despite the fact that the original version is filled with the stench of Argumentum Ad Infinitum (Argument By Repeated Assertion, etc.) and Avoiding The Question?

Green Hat: My Ah Ha came not despite the repetition, but because of it. I'm not a programmer, so I took the dialog more as a metaphor. That detachment let me extract the dialog's message and apply it to other areas, particularly writing and editing. Could it be that you're so focused on the dialog's domain (refactoring as a way to understand legacy code) that you've missed the larger lesson (tackling small parts of a problem first makes the bigger parts easier)?

Black Hat: No. The repetition is used as a blunt instrument in an attempt to hammer flat a problem containing curves. This is not to say that refactoring is not a powerful and valuable tool, just that it simply doesn't apply to all conditions of altering a design in its basic structure.

Discussion

[Note that much of this discussion refers to the original Id/Superego discussion on Refactor Low Hanging Fruit Original.]

This smacks of Argumentum Ad Infinitum. However, the idea of chasing down refactorings is valid in its own right, as long as it is not taken as an article of faith that this approach cures all poor design evils.

I'll tell you the truth, folks - if somebody did something like this to me in a real situation I might be tempted to apply some Parking Lot Therapy. Nobody in his right mind can believe that a simplistic approach like this cures every problem you will encounter. Nothing is that universal. Anybody who tries to tell you that is selling snake oil. -- Marty Schrader

Could expand on your objections?

First the "dialog" above would need to be "refactored" into a set of problem/solution statements. I simply can not live with the BS as it is currently stated. The drivel about other architects laughing is particularly grating. Eliminate the fluff and I'll be happy to write a rebuttal. Right now there is nothing to rebut other than the useless repetition. Where's the porpoise?

I've been very successful with refactoring towards high level architectural goals. But Refactor Low Hanging Fruit is still a very good piece of advice:

If you can't bring yourself to fix small problems, how will you ever be successful with the big problems?

You have to develop a habit that when you see a problem you fix it. Otherwise when you hit a big problem you'll (#1) not really know how to fight it and (#2) big refactorings always create lots of little problems anyway. You'll still have to Refactor Low Hanging Fruit.

-- Jeff Grigg

Unfortunately, there have been many, many hours of my time wasted in refactoring low hanging fruit when I was working on something that needed an axe taken to it. Lots of fixes to code and documents that later ended up in the ashcan. Had I been smart enough to take a longer view of the junk I was working on I would have seen that the particular code or doc simply wasn't needed for the solution I was striving for.

This is not a case of the goal posts shifting after the fact nor of clarity of vision due to more information. The information to analyze had been there all along. It was a matter of my wanting to "make it better" before actually making the Whole Thing work. My fault entirely. Still, not something you want to explain to your boss (or client) when it comes time to fill out a weekly report. -- Marty Schrader

This pattern makes Refactoring those big balls o' mud into a hill-climbing algorithm that flattens the hill as you go.

Joel Spolsky has an interesting article describing a refactoring exercise he did, which reads to me like a classic piece of refactoring low hanging fruit: www.joelonsoftware.com . -- Stephen Hutchinson

The Id makes it sound simple, but this pattern has two flavors: refactor the fruit that is easiest to find, to engage flow. Or take time to identify which fruit is truly lowest-hanging.

I thought that the term 'Low Hanging Fruit' meant that it was "patently obvious to the most casual observer" what was easy - it didn't need great study. Like replacing a lot of:

code with switch statements.

I think that's what the dialog above is trying to say. In other words, if you have a big ball of messy code, instead of trying to tackle the whole problem, just untangle a little bit. Keep at it, until the whole ball is gone.

Because, as was stated above, "Why should I bother with refactoring a bunch of code that may not address the true concerns of the overall system?...I'll refactor what I can use and toss what I can't." Don't refactor code you will be discarding later on. Suppose that whole if/then/else statement evaluates something I end up not using? It's a waste of time to refactor it. Eh?

True. However, I usually find out what's useless through the process of refactoring.

I'm really not trying to belabor the point, but the issue I was raising is that refactoring doesn't always help to define the real needs of a system, particularly when there is a lot of legacy code involved or if the system was never very clearly defined in the first place. Many clients have paid me a big piece of change to work on legacy systems that I cleaned up nicely, only to discover that the code simply wouldn't meet the New! IMPROVED!! needs of the system as the clueless client (a redundancy, I realize) specified. I have examples of this in mind. In almost all of these cases I could have saved the client moolah by analyzing the real system requirements first and then creating new code later. Had I been given the mandate to discover the real needs of the system I could have done that; in most of these cases the client whined about how many man-years of labor had been invested in the code base, so I had to start there.

In a few of these cases I completely ashcanned the original code and started over from scratchola. The new system was able to surpass the requirements in all cases where the underlying hardware had sufficient horsepower. Refactoring the original code forever simply wouldn't have achieved this. Period, end of statement.

-- Marty Schrader

Once again the vague definition of the word "architecture" threatens to derail a well-meaning discussion between programmers. When you say "architecture", Marty, do you mean "macro-structure", or "system requirements"? Because I get the feeling that whichever meaning you intend, somebody else here intends the other. -- francis

Also, remember, the purpose of refactoring is to maintain the current functionality and just change how the functionality is expressed. So, it is a true statement to say that refactoring will not cause code to meet new requirements. Refactoring will lead to changes in efficiency and performance, but whether those changes will be significant, improvements, or regressions is going to be almost impossible to predict. I think it is a separate discussion as to whether it is more appropriate to modify current code or rewrite the code from scratch when addressing new requirements. -- Wayne Mack

Refactoring techniques help when re-featurizing. If you do a Big Refactor and let behavior drift a little, it's a Big-Refeaturizing. But this applies much more to Deprecation Refactor than LHF. The latter is about design. -- Phl Ip

Francis, my comments about architecture are intended to talk about The Big Picture(tm), whether that entails the relationship of features to each other and to the system overall or to the relationships of components in the system or to integration of the system to the outside world. My point is that one needs to know how components will be used in The Big Picture(tm) before determining if they are usable or not. If a component is not usable I am not going to spend any amount of time at all refactoring the code; it simply gets tossed in the dustbin. -- Marty Schrader

Refactoring is merely a means to find a home for new or changed functionality. Refactor to create the appropriate place to make the change, make the lowest risk change possible, and then refactor to make the change clean. This provides a viable alternative to throw it out and rewrite from scratch. Some of the biggest project disasters I have seen were "rewrite from scratch" projects. The ugly code hid a lot of details that had to be rediscovered by the development team. The components of a system are rarely unusable (the system is currently using them), though they may be difficult to understand. Refactoring improves the understanding and understandability of the component while preserving the usability. -- Wayne Mack

Yeah, that's great for those cases where refactoring applies. The point is that refactoring doesn't apply to all cases of making structural changes to an application. Sometimes the required change is so great -- without appearing to be so -- that refactoring the code until Doomsday doesn't get you any closer to a solution. Architecture does.

At this point we may just have to Agree To Disagree; our experiences seem to be too different. I have not experienced applications that needed structural change. I have experienced applying large structural change to applications in a single step and seen the following struggle to recreate the original operating characteristics; the end result being a lot of time and effort with little clear benefit to the actual users of the application. Perhaps one day I will experience an application in need of major structural change and will be able to understand your argument. Until then, however, I will favor an incremental approach, which I see being formalized through the combination of Refactoring and Test First Design. --Wayne Mack

Concur. We will agree to disagree here because of the disparity of our experiences, I guess. If anyone is interested I can dredge up numerous projects over the last 25 years or so that needed architecture to fix the problems; refactoring had been attempted to some degree or another and had failed to bring about the necessary improvements to meet requirements, whether those requirements had changed or not. -- Marty Schrader

False Dichotomy: Perhaps this is more a matter of nuance and interpretation than anything else. Refactor Low Hanging Fruit is not intrinsically opposed to re-architecting. And the original page over-simplifies, but it does so to make a valid point - that the impact of continuous refactoring as part of the development process is often not appreciated sufficiently. Even if you do need to rewrite from scratch - and I have had to rewrite legacy subsystems more than once - you still need to refactor and change your view as you go. I quail when I see a bunch of people come in to sort something out with a Big Picture. There is no Big Picture(tm) in many business organizations. Or, at the least, the Big Picture(tm) doesn't give you a nice, neat up-front set of criteria and a gameplan. Better to start off with some High-Level But Still Meaningful Basic Principles(tm) that don't dictate exactly how they are to be fulfilled in software. You feel your way, building a rapport with the customers. You refactor and change your view of what system is required as you go along. Sometimes you junk a few day's worth of exploratory work. If you don't go this way, you'll just waste your time writing lots of nice shiny new programs that the customer doesn't actually need. After all, you might only need to rebuild half of that system. But without an iterative, refactoring-based approach, you're not going to be able to sort out which half.

Hmm. I suppose you may deal with clients who are so clueless as to not know what their plan is, but these must be firms without investors. Typically a firm doesn't get investment capital until after they pitch a business plan to a rather substantial number of investors, so many eyes get a chance to look the plan over and poke holes in it if they can. Clients are certainly clueless, but they can't afford to be stupid or wishy-washy if they want to use expensive resources like me. You go ahead and refactor their old crapola. I am going to meet their needs as they have specified to me, starting with what are the improvements of the new system over the old one.

You know, I am a basically lazy guy, so if I can use some old kaka out of the previous product then I will polish this turd up and use it. However, there are way too many times when no amount of turd polish will turn the old junk into a sparkling new widget. You gotta know when to hold 'em and when to fold 'em. Analysis gives you this information, and time wasted in refactoring old drivel won't help a whole lot.

-- Marty Schrader

RLHF is one of those rules that's very hard to go wrong. -- the TOOLS GUY

Generally, that's true. Refactoring is a powerful tool in cleaning up legacy code or adding simple feature enhancement to an existing system. The above discussion is all to act as a warning sign that refactoring isn't always the proper tool to use. In the cases of feature enhancement or performance increases where some fundamental change in architecture is required then refactoring SIMPLY DOESN'T APPLY TO THE PROBLEM DOMAIN!!! I really hope we were all listening that time.

Use the tool that fits. Don't pry open the paint can with a screwdriver. Don't hammer the nail with a wrench. Don't tighten the bolt with a wire cutter. DON'T REFACTOR CODE THAT WON'T BE USED IN A NEW DESIGN!! Do we get it now?

-- Marty Schrader

What you're saying makes a lot of sense, but I don't think it takes analysis separate from refactoring to find that dead code. When you want to clean up legacy code, you write some characterization tests to figure out what it does. If in doing so you find that the code isn't needed anymore, you just delete it. But if it is needed, you now have the tests in place to do refactor with confidence.

-- Pat Maddox

Yeah, I'm still not convinced that tests are any better than any other means of determining if legacy code is fitting to a particular altered architecture. (Of course, I am not fond of tests in the first place, being a Fagan convert and thoroughly enamored of the way formal inspections work.) In the progression of development architecture comes after specification and before design. Code is the complete tail end of the effort, so if there's code already there then you have to look at the design that stems from the architecture to see if any of the legacy code fits.

I am not interested in reusing existing code just because a client whines that he put so-and-so many man years of development into it. I have examples of products where multiple man years went into something that wouldn't work in the first place, not ever. Refactoring that trash forever gets you exactly zero. Let me see if I can whip up a design that incorporates the existing code into solving the defined architecture, and then I'll look into refactoring the code. Certainly not before that.

-- Marty Schrader

To try to extend that: when dealing with a design where the design fundamentally has to be changed at a basic level, one refactoring-based approach is to attempt to refactor by factoring out all the modules which you will be able to reuse. Then when you rip the basic architecture out and replace it, you have those modules ready and functional. I think once you know what your new architecture is, your perspective on low-hanging fruit tends to include only stuff which will still be needed in your new architecture, but that's not the normal meaning of low-hanging fruit, is it?

You have a good grasp of the point I was originally trying to make. -- Marty Schrader

Delete When Cooked: Regarding " [Um, Real Names Please.]", the Dramatic Identity THE TOOLS GUY comes from the narrative segment of this page. It's not an attempt to hide a real name. (And it wasn't me - please leave it alone.) --Phl Ip

Darn! You're right. I missed the one reference to this character from the cast of munchkins flogging Dorothy with their, um, "wisdom" above. Still think THE TOOLS GUY is as big an ass now as he was years ago when he first made that argument. -- Marty Schrader

See: Refactor Low Hanging Fruit Original

Contrast: Deprecation Refactor

Category Refactoring

See original on c2.com