Jack and Jill rewrite a system and then go on a dig

A software system is like a bucket of water.

Photo by Tobias Schlitt

But other than just the code and the persistence of data there's knowledge too. Knowledge about the system that is sometimes stored in documents, sometimes stored in tests but always stored with people.

I've just been watching Scott Bellware's "Beyond Agile" talk and he talks about software development of simply being knowledge transfer. A product sponsor transfers the knowledge to an analyst, who transfers to the business analyst, who transfers to the develop to tester into a product.

When we rewrite systems we will leak this knowledge. Exactly if you tried to pour a full bucket of water into another bucket or water. Chances are you'll spill a bit. Chances are the documentation is not comprehensive. Chances are the tests don't convey the meaning of the application well enough. Chances are the developers have forgotten what they were doing when they wrote that feature. Chances are the product sponsor has forgotten exactly why they wanted it 'that' way.

When system stakeholders leave the project the bucket springs a leak. You have lost some of that knowledge. Even with comprehensive documentation, tests and a full handover.  And your bucket is probably not a bucket any more. What once was a bucket is now probably a kettle that does credit card payments... and its rusty but only on a Thursday.

So how do I summarise this leaky abstraction? (go on groan)

A software system is is more than just code. It is the users, the stakeholders, the delivery team and the state of the system. When rewriting the system it is going to be impossible to relive the journey of knowledge accumulation that the team went through to get to the system that you are rewriting. Even though there will be a set of artefacts you will have discovered to reconstruct the system this will not be the full picture. That's ok because you only need to do what your stakeholder's need now. Reconstructing 100% of the old system when no one knows what 100% is wasteful (and impossible)

Rewriting software is like an archaeological dig.

Photo by Ben Garney

Except rather than rebuilding the Roman fort exactly we extract what we want from it and make it better.

DDDSW 3 - RE: the ReWrite session.

I went to DDDSW on Saturday( a conference based around Microsoft technologies) specifically to see one session.

"Rewriting software is the single worst mistake you can make - apparently" by Phil Collins.

This subject is dear to my mind as I've been involved in many rewrite projects over the years with varying degrees of success. Its also a big part of my studies at the minute.

Phil is a confident speaker and this session is an experience report rather than a "How to" session. Some of the choices that Phil has made I disagree with. I want to share the reasons why I disagree, not to have a pop at Phil but just to get clear in my mind why I do.
 
Language choice
Phil was moving from an old tightly coupled legacy system and he had a free choice of languages. He quite rightly had an R & D session on a few languages. However the basis of the choice he made given in the talk was problems with case insensitivity. There are many factors for choosing a language, tooling support & flexibility are two that spring to mind but the issues that you have in the first few weeks you will soon forget and so case insensitivity is a minor distraction in my opinion. Choose a language and platform that is right for the application not just the rewrite project.

Screen by screen rewrite.
Why? Sure you get all the features but this is an excellent time to find out exactly what you do use and more importantly what you don't. I know this from first had experience. I joined a rewrite project late that had done exactly this. There were dozens and I mean dozens of text boxes and fields in the application that were not used any more. Whole screens were devoted to the parts of the business that just didn't exist any more.

Take the time to talk with your users about each screen. Try and come up with the story behind each feature. If your users don't need it. Don't rewrite it. This saves you time and gives you opportunity to remove cruft from your application.

To be fair Phil did share a story where he added more features based on user interaction. I'm assuming that he may not have blindly rewritten all features.

What Phil has got right:
 
Continuous integration.
Phil lauds this and has a very funky set of build radiators. (BVC of the build status) and I agree. CI is instant feedback and build radiators are key to that feedback.

Slowly, carefully.
Phil tells his devs to go slowly. Any project should not be done a break neck speed. Otherwise you are under pressure to cut corners. If you are just in a race to rewrite a function you will make mistakes. A rewrite is an opportunity to cut tech debt. Don't make the mistake of adding more.

More Whiteboards
... I love just scribbling down stuff on white boards too. It is simply a great way to share ideas and concepts.

What I would do - ascertain the vision
Assuming that the rewrite has to be done. I would try and extract the vision for the project. Why is it that we have to do this rewrite? What are we trying to get out of the the rewrite? It may be to improve usability or performance. It may be for legal reasons. It may be the the cost of change is high in the legacy application or that the technology is just not supported. 


Observe system usage
Given we now have a vision, a set of woolly statements that we are heading towards we can now look at the current application and its features. Look at each of the features and see if they still align with how the users are using them. A good idea here is to sit with the users and observe their daily activity. Its an eye opener to see how your application is actually being used and its often not as designed. Of course you will not all of the usage in just one session but you will certainly get the core. 

Speak with your users
Next step is to interview your users find out what they think their processes are. At this point you should pick up things they do that you didn't pick up on the initial sweep. This is a great opportunity to find clunky procedure where you can improve upon. Many times I've done this and found hidden manual process too.


Collaborate to write automated specifications
Now that you have a vision, an idea of how the users are using the system and the processes they are trying to follow you can begin to look at the features the system needs to achieve the processes. These features can be broken down into user stories and then you can provide examples to specify the features. A tool like Cucumber (or SpecFlow) can work well in situations like this. Now that your specs are automated you can get that cosy feedback loop when you check with your CI system.


Iterate
Split your work up into chunks. Stop then reflect. Are we missing something? If so add it. Show your work to your users regularly. Rollout to them as soon as possible and as much as possible.

Summary
Phil's got some good ideas but his focus is different from mine. Check out his blog http://soyouthinkyouneedtorewrite.com/ for some views on rewrites and if he does his talk near you go see it. But also don't rewrite feature for feature. Take the opportunity to prune dead features and improve existing ones.

Actually all we care about.

Mary Poppendeick is a legend. She's got a fantastic paper about team rewarding, But I've just been reading her paper about lean thinking. "Wow John. Lean's so old now its in a nursing home." but hey I've just Mary's principles of lean thinking. Its nothing new if you aren't  on the agile slug trail but i kinda missed it.

So for everyone else;

here's what you are doing.

YOU ARE WASTING YOUR TIME
Mary and some guy from japan have noticed that:

The Seven Wastes of Software Development

Overproduction = Extra Features

Inventory = Requirements

Extra Processing Steps = Extra Steps

Motion = Finding Information

Defects = Defects Not Caught by Tests

Waiting = Waiting, Including Customers

Transportation = Handoffs

Can you see this in your organisation?

Nobody cares about agile but you

A poem :
I remember when I was concerned when agile was something that 'hippies did' (& me). I remember when people thought that agile meant no docs. I vaguely remember the acceptance. i don't remember the corporate adoption. I remember the scrum certification. I remember that now that the literati accept agile as a given. I remember moving jobs to an agile organisation. I remember how nothing had changed. I remember how this was not agile. I remember how we can do better. I remember the alt.net implosion. I remember the bright days of dev div. I remember the betrayal of people's hard work.  I remember getting my team to think about objects. I remember to get them to think about tests. I remember just writing code. I remember loving results. I remember loving the problem I remember  loving quality. I remember evolving design. I remember just wanting to make things better. I remember the stand ups, I remember the cynicism. I remember the hope. I remember the break through. I  try. We improve. we improve.

ORMs

There has been a backlash against ORMs. I missed it. While you were all messing around with Nhibernate. I avoided it. Why? because I can always write database queries much easier in SQL.

Always.

My mappings were much better when I mapped in with NH. I've always worked in smaller teams and much of the code has been written procedurally.

There I said it.

But this is why I like ORMs... because over several years into .net people are still writing procedural code. For the first time people were forced to think about domain objects.

and now we have EF. Ef is now getting people to think code first. (Wow welcome to the 1980s)

and now we have stuff simple.data.

ORMs were supposed to remove the impedance mistmatch between tables and code. This has only been solvee if your forward generate.

For those of us with legacy systems  all this backwards mapping is too much of a ball ache.
minimilist web frameworks don;t help and this is why I'm writing crap like SQl.data.
We don't need to worry what to call our call because its the name of the entity. we don't need to worry about joins because its the best SQL we can write.

Lets not worry about the mismatch because we can do what the devs did in the 80s. We write the best DSL to query data evar. S Q L.