Maurits thinks aloud

Writing software as easy as installing laminate flooring

At parties I usually try to avoid mentioning that I work in IT. There are a couple of reasons for this. First I often get a reaction like “Oh cool, I have this weird problem with Windows XP and my new printer. Since you are an expert, you can help me with that!” My answers range from anywhere between polite (trying to explain what I do for a living, which is not fixing Windows problems) and a simple “No, I can’t help you”.

The second question I often get is more like a remark of even an accusation: “why are software projects always late, expensive and unpredictable? By now everything is already available as standard libraries, so why is writing software not as simple a clicking existing components together?” These remarks mostly come from people whose software experience is limited to  small 100 line Visual Basic programs or from hobbyists who have tinkered a bit with Excel. Often they add examples like building a new house or bridge, arguing that this is way more difficult than building a simple piece of software and yet at the same time very predictable.

Lately I started telling those people (and others when giving a Scrum/Agile training) a story from my own experience, which is about installing laminate flooring in our house.

When I planned for this do-it-yourself activity, my first estimate was: 3 bedrooms and 1 hallway should take at most one weekend. Next I created an initial work breakdown structure (WBS):

1. Remove old carpet and floor panels: 1 hour
2. Install 50 m2 underlayment: 2 hour
3. Install 50 m2 laminate flooring: 8 hour

Nice, if I would work hard enough I could finish this on a single Saturday and have the Sunday for relaxing, spending time with my wife and kids or even write a blogpost! So I started and everything went more or less according to plan: first two steps took a little bit less, and in another hour I had installed the first 5 m2 of laminate. Since installing a floor is as easy as writing software I figured that I could scale 5 m2 to 50 m2, so that would take 10 hours instead of the planned 8. Well, not that bad.

But then I discovered that it would look way better if the laminate would be a bit under the skirting board, instead of against it which would leave some visible gaps. This was a bit of a setback since this would mean I probably couldn’t finish the job in one day. Luckily I still had the Sunday to finish the work. Then disaster hit: while removing the skirting board I discovered it was quite old and nailed into the wall with really long nails. So two things happened: part of the boards broke, while also part of the plaster felt off, damaging the walls.

Note: not my actual wall…

I decided to do a little bit of refactoring and reuse the old floor panels that I had removed from the first bedroom as skirting board. Without all the details my new WBS looked like this:

1. Remove old carpet and floor panels: 1 hour
2. Install 50 m2 underlayment: 2 hours
3. Remove old skirting board: 2 hours
4. Install 50 m2 limate flooring: 10 hours
5. Sawing 40 m floor panels into new skirting boards: 4 hours
6. Grinding 40 m skirting boards: 2 hours
7. Using a plunge router to add a nice profile to the skirting boards: 2 hours
8. Painting the skirting boards twice: 8 hours
9. Repair damaged walls: 2 hours
10. Remove remaining nails: 1 hour
11. Fixing new skirting boards to the walls: 4 hours
12. Some additional woodwork for the door posts: 8 hours

So my carefully planned 11 hours blew up to 46 hours! So that’s about 400 %. What’s worse, my initial lead-time of 1 day ultimately became 6 months. This simple seemingly  predictable set of tasks  behaved like a real software project after all with lots of unforeseen problems and new functionality during the project.

If I would have foreseen all those problems I might not have started at all. On the plus side I ended up with skirting boards that are way more beautiful than the original ones. And what’s more, I learned how to operate a plunge router, making me a better craftsman which will be useful in future projects.

Conclusion: writing software indeed is as predictable and easy as installing laminate flooring.

Generating release notes from JIRA with Google Docs

Recently I did a project using Scrum in short (one week) iterations. The acceptance testers weren’t part of the team and asked for (well, actually demanded) release notes with every increment we shipped. We told them that wasn’t a problem at all, since we keep track of all our user stories and issues in JIRA. We already had created an account for them at the start of the project, so end of story we thought. Almost.

This didn’t work out because they experienced JIRA as a bit too complicated and didn’t want to dig up all the information themselves every Friday. We realized that some basic introduction in JIRA might help but that we could help them even more by defining a couple of filters. So I asked what they needed, created the filters to come up with this information, showed them how to use them and again concluded: end of story and back to real work. Well, almost.

They still preferred to have a document containing the release notes attached to the email that announced every new release. Mainly because they had always done it like that and also because it was easier to print the document. We decided to take the path with the least resistance, use our own JIRA filters, and waste one or two hours per release to copy JIRA issues to Word, format them in tables, fight to get the layout somewhat correctly, etc. At least  some activities to give a project manager a reason for his existence. End of story. Almost.

Because as IT guys (and girls) we don’t like boring repetitive work, especially not when it has to be done late at night when we finally got that release shipped. And certainly not when we can’t see the added value of duplicating information from one format (JIRA) to another (Word). Our default solution is Yak shaving automating the work. So I came up with this set-up based on JIRA, Google Docs and some Google Apps Script:

The source is JIRA. In the past I already wrote a couple of blog posts on how to import data using Ruby (and Soap), using JavaScript (directly into a Google docs spreadsheet) or using ClojureScript (using REST). The report is based on a template that I created in Google docs. Here you can already include for example the company logo’s, the disclaimers, etc. etc.

Some sample code (note this doesn’t use JIRA) to create a document using a template:

function createDocFromTemplate() {
  var files = DocsList.find("my-template");
  return DocumentApp.openById(files[0].makeCopy("release-notes").getId());
}

As you can see in the picture I also use a Timed Trigger. This fires the script every Friday for example at 6:00 PM. I belong to the minority of people that think that distributing Word documents is not very professional (unless you want to co-author it with others) so I prefer to create a pdf. This is done in the next step. Some sample code on how to do this:

function createAttachments(doc) {
  var mimeType = "application/pdf";
  var blob = doc.getAs(mimeType);
  return [{fileName:"release-notes.pdf", mimeType:mimeType, content:blob.getBytes()}];
}

And finally we have to send the release notes to the right people. For this I created a new Group in Google Contacts. The next code snippet shows how I can read the email addresses from this group and how I create an email with the pdf as an attachment:

function sendDocumentAsPdf(doc) {
  var contacts = getRecipients();
  var attachments = createAttachments(doc);
  contacts.forEach(function(contact) {sendMail(contact, attachments);});
}

function getRecipients() {
  return ContactsApp.getContactGroup("MyGroup").getContacts();
}

function sendMail(contact, attachments) {
  var recipient = contact.getEmails()[0].getAddress();
  var subject = "Release notes";
  var body = "Please find the release notes as attachment.";

  MailApp.sendEmail(recipient, subject, body, {attachments:attachments});
}

This concludes my brief description on how to generate release notes. There is room for improvements. For example right now the email is scheduled at a fixed time. To make your reports look more ‘genuine’ (as in: a lot of work to create) you could use a ClockTriggerBuilder to generate your own triggers that fire at a more or less random time, preferably of course Friday late at night.

Final remark: it is almost always better to include testers in your team. Even at the cost of a lot of initial energy and frustration, it’s worth the end result. The solution I described is only a patch for a very bad process.

Why Scrum will never work

With such a slightly provocative title I will probably have to start with the disclaimer first: what is written here is my own opinion and not necessarily that of my employer. That is, if I still have one after posting this blog. What’s more: I’m a big fan of Scrum and other Agile methods. It pays my bills. Uh wait, let me rephrase that a bit more accurate: I’m totally 100 % convinced that Scrum works for software development.

Now with these formalities behind me let’s get a bit more serious: I really like Scrum. I have been using it for the last 5 years, I haven given several presentations about (distributed) Scrum at several conferences, I’ve written an article about it together with Scrum guru Jeff Sutherland, etc. However now that the Agile Manifesto is 10 years old I thought it would be fun to put on Edward de Bono‘s black hat and explain why Scrum is never ever going to work.

Reason 1: the cornerstone of Scrum is about trusting people. Creating a safe environment so that we can be open to each other, learn from our mistakes. And all that other touchy-feely hippy back to the 60’s stuff. That is not going to work! Did you notice my disclaimer when I started this blog? I had to put it there because quite a few people read my blog, including customers and my boss. There are a lot of pointy haired bosses out there (oh no, another disclaimer: by boss isn’t one, he is a nice friendly guy, bla bla bla). This world is full of alpha males (and females?) who are not at the least interested in you, your process, the outcome, etc. Being open is only going to hurt your career. Room for mistakes, taking risks? Don’t be naive!

Reason 2: according to Scrum ‘people do the best they can’ if you give them enough freedom. What the hell is this based upon? They don’t. They will probably do the least they can because in general most software developers are underpaid, especially compared to their managers. That’s why they want to become managers or software architects as soon as possible, since they can then still be lazy without anyone noticing and the added bonus that they are getting better paid.

Reason 3: because of the previous reason we still have to put project management on top of Scrum teams, so at least it has some output. So this is probably going to be business as usual. Assigning tasks to team members, micromanaging developers, demanding progress reports, etc. All the usual actions to slow your team down as much as possible.

Reason 4: Scrum is just a process. I have seen many processes (like CMM which is nowadays called CMMI) and I have seen them all fail and leave a lot of frustrated people behind. So if you are stuck (like most companies) with a bunch of average people then nothing is going to change. Scrum doesn’t improve your software, good people do! And by definition, you don’t have those good people since you just have Joe Average (or worse) as a programmer because your company doesn’t want to pay a decent salary.

Reason 5: Scrum delivers ‘business value’. Well no, actually it doesn’t. For many reasons. The guys or girls that know about business are not going to be involved in your project. They like to lunch with customers, not work on this weird thing called backlog to explain a bunch of introvert nerdy software developers what to do. So your team ends up with a junior help desk employee as a product owner. And besides, your whole ICT department is a cost center anyhow. So don’t start about business value.

Reason 6: an Agile team is supposed to continuously improve. That is why Scrum has retrospectives to see what went well, what can be improved and to define actions. Now do you really think people want to improve? First they have to think of possible improvement actions. Next they may even have to execute them, which might well take them way out of their comfort zone. People resist change, and therefor improvements. Your old working habits may suck, but at least they kinda work and it gets you through the day.

Reason 7: the Product Owner focusses on the ‘what’ and ‘why’ questions, the development team decides ‘how’. Nicely separated so the team can go for quality and thus high velocity on the long term. However, this is not going to work. Your product owner wants this functionality right now and doesn’t care the least about software quality. Just deliver those features as fast as possible because there always is a deadline, promises made to this important customer, etc. And don’t think you can blow away this junior product owner, because behind him is this business manager ranked high in the companies hierarchy. You as a developer are just part of a cost center and probably going to be outsourced soon anyhow. Now how is that for motivation and trust?

Reason 8: my previous point was about quality. There is some evidence that pushing productivity lowers the quality of software. On the other hand, when you focus on quality, you will get higher productivity. However Joe Average programmer doesn’t care about software quality. If the quality is poor, developing some piece of software takes more time, but why care? He is getting paid between 9 and 5. The project manager (or Scrummaster!) will take the blame for missed schedules. Even worse, if this developer is hired from another company it is in his (and his company’s) interest to stay as long as possible. So this all means that your productivity in Scrum isn’t going to be the least bit higher than with any other method.

Reason 9: “yes, but if we only build the necessary features, then at least we will have a lower total cost, right?” It never stops to amaze me how naive people are when they say something like that. You don’t build necessary features. Most of the time you are on a fixed-price contract for a major banking or insurance company. Or even worse: a government contract. They have selected you because you offered the cheapest bid (which is pretty naive in itself) but they are going to make sure that you will deliver all the requirements they have stated up-front. Of course at least 50 % of these requirements have no business value at all, but hey, you aren’t going to fool the project manager that handles the project from the customer side. He is an alpha male and you are going to deliver that last bloody feature as well!

Wow, wearing that black hat was even more fun than I thought! Must…. take…. it…. off.. now…

Effort estimates for software considered evil

Recently I had the opportunity to look at effort estimates for a big (about 4000 function points) software project. The interesting thing was that there were already quite elaborate specifications, so in theory that would be a good basis for an estimate. The estimates were done by three different parties:

1. An external consultancy firm. They counted function points based on a document with use-cases.
2. The internal business analysts.
3. The developers. They split-up the use-cases in user-stories and assigned story points.

Both the business analysts and the developers made relative estimates instead of absolute estimates in for example hours. What I had expected was a reasonable correlation (0.90 or higher) between the three estimates. This turned out to be quite different. Lets first have a look at a graph that plots the value of the estimations from the consultancy firm against the business analysts:

Consultancy firm versus business analysts

On the horizontal axis you can see the estimates in function points. On the vertical axis are the estimates from the business analysts in an arbitrary (but relative) unit. As you can see from this graph the correlation is pretty poor. In fact, when you do regression analysis, the correlation is only 0.73. This means that about 50 % of the estimates of the business analysts can be explained by the estimates from the consultancy firm. The consultancy firm at the same time claims about 15 % accuracy in their estimates.

When we look at the difference between the business analysts and the software development team we see a similar picture:

Development team versus business analysts

There are a lot less data points here because the development team only estimates those user stories that they will pick up in a next iteration of 2 weeks. Again the correlation between the two estimates is pretty poor: 0.71. So once again only 50 % of the estimates of the development team can be explained by the estimates from the business team, assuming there is a linear relation between the two.

If we correlate the estimates from the development team with the consultancy firm we get similar data. What worries me most is that the approved budget and lead time will be based on the estimates of the consultancy work which apparently bears hardly a relation with the effort that the development team has to invest. The estimated effort could easily be 100 % of more wrong. Guess who is going to be held responsible for that…