I remember the heyday of Joel on Software back in the early 2000s. It was a great site with excellent articles coming out every month. The forums were good too.
One of the major posts with remarkable sticking power was the Joel Test. At the time back in 2000 it was like wow a way to quantify how good your team was. And Microsoft, the top of the heap, does all 12. Joel was probably right that most teams were about a 2 or 3 at that time.
Today I'd say the industry has matured and with tools like Jenkins and Bugzilla and git available for free. Most teams I've been on over the years I'd say are around 7-10 on this scale. and scores have generally improved over the years. culturally I think a lot of that stuff is now standard and most teams, even mediocre teams, would just put a lot of that stuff in on their own now without having to be told to or getting resistance from management. as noted above since the stack is generally free it helps a lot that it only takes one good developer who cares about it go establish a pretty good stack off some spare Linux VM somewhere at no cost to the firm.
So using the Joel test as a starting point, what are some guidelines to today's software development teams. well here goes
1. do you use visual tools?
2. do you have automated testing?
3. do you use static source code analysis?
4. is all code reviewed?
5. do you build on every check in?
6. do you level the workflow?
7. do you use one piece flow?
1. do you use visual tools?
For your current project, how much stuff is in development? how much in code review? how much with the test team? how many requirements and defects are in the project scope but not yet assigned to a developer? at this moment how many issues on average are assigned to each developer?
With visual tools the process runs more smoothly as problems such as open code reviews piling up, or a negative find vs. fix ratio on defects are obvious and public. this allows issues to be identified and addressed much earlier, before it is a crisis.
2. do you have automated testing?
Since we work with computers let's get the computers to do the testing. there are some things computers are good at and software testing is one of them.
with powerful modern tools like powermock and mockito many of the historic barriers that made automated testing difficult or painful are no more. in general developers will write code. with automated testing we get development on board to write self-testing code
3. do you use static source code analysis?
Tools like Coverity and Klocwork are awesome. they find the really hard stuff, like did you know this API can return null on some obscure but possible scenarios. what happens in our code then? something the programmer overlooked but the static analysis found.
as well they find the "easy" stuff - uh this will NPE, oops. quick fix yay
4. is all code reviewed?
code review done properly is an effective and cost efficient way to find and fix runtime software errors at the early stage, when it is still very inexpensive to fix them.
5. do you build on every check in?
if someone checked in code that doesn't compile for whatever reason (such as forgetting to add a new file to the repo), then it should be made known so it can be fixed ASAP.
this checking is also part of the automated testing. after compiling the code it makes sense for the computer to also run the automated tests since it's free at that point. again if a developer broke or possibly broke something then everyone should know about it right away so it can be fixed right away
back at a previous company there was an informal rule "never sync before 10 AM" that was because to sync is to pick up whatever stuff that didn't compile and didn't work had been checked in since yesterday. by waiting until 10 you could be sure that the offender was in and after spending time manually investigating the problem you could ask him to look at it and fix. so in the old days when someone checked in bad code it typically wasn't noticed or fixed until the next day.
6. do you level the workflow?
a reason to use visual tools is to identify lumps or bottlenecks in the workflow. if code review is being neglected and backed up then announce "halt production of new code until the reviews are cleaned up". so many years I've heard people throw the term "waterfall" around like some kind of stick to hit people with. but the team has to be willing to do something to combat waterfall, which is something of a natural entropy state. using techniques like Kanban and lean to level out the amount of outstanding work at each station in the software development process
7. do you use one piece flow?
Queuing is toxic to software development. queues destroy value stream productivity. if you look at the lifecycle of some feature or defect reported, on many teams you will find that the work spent the vast majority of its time idle, sitting in some queue making zero progress while waiting for a tiny amount of attention from someone before going into the next queue and being idled again
some defect is reported. it sits until the 10 AM triage meeting the next day. assigned to some developer who already has multiple assigned issues. it sits idle for a few days until the developer gets to it. developer codes and tests a fix in a couple of hours. developer creates a code review. it waits for the reviewers to get to it, more than a day passes to do the review which was an hour of actual work. developer checks in. the official build isn't until next week so the work is stalled, done but not delivered to the test team. a week later there's the build. the item is one of a large batch of items delivered in the build. the test team will get to it when they get to it perhaps a few days later after running through the full regression first. it takes half an hour for the tester to retest and confirm the fix
now with one piece flow these disconnects, queues, asynchronous handoffs and large batches of work are sharply reduced or eliminated. once work on some item begins it proceeds expeditiously with little or no delays between each station.
Tuesday, September 17, 2013
Monday, September 09, 2013
cupcakes and software development
I've been hinting that software development and physical manufacturing processes are more closely related than we may have realized. To help illustrate this I'll describe a simplified cupcakes boutique and attempt to map that process to a software development shop.
In the cupcakes shop a customer such as say an accounting firm calls in with an order for a batch of custom cupcakes. They may order 20 or 60 or whatever. The customer chooses from a selection of icing colours and optional sprinkles topping. In this simplified shop orders come in complete and correct.
The order appears on the shop floor behind the walk in counter. A baker mixes the batch of batter into the cupcake trays and puts it in the oven. After it bakes the trays are moved to the icing station. For quality purposes a different baker checks the batch, then adds the icing and sprinkles. Every baker is trained on both the mixing and icing phases and works interchangeably on both.
From there the cupcakes go to the wrap station where another quality check is done on the work. the cellophane wrap is added and from there onto the pick up area. the customer is phoned to alert them their order is ready to be picked up.
at any stage if a quality error is detected then the batch is sent back to the previous station, or the batch may need to be discarded and redone if the error is found late and not easily fixed.
--
This maps to my development site as follows. customer orders represent change requests from product management and the executive, as well as defects found by testers.
The mixing phase represents design and coding. The icing station is code review. The wrapping is testing by the test team.
So far so good. Now with this physical model we can hopefully better analyze the more virtual world of software, identify waste and expose the real value stream.
Suppose work comes into development from a triage process. Suppose triage is only done once a day at 10 AM. hmmm that means that from the time a customer calls in the order, up to 23 hours may go by with no progress and nobody even looking at or starting on the order. Perhaps the team should move closer to continuous triage using the 80/20 rule as most of triage is routine and can be done by a senior developer. only a fraction of the issues would need the daily conference call among several principles
The baking and icing. Suppose for whatever reason the bakers in the kitchen enjoy the mixing and baking part, while they enjoy the icing station less. perhaps it's about what they see as "their" work vs. "others" work. perhaps management is much more diligent about monitoring and measuring and rewarding batches baked, while the icing phase is watched far less closely.
now in the bakery we have "visual tools" in that a foreman can see if things are getting messy and backed up at the icing station and cupcakes are going stale. he can order bakers to stop baking new batches until the situation in the icing is cleaned up.
in software over the years I've observed a tendency to really focus on coding and the assignment and delivery of code, while code review is generally tackled reluctantly, at the end, until ordered by management. I've never met a developer who had to be ordered to write code. so in software the "neglected code review" anti patten can be addressed by the use of visual tools, as well as management placing equal emphasis and attention on code review as they do on the writing of code.
onto the wrapping. suppose the baking and icing phase tends to send the batches through based on each order. now in wrapping they prefer to let a few baked orders accumulate and do a big batch wrap as that is more efficient for them. so again the work is done and just stalled in some queue making no progress and getting staler. from the value stream perspective the company only gets paid when the work completes and is delivered. so having cupcakes stuck in wrap and not moving is costing the company money in the short term and customers in the long term.
to the software developer, once the code is checked in they want it tested as soon as possible. that way if there is testing feedback the developer gets it while the matter is still fresh in his mind. also again the dev team as a whole including test only gets credit for delivery when it's all done. so in software there's an opportunity to again use visual tools to see if there are a large number of items stuck in test awaiting verification. also to measure the takt time from when some item is available from development into test, and when the testing completes. this takt time analysis is a powerful tool as it can also be used in code review to measure how long in clock time from when the review was created to when it completed.
Pulling it together, suppose your software development shop was instead a cupcakes boutique. A customer calls at 11 AM on a Monday with an order for 36 cupcakes with powder blue icing and red sprinkles. How long would it take your shop to call the customer to come pick up his finished order. later that same day? the next day? a week, usually stale, with 20% deviation rate from the order instructions? at least a month? would your software shop be able to survive in a type of business where swift delivery with high quality is required?
In the cupcakes shop a customer such as say an accounting firm calls in with an order for a batch of custom cupcakes. They may order 20 or 60 or whatever. The customer chooses from a selection of icing colours and optional sprinkles topping. In this simplified shop orders come in complete and correct.
The order appears on the shop floor behind the walk in counter. A baker mixes the batch of batter into the cupcake trays and puts it in the oven. After it bakes the trays are moved to the icing station. For quality purposes a different baker checks the batch, then adds the icing and sprinkles. Every baker is trained on both the mixing and icing phases and works interchangeably on both.
From there the cupcakes go to the wrap station where another quality check is done on the work. the cellophane wrap is added and from there onto the pick up area. the customer is phoned to alert them their order is ready to be picked up.
at any stage if a quality error is detected then the batch is sent back to the previous station, or the batch may need to be discarded and redone if the error is found late and not easily fixed.
--
This maps to my development site as follows. customer orders represent change requests from product management and the executive, as well as defects found by testers.
The mixing phase represents design and coding. The icing station is code review. The wrapping is testing by the test team.
So far so good. Now with this physical model we can hopefully better analyze the more virtual world of software, identify waste and expose the real value stream.
Suppose work comes into development from a triage process. Suppose triage is only done once a day at 10 AM. hmmm that means that from the time a customer calls in the order, up to 23 hours may go by with no progress and nobody even looking at or starting on the order. Perhaps the team should move closer to continuous triage using the 80/20 rule as most of triage is routine and can be done by a senior developer. only a fraction of the issues would need the daily conference call among several principles
The baking and icing. Suppose for whatever reason the bakers in the kitchen enjoy the mixing and baking part, while they enjoy the icing station less. perhaps it's about what they see as "their" work vs. "others" work. perhaps management is much more diligent about monitoring and measuring and rewarding batches baked, while the icing phase is watched far less closely.
now in the bakery we have "visual tools" in that a foreman can see if things are getting messy and backed up at the icing station and cupcakes are going stale. he can order bakers to stop baking new batches until the situation in the icing is cleaned up.
in software over the years I've observed a tendency to really focus on coding and the assignment and delivery of code, while code review is generally tackled reluctantly, at the end, until ordered by management. I've never met a developer who had to be ordered to write code. so in software the "neglected code review" anti patten can be addressed by the use of visual tools, as well as management placing equal emphasis and attention on code review as they do on the writing of code.
onto the wrapping. suppose the baking and icing phase tends to send the batches through based on each order. now in wrapping they prefer to let a few baked orders accumulate and do a big batch wrap as that is more efficient for them. so again the work is done and just stalled in some queue making no progress and getting staler. from the value stream perspective the company only gets paid when the work completes and is delivered. so having cupcakes stuck in wrap and not moving is costing the company money in the short term and customers in the long term.
to the software developer, once the code is checked in they want it tested as soon as possible. that way if there is testing feedback the developer gets it while the matter is still fresh in his mind. also again the dev team as a whole including test only gets credit for delivery when it's all done. so in software there's an opportunity to again use visual tools to see if there are a large number of items stuck in test awaiting verification. also to measure the takt time from when some item is available from development into test, and when the testing completes. this takt time analysis is a powerful tool as it can also be used in code review to measure how long in clock time from when the review was created to when it completed.
Pulling it together, suppose your software development shop was instead a cupcakes boutique. A customer calls at 11 AM on a Monday with an order for 36 cupcakes with powder blue icing and red sprinkles. How long would it take your shop to call the customer to come pick up his finished order. later that same day? the next day? a week, usually stale, with 20% deviation rate from the order instructions? at least a month? would your software shop be able to survive in a type of business where swift delivery with high quality is required?
Monday, September 02, 2013
code review: runtime errors only
I've continued to think about code review and how it should fit into the software development process. By applying concepts like value stream and one piece flow we can determine what the useful outputs of code review are. From there we can determine how to better focus code reviews, and even if we should be spending the time and effort to have code review at all.
Probably the first question is if we should even be doing code review. Would we be better off to just skip code review and thus be able to move faster in development? I still believe that done right there is positive return in the time and effort spent on code review.
The value is in identifying and fixing runtime errors. As is well documented the cost of dealing with defects rises exponentially the further along the code is before the error is found. With code review we can still fix defects very quickly and inexpensively. The code isn't in a build yet and it hasn't gone to any testers or users outside development. This is probably the last point where real errors can be quickly fixed with a minimum of cost and drama.
That is the entire value of code review. Anything else is waste. I've seen perfectly good code that works rejected and sent back for rework with bogus subjective "defects" that are just noise. If it's not a runtime error then it's not a defect.
Code review can be guided by these two simple bullet points. There are two purposes to code review
thus rejecting code and sending back for rework should be avoided. it should only be done for runtime errors, where the cost of a code review iteration is much less than if the error is exposed at a later stage.
I know that some believe that code review is an opportunity for coaching or teaching. This is misguided. On the teams I've spent my career on everyone is always experienced. 2, 5, 10, 15+ years of software development experience. at this point if the developer hasn't figured out or isn't motivated to write good, clean, fast, maintainable code then he's just never going to. and there's nothing code review can do about it.
maybe if you're dealing with interns, entry level people in their first year, or people working with some new to them technology that's about the only time there's use in pointing out a better (in your subjective opinion) way to do something.
Probably the first question is if we should even be doing code review. Would we be better off to just skip code review and thus be able to move faster in development? I still believe that done right there is positive return in the time and effort spent on code review.
The value is in identifying and fixing runtime errors. As is well documented the cost of dealing with defects rises exponentially the further along the code is before the error is found. With code review we can still fix defects very quickly and inexpensively. The code isn't in a build yet and it hasn't gone to any testers or users outside development. This is probably the last point where real errors can be quickly fixed with a minimum of cost and drama.
That is the entire value of code review. Anything else is waste. I've seen perfectly good code that works rejected and sent back for rework with bogus subjective "defects" that are just noise. If it's not a runtime error then it's not a defect.
Code review can be guided by these two simple bullet points. There are two purposes to code review
- identify runtime errors
- move the unit of work along as quickly as possible
thus rejecting code and sending back for rework should be avoided. it should only be done for runtime errors, where the cost of a code review iteration is much less than if the error is exposed at a later stage.
I know that some believe that code review is an opportunity for coaching or teaching. This is misguided. On the teams I've spent my career on everyone is always experienced. 2, 5, 10, 15+ years of software development experience. at this point if the developer hasn't figured out or isn't motivated to write good, clean, fast, maintainable code then he's just never going to. and there's nothing code review can do about it.
maybe if you're dealing with interns, entry level people in their first year, or people working with some new to them technology that's about the only time there's use in pointing out a better (in your subjective opinion) way to do something.
Subscribe to:
Posts (Atom)