Friday, November 18, 2005

Split the team for code and test

My previous post about JUnit got me thinking some more about the subject.

Traditionally in projects I've worked on the developers write both the code and the JUnit tests. The developer will write some code, then write tests for the code he just wrote. This means that a developer will spend a portion of a project schedule writing the code, and a part of the schedule writing the JUnit tests.

I wonder if that approach can be improved. I'll use an example. Consider a software development project with six developers. One of the developers is also the team leader. The team leader both develops code himself and manages the other developers on the project. This example is somewhat based on a project earlier this year where I was the team leader with around six developers. Although this example refers to Java and JUnit, it applies to the general issue of code and formal code testing.

In the traditional model described above, all six of the developers would both write code and write JUnit tests for that code. I can immediately see drawbacks to that approach.

  • Many developers at the code. That means more communication links, more coordination, more management to do. With more developers the team leader has to spend a lot of time keeping track of who is working on what, and managing the dependencies, and making sure people who depend on one another understand what the other person is expecting.
  • When the team leader spends a lot of time managing, then he has little time for writing code. This is generally a bad thing. Typically a developer is made a team leader because he has shown technical ability in the past. If the better developers spend their time not programming and the lesser developers write most or all of the code that's not good. There's probably an antipattern around this. You want the most capable developers to write as much of the code as possible. It's hard to write code when you're managing people.
  • With split responsibility of both code and JUnit, the JUnit necessarily cuts into the time a developer can spend writing code and informal scratch testing. The code is the most important product of a development project. I'm suspicious of activities that compete with and distract from writing code.
  • Lack of visibility. If a developer writes some code, then writes some JUnit, and the code builds and the JUnit runs through successfully, well he's done right? I guess so, things seem to be done. However the developer has worked in isolation. You don't really know until you pull the pieces together if he did a good job with the code and the JUnit. After all if someone writes bad code, then the JUnit will likely also be low quality and fail to detect the problems in the bad code.
So the question is, is there a way to address these problems? The solution would need to produce a higher quality software product within a software project that runs more smoothly within the original schedule and headcount resources.

Well I think there may be a better approach than that described above. Instead of having everyone do both code and test, split the development team into two. Have a code team and a test team. The code team does all the code. They should do informal scratch and feature testing as they go, but not formal JUnit testing. The testing team does the formal JUnit tests on the code the code team writes.

Using the example above, the team of six would be a three person code team and a three person test team. The original team leader is the subteam leader code team and remains the overall team leader. One of the JUnit test team developers is appointed subteam leader of the JUnit team.

I can immediately see benefits to this.

  • There are now only three people at the code instead of six before. This is good because there is less communication and dependencies between developers. There will be less misunderstandings and disconnects. There is less communication and synchronization overhead. The code development should go more smoothly.
  • With less code developers, the team leader will spend less time managing and coordinating individuals. Thus he can spend more time writing code himself. This is good because typically the team leader is also one of the better programmers. You want your good programmers to spend as much time as possible programming.
  • With a separate JUnit team, the management burden can be partly offloaded from the overall team leader to the JUnit subteam leader. This frees up the team leader's time to focus on writing code himself and managing the more important code project.
  • There is more focus. The code people are not distracted with JUnit and are able to spend more time on the code development. The JUnit people are able to focus on doing a good job testing instead of just having it cut into more important code activity.
  • Most software development sites aren't like Google. Instead of programming superstars everywhere there's more of a mix of strong developers, reliable middle class type developers, and people who tend to require more supervision (you need to keep this third group to a minimum, max 10% of developer head count, or you're in big trouble). By splitting the team up you can put the most capable people on the code team. Thus assuring a higher quality of delivered code. The rest can do the JUnit. If they do a mediocre job it doesn't really matter since the code will be solid anyway.
  • Do you still need six people? With less synchronization and coordination among individuals, and the team leader able to spend more time programming, I don't think you do any more. You can now get the same job done with higher quality and the same schedule using only five people instead of six. Two JUnit people will be enough to do a good job. So that's a financial benefit to the company, being able to get the project done with five people instead of six in the original schedule.
  • Finally, this way actually forces a real budget to be allocated for the JUnit. I've been on projects where the schedule was overambitious, and we were just told to "fit in" the JUnit on top of doing the code, even though the schedule didn't really even have enough time to do a proper job of the code. You can guess how much JUnit got done on those projects and the quality of it. By allocating a proper budget and resources for JUnit, it is much more likely to actually be done and a better job will be done with it.

Sounds good to me so far. Of course there are some hazards to this type of plan. I can see some political type problems from this. Some people are suspicious of change in general so a departure from established ways may meet resistance just because it is a bit different.

The other issue may be in the assignment of developers between the code and JUnit teams. Some people might not like the idea of developers being on a software project but being "relegated" to unit testing and not writing production code. One way to deal with this is to mix it up over different projects and have developers be code team on some projects and JUnit team on others. Or just deal with it and use it as an incentive to the mediocre developers to improve themselves to get on the code teams.

I think it's worth trying.

Thursday, October 27, 2005

Cutdown day

Update August 2006, survived another one.

---

After a tough quarter, the expected layoff happened at the office. I'm glad that I was fortunate to survive this downsizing.

I've never actually been downsized in high tech. That is something of a rarity in this business. I've managed to survive quite a few though. Here they are that I can remember.

Date Company
======== =======
Jul 2000 xwave
Jul 2001 xwave
Oct 2001 Core Networks
Mar 2002 Core Networks - huge one
Sep 2002 Core Networks
Apr 2003 Core Networks
Aug 2004 Core Networks
Oct 2005 present job
Aug 2006 present job
Oct 2007 present job

I've been pretty fortunate to survive that many. For some reason I've wondered what it's like to be fired. What does the manager actually say to you. Then what happens? Still, I don't really want to find out first hand.

In my career when it happens it's been kind of like with Syme in 1984. People just kind of disappear during the day and are never seen again. They are generally spoken of little afterward, however they don't become full unpersons.

It was expected that this action was going to happen. We've preannounced twice in the last year, so something had to give.

When a layoff is imminent or has just happened, it makes me think of a story I heard about a guy who was a programmer for Corel. I lived in Ottawa around 1994, and at that time Corel was high flying. It seemed they could do wrong.

Well fortunes can change quickly in high tech. Corel made some strange forays into WordPerfect and Linux. Add in an aging main product line and tough competition from Microsoft. By the early 2000s Corel was struggling.

After another tough quarter, it was known that there was going to be a big layoff in the Spring for the company to survive. Now "Bill" was a programmer at Corel. He'd been there through the high flying times and was still around as the company struggled. For Bill, he'd had just about enough of life at Corel. The struggling, the abrupt changes in direction, the increasingly dysfunctional environment took their toll.

So with the big layoff coming Bill realized that he wanted to be fired. He figured he'd be better off after it happened.

  • If he was sacked he'd get severance. Severance cash in hand today was better to him than underwater stock options which might never get into the black in the future.
  • It was Springtime in Ottawa. Ottawa is nice in the Spring. After he got laid off, Bill could go on unemployment insurance (called UI in Canada) . That meant Bill would get to have a "UI Summer". In Canada the UI summer is an unusual and coveted thing. Most people get to do this never, or at most just once.
  • Bill picked up A list tech skills working at Corel. Plus he was a good worker and would have good references. He knew some people at other companies who would be willing to help him out. He wasn't concerned about being able to find another job after Corel.

Yup, Bill was looking forward to getting laid off. Severance, followed by a UI summer. Plus he wouldn't have to work at Corel any more. He was going to relax, hit the beach, not even think about going back to work until the NFL kicked off in September.

In the last three weeks Bill cut back from his regular 45-50 hours a week to a straight 40 hours a week. That was Bill's signal to managment that he wasn't going to fight to save his job and they should let him go.

Finally the expected cutdown day arrived. But then something unexpected happened. Bill survived the layoff. Well that was disappointing. Instead of severance and UI like many of his co-workers, Bill got left doing extra work to cover for those who were now gone. Plus the company was now struggling to survive so he had to work under all that uncertainty. The environment at Corel was even more unenjoyable after the layoff.

So that's where I came up with the term Corel'ed. That's when the big layoff is known to be coming. You're looking forward to getting fired because you think you'll be personally better off after you're gone, and you won't have to work at the crummy old company any more. But when the layoff does happen, you unexpectedly survive it. Then you're unhappy to survive it because you still have to work there and you miss out on the severance and other stuff you were looking forward to.

I don't know what happened to Bill after he got Corel'ed. Perhaps he's still working there.

Thursday, September 22, 2005

Google desktop search

If you don't have this then get it. It's great. No more wasting time on interminable Outlook "advanced" searches or Windows explorer file searches. Although in win explorer I was able to turn off the stupid dog, I couldn't fix the rest of that horrid user interface, nor make it any faster. I was surprised this week at work to learn that quite a few people aren't using this.

You can get it at http://desktop.google.com.

Also on Google, I've got a stack of Gmail invites if anyone is interested in getting Gmail. Go to my profile and click on the e-mail link and I'll give you a Gmail invite.

Sunday, August 28, 2005

The average user

Last night I went over to my neighbours house. He was at our house about a week ago, picking up his son who was visiting my 7 year old. Anyway he mentioned that he had saved some pictures from his digitial camera to his computer. He was trying to get them from his computer onto some blank CDs he'd bought but was having problems and getting errors. He'd heard that I know about computers and asked if I'd drop by his house and take a look.

So I went over yesterday evening. He'd been trying to burn using some Nero software or something. I stepped through the wizard with him and confirmed that it didn't work. All I did was show him how to burn his pictures to CD directly using the Windows Explorer. The burn stuff that comes free, built in with XP, is not at all bad with an acceptable enough wizard. And it worked fine and he was able to save his pictures to CD successfully. So he basically paid for this junk Nero thing that didn't work when XP already has what he needed. He'd had the computer for over two years and until last night had not been able to save to CD successfully.

So that wasn't too bad. It only took about a half hour to get his pictures burned onto CD. Most of that time was spent training him so he could do it himself the next time when I wasn't around.

All done right? Not quite. He mentioned that his computer was starting slower and slower and he was getting these annoying pop ups. Could I look at that too? Hmmm. So anyway it was not a good scene.

  • his IE start page had been changed on him to some Oemji crap
  • there was some garbage IE "search" toolbar
  • he was at XP SP1. He wasn't receiving automatic updates
  • his firewall was not turned on
  • he was getting billed every month on his credit card from classmates.com and wasn't able to cancel it
  • he was getting these popups every few minutes for some junk or other
The very first thing I did was download and install Firefox. Then I could at least have a usable browser that would bring up Google successfully without being redirected.

From there I was able to squash Oemji (there was all kinds of it), fix his IE start page, get rid of the unwanted junk toolbar, squash the periodic popups, stop the scam "spy scan" program that ran at startup. Then I put on the Google toolbar - he'd had all kinds of problems with popups in IE. I also put him on windows automatic update (not sure how it got turned off), so hopefully he'll get SP2 shortly, and turned on his firewall.

For the classmates.com thing I told him to just call his credit card company and cancel the preauthorization with them. Don't bother trying to figure out how to cancel with classmates.

So all this extra stuff took a couple of more hours, not the most enjoyable way to spend an evening. Anyway he's kind of handy so I'll have a favor to call him for where I can use some of his skill in the physical world in exchange for the big electronic favor I did him.

I strongly hinted to him that he should switch to Firefox. I've been using Firefox for over two years and during that time my start page has never been changed on me, no unwanted toolbars have ever appeared, and there's never been a popup. I can only hope he took the hint.

He showed me the bill of sale from the computer. It was from a prominent local dealer, the ones who advertise on the radio all the time. I was highly unimpressed by what I saw. They charged him over $2000 for a mediocre system. It was basically a white box clone, not tier one. Also I noticed his CD player was an HP. I doubt it was brand new since the dealer could surely have bought a clone CD player cheaper - does HP even sell CD/R direct to small shop vendors? So it was very likely a used player cannibablized off some other HP system.

Still one more thing. At one point he took his PC back to the dealer about the inability to write his pictures to CD using Nero. The dealer gave him some mumbo jumbo tech talk he didn't understand, charged him $200, and didn't even fix it. The only difference before and after taking it in for "repair" was that he was out the $200.

I wonder what would happen with my neighbour if he had a leaky faucet, then called a professional plumber to come over to fix it. I doubt he'd pay that plumber $200 if he left his house without actually fixing the faucet. Yet that's just what he did with his PC.

It's a shame because for what he paid he could have had a Mac or a nice Dell workstation. Instead he got a white box clone that for two years didn't even write his pictures to CD, which is what he bought it for. I felt a bid bad for him for what he got for what he spent. Maybe if they didn't have so many radio ads their prices could come down some.

I'm pretty sure that most computer users are around his skill level. I'm also pretty sure a person who has only basic tech skills and knows how to use Google could make an OK living doing what I did, coming into people's houses and fixing IE start pages. Not that I would want to do that for money, no thanks.

The problem with doing it for money would be that you would go in and fix all the stuff, then two months later the user would be calling you saying that their IE start page has been changed again, the popups are back, and they're getting redirected off Google. Except when they call you back they'll say it's your fault and want you to come and fix it again, this time for free. So no, I'll stick with professional programming.

One thing I found interesting was my neighbour could not understand the difference between the Google start page and the IE address bar. He couldn't understand what the address bar was for or how it was different than typing something into Google. I tried to explain that the address bar will go straight to the site you want. But he just wasn't grasping it, I guess it is a subtle thing to many users. When I started his IE last night, the address bar wasn't even turned on.

What he likes to do is use Google as his IE start page. From Google he just types what he wants, then clicks the first link in the results. So if he wanted to go to Ebay, he would type ebay.com in Google, then use that to get to Ebay.

I once read that people go to Google, then type amazon.com and click through from there to Amazon. A guy from Google said he couldn't understand why people do that. I've now explained here why. People either have their address bar in IE turned off or pushed to the right, or don't know what the address bar is or how to use it. So they use trusty Google as their default IE start page, then type in where they want to go then click through from the Google results page. I can't imagine how much money Google must be making off selling that top link in the results. I can tell you that probably most users don't realize it's a sponsored link.

Wednesday, August 24, 2005

More Python

We're doing some stuff at work where we need to get the Base64 encoding of the SHA-1 hash of a password. Here's how to obtain it in Python


import sha1
import base64

print base64.encodestring(sha.new('password').digest())


That's it. One line of code. No download of 3rd party. No worrying about licensing or purchasing something.

Try it in C or Java. In Java the JDK sees fit to provide neither SHA-1 or Base64. So write your own or try to download or purchase something.

In C there's RFC 3174 for SHA-1. You can try to get it to compile on your system and get it into your build. Have fun with that, I did a couple of years ago. Like Java, you'll have to thrash around the Internet for Base64 or write it yourself.

Compare all that hassle with C/Java, compared to one line in Python using built in components. Python is just better.

Saturday, August 20, 2005

Value of JUnit?

I just wrapped up my part in a project which has now gone into testing. It went pretty well. I didn't do any JUnit stuff. I haven't done any JUnit in more than a year.

I actually like JUnit. It's a good way to generate automated tests. It is pretty easy to learn and use. So if there was more time then I'd be motivated to use it.

What should you do if there isn't quite enough time for JUnit? Don't get me wrong, I test and the code I write tends to have a low defect rate.

JUnit is kind of the "formal" test for the code. This type of test is a permanent type of test involving planning, source control of test code, potential peer review, ongoing maintenance. So there's overhead involved.

For me, JUnit is the third level of testing. The first level is to print the code after I write it, then go over it carefully looking for errors. I find most errors through this static self inspection. For schedule reasons we don't do peer review, which is unfortunate because I am a fan of peer review.

The second level is informally running the code. Create a scratch J2SE client program that exercises the new code. Or click on the screen that will invoke the new code and check that it works as expected. The important thing is to exercise the new code in scenarios that are as close to possible to how it will be invoked in production. This will give you confidence that the code will be OK in black box integration testing.

The third level is formally testing the code within a controlled environment like JUnit, where the formal tests go into source control and are maintained along with the actual code.

So the JUnit level is basically a formal retest of code that you know works. My experience has been that the effort for levels 1 and 2 (static code inspection, informal scratch testing) is a fraction of that required to do JUnit. So it is possible to ship robust, tested code without requiring the overheads of JUnit. So focus on checking your code and scratch testing, and only do the JUnit at the end if there is time.

An issue I do have with JUnit is that it tests code in unrealistic scenarios that will not happen in production, while missing scenarios that will happen in production. Earlier this year, I was team lead on a project. This project involved a Tomcat WAR that processed a formatted URL and sent a message over JMS to a J2EE server that invoked an EJB call that did a bunch of stuff. The code was written and ready to test.

The developer responsible for that section wanted to set up JUnit for the different pieces. I said don't bother with JUnit, do an end to end test instead. Set up the components, invoke the Tomcat URL, and verify from the database and the logs that it works end to end. Well he did what I asked, then he spent several days sorting out deployment issues, permission issues and glitches between the components. But he got it working and it worked when we went to production. We never did do the JUnit for that, and I'm glad we didn't. The issues the developer found would only have been uncovered by doing an end to end test of the running system. If we'd done JUnit instead, we would have looked bad as the problems would have been found by the test team.

In my opinion the bulk of value in JUnit is for refactoring. It's also valuable for automated test of working code. The automated test is basically a form of refactoring test. You take something that is verified to work with JUnit. Then if someone inadvertantly breaks your code by changing something you call, then the automated test will catch it. But code has to exist and work before it can be refactored. And refactoring of new code generally only happens after the current project ends. Also let the person doing the refactoring absorb the cost of JUnit, since he is the one who benefits from it.

My observation is that there is no link at the individual level between use of JUnit and defect rate. Some mediocre developers are JUnit fans, yet it doesn't seem to improve the quality of code they produce. Some developers write lots of good, low defect code while spending little time on JUnit.

My theory is that strong developers will write good code and if they do JUnit then they'll do a good job of that too. But for the good developers the JUnit is not really necessary as the code is high quality regardless. On the other hand, weak developers will write mediocre code. The JUnit will be of similar lower quality and there will be more defects regardless of how much time they spend on JUnit.