the trouble with JSF

At work the last year has been back to the future in development. After a reorg in 2012 the DI team morphed into a disastrous and miserable time with some larger devops team who didn't want us, and we didn't want to be with them. I returned to Java server development in January of this year. In March of this year there was another reorg and devops vanished and the Halifax branch was swept out the door. So I narrowly avoided that fate. devops is not missed.

It's been a good year back in the saddle of development. It's where I belong and I can still produce quality design and code. I've always liked Java and I had some good Java years back at SupportSoft. This past year was a good year.

At this point my plan is pretty much to try stick with Java as the go forward plan. Even if the creation of new Java code suddenly and unexpectedly ceased the existing Java base is massive and there should be enough legacy out there to allow me to run out the remaining years of my development career. We'll see. There's still quite a few years to go on that!

So my dev team has a .java backend with a front end website. Most of the action happens in the backend but the website isn't exactly small. The website has been around for several years with the typical nooks and crannies and special features. It is built using Seam and JSF. I was at the website a bit but mostly I worked in the .java part. It's not an area of interest to me and there were other developers who were more interested in UI.

So after observing and being around the JSF for a year, writing some JSF code and being on code reviews, I can finally articulate some things that had been in the back of my mind about Java web.

The thing about JSF is this: where is the Java? You try to do anything and it's all .xml, .xhtml, .js. .css. etc. You virtually never touch a .java file. Why would Sun/Oracle spend all these years promoting a non-Java technology stack within the Java standards? It doesn't make any sense.

There are so many things wrong with this strategy. Java developers don't really mind doing UI. What they mind is being required to leave behind their years of skill and experience in Java to work with this completely different stack(s) that has nothing to do with Java. So we just don't really want to leave Java and why would we. And it is so painful having to learn all these multiple and large foreign syntaxes and terminologies just to get a basic screen to come up. And don't even start on "this code is correct but it doesn't work on version X of browser Y running on desktop OS or mobile phone Z"

So what happens is the Java development team becomes split. Those who do UI and those who don't. That's what happened at SupportSoft. This should be so avoidable. The strength of Java is that it's generally easy for a Java developer to move into a new area such as database, crypto, XML processing, business logic, etc. With JSF because it's not really Java there's this huge barrier with large up front learning curve and no easy transition.


For a long time this JSP/JSF situation was something that developers were just expected to shrug and passively accept. Kind of like the way it was with EJB back in J2EE. After all it's the "standard" way so too bad. Luckily some others realized before I did that the JSF approach is fundamentally flawed and going in a new direction away from a broken standard is better.

You can see some of that on the GWT overview page. This is basically the first words out of their mouth

... goal is to enable productive development of high-performance web applications without the developer having to be an expert in browser quirks, XMLHttpRequest, and JavaScript

It's like, wow, the anti JSF. productive development. not having to be expert in browser quirks and JS. you also don't have to leave Java. you know, Java, that thing you invested years of your career mastering and promoting. It's so bizarre that Oracle/Google lawsuit over Java. Google has done more than pretty much anyone to preserve, promote and advance Java.

thoughts on the Obamacare rollout

Along with the "tech surge" I heard Google and Oracle have volunteered personnel to help with the healthcare.gov site. hmmm this seems to violate Brooks's law which states that "adding manpower to a late software project makes it later." So now time will be lost answering questions and bringing all these new people up to speed.

I read Obamacare came in at 500 million lines. what is the system doing that requires so much code? By comparison Windows8 came in around 80 million lines. So this system is really six times bigger and more complex?

I'd like to see the looks on the Google and Oracle developers faces when they see the Obamacare source code for the first time. Welcome to corporate/government IT!

I'm confident CGI already has a very deep understanding of agile. The probability that the issues can be traced to a lack of agile, or misimplementation of agile, is basically about zero.

A lot of the problem can be traced to the immovable deadline. If X million must be signed up by end of March 2014, then the site must go live no later than end of September 2013. So they went live because they had to and they obviously weren't ready or particularly close to ready. So it's just another big government IT project, late and over budget. But with the hard launch date there's nowhere to hide and no way to slip on the delivery.

It's easy to vilify CGI. To be fair, the go live deadline wasn't their estimate. It was Congress who dictated when the law was enacted that a 500 million line system would be built and ready in Z months - with apparently no consultation with the people who would build this system if this was realistic. CGI tried but it may have been an impossible mission.

sent home, to work

There was a big local downsizing last week. The entire office is being closed, the remaining four occupied floors from an office built in 2008. ah the brand new building, the high tech kiss of death. I guess it was to be our contribution to an overall 40% global RIF.

A fraction of the locals, around 10%, were told they will be working from home in January after the office closes. I was among that group. Statistically at least it was a bit of a surprise. Although it's more like still in the running for January rather than a lock. There's new management scheduled to come in early November so who knows what decisions they will make. At this point the plan is to start work from home in January.

In some ways I'm looking forward to it. I won't miss the commute in winter. To think about it, how much time and money has been wasted on commuting over the years. I know some people who work from home and they speak well of it. In some ways work from home has executive perks including

• work location chosen to optimize your commute
• private office
• personal printer
• private bathroom
• own coffee machine
• comfortable high backed manager's chair (go for it)

The company has been pretty good about it helping to get the space set up. That will still take some thinking where to fit the new desk at my place. Overall I see it as a positive change. Some things will be different and a bit harder but overall to me it seems like it's the same work so why not do it from home.

a modern software team test

I remember the heyday of Joel on Software back in the early 2000s. It was a great site with excellent articles coming out every month. The forums were good too.

One of the major posts with remarkable sticking power was the Joel Test. At the time back in 2000 it was like wow a way to quantify how good your team was. And Microsoft, the top of the heap, does all 12. Joel was probably right that most teams were about a 2 or 3 at that time.

Today I'd say the industry has matured and with tools like Jenkins and Bugzilla and git available for free. Most teams I've been on over the years I'd say are around 7-10 on this scale. and scores have generally improved over the years. culturally I think a lot of that stuff is now standard and most teams, even mediocre teams, would just put a lot of that stuff in on their own now without having to be told to or getting resistance from management. as noted above since the stack is generally free it helps a lot that it only takes one good developer who cares about it go establish a pretty good stack off some spare Linux VM somewhere at no cost to the firm.

So using the Joel test as a starting point, what are some guidelines to today's software development teams. well here goes

1. do you use visual tools?
2. do you have automated testing?
3. do you use static source code analysis?
4. is all code reviewed?
5. do you build on every check in?
6. do you level the workflow?
7. do you use one piece flow?

1. do you use visual tools?
For your current project, how much stuff is in development? how much in code review? how much with the test team? how many requirements and defects are in the project scope but not yet assigned to a developer? at this moment how many issues on average are assigned to each developer?

With visual tools the process runs more smoothly as problems such as open code reviews piling up, or a negative find vs. fix ratio on defects are obvious and public. this allows issues to be identified and addressed much earlier, before it is a crisis.

2. do you have automated testing?
Since we work with computers let's get the computers to do the testing. there are some things computers are good at and software testing is one of them.

with powerful modern tools like powermock and mockito many of the historic barriers that made automated testing difficult or painful are no more. in general developers will write code. with automated testing we get development on board to write self-testing code

3. do you use static source code analysis?
Tools like Coverity and Klocwork are awesome. they find the really hard stuff, like did you know this API can return null on some obscure but possible scenarios. what happens in our code then? something the programmer overlooked but the static analysis found.

as well they find the "easy" stuff - uh this will NPE, oops. quick fix yay

4. is all code reviewed?
code review done properly is an effective and cost efficient way to find and fix runtime software errors at the early stage, when it is still very inexpensive to fix them.

5. do you build on every check in?
if someone checked in code that doesn't compile for whatever reason (such as forgetting to add a new file to the repo), then it should be made known so it can be fixed ASAP.

this checking is also part of the automated testing. after compiling the code it makes sense for the computer to also run the automated tests since it's free at that point. again if a developer broke or possibly broke something then everyone should know about it right away so it can be fixed right away

back at a previous company there was an informal rule "never sync before 10 AM" that was because to sync is to pick up whatever stuff that didn't compile and didn't work had been checked in since yesterday. by waiting until 10 you could be sure that the offender was in and after spending time manually investigating the problem you could ask him to look at it and fix. so in the old days when someone checked in bad code it typically wasn't noticed or fixed until the next day.

6. do you level the workflow?
a reason to use visual tools is to identify lumps or bottlenecks in the workflow. if code review is being neglected and backed up then announce "halt production of new code until the reviews are cleaned up". so many years I've heard people throw the term "waterfall" around like some kind of stick to hit people with. but the team has to be willing to do something to combat waterfall, which is something of a natural entropy state. using techniques like Kanban and lean to level out the amount of outstanding work at each station in the software development process

7. do you use one piece flow?
Queuing is toxic to software development. queues destroy value stream productivity. if you look at the lifecycle of some feature or defect reported, on many teams you will find that the work spent the vast majority of its time idle, sitting in some queue making zero progress while waiting for a tiny amount of attention from someone before going into the next queue and being idled again

some defect is reported. it sits until the 10 AM triage meeting the next day. assigned to some developer who already has multiple assigned issues. it sits idle for a few days until the developer gets to it. developer codes and tests a fix in a couple of hours. developer creates a code review. it waits for the reviewers to get to it, more than a day passes to do the review which was an hour of actual work. developer checks in. the official build isn't until next week so the work is stalled, done but not delivered to the test team. a week later there's the build. the item is one of a large batch of items delivered in the build. the test team will get to it when they get to it perhaps a few days later after running through the full regression first. it takes half an hour for the tester to retest and confirm the fix

now with one piece flow these disconnects, queues, asynchronous handoffs and large batches of work are sharply reduced or eliminated. once work on some item begins it proceeds expeditiously with little or no delays between each station.

cupcakes and software development

I've been hinting that software development and physical manufacturing processes are more closely related than we may have realized. To help illustrate this I'll describe a simplified cupcakes boutique and attempt to map that process to a software development shop.

In the cupcakes shop a customer such as say an accounting firm calls in with an order for a batch of custom cupcakes. They may order 20 or 60 or whatever. The customer chooses from a selection of icing colours and optional sprinkles topping. In this simplified shop orders come in complete and correct.

The order appears on the shop floor behind the walk in counter. A baker mixes the batch of batter into the cupcake trays and puts it in the oven. After it bakes the trays are moved to the icing station. For quality purposes a different baker checks the batch, then adds the icing and sprinkles. Every baker is trained on both the mixing and icing phases and works interchangeably on both.

From there the cupcakes go to the wrap station where another quality check is done on the work. the cellophane wrap is added and from there onto the pick up area. the customer is phoned to alert them their order is ready to be picked up.

at any stage if a quality error is detected then the batch is sent back to the previous station, or the batch may need to be discarded and redone if the error is found late and not easily fixed.

--

This maps to my development site as follows. customer orders represent change requests from product management and the executive, as well as defects found by testers.

The mixing phase represents design and coding. The icing station is code review. The wrapping is testing by the test team.

So far so good. Now with this physical model we can hopefully better analyze the more virtual world of software, identify waste and expose the real value stream.

Suppose work comes into development from a triage process. Suppose triage is only done once a day at 10 AM. hmmm that means that from the time a customer calls in the order, up to 23 hours may go by with no progress and nobody even looking at or starting on the order. Perhaps the team should move closer to continuous triage using the 80/20 rule as most of triage is routine and can be done by a senior developer. only a fraction of the issues would need the daily conference call among several principles

The baking and icing. Suppose for whatever reason the bakers in the kitchen enjoy the mixing and baking part, while they enjoy the icing station less. perhaps it's about what they see as "their" work vs. "others" work. perhaps management is much more diligent about monitoring and measuring and rewarding batches baked, while the icing phase is watched far less closely.

now in the bakery we have "visual tools" in that a foreman can see if things are getting messy and backed up at the icing station and cupcakes are going stale. he can order bakers to stop baking new batches until the situation in the icing is cleaned up.

in software over the years I've observed a tendency to really focus on coding and the assignment and delivery of code, while code review is generally tackled reluctantly, at the end, until ordered by management. I've never met a developer who had to be ordered to write code. so in software the "neglected code review" anti patten can be addressed by the use of visual tools, as well as management placing equal emphasis and attention on code review as they do on the writing of code.

onto the wrapping. suppose the baking and icing phase tends to send the batches through based on each order. now in wrapping they prefer to let a few baked orders accumulate and do a big batch wrap as that is more efficient for them. so again the work is done and just stalled in some queue making no progress and getting staler. from the value stream perspective the company only gets paid when the work completes and is delivered. so having cupcakes stuck in wrap and not moving is costing the company money in the short term and customers in the long term.

to the software developer, once the code is checked in they want it tested as soon as possible. that way if there is testing feedback the developer gets it while the matter is still fresh in his mind. also again the dev team as a whole including test only gets credit for delivery when it's all done. so in software there's an opportunity to again use visual tools to see if there are a large number of items stuck in test awaiting verification. also to measure the takt time from when some item is available from development into test, and when the testing completes. this takt time analysis is a powerful tool as it can also be used in code review to measure how long in clock time from when the review was created to when it completed.

Pulling it together, suppose your software development shop was instead a cupcakes boutique. A customer calls at 11 AM on a Monday with an order for 36 cupcakes with powder blue icing and red sprinkles. How long would it take your shop to call the customer to come pick up his finished order. later that same day? the next day? a week, usually stale, with 20% deviation rate from the order instructions? at least a month? would your software shop be able to survive in a type of business where swift delivery with high quality is required?

code review: runtime errors only

I've continued to think about code review and how it should fit into the software development process. By applying concepts like value stream and one piece flow we can determine what the useful outputs of code review are. From there we can determine how to better focus code reviews, and even if we should be spending the time and effort to have code review at all.

Probably the first question is if we should even be doing code review. Would we be better off to just skip code review and thus be able to move faster in development? I still believe that done right there is positive return in the time and effort spent on code review.

The value is in identifying and fixing runtime errors. As is well documented the cost of dealing with defects rises exponentially the further along the code is before the error is found. With code review we can still fix defects very quickly and inexpensively. The code isn't in a build yet and it hasn't gone to any testers or users outside development. This is probably the last point where real errors can be quickly fixed with a minimum of cost and drama.

That is the entire value of code review. Anything else is waste. I've seen perfectly good code that works rejected and sent back for rework with bogus subjective "defects" that are just noise. If it's not a runtime error then it's not a defect.

Code review can be guided by these two simple bullet points. There are two purposes to code review

• identify runtime errors
• move the unit of work along as quickly as possible

the cost of sending code back for rework is very high. each iteration requires more design, more dev testing, another iteration of code review. and the time spent on the original rejected material is partially or fully wasted. additionally there is a corrosive hidden cost that if a developer has to revisit something then of course he is pulled away from some other value-add activity which could advance the product

thus rejecting code and sending back for rework should be avoided. it should only be done for runtime errors, where the cost of a code review iteration is much less than if the error is exposed at a later stage.


I know that some believe that code review is an opportunity for coaching or teaching. This is misguided. On the teams I've spent my career on everyone is always experienced. 2, 5, 10, 15+ years of software development experience. at this point if the developer hasn't figured out or isn't motivated to write good, clean, fast, maintainable code then he's just never going to. and there's nothing code review can do about it.

maybe if you're dealing with interns, entry level people in their first year, or people working with some new to them technology that's about the only time there's use in pointing out a better (in your subjective opinion) way to do something.

The Toyota Way by Jeffrey K. Liker

I finished the second suggested reading from work. It was The Toyota Way by Jeffrey K. Liker. This is the second book I got from a former VP at work, after Scaling Lean & Agile Development.

It was a pretty good book. As advertised on the cover there are 14 Toyota Way principles. The author describes all fourteen. The author knows his stuff and demonstrates several of the Toyota principles in writing the book such as "go see", and "towering technical knowledge".

It comes in at around 300 pages. With 14 principles to cover in detail it was lot of ground to cover. The first 100 pages were pretty good. The next 150 or so dragged a bit and it seemed a bit repetitive and hard to follow all of these specific Japanese terms getting through all 14 principles. The author finishes pretty strong in the last 50 pages.

All in all it was a good book. For the first half or so I struggled a bit to see the linkage between automobile design and factory assembly, and software development. However on thinking about it some more I think I may finally be grasping what that executive may have been getting at when he commissioned a copy of these two books for his entire organization.

I'm starting to finally see the linkage between software development and physical manufacturing/assembly processes. There's a lot to be gained in software development by applying ideas from Toyota and modern manufacturing such as lean, low inventory, levelling, value streams, tracing and counting the number of "steps" from idea to production code in the field, one piece flow, finding and eliminating queuing and other waste.

I've got some more to say about it and I'm pulling some thoughts together and I'll have some more to post on it coming up.

coffee and a smoke

At work there's a new tenant in our building. Originally the site was built and fully occupied by my company. It's a five storey building. After several rounds of restructuring and such a decision was made to seek tenants to rent out some of the space.

It took a few months but IBM has now moved in and taken over the 5th floor entirely. Welcome IBM! We've seen them around the elevators and 1st floor cafeteria a bit. They seem like decent blokes. There were a few stragglers up on 5 and they were just absorbed into the lower floors a few months ago. There's still a good amount of space so it wasn't disruptive in my area.

Someone mentioned to me last week that the Halifax site is the only IBM facility in the world with free coffee. Apparently that is a global IBM policy. This is because the cafeteria has free coffee from the existing contract with my company. Don't be upset IBMers worldwide, the free coffee is awful, sludge, pretty much undrinkable. It's free and you get what you pay for.

Still something occurred to me and IBM may have already done this. What they could do is set up a jar on their floor and anytime someone gets a free coffee downstairs they have to put say 25 cents or whatever into the jar. Every few months or so the jar is counted up and donated to some local charity. This would then respect the spirit of the no free coffee rule while being a feel good initiative for the employees.

There was another change in the wake of IBM's arrival. The outside smoking area has been restored. There was an original smoking area about 100 feet outside the side doors. One day a couple of years ago or so HR announced that worldwide all staff smoking areas were ended and the employees had to go off-campus to smoke. But now with shared space IBM cares more about the tiny percentage of employees who are smokers and requires them to be accommodated on site. So the designated smoking area is back.

I'm glad the smoking area is back. I don't smoke myself so it might seem irrelevant to me one way or the other. I was concerned about the safety and security of my coworkers while at work. After all let's consider the unintended consequences of the HR decision to kick the smokers off campus.

Well first of all by going off campus they immediately lose the protection of the corporate security guard patrols. By definition security's realm is limited to the campus grounds. When employees step off campus they are on their own and thus exposed.

The other major issue with off campus smoking is that they were moved out of sight from the cubicles. With the on site smoking area we can see them from where we sit. Thus if there's some disturbance or security problem we can raise security from our desks or rush outside ourselves. With offsite smoking a bad actor can easily walk up to the smokers unseen from the building and create trouble. Again the smokers are exposed to additional risk.

So I'm glad IBM put the safety and security of their employees ahead of the agenda of certain anti-smoking busybodies in HR.

don't have time

Once in a while something will come up. Perhaps a code review you need someone to do, or something like that. Now sometimes you'll find there's a snag that the person doesn't have time for it.

So I got thinking about that a bit "I don't have time". First of all the statement is obviously false. You have plenty of time. There's eight full hours available each working day and the task would take less than one hour. What the person is really saying is "I don't have priority" - this isn't important enough to bump some other task.

It might seem unimportant to make this distinction, but it is valuable to describe the situation accurately. Because by describing the problem properly then it can be a start toward some resolution. After all saying "don't have time" is absolute, final, intractable, unrecoverable. After all it's not possible to create time.

Now "don't have priority" is much more manageable. For example it could be possible to de-prioritize some other task. But more likely it allows an honest conversation. If your task isn't important to the other person, then great. Eliminate the dependency, get someone else to do the code review or whatever, and move on. Accept that roles have changed and someone has bigger fish to fry now. Realize that an error in your area is less catastrophic than an error in some other area, thus we assume the lower risk to focus resources on more critical things. In any case by breaking the dependency on your priority isn't my priority it allows everyone to move ahead.

interviewing again

I attended an interview a little while back. No I'm not seeking new opportunities. I'm getting settled in with a new, better team since January and things are going pretty well so far.

The team is looking to add a new person to backfill for turnover. A candidate got through the first interviews with HR and the team lead. The TL asked the developers to do the next interview. It was a good interview I thought. The candidate was pretty solid and I thought he came across pretty well. Myself and the other developers had some probing questions. After the interview the team lead escorted the candidate out and we had a quick meeting to give the instant feedback and offer a thumbs up or thumbs down hiring recommendation.

It's a good experience to be on the other side of the desk. I didn't think I'd get to do any more interviews. In the first couple of years with the company I did a number of interviews for everything from co-op student through to team lead. I enjoyed doing them and I believe I was good at it. Then one day HR announced that in order to do interviews we had to go to some sensitivity / political correctness training class. I of course declined and that was the end of interviewing. I'm not sure what happened between three years ago and today, I didn't bring it up when the TL mentioned about us doing an interview.

--

I'm glad the disastrous HR policy is apparently no more. The closest I heard to a rationale at the time was something about what if a rejected candidate comes back with a lawsuit? yes let's talk about that. The company is large, with thousands of employees. So it's safe to say there have been around 100,000 interviews. Of these how many lawsuits have there been? Probably zero, none that I ever heard of. In the worst case even if some settlement ended up being paid to a troll lawyer then we can quantify the cost of these settlements over the population of interviews. And in the worst case the cost is probably about $1 an interview for these theoretical lawsuits.

Now let's compare that to the cost of a bad hire. Even for an entry level programmer or tester the cost of a single bad hire is staggering, well into the tens of thousands. And at the intermediate level and above it's certainly upwards of and into six figures. So what's more important financially, avoiding some imaginary theoretical small lawsuit that legal/HR would have to deal with, or avoiding a bad hire that development and the entire company would have to deal with.

Obviously the value of avoiding bad hires is orders of magnitude more important (and hazardous, as bad candidates are many and can often get quite deep into the recruiting process and even all the way on board). Yet HR/legal had the power they added this policy in order to make their lives theoretically easier. The term for this business antipattern is local optimization, where some department will put in some new rule or procedure that makes their workflow easier or faster while the overall cost to the business as a whole is worse off.