I've been doing some JSP work the last few months. We use struts. It's not too bad I've found.
One thing struts does pretty well is supporting translation to different languages through the use of keys [aka tokens] into a properties file. This generally works well for the programmer and we generally don't have to worry a whole lot about translation of the user interface screens to different languages.
A challenge can occur when developing a new screen or adding a section to an existing screen. The developer wants to focus on getting the JSP, JavaScript and HTML right. He will typically just hard code the labels right on the JSP and leave it to later when things are working to cycle back and convert the hard coded labels in the GUI to proper keys which have been added or already exist in the .properties file.
The problem can be that under tight deadlines the developer can forget to do this or miss a label or two, especially when there is conditional processing and everything doesn't necessarily appear every time. This can be a big problem because if the test team doesn't specifically test for this then the application can be in the field before the problem is detected. That could cause an expensive patch into the field.
One thing the developer can to do deal with this is to not use the true labels during the screen development when hard coding them. What I use is the prefix "zz". So Name, Address, Number becomes zzName, zzAddress, zzNumber in the hard coded development stage GUI. That way it's obvious looking at the screen I'm not using the tokenized labels and it's a reminder to fix it before shipping. For the programmer you can be sure you have everything by doing a search in Eclipse on "zz" in your code base and it will be obvious if you've missed anything.
For the test team, this can be a somewhat difficult problem to detect. One test case the test team can do is open up the tokens .properties file and prepend "zz" to every single label - this is pretty fast to do using a macro or script. Then restart the app and go through all the screens, including the dropdown lists. If you see anything appear on the new GUI without the zz prefix you know it's a raw, untokenized GUI element, a defect. This is important to test especially in screens where some elements may only occur under certain conditions, or when the screen content can be generated dynamically. Also error messages are a common source of untokenized GUI labels.
Friday, August 15, 2008
Monday, July 14, 2008
technical debt
Technical debt is an interesting term we seem to hear a bit more lately.
I think anyone with experience in software development knows instinctively at least what it is. It's good that people are finally attaching a label to it. It's an important start.
Sometimes a team will seem to make incredible progress on early releases of some product, then may seem to slow down later. There can be a number of reasons for this slow down. One of them is productizing. Another is that with more users people will find more bugs and the team has to backtrack to address them, impeding progress on new features in the process.
But I think what often slows a team down later on is technical debt. In the early releases the team may have used unsound practices such as poor to no source documentation, poor to no unit testing, copy and paste code, heavily intertwined modules, death march project schedules. These shortcuts can yield seemingly rapid progress on a small team with a small code base over a small period of time.
However that progress comes at a price. Like an irresponsible man who lives beyond his means and uses credit cards to fund a flashy lifestyle. He seems prosperous to an outside observer but actually he's heading for a crash and what he has been doing is unsustainable. Technical debt is very similar to that type of financial debt. Much of the early progress is illusory, financed through technical debt. The result is an unmaintainable code structure that will require costly and time consuming maintenance and rewrites.
There are some hazards around technical debt, especially when a new team is asked to take over code originally developed by a different team. If the original team is too willing to part with the original code base then that may mean the code has a lot of technical debt which is coming due and the original team wants to put it on someone else to deal with it. Also with technical debt, if the later team seems to make less progress than the original team, that may be because the new team is being badly slowed by the burden of servicing the technical debt the original team ran up.
One good thing about technical debt now we have defined this "problem with no name" and put a name to it. From there it would be good if there was a way to take the next step and devise ways to measure the technical debt in a code base and in an automated way assign some debt level to the code and estimate the costs of servicing it. I think for some graduate students there may be some interesting and valuable original research that can be done in this area.
I think anyone with experience in software development knows instinctively at least what it is. It's good that people are finally attaching a label to it. It's an important start.
Sometimes a team will seem to make incredible progress on early releases of some product, then may seem to slow down later. There can be a number of reasons for this slow down. One of them is productizing. Another is that with more users people will find more bugs and the team has to backtrack to address them, impeding progress on new features in the process.
But I think what often slows a team down later on is technical debt. In the early releases the team may have used unsound practices such as poor to no source documentation, poor to no unit testing, copy and paste code, heavily intertwined modules, death march project schedules. These shortcuts can yield seemingly rapid progress on a small team with a small code base over a small period of time.
However that progress comes at a price. Like an irresponsible man who lives beyond his means and uses credit cards to fund a flashy lifestyle. He seems prosperous to an outside observer but actually he's heading for a crash and what he has been doing is unsustainable. Technical debt is very similar to that type of financial debt. Much of the early progress is illusory, financed through technical debt. The result is an unmaintainable code structure that will require costly and time consuming maintenance and rewrites.
There are some hazards around technical debt, especially when a new team is asked to take over code originally developed by a different team. If the original team is too willing to part with the original code base then that may mean the code has a lot of technical debt which is coming due and the original team wants to put it on someone else to deal with it. Also with technical debt, if the later team seems to make less progress than the original team, that may be because the new team is being badly slowed by the burden of servicing the technical debt the original team ran up.
One good thing about technical debt now we have defined this "problem with no name" and put a name to it. From there it would be good if there was a way to take the next step and devise ways to measure the technical debt in a code base and in an automated way assign some debt level to the code and estimate the costs of servicing it. I think for some graduate students there may be some interesting and valuable original research that can be done in this area.
Sunday, June 29, 2008
Query results as SQL function arguments
I kind of discovered this by accident. I was going through a manual multi step process to convert some numeric IP addresses in the database to a text dotted quad format, for input into a command line load testing script. I was finding it tedious.
The thing is I had SQL functions available to convert numeric to dotted quad and dotted quad to numeric IP addresses. In the database device table we have the IP address as numeric.
The device table has a unique index on serial number. I was using serial number to identify the devices. What I wanted to do was for a given serial number get the IP address in dotted quad format.
Originally I used a two step approach. First get the numeric IP address for the device serial number, in this example "serial1000".
Then copy that numeric IP address query result into a second query.
That worked well enough but it was slow, manual and tedious all of the copying and
pasting. I wanted to combine them into a single query so I only had to enter the serial number and could get the dotted quad IP from that.
I couldn't figure out how to get the original query result to be the argument to the numberToDottedQuad() function. I tried a bit then gave up. This is what the DB wouldn't allow, syntax error.
After a while though it kind of just came to me. Use the same style syntax as insert .. select statements. Nest the inner result in parenthesis. That was the aha moment and I set back to it. This worked.
The trick is that you have to use parenthesis on the inner query so the SQL engine evaluates that first, then it can feed it as the input to the outer query. Which makes sense. This was nice because essentially the same syntax can be used for both Oracle and SQL Server [except the dual part but that's trivial to take out].
If I had created a separate stored function to run the first query and feed the result into the second function then it would have meant separate code for Oracle and SQL Server. This way I didn't have to create a stored function (get the database to do the work) and I can use the same SQL query on Oracle and SQL Server.
The thing is I had SQL functions available to convert numeric to dotted quad and dotted quad to numeric IP addresses. In the database device table we have the IP address as numeric.
The device table has a unique index on serial number. I was using serial number to identify the devices. What I wanted to do was for a given serial number get the IP address in dotted quad format.
Originally I used a two step approach. First get the numeric IP address for the device serial number, in this example "serial1000".
select ipaddress from device where serial_number = 'serial1000';
Then copy that numeric IP address query result into a second query.
select numberToDottedQuad( result_from_above_query ) from dual;
That worked well enough but it was slow, manual and tedious all of the copying and
pasting. I wanted to combine them into a single query so I only had to enter the serial number and could get the dotted quad IP from that.
I couldn't figure out how to get the original query result to be the argument to the numberToDottedQuad() function. I tried a bit then gave up. This is what the DB wouldn't allow, syntax error.
select numberToDottedQuad(select ipaddress from device where serial_number = 'serial1000') from dual;
After a while though it kind of just came to me. Use the same style syntax as insert .. select statements. Nest the inner result in parenthesis. That was the aha moment and I set back to it. This worked.
select numberToDottedQuad((select ipaddress from device where serial_number = 'serial1000')) from dual;
The trick is that you have to use parenthesis on the inner query so the SQL engine evaluates that first, then it can feed it as the input to the outer query. Which makes sense. This was nice because essentially the same syntax can be used for both Oracle and SQL Server [except the dual part but that's trivial to take out].
If I had created a separate stored function to run the first query and feed the result into the second function then it would have meant separate code for Oracle and SQL Server. This way I didn't have to create a stored function (get the database to do the work) and I can use the same SQL query on Oracle and SQL Server.
Thursday, June 12, 2008
Struts and JSP
I've spent the last couple of weeks doing something that I'd avoided for the last four years. I developed some Web pages in Java using Struts and JSP.
You have to know the history on it. Back at Core Networks in 2004 there was this core of around 5 Java developers who had been with the company for a little over a year on a next generation product. Additionally there were around 12 existing pre Java team who were working on various legacy products like CoreOS and point solutions like CoreMeter and others.
In 2004 we had this important business opportunity around a big Java/J2EE project. So the old guard joined together with the newer Java developers to create a unified team. What then happened was some of the people who joined the Java team had to create the GUI pages. One of the new Java people had some stuff set up using this exciting technology we'd heard of called Struts.
Well the Struts/JSP web pages were a disaster for the new developers. Previously we used PHP to create pages for our GUIs and it took around 1 developer day to create a basic page with data and a form. We used SOAP calls to both obtain the data to display and to invoke the change actions when the user clicked Submit on the HTML forms. It worked pretty well. The PHP UI developers and SOAP API backend developers could work independantly and in parallel.
Anyway the switch to Struts has horrible for the new Java developers. It was now taking a week or more to create each page in the GUI! The output was flakey and nobody seemed able to understand how it worked or how to get anything that was broken fixed. I would go by the printer for something of my own and see these e-mail printouts with the title "Struts hell"
But on this project with a new company I had to do some web pages so I delved into struts for the first time with some trepidation. I started off by reading the Struts Survival Guide which was very good, especially chapter 2. From there I was able to grasp enough about MVC to get started and understand that Struts is generally not concerned with the presentation aspect of things.
There were some rough parts and it was hard at times but I was able to get the pages to come up. It took a while due to all of the learning curve. Besides the struff you need to learn around struts like Actions, Forms, execute(), validate(), struts-config.xml there is so much more you have to grasp at least some of like JSP, HTML, JavaScript. We aren't using JSTL a whole lot yet in the existing code which is a bit surprising but at least I didn't have to grasp that as well.
I'd forgotten how enjoyable JavaScript is. Ahhh, scripting. It's so nice at times to be running the code right inside the browser and JavaScript is great to work with. I can understand why Google is so up on it.
So all in all it was a positive first real experience with struts. It was quite satisfying when I entered some stuff with known bad data and the page redisplayed properly with the error message and the original input data preserved.
I can't imagine how things were so messed up back in 2004 at Core that it was so painful but I'm glad I missed that. Historically I always thought I liked being in the back end server side processing away from the user interface. However after doing some GUI stuff it was kind of good in a way. We had a nice separation of concerns on this project where I only had to get the JSP and navigation working and another guy who wasn't interested in the GUI side had to do the validation and business processing part. That back end part now seems less interesting than it might have in the past.
You have to know the history on it. Back at Core Networks in 2004 there was this core of around 5 Java developers who had been with the company for a little over a year on a next generation product. Additionally there were around 12 existing pre Java team who were working on various legacy products like CoreOS and point solutions like CoreMeter and others.
In 2004 we had this important business opportunity around a big Java/J2EE project. So the old guard joined together with the newer Java developers to create a unified team. What then happened was some of the people who joined the Java team had to create the GUI pages. One of the new Java people had some stuff set up using this exciting technology we'd heard of called Struts.
Well the Struts/JSP web pages were a disaster for the new developers. Previously we used PHP to create pages for our GUIs and it took around 1 developer day to create a basic page with data and a form. We used SOAP calls to both obtain the data to display and to invoke the change actions when the user clicked Submit on the HTML forms. It worked pretty well. The PHP UI developers and SOAP API backend developers could work independantly and in parallel.
Anyway the switch to Struts has horrible for the new Java developers. It was now taking a week or more to create each page in the GUI! The output was flakey and nobody seemed able to understand how it worked or how to get anything that was broken fixed. I would go by the printer for something of my own and see these e-mail printouts with the title "Struts hell"
But on this project with a new company I had to do some web pages so I delved into struts for the first time with some trepidation. I started off by reading the Struts Survival Guide which was very good, especially chapter 2. From there I was able to grasp enough about MVC to get started and understand that Struts is generally not concerned with the presentation aspect of things.
There were some rough parts and it was hard at times but I was able to get the pages to come up. It took a while due to all of the learning curve. Besides the struff you need to learn around struts like Actions, Forms, execute(), validate(), struts-config.xml there is so much more you have to grasp at least some of like JSP, HTML, JavaScript. We aren't using JSTL a whole lot yet in the existing code which is a bit surprising but at least I didn't have to grasp that as well.
I'd forgotten how enjoyable JavaScript is. Ahhh, scripting. It's so nice at times to be running the code right inside the browser and JavaScript is great to work with. I can understand why Google is so up on it.
So all in all it was a positive first real experience with struts. It was quite satisfying when I entered some stuff with known bad data and the page redisplayed properly with the error message and the original input data preserved.
I can't imagine how things were so messed up back in 2004 at Core that it was so painful but I'm glad I missed that. Historically I always thought I liked being in the back end server side processing away from the user interface. However after doing some GUI stuff it was kind of good in a way. We had a nice separation of concerns on this project where I only had to get the JSP and navigation working and another guy who wasn't interested in the GUI side had to do the validation and business processing part. That back end part now seems less interesting than it might have in the past.
Saturday, May 24, 2008
Problems with auto increment and foreign keys
A while ago we came across a puzzling problem. We had a situation at a customer site where under some seemingly random circumstances some database records were not getting cleaned up as they should.
It was hard to reproduce the problem either on the customer side or in our own test sites. After some investigation we realized the problem. We had two tables, which I'll call "site" and "site_control". When a "site" is created the associated "site_control" is created at the same time. Both tables had integer id key fields which were both based on an auto increment number (which is a sequence in Oracle) starting at 1000.
Typically the id fields in the site and associated site_control records are the same because of the auto increment. The problem occurred when in certain unusual circumstances they can get out of sync in valid ways where a site record is created but not site_control.
The defect was in our code and it had been in the code undetected for several years. When a site is to be deleted the code is supposed to call deleteSite(int siteId). Deleting a site also deletes site_control as designed due to foreign key referential integrity.
What we were doing was incorrectly passing the site_control id, calling deleteSite(siteControlId). Although incorrect, this still works most of the time since the site and site_control id's were the same. When they got out of sync, the problems occurred. The fix was pretty simple, just call deleteSite() using the correct siteId which we were holding.
This demonstrates one of the real problems with integer auto increment key fields. If the key for the wrong table is used in the code then the DB selects, inserts, updates, and deletes, may still work or seem to work in terms of foreign key checks and such. However the code is wrong and it can lead to very difficult to diagnose errors and problems after the code ships.
If there are bugs in the code then we definitely want to find them as soon as possible, certainly before the code ships. The use of auto increment with possibly shared ids is dangerous as it can allow incorrect code to pass testing and thus for defects to escape detection until the code is in the field and the bugs become extremely expensive to address.
There are a few strategies to counter this. One approach which I like is to use GUID instead of numeric keys. That way every key is unique not just within tables but across all tables. So if the code is using they key from the wrong table then the foreign key constraint will fail and it will be obvious that there is a problem.
Another approach is to "stagger" your starting point with the auto increment ids. That is a good approach too because if there is an error that it will very rarely just happen to work since the parent and child tables virtually never have the same id values. Again if there is a code issue where it is using the id from the wrong table then it will fail immediately in developer or QA testing and the code defect will be fixed before the product ships.
So in your schema definition use an offset, even 50 would be plenty. So the site table ids would start at 1000, site_control at 1050, table_c at 1100, table_d 1150, etc.
It was hard to reproduce the problem either on the customer side or in our own test sites. After some investigation we realized the problem. We had two tables, which I'll call "site" and "site_control". When a "site" is created the associated "site_control" is created at the same time. Both tables had integer id key fields which were both based on an auto increment number (which is a sequence in Oracle) starting at 1000.
Typically the id fields in the site and associated site_control records are the same because of the auto increment. The problem occurred when in certain unusual circumstances they can get out of sync in valid ways where a site record is created but not site_control.
The defect was in our code and it had been in the code undetected for several years. When a site is to be deleted the code is supposed to call deleteSite(int siteId). Deleting a site also deletes site_control as designed due to foreign key referential integrity.
What we were doing was incorrectly passing the site_control id, calling deleteSite(siteControlId). Although incorrect, this still works most of the time since the site and site_control id's were the same. When they got out of sync, the problems occurred. The fix was pretty simple, just call deleteSite() using the correct siteId which we were holding.
This demonstrates one of the real problems with integer auto increment key fields. If the key for the wrong table is used in the code then the DB selects, inserts, updates, and deletes, may still work or seem to work in terms of foreign key checks and such. However the code is wrong and it can lead to very difficult to diagnose errors and problems after the code ships.
If there are bugs in the code then we definitely want to find them as soon as possible, certainly before the code ships. The use of auto increment with possibly shared ids is dangerous as it can allow incorrect code to pass testing and thus for defects to escape detection until the code is in the field and the bugs become extremely expensive to address.
There are a few strategies to counter this. One approach which I like is to use GUID instead of numeric keys. That way every key is unique not just within tables but across all tables. So if the code is using they key from the wrong table then the foreign key constraint will fail and it will be obvious that there is a problem.
Another approach is to "stagger" your starting point with the auto increment ids. That is a good approach too because if there is an error that it will very rarely just happen to work since the parent and child tables virtually never have the same id values. Again if there is a code issue where it is using the id from the wrong table then it will fail immediately in developer or QA testing and the code defect will be fixed before the product ships.
So in your schema definition use an offset, even 50 would be plenty. So the site table ids would start at 1000, site_control at 1050, table_c at 1100, table_d 1150, etc.
Friday, May 02, 2008
Save as PDF
A little while back I had to create a PDF version of an install guide for a maintenance patch release I was putting together.
I hadn't generated a PDF before. Historically the tech writers just took care of it in some mysterious process. Due to some company reorganization it now fell to the developer who was creating the code patch to also put together the PDF patch install guide.
The document was already done and ready to convert from Word. I asked a couple of developer coworkers about how to save as PDF. One of them pointed me in the direction of PDF 995.
It turns out this PDF 995 thing is a very handy utility. Install is fast and easy. It works as a printer driver of all things. So when you click Print the PDF 995 comes up as a "Printer" in your printer list. That's very clever.
It works fine and it generates a nice clean glossy PDF just like you expect. Then you click Save As to save the PDF on your file system and you're done. It's very easy to use.
So +1 for PDF 995, a well designed product.
I hadn't generated a PDF before. Historically the tech writers just took care of it in some mysterious process. Due to some company reorganization it now fell to the developer who was creating the code patch to also put together the PDF patch install guide.
The document was already done and ready to convert from Word. I asked a couple of developer coworkers about how to save as PDF. One of them pointed me in the direction of PDF 995.
It turns out this PDF 995 thing is a very handy utility. Install is fast and easy. It works as a printer driver of all things. So when you click Print the PDF 995 comes up as a "Printer" in your printer list. That's very clever.
It works fine and it generates a nice clean glossy PDF just like you expect. Then you click Save As to save the PDF on your file system and you're done. It's very easy to use.
So +1 for PDF 995, a well designed product.
Subscribe to:
Posts (Atom)