A while ago we came across a puzzling problem. We had a situation at a customer site where under some seemingly random circumstances some database records were not getting cleaned up as they should.
It was hard to reproduce the problem either on the customer side or in our own test sites. After some investigation we realized the problem. We had two tables, which I'll call "site" and "site_control". When a "site" is created the associated "site_control" is created at the same time. Both tables had integer id key fields which were both based on an auto increment number (which is a sequence in Oracle) starting at 1000.
Typically the id fields in the site and associated site_control records are the same because of the auto increment. The problem occurred when in certain unusual circumstances they can get out of sync in valid ways where a site record is created but not site_control.
The defect was in our code and it had been in the code undetected for several years. When a site is to be deleted the code is supposed to call deleteSite(int siteId). Deleting a site also deletes site_control as designed due to foreign key referential integrity.
What we were doing was incorrectly passing the site_control id, calling deleteSite(siteControlId). Although incorrect, this still works most of the time since the site and site_control id's were the same. When they got out of sync, the problems occurred. The fix was pretty simple, just call deleteSite() using the correct siteId which we were holding.
This demonstrates one of the real problems with integer auto increment key fields. If the key for the wrong table is used in the code then the DB selects, inserts, updates, and deletes, may still work or seem to work in terms of foreign key checks and such. However the code is wrong and it can lead to very difficult to diagnose errors and problems after the code ships.
If there are bugs in the code then we definitely want to find them as soon as possible, certainly before the code ships. The use of auto increment with possibly shared ids is dangerous as it can allow incorrect code to pass testing and thus for defects to escape detection until the code is in the field and the bugs become extremely expensive to address.
There are a few strategies to counter this. One approach which I like is to use GUID instead of numeric keys. That way every key is unique not just within tables but across all tables. So if the code is using they key from the wrong table then the foreign key constraint will fail and it will be obvious that there is a problem.
Another approach is to "stagger" your starting point with the auto increment ids. That is a good approach too because if there is an error that it will very rarely just happen to work since the parent and child tables virtually never have the same id values. Again if there is a code issue where it is using the id from the wrong table then it will fail immediately in developer or QA testing and the code defect will be fixed before the product ships.
So in your schema definition use an offset, even 50 would be plenty. So the site table ids would start at 1000, site_control at 1050, table_c at 1100, table_d 1150, etc.
Saturday, May 24, 2008
Friday, May 02, 2008
Save as PDF
A little while back I had to create a PDF version of an install guide for a maintenance patch release I was putting together.
I hadn't generated a PDF before. Historically the tech writers just took care of it in some mysterious process. Due to some company reorganization it now fell to the developer who was creating the code patch to also put together the PDF patch install guide.
The document was already done and ready to convert from Word. I asked a couple of developer coworkers about how to save as PDF. One of them pointed me in the direction of PDF 995.
It turns out this PDF 995 thing is a very handy utility. Install is fast and easy. It works as a printer driver of all things. So when you click Print the PDF 995 comes up as a "Printer" in your printer list. That's very clever.
It works fine and it generates a nice clean glossy PDF just like you expect. Then you click Save As to save the PDF on your file system and you're done. It's very easy to use.
So +1 for PDF 995, a well designed product.
I hadn't generated a PDF before. Historically the tech writers just took care of it in some mysterious process. Due to some company reorganization it now fell to the developer who was creating the code patch to also put together the PDF patch install guide.
The document was already done and ready to convert from Word. I asked a couple of developer coworkers about how to save as PDF. One of them pointed me in the direction of PDF 995.
It turns out this PDF 995 thing is a very handy utility. Install is fast and easy. It works as a printer driver of all things. So when you click Print the PDF 995 comes up as a "Printer" in your printer list. That's very clever.
It works fine and it generates a nice clean glossy PDF just like you expect. Then you click Save As to save the PDF on your file system and you're done. It's very easy to use.
So +1 for PDF 995, a well designed product.
Wednesday, April 16, 2008
Changing jobs
I'm doing something that many people in high tech do. I'm leaving my job to join a new company. I've been with SupportSoft and its acquired predecessor Core Networks since 2001. I'm joining Research In Motion later this month.
For me a job switch is a big move. I've only done one before, going from xwave to Core Networks in 2001 after 3 years at xwave and its acquired predecessor PRIOR Data Sciences. So many acquisitions in tech. Some people jump all the time, around every two years on average. But that's not my style and I'm personally a little bit suspicious of serial jumpers.
It's pretty wacky how networking and chance events can lead to new things. Last Christmas break I went to an annual road hockey game. There I met a guy who I knew over the years and worked with at PRIOR in the late 1990s. He's director level now. He was finishing up at his job after several years and was looking for a new job. We agreed to keep in touch over LinkedIn when he found a new job in case he was hiring additional people. As it turns out he landed at RIM in the first part of the year and that lead to me getting interviewed. I don't normally go to road hockey games expecting a new job to come of it but it just shows what can happen.
I don't have anything bad to say about SupportSoft. The local job market has been pretty good since around 2004 so if I was unhappy then I could have chosen to leave at any time. This just looks like a promising opportunity which I believe is in my best interest to take.
Every job has its ups and downs. Good things and little annoyances. I think expatriates mostly look back well on previous employers; even if they grumbled a bit when they worked there. I look back kindly on my time with PRIOR and xwave and I'll look back well on my seven years with Core and SupportSoft.
For me a job switch is a big move. I've only done one before, going from xwave to Core Networks in 2001 after 3 years at xwave and its acquired predecessor PRIOR Data Sciences. So many acquisitions in tech. Some people jump all the time, around every two years on average. But that's not my style and I'm personally a little bit suspicious of serial jumpers.
It's pretty wacky how networking and chance events can lead to new things. Last Christmas break I went to an annual road hockey game. There I met a guy who I knew over the years and worked with at PRIOR in the late 1990s. He's director level now. He was finishing up at his job after several years and was looking for a new job. We agreed to keep in touch over LinkedIn when he found a new job in case he was hiring additional people. As it turns out he landed at RIM in the first part of the year and that lead to me getting interviewed. I don't normally go to road hockey games expecting a new job to come of it but it just shows what can happen.
I don't have anything bad to say about SupportSoft. The local job market has been pretty good since around 2004 so if I was unhappy then I could have chosen to leave at any time. This just looks like a promising opportunity which I believe is in my best interest to take.
Every job has its ups and downs. Good things and little annoyances. I think expatriates mostly look back well on previous employers; even if they grumbled a bit when they worked there. I look back kindly on my time with PRIOR and xwave and I'll look back well on my seven years with Core and SupportSoft.
Saturday, April 05, 2008
Java GUI coming of age
Somewhat quietly the Java JDK GUI has improved. Especially since Java 5.
I'd kind of given up on swing over the years. Everyone has bad memories of traditional Java GUIs. Battleship gray, clunky. Control-C didn't work on Windows, instead it used Unix semantics like Control-Insert for copy and paste. Select a piece of text and right click the mouse. Nothing happens.
If there was one thing which made Eclipse it was developing SWT as a much superior end user experience for Java GUI. With native widgets Eclipse was so much better. Suddenly Ctrl-C works properly on Windows. Select a piece of text and right click the mouse and the context menu comes up. Finally the GUI just looks and works the way we expect. At the time it came out SWT was the obvious and superior choice.
I have to give Sun credit. They didn't give up on JFC. They obviously worked hard to improve it. Now in Java 1.5 they pass the "10 foot test" for the first time. That is, standing 10 feet away from the computer, you can't tell that it's a Java GUI. It looks and acts much smoother now.
One of my favorite programs that I use most every day is Oracle SQL developer. This uses the JDK GUI and it is just fine to work with. It looks and feels great. The standard keyboard and mouse actions all work the way you expect. It's plenty fast.
For the first time developers can consider using swing for serious Java GUI applications.
I'd kind of given up on swing over the years. Everyone has bad memories of traditional Java GUIs. Battleship gray, clunky. Control-C didn't work on Windows, instead it used Unix semantics like Control-Insert for copy and paste. Select a piece of text and right click the mouse. Nothing happens.
If there was one thing which made Eclipse it was developing SWT as a much superior end user experience for Java GUI. With native widgets Eclipse was so much better. Suddenly Ctrl-C works properly on Windows. Select a piece of text and right click the mouse and the context menu comes up. Finally the GUI just looks and works the way we expect. At the time it came out SWT was the obvious and superior choice.
I have to give Sun credit. They didn't give up on JFC. They obviously worked hard to improve it. Now in Java 1.5 they pass the "10 foot test" for the first time. That is, standing 10 feet away from the computer, you can't tell that it's a Java GUI. It looks and acts much smoother now.
One of my favorite programs that I use most every day is Oracle SQL developer. This uses the JDK GUI and it is just fine to work with. It looks and feels great. The standard keyboard and mouse actions all work the way you expect. It's plenty fast.
For the first time developers can consider using swing for serious Java GUI applications.
Tuesday, March 18, 2008
The computer and the network
Social networks are interesting. They represent the evolution of the computer and network.
First there were PCs. Then came local networks with printer and file sharing and e-mail. Originally the networks were to extend the capabilities of the PC. They were originally called "computer networks", i.e. computers linked together. This changed over time and the network became more important. Instead of the network being useful to the PC, the PCs purpose was to enable a network.
This continued as the networks extended out to the Internet and Web by the late 1990s.
Today we are seeing the next step in the evolution. The underlying network is becoming less important. The social network is becoming more important. Instead of social networking being an application of the Internet network, the network's purpose is becoming to enable social networking.
Social networking like Facebook is interesting because the network on its own is just inanimate technology. Instead of connecting computers to other computers, social networks are about enabling people to connect with other people. It becomes less about the technology and more about people meeting and interacting.
First there were PCs. Then came local networks with printer and file sharing and e-mail. Originally the networks were to extend the capabilities of the PC. They were originally called "computer networks", i.e. computers linked together. This changed over time and the network became more important. Instead of the network being useful to the PC, the PCs purpose was to enable a network.
This continued as the networks extended out to the Internet and Web by the late 1990s.
Today we are seeing the next step in the evolution. The underlying network is becoming less important. The social network is becoming more important. Instead of social networking being an application of the Internet network, the network's purpose is becoming to enable social networking.
Social networking like Facebook is interesting because the network on its own is just inanimate technology. Instead of connecting computers to other computers, social networks are about enabling people to connect with other people. It becomes less about the technology and more about people meeting and interacting.
Sunday, March 02, 2008
Application performance testing and optimization
I've been assigned to do performance testing for an upcoming application release. We made some architectural and database changes so we want to be sure our hardware estimation process is still valid. Also we want to try it out on the newer Sun T Series servers.
Along with measuring performance I can identify areas for performance improvement and optimize where possible.
Application performance is a bit like navigating through water of unknown depth. With a canoe you can paddle happily along on a shallow river. With software the canoe corresponds to one developer or tester clicking along through screens with very small data sets.
Moderate load is like navigating a 40 foot yacht. With the yacht there's more draft so if the water is very shallow then you'll run aground. The 40 foot boat would be like around 5-10 developers using the application at the same time with a modest size data set.
Heavy load is like the aircraft carrier. You need to be very deep to be able to handle this very large boat. Heavy load is when you simulate large numbers of simultaneous users and have a large data set. The load and data set size should be the same as what you plan to use in production using the same hardware.
With software like with the waterway, you don't really know about how it performs until you test it under load. You can't tell by looking at it. Taking a canoe or a small pleasure craft through a harbour does not tell you if the water is deep enough for a massive freighter ship.
Optimizing performance is fairly straightforward especially at the start. It is an 80/20 situation. 80% of the resources are consumed by 20% of the features. So using tools like JProbe it is easy to find the hot spots. Typically optimizing the small number of trouble spots will dramatically improve performance and then you're done.
This can be frustrating to the programmer because the same optimization patterns can be applied throughout the code base but the other areas don't use enough resources to justify the investment required to refactor for performance.
Although after the first iterations when the biggest resource hogs are dealt with some of the other problem areas that were hidden by the original optimized modules now become part of the 80 in the 80/20 rule and they can then be optimized.
In most applications, the biggest performance issues are around the database. This can be caused by inefficient queries that the DBMS cannot execute quickly. It can also be caused by a poor indexing strategy (or no indexes!) on tables which have large data sets.
A good free tool for DB analysis is Oracle SQL developer. I find you can learn a lot by just copying and pasting application queries into SQL developer. In addition to seeing the execution time you can also get the explain plan in an excellent graphical view.
Any intermediate level or higher professional software developer should be aware of database execution plans and how to interpret them and optimize them. Even at the junior level a programmer should understand how indexes impact query performance in large datasets.
One of the many eternal performance headaches with EJB is around static data from the DB that is requested often but changes infrequently or never. If a trip to the DB is required every single time then this redundancy will consume a lot of resources and really slow down system performance and responsiveness.
While caching is an apparent answer to this, EJB and caching basically don't seem to go together. We've had good success though using ehcache to deal with this shortcoming of EJB. I recommend ehcache based on my experience with it in this project.
Along with measuring performance I can identify areas for performance improvement and optimize where possible.
Application performance is a bit like navigating through water of unknown depth. With a canoe you can paddle happily along on a shallow river. With software the canoe corresponds to one developer or tester clicking along through screens with very small data sets.
Moderate load is like navigating a 40 foot yacht. With the yacht there's more draft so if the water is very shallow then you'll run aground. The 40 foot boat would be like around 5-10 developers using the application at the same time with a modest size data set.
Heavy load is like the aircraft carrier. You need to be very deep to be able to handle this very large boat. Heavy load is when you simulate large numbers of simultaneous users and have a large data set. The load and data set size should be the same as what you plan to use in production using the same hardware.
With software like with the waterway, you don't really know about how it performs until you test it under load. You can't tell by looking at it. Taking a canoe or a small pleasure craft through a harbour does not tell you if the water is deep enough for a massive freighter ship.
Optimizing performance is fairly straightforward especially at the start. It is an 80/20 situation. 80% of the resources are consumed by 20% of the features. So using tools like JProbe it is easy to find the hot spots. Typically optimizing the small number of trouble spots will dramatically improve performance and then you're done.
This can be frustrating to the programmer because the same optimization patterns can be applied throughout the code base but the other areas don't use enough resources to justify the investment required to refactor for performance.
Although after the first iterations when the biggest resource hogs are dealt with some of the other problem areas that were hidden by the original optimized modules now become part of the 80 in the 80/20 rule and they can then be optimized.
In most applications, the biggest performance issues are around the database. This can be caused by inefficient queries that the DBMS cannot execute quickly. It can also be caused by a poor indexing strategy (or no indexes!) on tables which have large data sets.
A good free tool for DB analysis is Oracle SQL developer. I find you can learn a lot by just copying and pasting application queries into SQL developer. In addition to seeing the execution time you can also get the explain plan in an excellent graphical view.
Any intermediate level or higher professional software developer should be aware of database execution plans and how to interpret them and optimize them. Even at the junior level a programmer should understand how indexes impact query performance in large datasets.
One of the many eternal performance headaches with EJB is around static data from the DB that is requested often but changes infrequently or never. If a trip to the DB is required every single time then this redundancy will consume a lot of resources and really slow down system performance and responsiveness.
While caching is an apparent answer to this, EJB and caching basically don't seem to go together. We've had good success though using ehcache to deal with this shortcoming of EJB. I recommend ehcache based on my experience with it in this project.
Subscribe to:
Posts (Atom)