Last week I toured the West coast with our friends from VersionOne, Perforce, and Electric Cloud on our Agile roadshow hitting the cities of Seattle, Santa Clara, and San Diego. In one of the after meeting discussions, one of the attendees asked me what the differences were between Agile and Lean. Having only been involved with Lean from an outside perspective, I didn’t really think there were huge differences and that they shared many of the same beliefs.

Luckily, it looks like others believe this to be the case too. So rather than me trying to explain this, this timely blog does a great job of explaining Agile and Lean, and why there is a lack of understanding and acceptance of Agile practices in many companies that practice Lean.

Also, as part of this Agile roadshow, we had a bit of a celebrity in our midst--our illustrious keynote speaker David Hussman of DevJam consulting has a past that most of us weekend musicians dream about. He was part of a big-hair metal band! Not only can he play a mean guitar, the dude knows his stuff about Agile and gave one of the best keynotes I’ve ever seen. Check out his website when you get a chance, and see if you can find him in this video.

There was a recent article from Mark Pitchford titled: “Think static analysis cures all ills? Think again.” Obviously being biased working here at Klocwork, I take a major exception to what Mark has to say.

This article makes ridiculous claims. About the only thing Mark got right was that static analysis has been around for a long time. However it’s ludicrous to think that they’re the same as they were in the past. That’s like saying computers from decades ago are the same as today. The advancement has been huge for static analysis tools, especially in the last couple of years.

The author is really selling the merits of dynamic testing, which is great. Everyone should have the proper testing procedures in place. But static analysis is complementary–it’s another tool that will help you go through ALL the paths of your code to help you find bugs you’d otherwise miss.

One of the biggest reasons static analysis tools have taken off, in my opinion, is the level of integration. It’s quite simple to get results with static analysis tools with little effort. Especially compared with dynamic analysis tools.

Paul Anderson, a fellow competitor, sums it up very nicely in the comments. Check it out.

It happened again.  For what seems like the 100th time, someone reports to me that they are seeing a number of false positive reports on the resource leak checker.  For those not familiar with a resource leak, take a look at a previous post.  Although resource leaks apply across most languages, the place where this question keeps coming  up seems to always be in Java or C# code.  My last query came from Java code, so we will use that as an example.  Here was a report where the FileInputSteam is not closed on exit.

FileChannel sch = new FileInputStream(...);
FileChannel dch = new FileOutputStream(...);
try{
   ...
} catch (...) {
   ...
} finally {
   try {
      sch.close();
   }
}
...

Do you see this issue?  Clearly, they close the FileInputStream in the finally block.  The problem is that you can still throw an exception when you try to create a FileOutputStream.  The simple fix is to encompass the FileInputStream and FileOutputStream in the try/catch blocks.

Static analysis tools are great for finding bugs like this resource leak.  In this case, the report that a FileInputStream is not closed on exit wasn’t enough information to debug this.  This brings us to a very important discussion with static analysis tools–trace.

Gone are the days where a static analysis tool reports a bug at line X with no context.  Today you get detailed traces.  The trace contains conditions and assignments to give you clues about why the tool is identifying a bug.  In this example below, identifying the “source” and “sink” of any defect will help you trace through the start/end of the control flow.

More importantly in the case of Java, here we have a clear understanding where exceptions are thrown.  This would have helped tremendously for the example above.

Detailed traces are very important and can do many things for individual checkers.  So whatever you do, don’t just look at the bug description, pay attention to the trace!

As a product manager, the only backlog I typically care about is my product backlog. Do I have the right stories in there? Do the stories have enough detail? Are they properly prioritized? You know, that kind of stuff. Today, however, I’m going to write about a very different backlog, that is the static analysis defect backlog.

A static analysis backlog is created when you run a static analysis product on your code base for the very first time. Chances are pretty good that the first analysis is going to list a large number of defects, some that are without question real, and some that perhaps are not. Do not freak out! This is the first time that analysis engine has ‘laid eyes’ upon your code and it is going to flex its muscles and show you any weaknesses it believes exist. So how does one deal with this? Here are a few strategies to help you:

1) Don’t boil the ocean. Before you even run that first analysis, don’t have a “wouldn’t it be cool” moment, where you decide to turn on every single rule the analysis engine has. There is a reason why static analysis tools haven’t turned on everything.  They are showing the most accurate and critical issues first.  So unless you have unlimited time and resources, your best bet is to start with a core set of rules and run the analysis based on that set. This core set of rules should include things such as memory/resource leaks, buffer overruns, null pointer dereferences, uninitialized variables, and so on. Add other rules once you have this core set under control.

Is your issue backlog making you cross eyed? Try these coping strategies.

2) Baseline your defects. Consider that first analysis your baseline and choose to ‘park’ them for the time being. Chances are the product that the analysis was run on is one that has already been released to the public, and in good working order. Zero out these defects for now, and start to triage them, which leads into strategy #3.

3) This is going to sound pretty obvious, but when it comes to managing your issue backlog start looking at the most critical issues first. These are the ones that are most likely to cause a failure of some sort, so determine if these issues are real, and if so, fix them immediately. Once you’re done with the most critical issues, move to the next level of severity, and continue on that way.

4) Finally, tune your analysis. Any good vendor will allow you to tune your analysis. The benefits of tuning are twofold; 1) you can find code issues that would otherwise go undetected and, 2) reduce the number of issues that the engine reports incorrectly in the context of your source code. You should think of ways to give the tool more context about your code base to increase accuracy.

If you follow these suggestions, you’ll definitely have a better grasp of your bug backlog, and you’ll be able to execute on reducing that backlog quickly and efficiently. If you don’t, then at some point, you may feel a little like the critter pictured here.

If there are any other strategies you’ve tried to deal with your bug backlog, leave a comment or two. I’d love to hear about them.

If you’ve ever gone through the process of porting an application, you know the pain.  Porting can be difficult if you’re not vigilant from the outset.  There are tons of written guidelines and best practices for specific platforms or architectures, such as those going to 64 bit for Windows apps or Intel architecture and Mac OS.

In the past, we have talked about Endian issues, which are very specific to porting from different architectures (big-endian vs little-endian).  This time I want to take you through some general porting issues to show you how you can now use static analysis tools to help you through this process.

A very common gotcha are situations in which an expression is cast to a type of potentially different size.  Code written for a particular architecture’s data model can face significant challenges in porting when the new architecture imposes a new data model. For example, when porting between 32 bit and 64 bit data models, the new data model typically leaves ‘int’ at 32 bits, but uses ‘long’ as a 64 bit value, breaking any assumed equality in data type size that worked under the 32 bit data model.  Using static analysis tools can help you find this automatically.  In the example below all casts will throw an error:

void bad(){
 unsigned char uc = 0;
 unsigned short us = 0;
 unsigned int ui = 0;
 unsigned long long ull = 0;
 long l=0;

 l = (long)ull; //porting issue
 uc = (unsigned char)l; //porting issue
 us = (unsigned short)l; //porting issue
 ui = (unsigned int)l; //porting issue
}

Another gotcha is using bit fields within a structure.  When you define a data structure using bit fields, you are relying on the endian representation of a particular compiler. Moving to a different compiler can modify this layout, resulting in problems when that data structure is projected onto a byte buffer.  Again static analysis tools can be used to identify where the bit field definition exists and NOT properly covered by a big-endian or little-endian macro.  For example flag this:

struct mystruct{
   unsigned short x : 4;
   unsigned short y : 5;
   unsigned short z : 7;
} test;

But don’t flag the same thing with conditional compilation:

#ifdef BIG_ENDIAN
struct mystruct{
   unsigned short x : 4;
   unsigned short y : 5;
   unsigned short z : 7;
} test;
...

There are tons of other gotchas as well, such as using a custom byte swap macro without checking endianness, an incompatible type is used with a network macro such as ‘ntohl’, compiler-dependent option is used, macro describing a builtin numeric type is used, and many more.

Using static analysis tools not only detects porting issues, but provides metrics and reporting capabilities that allow you to prioritize and identify the scope of work involved.  For example, Top 10 porting issues (see the diagram above) helps you identify the bulk of the issues that need to be addressed.  Or you can use the component diagram report to identify specific components of your software code base you may have to concentrate on going forward.

Nice blog post from Walter Bright over at Dr. Dobbs on the Patterns of Bugs. He ties together bug patterns, recommended process changes, and the resulting productivity payoff from making these improvements. He recommends a bunch of process changes, including static analysis, code reviews, and coding standards, then goes on to review examples of different bug patterns. A few can be detected with static analysis (coding mistakes as written) but many are errors with the code as intended (something static analysis doesn’t check… that’s what testing is for). His main recommendation seems to be that bugs can often be pattern based, so once a bug is identified, take steps to remove that pattern from your code through process or tool changes.

Once the pattern is identified, then look for changes in process that will permanently eliminate that bug. Eliminate enough of the bug patterns, and you should enjoy a substantial increase in productivity.

His experience is that, over time this kind of systematic approach to fixing bugs makes developers better.

I’ve noticed in my decades of writing programs that I just don’t make the kinds of mistakes I used to. Apparently I’ve unconsciously evolved coping strategies to avoid them. Identifying and building such strategies into the process means everyone can benefit from that experience.

We certainly see that in the area of static analysis bug detection; lots of customers report that their developers make fewer mistakes, defect injection rates go down, and overall productivity is improved.

Yesterday we kicked off our first Medical Devices specific seminar in Boston, with our friends from SterlingTech and Vector Software. The day was all about software development for medical devices and more specifically, about managing software risk to help ensure you are compliant with all of the FDA regulations specific to software code. We had a great turnout, with over 75% of registrants showing up for this session. A couple of observations I found interesting from this event:

• The medical devices community seems to be fairly tight-knit. Everyone at the event seemed to know everyone else, and there was definitely a strong support system in place among the companies in attendance.
• And secondly, there appears to be fairly high adoption of Agile practices, particularly Scrum (albeit somewhat modified), with these Medical device companies, and the healthcare industry in general for that matter. From everything I heard, this adoption curve shows no signs of slowing down.

If IEC 62304 is important to your organization, you might want to check out this half-day seminar. The next one will be in Minneapolis in January. I’ll be packing my woolies for that one!

Part of my life has been spent in the manufacturing sector working with industrial automation devices, but the discovery of the Stuxnet virus is the first time I’ve ever heard of specifically virus targeting and even rootkitting a PLC (programmable logic controller) or  SCADA (supervisory control and data acquisition) network.

When working in industrial plants, we took the standard precautions with regard to Windows viruses and even started to add virus protection for Linux, but never did it occur to any of us that the industrial automation equipment might be at risk. Whenever the subject was even brought up, which was rare in itself, there were the standard arguments:

• Oh, it’s on a physically separate network (or VLAN configuration), only USB (thumb/flash) drives are allowed and they’re virus checked before use.
• Oh, it’s running a completely different processor/operating system/architecture – there’s no way it can be infected.

The consequences of infection are severe.   These devices run everything from our nuclear power plants to complex manufacturing assembly lines, aircraft controls (FADECs) and chemical refineries, just to name a few.  In its most basic of functions, industrial automation is used for two purposes: to keep humans safe and to produce products for less cost.  Interrupting either of these is going to kill someone or cost a company a large chunk of change.

So, what does this all mean?  It means that industrial automation and PLC vendors had better start hardening their solutions for security vulnerabilities and elevate the quality of their firmware and software components using security vulnerability tools such as Klocwork’s static analysis just as the general computing industry has done for the past 30 years.

For an in-depth analysis and timeline, refer to either Symantec’s whitepaper on their Stuxnet analysis or the work done by ESET on their version of Stuxnet.

Endianness refers to the ordering of bytes into memory. As many of you are aware, computers do ordering differently. You can have the representation of Big-endian or Little-endian.

Essentially Big-endian stores data big-end first, meaning the first byte is the biggest and Little-endian stores data little-end first, meaning the first byte is smallest.

Because all machines are different and write data either as big or little-endian, a computer could read this data incorrectly.  If you are not prepared ahead of time for heterogeneous processor architectures, then you might be in for a world of hurt. This article describes the problem in detail and also provides solutions when you’re starting off.

But what if you’re not prepared?  This can be a costly and complicated problem when developing on large systems with multiple processors or during a porting effort.  You need to be able to verify the developers provide data interaction with the target processor in the proper endian format.  Here’s a good post that talks about this issue in more details, and references a bunch of good third party articles, some of which outline how source code analysis can help; the basic idea being that source code analysis can flag instances where data is being transmitted to or from the target without being transformed.

We’ve just introduced some capabilities in this area which are described in this white paper authored by our CTO Gwyn Fisher. Check it out and let us know what you think.

We get many questions from medical devices customers on how they should validate the use of Klocwork’s static analysis tools for the FDA. I suspect the situation would be similar for most vendors of software development tools. As we’ve done before, we thought it would be a good idea to ask Bruce Swope from SterlingTech Software to clarify this whole topic for us.

[Brendan] First, what is tool validation?
[Bruce] Tool validation is the act of demonstrating that a tool will consistently produce expected results.

[Brendan] How can a medical device company know whether they should validate a tool or not?
[Bruce] From 21CFR820.75, “Where the results of a process cannot be fully verified by subsequent inspection and test, the process shall be validated with a high degree of assurance and approved according to established procedures”. For example, if you are using a tool to perform work and you do not plan on using any other method to verify that the work was done properly then the tool will need validation. Please note that you must validate the tool for your intended use, not the entire tool.

FDA Tool Validation Checklist

[Brendan] Ok, let’s take a specific example. What would validating a static analysis tool involve? 
[Bruce] Here’s a basic list of what needs to be completed.

a) Develop a Tool Validation Plan. This should include the test environment and methods to be used
b) Develop a set of requirements that the tool is intended to meet
c) Develop a test protocol to verify that the requirements have been met
d) Develop a traceability matrix and verify that all requirements have been tested
e) Execute the test protocol
f) Write a test report summarizing the results

[Brendan] Do most companies do this themselves or can they outsource this activity? 
[Bruce] If the company has the internal trained resources available then they can certainly do the tool validation themselves. Many companies lack the staffing, or expertise, to perform validation of a software tool. It is common for these companies to outsource the documentation and testing activities to a firm like SterlingTech. We’ve completed this process numerous times before and it’s a good way to reduce the cost and effort around the validation process.

[Brendan] Thanks Bruce.