47 posts
« Previous 1 / 2 / 3 / 4 / 5 Next »

Archive for the ‘Software Quality’ Category


JSR 305: a silver bullet or not a bullet at all?

Posted by Mikhail Ksenzov   March 30th, 2009

JSR-305 is a Java Specification Request intended to improve the effectiveness of static analysis tools operating in Java 5+ environments. The idea here is that one can use special purpose annotations in order to provide static analysis tools with hints regarding the behaviour and side effects of methods.

An example of such annotations can be found in the presentation ‘Annotations for Software Defect Detection’ by William Pugh, who is masterminding the whole spec. Here we go:

 1: void test() {
 2:    if (spec != null) fFragments.add(spec);
 3:    if (isComplete(spec)) fPreferences.add(spec);
 4: }
 6:
 5: boolean isComplete(AnnotationPreferences spec) {
 6:    return spec.getColorPreferenceKey() != null
 7:        && spec.getColorPreferenceValue() != null
 8:        && spec.getTextPreferenceKey() != null
 9:        && spec.getOverviewRulerPreferenceKey() != null;
10: }

What’s wrong with the snippet above? Well, the check for null on line 2 shows that the developer expects that the value of ‘spec’ can potentially be null, but it is still passed to method ‘isComplete’. Later, ‘isComplete’ attempts to dereference the value, which causes a NullPointerException.

According to Dr. Pugh, the best way to detect this issue statically is to force a developer to add the annotation @Nonnull to the method signature like this:

5: boolean isComplete(@Nonnull AnnotationPreferences spec) {

In this way, a basic static analysis tool that can minimally track ‘spec’ as a potential suspect for ‘null’ can issue a warning when the @Nonnull annotation is contradicted (that is, when ‘spec’ is passed to ‘isComplete’ as a parameter).

There are two problems with this approach:

  • it forces the developer to do work that rightly should be performed by a static analysis engine
  • it takes time to write the annotation for static analysis, but it takes even more effort to maintain the annotations and the actual code base in a consistent state.

In reality, the proposals behind JSR-305 exist to enable a tool intended for single function analysis (so-called intra-procedural analysis) to act as if it were performing whole program analysis by requiring the developer to state expected behaviour up front (whether or not that behaviour is actually expressed correctly in the developer’s code).

In contrast, this same scenario is supported by a whole program static analysis tool (so-called inter-procedural analysis) without developer intervention:

  1. First, a complete call-graph of the system is built, and then all the methods are ordered so that called methods are processed prior to callers — such an ordering allows the tool to generate all the necessary information about, in this instance, the method ‘isComplete’ by the time the analysis of the method ‘test’ begins.
  2. During the analysis of ‘isComplete’, the tool records the fact that the incoming argument ‘spec’ is dereferenced.
  3. Next, the method ‘test’ is analyzed. In this method, the variable ‘spec’ is checked for null, so it is tracked as a potential suspect for an exception. Using the information generated about ‘isComplete’ the tool can reliably issue a warning on line 3, since it already knows that ‘isComplete’ dereferences the incoming argument.

So that example applies to a simple unconditional dereference scenario. In more complicated cases, Dr. Pugh proposes to use the annotation parameter ‘when’, with one of the following values:

  • ALWAYS
  • NEVER
  • MAYBE
  • UNKNOWN

For example: ‘@Nonnull(when=When.NEVER)’ means that a value is always null in the given context.

This specification seems to be a compromise between the amount of information provided by a developer to a static analysis tool and the amount of effort a developer has to put into it, a compromise that does not seem to be a particularly good solution here. First of all, the amount of information provided in such a manner is insufficient to provide accurate analysis, and secondly this seems to be too much work for the developer, especially when this work can be avoided.

Let’s examine how conditional value dereferencing is supported by whole program static analysis tools:

 1: void test() {
 2:     entity.qualifiedName = null;
 3:     saveName(entity, false);
 4: }
 5:
 6: boolean    saveName(Entity entity, boolean qualified) {
 7:    String name;
 8:    if (qualified)
 9:        name = entity.qualifiedName.trim();
10:    else
11:        name = entity.name.trim()
12:
13:     save(name);
14: }

In this example, an inter-procedural static analysis tool would first analyze the method ‘saveName’. A good analysis engine should be able to record the fact that this method only dereferences ‘entity.qualifiedName’ if the second parameter, ‘qualified’, is set to ‘true’. This, it would appear, is a deal more detailed than one can practically achieve by adding @Nonnull(when=When.XXX) annotations, even with all the work the annotation implies for the developer.

Next, the method ‘test’ would be analyzed. A good static analysis tool will naturally keep track of ‘entity.qualifiedName’ because of the assignment to ‘null’ on line 2 and its therefore potential use in an exception causing context. However, given that the actual call to ‘saveName’ on line 3 uses ‘false’ as its second argument, such a tool will not issue a warning that would in reality be a false positive, since the knowledge gained from analyzing ‘saveName’ disqualifies any potential warning due to the conditional relationship between arguments.

In summary, JSR-305 proposes a whole roster of interesting ideas for using annotations to enhance static analysis of Java code, and NPE detection seems to be only one aspect of this specification request. In upcoming blog posts, we shall continue the discussion of proposed annotations as well as offering our own ideas about how and when annotations should be used in static analysis.


Avionics Software Development and DO-178B

Posted by Brendan Harrison   March 18th, 2009

Today, I had a chance to connect with Connie Beane, the Director of Certification and Safety Critical Software for ENEA Embedded Technology, Inc. Connie has a deep background in safety-critical avionics systems development as a Federal Aviation Administration Designated Engineering Representative (DER) with authority for design assurance level A systems, software and complex electronic hardware. Her additional experience includes 12 years with the FAA in the Transport Airplane Directorate as a Project Officer, Federal representative and Secretary of the RTCA committee SC-180, which produced DO-254, as well as 8 years at Boeing as a Lead Engineer in Boeing Commercial and Boeing Aerospace Divisions.

I wanted to talk to her more about avionics software development, specifically give Kloctalk readers some additional background on the DO-178B guidance that’s used in avionics software development.

[Brendan]:  Can you describe the short version of the history behind this guidance, its purpose, and use in the industry?
[Connie]: In the early 1990’s Boeing was developing the 777 aircraft, which included very software intensive systems.  As a result, industry began using software development and verification tools as a means of developing software more efficiently. The FAA saw this trend as a concern since many of these tools were being used with very little human oversight. When DO-178B was published in 1992, a section on Tool Qualification was included.

[Brendan]:  Is DO-178B guidance only used by teams developing software for commercial aircraft or is it used elsewhere?
[Connie]: DO-178 was developed for commercial aircraft, however, today DO-178B is being used for much more.  It is being used for commercial systems which aid aircraft such as ground based navigation and communication systems. Interestingly, it’s also now being used in military applications both airborne and ground based.

[Brendan]: What’s the most common misunderstanding people have about DO-178B?
[Connie]: That it involves much more than good engineering practices.  DO-178B was developed by government and industry.  They incorporated good engineering practices and some additional activities to ensure safe, reliable software.

[Brendan]: What are a few of the core software development best practices that are required by DO-178B?
[Connie]: Quality Assurance audits, requirements traceability, robustness testing, standards for design, coding and verification.

[Brendan]:  Sounds like a pretty rigorous approach to software development, which of course is a good thing given its safety-critical nature. What kind of cost burden does DO-178B add to a typical project?
[Connie]: That depends on whether the software being developed is performing highly critical functions or not.  For highly critical software, the cost of developing to DO-178B could double the average development cost.

[Brendan]:  Switching gears a bit to your role at ENEA. Out of the full range of DO-178B service offerings you provide, what’s the most common problem you’re brought in to solve?
[Connie]:  Probably the most common issue is a project that is behind schedule and needs to be certified to meet a customer deadline.

[Brendan]:  What’s the biggest change you’ve seen in aircraft systems development during your career?
[Connie]:  Growth in use of software and complex electronic hardware to develop highly integrated avionics systems.

[Brendan]: What do you think the future holds for this guidance? Do you see it evolving in a particular direction that’s different from where it is now?
[Connie]: There is an international committee currently working on the next version of DO-178. This committee is addressing issues such a object-oriented design, model based development, and further defining qualification of software tools.  This committee is also developing a companion guideline to DO-178 which will be applied to ground based software.

[Brendan]: Any final thoughts?
[Connie]:  Companies resist embracing DO-178.  However, once a company has the processes and procedures in place to ensure compliance to DO-178, their software development process is easily understood, repeatable, and maintainable.  This translates to software which is reliable, understandable, traceable and maintainable.


On Test Strategy

Posted by Johanne Leduc   February 24th, 2009

Testing can have many goals: to assess quality, to assess conformance to specification, to help managers decide whether or not to ship, etc. These all affect a test team’s approach.  Very often, it seems that “Finding Important Bugs Early” is the tester’s primary goal during a development cycle in an agile environment. It’s the ideal: find a quality impacting issue as quickly as possible after it was introduced so that the code is fresh in developer’s minds. “Finding Important Bugs Early” is a goal shared by countless test teams.

It struck me as odd, then, that James Whittaker of JW on Test in his excellent “The Future of Software Testing” series (Part 4), stated that in the future, we need to “close the gap between when a bug is created and when that same bug is detected.” In my opinion, why wait? There are plenty of things we can do to close that gap right now!

The most efficient tactics I’ve seen are the following: early involvement of testers in the development process, good unit testing, code inspection, static code analysis, continuous integration, and effective use of stubs.

Early involvement of testers is not a new concept; it’s already a part of good test process practices. For example, it is a key area of TPI (Test Process Improvement) known as the “moment of involvement”. As a bonus, I’ve always found that testers make great requirement appraisers; not only can they help assess that a requirement is verifiable, but also that it is complete, consistent and unambiguous.  Keeping testers in the loop is especially important in an agile setting where it can help make the documentation barely sufficient, as recommended by this type of development process.

We all know that unit tests are your “first line of defence” for testing executable code. To the developers to whom this task most often falls, I encourage you to ask your test team for help should the need arise. A good test team member has a way of seeing testing from a unique perspective. Use this hidden resource! You can pay the favour back when it’s time for them to use a new automation framework. It’s worth it: a good unit test suite can find many problems early that can be hard to diagnose later downstream.

Code inspection is another early detection tool. Used effectively, it can find logic and design problems at the source, not as a symptom (i.e. when system testing will stumble upon the issue). Moreover, there is no prerequisite of compilation to start it! The same is true for static code analysis. We all know the benefits so apply early, apply often, and save a lot of trouble later on.

For environments that are not of that kind where you “throw the product over the wall” to be tested, continuous integration is necessary. Otherwise, your process may be hindering the test team’s effort to Find Bugs Early. Should your testers be validating last week’s code? Last month’s? If they are, they have no chance of finding your most immediate problems. Don’t forget about bug interactions. It frequently happens that a bug is hidden behind another bug; fix one and possibly expose a slew of others. I heard a question asked once that truly captures the idea: “Why is it that when a bug dies, all the others come to its funeral?” You won’t know who’s in attendance if you’re not testing the latest code.

You may recall that I mentioned the effective use of stubs at the beginning of this post. I hope it piqued your curiosity. It’s something I learned by observing effective testers over the years. When a widget comes off the conveyor belt, they don’t wait for the thingamajig it screws into to start testing. The thingamajig may not be ready for another week! They instead whip up a mock-up of the thingamajig that works somewhat like the real thingamajig and try out their widget, pronto! Very often, the combination doesn’t work as expected and they must ask themselves: are their assumptions about the thingamajig wrong or is it the widget itself? It’s a good thing they didn’t wait until next week to discover this potential problem! This type of experiment enables early integration testing: the component didn’t work correctly in its intended environment. This prompts an investigation into the root cause: is it a bug in the component or a poorly defined interface between two components? Beware however, software stubs can fall into the same trap as a test oracle: you can make it as complex as the real thing if you’re not careful.

These are but a few of the tools for Finding Important Bugs Early. There are many other testing approaches that are not addressed here (so don’t throw away your regression tests!) This post focussed on the “early” part of the goal. What about tactics for the “important” part? Good question! I might be coerced into a follow up post one day…


In-phase defect containment

Posted by Brendan Harrison   February 16th, 2009

Here’s Gwyn chatting about general software development challenges, in particular the whole goal of “in-phase defect containment” – i.e. identifying and correcting defects in the same development phase they’re created. Near the end of the video, there’s a short discussion on how this objective fits in an Agile context. With Agile’s focus on the frequent delivery of working software, in-phase containment becomes even more important, even though it’s more often associated with more formal methodologies such as CMMI and Six Sigma.


Java source code vs bytecode analysis

Posted by Alen Zukich   January 6th, 2009

David posted an interesting discussion on the usage of static analysis tools by developers to find security vulnerabilities.  As always the discussion with static analysis tools lean towards the false positive and false negative discussion.  But also David mentions their results are sometimes difficult to understand.  

This is one of the reasons Klocwork switched from a bytecode analysis tool for Java to a source code analysis tool.  As both have their advantages and disadvantages (and I admit I’m very biased here) we have certainly found that we have been able to reduce our false positive rates, find more issues (not to mention add new issues quickly) and provide more details on the results.

The reason for this is that the bytecode started to get noisy in Java 5 and even more so in Java 6.  Specifically one of the great advantages that static analysis tools have today is being able to show you the details of any issue it finds.  For example

public class Source {
     public static Source getInstance() { return null; }
     public int getValue() { return 32; }
     public Source() { getInstance().getValue(); }
}

This is a Null Pointer Exception issue. Although this is a very small example it is an inter-procedural issue and something that is missed by other defect detection tools. The important thing here is to help the developer understand this issue by tracing through the paths that make this happen.  In other words you need to know that getInstance() has the potential to return null and what paths bring us to the actual error.  This can get quite complex where it starts to not only span the same class but different methods in different classes.

Because of this extra noise this could be part of the reason your seeing new Java annotations being introduced.   Many of you may be familiar with the Java annotation that were introduced in Java 5.  This was part of a specification called JSR-250.  There is a new specification called JSR 305 for the purposes of allowing annotations to help assist tools that detect software defects.  Interestingly enough it is Findbugs leading the charge to add this in for Java 7.  Interesting to see if this specification happens.


How Pervasive is Agile?

Posted by Todd Landry   December 15th, 2008

I like statistics. I always have and always will. I remember manually keeping track of all the statistics for my favourite baseball team by going through the box scores everyday in the newspaper (obviously this was well before the internet came around).  I would manually calculate batting average, slugging percentage, strikeouts per 9  innings and so on. Yup, I lived a pretty exciting youth! Anyways, that love for statistics continued with me, but now I like to know things like how much snow has fallen overnight, and how much more we can expect over the next few days, or what my accuracy is in Guitar Hero, so I can try to beat my 9 year old daughter on Black Sabbath’s Paranoid.

Why am I telling you about all this? Well, I was recently participating in a tour of 5 different cities in Scandinavia over a 5 day period, delivering a presentation on how Source Code Analysis fits in Agile development environments. Fun topic really, but one of the points I raised in each of the 5 sessions I did was that more and more organizations using Waterfall development techniques are moving to Agile.

After being asked by one of the participants to provide an example of this, I brought up some things that I have personally seen over the past couple of years. So I rattled off how in a previous “life” I worked in a mid-sized company where my team was the first Agile team formed. Over the course of 6 months I saw a few more teams show up, and that continued on (and still does from what I hear). I then pointed to the Agile 2008 Conference which, from what I understand, grew in both attendees and vendors, from the previous year, and the expectation is for that trend to continue for 2009. Finally, in talking to people at different trade events (non Agile), training sessions, or customers, etc. I just keep hearing that more and more teams are adopting Agile. But I didn’t have any concrete numbers that illustrated that Agile really was becoming widely adopted.

So I spent a couple hours later that evening (after the usual trains, planes, trains, taxis routine) to try and find some real statistics on this. Well, to continue with the baseball metaphors, I probably hit a solid double. I was able to find a lot of good statistics on typical length of iterations, whether an organization has adopted one or more Agile “techniques”, average Agile team size, and so on, but I just could not find a definitive statistic saying that Agile now is being done in x% of development organizations worldwide. What is Agile’s worldwide market share? That is all I really want. Then I’ll know without question or hesitation that Agile is the approach of choice for software development.

So, from everything I’m *hearing*, I have to believe that Agile is really becoming more and more prevalent. It would just be nice to open up the paper, and see in black and white if Agile is a singles hitter or a home run hitter.

BTW, here are a few other statistics I gathered while on my trip:
•    93% of plane/train food is bad regardless of the carrier.
•    I forgot my hotel room number 60% of the time while over there.


ISV software quality; tortology or oxymoron…

Posted by Gwyn Fisher   December 10th, 2008

It’s kind of bizarre, but in my pre-Klocwork experience of running ISV development groups, from small teams to global enterprises, it never struck me as wrong that we would routinely ship software containing critical bugs. We knew we were doing it. We knew, on some abstract underground never-to-be-admitted layer of our deepest darkest souls, that this was a “bad thing.”

But mostly, we knew that when somebody found a bug we could just send them a patch.

And what’s more, we knew that customers expected this behavior. We got requests to “send us a patch quickly, this is causing me {insert unpleasant business scenario here}.” We had customers telling us that we were a great company to deal with, because we responded to patch requests quickly.

We weren’t the bad kids on the block, we were right up there with the upstanding corporate citizens of the software world!

Why are software companies allowed, perhaps even expected, to mess up on this scale? If we were engineers, we’d be sued. If we were doctors, we’d be in court before you could say something snappy and relevant like “most software sucks.” But for ISV developers, hey, it’s just business.

Talk to developers in the embedded space, or mil/aero, or telecoms, or a bunch of other places where this approach to business isn’t acceptable and it’s like talking to engineers. They understand what their tolerances are, they understand what it takes to make quality software, and they fundamentally understand on a basic level what happens to them and their company if they get it wrong. Inventory turnover is one thing, killing people is something else.

So is it just that ISV software is unimportant? That ISV software won’t ever be put in a situation where it could be life-threatening (or life-enhancing), so it doesn’t matter if it crashes all the time?

Obviously that’s not the case. Software created by ISVs is used all over the world, in all manner of situations, some dire, some not, but at the end of the day, in every situation, there’s a name on that software.

Your name.