That may sound counter-intuitive. Shouldn’t a zero noise point on the spectrum be accompanied by an infinitely strong signal? Perhaps in the world of DSP this is true, but in the world of SCA reducing noise comes right along with a reduction in detection capability – it’s unfortunately almost a straight-line correlation.

So if we assume that we’re trying to balance a couple of dials on our theoretical tuner, we might start by reducing or dampening noise – it’s the most obvious place to start, after all. Nobody likes to listen to their favorite FM station through the curtain of hissing and popping that accompanies the act of driving through a major city.  Likewise no developer likes sifting through a long list of bogus detection errors in order to find the hidden gems. But to drag out the analogy, assume that the only way of reducing hiss on your FM signal is to turn down the volume… now you’ve got less hiss, but also less Bruce Springsteen goodness to accompany it.

Balance is what we need here, obviously. Enough Boss to make us ignore the hiss, or to put it in a more SCA-like context, enough interesting bugs to make us ignore the incorrect, or the irrelevant (correct detections on the part of the engine that the developer just doesn’t care about, e.g. low memory conditions in a memory-insensitive environment).

Consider the following simple example that clearly lies “on the line”:

    void foo(char* s, int a)
    {
        char* s1 = s;
        if( a > 0 )
            *s1 = 'a';   // potentially use an uninitialized ‘s1’
    }

    void bar(int m)
    {
        char *s;
        foo(s, m);       // s is not initialized prior to calling ‘foo’
    }

So… to report, or not to report?

Lacking any other information, it is obvious that function ‘foo’ interacts under certain situations (when parameter ‘a’ is positive) with parameter ‘s’ (aliased as local variable ‘s1’). As we have no knowledge about the provenance of parameter ‘s’ when analyzing ‘foo’, however, there’s nothing here to cause a report and so we squirrel away the knowledge of what ‘foo’ does for later use.

When analyzing ‘bar’ we know what ‘foo’ does, and we know we’ve got an uninitialized local pointer, ‘s’. But again we’re lacking enough knowledge to know the valid values, or ranges, that parameter ‘m’ may take. There are definitely a set of circumstances here in which we know a problem will occur (if parameter ‘m’ is positive), and a set of circumstances in which we know a problem will not occur (if parameter ‘m’ is zero or negative) – this much is encoded in the functional behavior of ‘foo’. But is it a defect, or should we filter out the report in favor of providing only those situations in which we can be “sure” the bug not only exists, but can be proven to be exercised?

There’s the art of balance in a nut-shell, and it revolves around the phrase “lacking any other information.” In the ideal world, lacking any restrictions in terms of time, memory or computing power (or indeed actual from-the-wall power, as we have to worry about now), we might defer all such decisions until we categorically know that a particular data value is passed down the call graph far enough to get to ‘foo’. But in the real world of multi-million LOC projects, that approach simply can’t scale.

And so, calling on balance as our friend, we can bias a localized decision to report or not, given that we know to at least one order of approximation that bad things could happen here. Different engines pronounce that bias differently, leading to one of the greatest divides between prevalent solutions.

Now ask yourself, as the developer, is it a worthy report if you know that 10 levels up the call graph there’s a check on what eventually becomes parameter ‘m’ to ensure that it’s never positive? Perhaps you’d automatically classify this as a false positive and, annoyed at the tool, move onto the next report. Or perhaps, seeing the size of the gap in the call graph, you might just choose to code defensively, initializing ‘s’ to NULL in ‘bar’ and adding guard code to ‘foo’ because, hey, you never know.

And as we’ve all seen so many times over the years, “you never know” might just as well be written “and so it came to pass…”

#include <stdio.h>;

void foo(char * v) {
    char buffer[10];
    if(v != NULL)
   {
            memcpy(buffer, v,strlen(v));
    }
}

int main (int argc, char ** argv)
{
    foo(argv[1]);
    foo("the longest string you can find");
    return 0;
}

Just like the memory overflow post this is another form of a buffer overflow.  So there you have it, just more terminology to describe bad things in your code.  Gwyn promises to give a follow up to these posts with some details on how this general area (stack corruption, tainted sources, etc.) can be exploited.  Can’t wait to see that.

Yes, except it is probably not what you’re thinking, it’s not the same as a memory leak;  a memory overflow is quite different.  A memory overflow is really just a form of a buffer overflow.  The impact of memory overflow is unexpected behavior or program failure.   Take this example:

#include <stdio.h>
typedef struct s1_ {
   int i;
   int j;
   char arr[10];
}s1;

typedef struct s2_ {
   char b[20];
   char c[40];
}s2;

main()
{
   s1 block1;
   memset(&block1, 0, sizeof(s2));
   block1.i =1;
}

Here we have the case of incorrectly using ‘memset’ at line 16 where ‘sizeof(s2)’ is bigger than ‘block1′.  In fact, going back to this customer revealed that the issue was due to memset clearing much more area than intended.  If you’re using static analysis or source code analysis tools then you are probably covered by this.  You will find this type of issue usually in the “buffer overflow” category.

So, are you free of your memory overflows?

FileChannel sch = new FileInputStream(...);
FileChannel dch = new FileOutputStream(...);
try{
   ...
} catch (...) {
   ...
} finally {
   try {
      sch.close();
   }
}
...

Do you see this issue?  Clearly, they close the FileInputStream in the finally block.  The problem is that you can still throw an exception when you try to create a FileOutputStream.  The simple fix is to encompass the FileInputStream and FileOutputStream in the try/catch blocks.

Static analysis tools are great for finding bugs like this resource leak.  In this case, the report that a FileInputStream is not closed on exit wasn’t enough information to debug this.  This brings us to a very important discussion with static analysis tools–trace.

Gone are the days where a static analysis tool reports a bug at line X with no context.  Today you get detailed traces.  The trace contains conditions and assignments to give you clues about why the tool is identifying a bug.  In this example below, identifying the “source” and “sink” of any defect will help you trace through the start/end of the control flow.

More importantly in the case of Java, here we have a clear understanding where exceptions are thrown.  This would have helped tremendously for the example above.

Detailed traces are very important and can do many things for individual checkers.  So whatever you do, don’t just look at the bug description, pay attention to the trace!

A static analysis backlog is created when you run a static analysis product on your code base for the very first time. Chances are pretty good that the first analysis is going to list a large number of defects, some that are without question real, and some that perhaps are not. Do not freak out! This is the first time that analysis engine has ‘laid eyes’ upon your code and it is going to flex its muscles and show you any weaknesses it believes exist. So how does one deal with this? Here are a few strategies to help you:

1) Don’t boil the ocean. Before you even run that first analysis, don’t have a “wouldn’t it be cool” moment, where you decide to turn on every single rule the analysis engine has. There is a reason why static analysis tools haven’t turned on everything.  They are showing the most accurate and critical issues first.  So unless you have unlimited time and resources, your best bet is to start with a core set of rules and run the analysis based on that set. This core set of rules should include things such as memory/resource leaks, buffer overruns, null pointer dereferences, uninitialized variables, and so on. Add other rules once you have this core set under control.

Is your issue backlog making you cross eyed? Try these coping strategies.

2) Baseline your defects. Consider that first analysis your baseline and choose to ‘park’ them for the time being. Chances are the product that the analysis was run on is one that has already been released to the public, and in good working order. Zero out these defects for now, and start to triage them, which leads into strategy #3.

3) This is going to sound pretty obvious, but when it comes to managing your issue backlog start looking at the most critical issues first. These are the ones that are most likely to cause a failure of some sort, so determine if these issues are real, and if so, fix them immediately. Once you’re done with the most critical issues, move to the next level of severity, and continue on that way.

4) Finally, tune your analysis. Any good vendor will allow you to tune your analysis. The benefits of tuning are twofold; 1) you can find code issues that would otherwise go undetected and, 2) reduce the number of issues that the engine reports incorrectly in the context of your source code. You should think of ways to give the tool more context about your code base to increase accuracy.

If you follow these suggestions, you’ll definitely have a better grasp of your bug backlog, and you’ll be able to execute on reducing that backlog quickly and efficiently. If you don’t, then at some point, you may feel a little like the critter pictured here.

If there are any other strategies you’ve tried to deal with your bug backlog, leave a comment or two. I’d love to hear about them.

Once the pattern is identified, then look for changes in process that will permanently eliminate that bug. Eliminate enough of the bug patterns, and you should enjoy a substantial increase in productivity.

His experience is that, over time this kind of systematic approach to fixing bugs makes developers better.

I’ve noticed in my decades of writing programs that I just don’t make the kinds of mistakes I used to. Apparently I’ve unconsciously evolved coping strategies to avoid them. Identifying and building such strategies into the process means everyone can benefit from that experience.

We certainly see that in the area of static analysis bug detection; lots of customers report that their developers make fewer mistakes, defect injection rates go down, and overall productivity is improved.

In my first Agile team, many years ago, it was tabled (in our retrospective meeting after a couple of Sprints) that code reviews should be added to our definition of “Done”.  Let’s just say my initial response was less than enthusiastic… but why was that?  Well, in my opinion (perhaps uneducated on this topic), doing code reviews seems to add more to the time it takes to finish stories, so that means less stories are getting done per iteration, which potentially means longer release times, or releases with less functionality than hoped for. This is not something a Product Manager is usually receptive to. After some debate, we put it to a vote where the “yays” defeated the “nays” by a fairly healthy margin (okay, it missed being unanimous by one vote).  So we updated our “Done” criteria and moved into our next Sprint.

Our next couple of sprints went off similar to our earlier sprints, I didn’t really notice any differences. We seemed to have about the same number of stories being started and completed, and I for one was mildly surprised that we were able to maintain the same velocity, even with the extra process of doing code reviews for each story. Curious, I decided to talk to one of the more senior developers about what was going on. He walked me over to our Scrum board and asked me if anything looked different. Nothing jumped out at me initially, until he pointed out that the number of ‘bug’ cards (the dreaded red cards) were significantly less than in those early iterations. He proceeded to tell me that the code reviews were playing a major role in this. Developers were finding things early and fixing them before passing the code onto the testers, leaving the testers to focus on testing the actual features …crazy, I know.

It really appeared as though the code reviews were producing better code, without actually slowing down the development process. My opinions of code reviews did a complete 180…now they were helping to contribute to better quality code that I could show our customers, without having to sacrifice anything in the way of release delays or velocity degradation. I had become a believer!

 I think I have something to Twitter about now…

Deadlock: frog vs. snake
Photo: David Maitland / National Geographic

Continuing the discussion about the embedded community moving to muticore/hetrogeneous hardware from watch out here comes multicore, embedded software development teams have historically been shielded from mulitcore issues. This is due to the specialized functionality of many embedded application classes and the inherent serialized nature of the C language.^[1]

Muticore is an ambiguous term for software developers and one they don’t really use; software developers think in terms of threads/processes and concurrency, not how many cores or processors are available on the target. Concurrency is not a new topic either as Mark Smotherman captured in a history of multithreading, it has been a subject in computer science since its early beginnings in the 1950s.

What has changed is the rapidly increasing use of multicore technologies for embedded devices. One of the prominent software challenges that moving to multicore execution exposes is latent deadlocking bugs as true parallel execution comes into play, instead of a single core’s task scheduling/context switching techniques.

As an example, consider the following code snippet, which has been paraphrased from a deadlock discovered in a real-world open source multithreaded project.

Can you spot the deadlock?

lock_t lock1, lock2;
int refCount = 0;

void enter() {
   reserve_lock(lock1);
      if( refCount == 0 )
         reserve_lock(lock2);
       release_lock(lock1);
   refCount++;
}

void leave() {
   reserve_lock(lock1);
   refCount--;
   if( refCount == 0 )
      release_lock(lock2);
   release_lock(lock1);
}

To see the answer and understand the conditions that lead to the deadlock, download Klocwork’s whitepaper on Developing software for multicore.

A little about the picture in this post.  I found it when searching for pictures of deadlocks.  The photographer, David Maitland, titled his image “Deadlock” and describes it as a continuing struggle between a Morelet’s tree frog and cat-eyed snake. After three hours in the high-stakes cage match, Maitland said there was no clear winner.

VDC Research, “Next Generation Embedded Hardware Architectures: Driving Onset of Project Delays, Costs Overruns, and Software Development Challenges”, September 2010.

In the Chromium Blog, Chris Evans of Chrome Security announced a cash for bugs initiative, paying between 500 and 1337 USD depending on the severity for any previously undiscovered flaw.  I am glad to see Google encouraging the community at large to participate in hardening my current browser of choice.  As Chris points out, Mozilla was one of the first to embark on this type of program, but I am happy to see Chrome following suit.  Me and my online transactions appreciate it.

Hmm.  Maybe I should roll-up the sleeves and  invoke the  ”I’m gonna write me a minivan” approach and get the driveway cleared for the armored cash trucks.

But seriously, if you’re interested in helping out and getting a small reward for your efforts, visit the Chromium Security project.

The next 2 weeks were a roller coaster as we cut our teeth with Scrum. First the good:

• Communication: the interaction amongst the team members was definitely improved. If someone needed an answer to something, they immediately sought out help. The team realized that if they didn’t get timely answers, tasks wouldn’t get done. They really didn’t want to say those dreaded 2 words, “nothing finished”, in the daily scrum meeting.
• Meetings: The daily Scrum meetings were kept short and  sweet as everyone said what tasks they had finished, what they were working on, and if there were any roadblocks in the way. If something required further discussion, a break out meeting with the appropriate people was held.
• Energy: This was a high performing team to begin with, but there was now a newfound energy and buzz. This was a fun team to be around!

As the title suggests, there certainly was some bad in that first iteration.

• Testing and documentation: These were the 2 areas that struggled the most in the first iteration (and the next couple as well). They felt that their work was too heavily back loaded, that is, they would receive their stuff too late in the iteration to either test or document properly. Many of the stories were not totally Done because they were either not tested properly or documented with the time they were given.
• Defects and bugs: Because testing happened so late in the iteration, many of the bugs they found could not be addressed in that iteration. These bugs would have to be carried over to the next iteration, meaning the number of new stories would have to be reduced.

Now for the ugly.

• After just a day or so into the iteration, a plethora of unplanned tasks starting showing up on the Scrum board for many of the stories. These stories now had many new hours of tasks added to them, and we fell behind very quickly. This leads into the next ugly…
• The Burndown chart: Talk about a misnomer! We started to affectionately call our chart the burn-up chart, because there was very little down direction going on with it. Our chart would have looked great at a sales meeting, but in our Scrum meeting, not so much.

So as you can see our 1^st iteration had its share of warts, and in fact, the next couple did as well. But we didn’t get frustrated. We learned from our mistakes and changed/added things based on those mistakes. The Retrospective meetings were incredibly useful because they made us all take a hard, honest look at what went well, and what didn’t. The next, and last entry in my Going Agile series will look at the Retrospective meeting.

Jeremy Brown posted an interesting article on Jeremy’s Computer Security blog where he uses his security know-how to draw a straight line between a software vulnerability found with static analysis and a real 0day exploit on an open source project called gAlan.

Jeremy takes us on a short journey where he finds an unprotected buffer with static analysis, creates an exploit payload to cause a buffer overrun, rewrites the instruction pointer and executes a telnet session, demonstrating how easy it is to turn a run of the mill application into a tunnel into the OS.

One of my colleagues did a similar presentation like this a few years back with a Firefox vulnerability but this is a much better example! Very cool work Jeremy.

Enjoy…

I have always loved “I’m gonna write me a new minivan”  from Scott Adams.  To me, it never gets old.  Originally published in 1998, the theme that applied then still does today: driving 100% of defects or bugs out of the code-base is a laudable goal, but is it really the right one?   I would have to argue no.  There’s no silver bullet out there that will find all software defects and solve issues automagically, and until there is, software development will continue to struggle with prioritization.  Unfortunately, we live in a world of finite resources and constantly evolving demands, but we can always dream about being Wally for a little while.

While C dominates as the programming language of choice for embedded development, the use of Java is definitely on the rise. In fact, according to a recent VDC survey, 12.3% of respondents currently use Java in the embedded space, and 17.9% expect to be using Java in the next two years.

For those transitioning from embedded development using C, you might find yourself falling into the hype that Java is a “safe” language. For example, Java developers face no requirement for managing memory associated with objects. However, this is where the trap may be laid. Even though there’s no need for memory management, developers may need to keep track of specific resources the object allocates. This is especially true in an embedded context where resources are often constrained. Even for experienced developers, these traps pop up time and again and can easily jeopardize your code quality and security.

Here’s a round-up of the top five programming issues developers should be aware of in embedded Java development [More...]

1.    Null pointer exceptions from a method

1                  public class A {
2                      public A foo() {
3                          if (flag)
4                              return null;
5                          return new A();
6                      }
7
8                      public void var() {
9                          A a = foo();
10                         if (flag)
11                             a.foo();
12                     }
13
14                     private int flag;
15                 }

This is the most common issue I see.  In this example a warning is issued at line 11 for a possible null pointer exception.  Essentially there is a potential for a null value from the method foo().  I believe I see these issues more than any others  because of the inter-procedural context of this issue.  This example is quite simple, but where you can easily get lost is when that null value is coming from a long method call chain in multiple classes.

2.    Resource leaks

1  using System;
2  using System.IO;
3
4  namespace LeakExample
5  {
6      class Test
7      {
8          public String Run(String name)
9          {
10             StreamReader reader = new StreamReader(name);
11             String result = reader.ReadToEnd();
12             reader.Close();
13             return result;
14         }
15     }
16 }

I think for most of us this is pretty easy to see.   An object of ‘StreamReader’ type is allocated and its reference is assigned to the ‘reader’ member. If the call to ‘ReadToEnd’ throws an exception, control is transferred out of method ‘Run’, variable ‘reader’ goes out of scope, the object referenced by it becomes lost, but related resources are not disposed.  See my previous blog post on how some people miss this and what you need to do to fix this.

3.    Forward null pointer exception

1                  public class A {
2                      public void foo() {
3                          A a = new A();
4                          if (a == 0)
5                              if (flag)
6                                  a.foo();
7                      }
8                      private int flag;
9                  }

This is another example of a null pointer exception with a twist.  In this example the class data member ‘a’ is compared with 0 value at line 4, and therefore may still be expected to be null when it is dereferenced at line 6.

4.    Reverse null pointer exception

1                  public class A {
2                      public void foo() {
3                          A a = null;
4                          a.foo();
5                          if (a == null)
6                              a = new A();
7                      }
8                  }

Yes, another null pointer exception (getting a theme here?).  Same idea at #3 but in reverse order.  Here they dereference the data member ‘a’ at line 4 but later check for null at line 5.

5.    Empty catch clause

1  class FileHandler {
2      public void Open(String name) {
3          try {
4              // opening file ...
5          } catch (FileNotFoundException e) {   // defect - no statements in the 'catch' clause
6          }
7      }
8  }

Okay, so not exactly the big impact as the top 4, but I just had to mention it.  I’m always guaranteed to see some of these.  Granted they are maintainability issues more than anything else, but c’mon they are there for a purpose.
As promised, I did say that I would post the Java version soon.  It is next…

Truepath Analysis

Ever been faced with that glassy-eyed expression, the look of unthinking, unwholesome fear when some long, incomprehensible word escapes your geeky mouth and upsets the maiden aunts around the once-a-year, wear-your-best-tie, try-not-to-die-before-the-cake’s-all-gone tea table? OK, so this paper won’t help you in that situation whatsoever, but if you replace your maiden aunts with a bunch of your best geek friends, and replace the tea with a sturdy helping of Dew, knowing how a real whole program analysis solution works might just conceivably come in handy. Some day. “Dude, I was totally stoked when I read this thing, trust me it’s ahh-some.” Maybe.

Anyway, in the best traditions of self-serving corporate PR blogs everywhere, I give you… drum roll please… Whole Program Analysis, the Klocwork Way.

Enjoy.

1.    Null Pointer dereference

This is far and beyond the most common issue that I see time and time again.

void npd_gen_must() {
int *p = 0; // NULL is assigned.
*p = 1;  // pointer is dereferenced
}

Now this example is pretty basic and if you ever did something this obvious, maybe it was time to re-evaluate your development skills.  The idea is simple, you assign NULL somewhere then dereference it at some point later.  This is usually missed under a complicated control flow (many conditionals).  Or even more common is the fact that I see memory is allocated, but is never checked against NULL.  Now, some organizations don’t care about this but I would hope anyone doing embedded development is all over it.

2.    Null pointer dereference from function

This is really the same thing but with one very important difference.  This deals with issues from functions.

void xstrcpy(char *dst, char *src) {
if (!src) return;
 dst[0] = src[0];
}

char global;

char *xmalloc() {
  if (global) return &global;
  return 0;
}

void npd_func_might(int flag, char *arg) {
  char *p = &arg;
  if (flag) p = xmalloc(); // xmalloc() may return NULL
  if (arg) { p = arg; } // p may get a new value here
  xstrcpy(p, "Hello"); // p will be dereferenced in xstrcpy()
}

It is this inter-procedural (spanning multiple files/functions) context that is often overlooked.

3.    Memory leaks

I have yet to find a programmer in the C/C++ world who doesn’t know this intimately.  Sadly they happen, a lot.

void foobar(int i) {
  char* p = new char[10];
  if(i) {
    p = 0;
  }
  delete[] p;
}

Here we have dynamic memory stored in ‘p’ and allocated through the function ‘new[]‘ at line 3 and is ultimately lost at line 5.

4.    Array index out of bounds

Again, most people know what these are but there are so many variations of this that they are always inevitable.

int main() {
  char fixed_buf[10];
  sprintf(fixed_buf,"Very long format string\n");
  return 0;
}

The string is 24 characters so at line 4 the array index of ‘fixed_buf’ may be out of bounds.

5.    Uninitialized variables

int foo(int t) {
  int x;
  if (t > 16) {
    x = 1;
  } else if (t > 8) {
    x = 2;
  }
  return x + 1;
}

The value of variable ‘x’ can be used at line 8, when it might be uninitialized.  I always found these surprising that these come up as they are pretty basic.  But I tend to only see these in complex control flow paths.  So the developer might check for these under normal conditions but forgot on some path.  Especially for legacy code this might not bite you until you change something later on.

So that’s it.  These examples are pretty simple and certainly not reflective of the real world (or at least I hope not).  Later I will post the same idea for Java code.

Since we are in the bug-finding business, we take bugs very seriously and adding them to our backlog (and as a part of our iteration planning) helps ensure we address them quickly.

Although most C/C++ developers recognize explicit resource management as an issue, I’ve recently found out talking to some of our customers that they are totally unaware of the need to worry about such things in Java and especially C#. I even had one customer tell me in the context of C# that the CLR will “take care” of the issue. While it’s perhaps true that eventually the Finalize() method might take care of many resources, professional developers really should take the utmost care in dealing with precious unmanaged resources like file handles, graphics handles, or database connections.

A simple but surprisingly common example in C# illustrates the point:

using System;
using System.IO;

class Blah
{
static void Main()
{
StreamReader r = null;
r = File.OpenText (“blah.txt”);
if (r.EndOfStream) return;
Console.WriteLine (r.ReadToEnd());
}
}

The StreamReader class used here encapsulates an unmanaged resource, the underlying file handle that the O/S allocates to the CLR during the call to OpenText(). As you can see, that unmanaged resource is never released back to the O/S (until such time as the application terminates, of course).

A simple statement:

r.Close();

in a finally block would clean it up.  Calling Close directly is okay but it is crucial to make sure you put that in the finally block.  The most common resource leaks that I see static analysis tools find are the ones where you see a developer has been smart enough to dispose of his resource but there is another path (such as an exception that can be thrown on a method) where the resource is not disposed.

Another easier way to dispose of the resource is to apply the “using” statement, which calls Dispose on an IDisposable object automatically for you. This code:

using (Streamreader r = File.OpenText (“blah.txt”))
{
…
}

is the same as writing:

Streamreader r = File.OpenText (“blah.txt”);
try
{
…
}

finally
{
if (r != null)
((IDisposable)r).Dispose();

}

So make sure you know that you need to dispose of certain resources.  In fairness, these can be hard issues to detect, particularly in an inter-procedural context, but good regimented use of code reviews, coupled with a good static analyzer should get you well on your way.