Testing can have many goals: to assess quality, to assess conformance to specification, to help managers decide whether or not to ship, etc. These all affect a test team’s approach.  Very often, it seems that “Finding Important Bugs Early” is the tester’s primary goal during a development cycle in an agile environment. It’s the ideal: find a quality impacting issue as quickly as possible after it was introduced so that the code is fresh in developer’s minds. “Finding Important Bugs Early” is a goal shared by countless test teams.

It struck me as odd, then, that James Whittaker of JW on Test in his excellent “The Future of Software Testing” series (Part 4), stated that in the future, we need to “close the gap between when a bug is created and when that same bug is detected.” In my opinion, why wait? There are plenty of things we can do to close that gap right now!

The most efficient tactics I’ve seen are the following: early involvement of testers in the development process, good unit testing, code inspection, static code analysis, continuous integration, and effective use of stubs.

Early involvement of testers is not a new concept; it’s already a part of good test process practices. For example, it is a key area of TPI (Test Process Improvement) known as the “moment of involvement”. As a bonus, I’ve always found that testers make great requirement appraisers; not only can they help assess that a requirement is verifiable, but also that it is complete, consistent and unambiguous.  Keeping testers in the loop is especially important in an agile setting where it can help make the documentation barely sufficient, as recommended by this type of development process.

We all know that unit tests are your “first line of defence” for testing executable code. To the developers to whom this task most often falls, I encourage you to ask your test team for help should the need arise. A good test team member has a way of seeing testing from a unique perspective. Use this hidden resource! You can pay the favour back when it’s time for them to use a new automation framework. It’s worth it: a good unit test suite can find many problems early that can be hard to diagnose later downstream.

Code inspection is another early detection tool. Used effectively, it can find logic and design problems at the source, not as a symptom (i.e. when system testing will stumble upon the issue). Moreover, there is no prerequisite of compilation to start it! The same is true for static code analysis. We all know the benefits so apply early, apply often, and save a lot of trouble later on.

For environments that are not of that kind where you “throw the product over the wall” to be tested, continuous integration is necessary. Otherwise, your process may be hindering the test team’s effort to Find Bugs Early. Should your testers be validating last week’s code? Last month’s? If they are, they have no chance of finding your most immediate problems. Don’t forget about bug interactions. It frequently happens that a bug is hidden behind another bug; fix one and possibly expose a slew of others. I heard a question asked once that truly captures the idea: “Why is it that when a bug dies, all the others come to its funeral?” You won’t know who’s in attendance if you’re not testing the latest code.

You may recall that I mentioned the effective use of stubs at the beginning of this post. I hope it piqued your curiosity. It’s something I learned by observing effective testers over the years. When a widget comes off the conveyor belt, they don’t wait for the thingamajig it screws into to start testing. The thingamajig may not be ready for another week! They instead whip up a mock-up of the thingamajig that works somewhat like the real thingamajig and try out their widget, pronto! Very often, the combination doesn’t work as expected and they must ask themselves: are their assumptions about the thingamajig wrong or is it the widget itself? It’s a good thing they didn’t wait until next week to discover this potential problem! This type of experiment enables early integration testing: the component didn’t work correctly in its intended environment. This prompts an investigation into the root cause: is it a bug in the component or a poorly defined interface between two components? Beware however, software stubs can fall into the same trap as a test oracle: you can make it as complex as the real thing if you’re not careful.

These are but a few of the tools for Finding Important Bugs Early. There are many other testing approaches that are not addressed here (so don’t throw away your regression tests!) This post focussed on the “early” part of the goal. What about tactics for the “important” part? Good question! I might be coerced into a follow up post one day…

Here’s Gwyn chatting about general software development challenges, in particular the whole goal of “in-phase defect containment” – i.e. identifying and correcting defects in the same development phase they’re created. Near the end of the video, there’s a short discussion on how this objective fits in an Agile context. With Agile’s focus on the frequent delivery of working software, in-phase containment becomes even more important, even though it’s more often associated with more formal methodologies such as CMMI and Six Sigma.

Have just stumbled across the lamda module in boost (popular C++ general-purpose library known for extensive usage of templates and influence on C++ standard committee).

A quote:

The primary motivation for the BLL (Boost Lambda Library) is to provide flexible and convenient means to define unnamed function objects for STL algorithms …

for_each(a.begin(), a.end(), std::cout << _1 << ' ');

My first thought was: "Hmm ... a macro?" It appears it is not. The <code>_1 object is a lambda placeholder, and should be read as first parameter of lambda expression (a.k.a. unnamed function). In fact the std::cout << _1 << ' ' expression is automatically converted into a function-like object, that can be used with most STL algorithms (like for_each, find_if, etc.) So, instead of writing this (a traditional STL way): template <typename T> struct my_printer : public std::unary_function<void, T> {      void operator()(const T &x)      { std::cout << x << ‘ ‘; } }; // … for_each(a.begin(), a.end(), my_printer<my_element_type>());

You can use the above expression, and C++ compiler automatically creates anonymous class that represents an unnamed function. The trick is that all this "behind-the-scene" work is expressed in terms of C++ templates and heavily relies on automatic type inference, and operator overloading. Essentially the module has to overload every possible C++ operator that can be applied to the lambda placeholder, "delay" the evaluation of expression and "wrap" the computation into a callable object.

This approach apparently simulates Lisp lambda expressions (hence the name of the module) and Smalltalk/Ruby code blocks:

a do: [ :⁣x | Transcript show: x; show: ' ' ]

a.each { |x| print x, " " }

(for-each (lambda (x) (display x) (display " ")) a)

Lambda expressions and code blocks are essential part of these languages and one of the things that make them consistent and fun to use. Now, this library is trying to adopt this concept into C++.

Pretty cool, but the following limitations lead me to think that this C++ implementation of lambda expressions is far from complete:

• Boost lambda expressions are not true closures (you can not refer to "non-local" data from the block)
• Some operators (most notably "." and assignment) can not be overloaded this way, so i = _1 and _1.my_field does not work and requires some extra C++ tricks
• Can not use statements in this kind of code blocks, works only with expressions
• If you make a simple mistake in this kind of lambda-expression (or change some related interface), the C++ compiler will gladly present you with a pile of human-unreadable error messages with about a dozen template instantiations with names that do not fit a line of text (no matter how wide your LCD panel is), using each other ... that's a common problem of all non-trivial template libraries

After all, this demonstrates that C++ templates and type inference are indeed powerful concepts, but still not powerful enough to redefine/extend the language in consistent way.

Another problem with this kind of smart C++ tricks is that it creates an illusion of simplicity. As always with C++ - you can not rely on this simplicity, unless you know the details of what's under the hood. If you dare to ignore implementation details you risk losing control over your own code one day.

I’m picking up the theme of the CWE Top 25 today (see posts below for more detail on the list itself, or check out this blog posting for a more exhaustive description) as we run into what is described as CWE-404 all the time in managed code environments.

Although most C/C++ developers recognize explicit resource management as an issue, I’ve recently found out talking to some of our customers that they are totally unaware of the need to worry about such things in Java and especially C#. I even had one customer tell me in the context of C# that the CLR will “take care” of the issue. While it’s perhaps true that eventually the Finalize() method might take care of many resources, professional developers really should take the utmost care in dealing with precious unmanaged resources like file handles, graphics handles, or database connections.

A simple but surprisingly common example in C# illustrates the point:

using System;
using System.IO;

class Blah
{
static void Main()
{
StreamReader r = null;
r = File.OpenText (“blah.txt”);
if (r.EndOfStream) return;
Console.WriteLine (r.ReadToEnd());
}
}

The StreamReader class used here encapsulates an unmanaged resource, the underlying file handle that the O/S allocates to the CLR during the call to OpenText(). As you can see, that unmanaged resource is never released back to the O/S (until such time as the application terminates, of course).

A simple statement:

r.Close();

in a finally block would clean it up.  Calling Close directly is okay but it is crucial to make sure you put that in the finally block.  The most common resource leaks that I see static analysis tools find are the ones where you see a developer has been smart enough to dispose of his resource but there is another path (such as an exception that can be thrown on a method) where the resource is not disposed.

Another easier way to dispose of the resource is to apply the “using” statement, which calls Dispose on an IDisposable object automatically for you. This code:

using (Streamreader r = File.OpenText (“blah.txt”))
{
…
}

is the same as writing:

Streamreader r = File.OpenText (“blah.txt”);
try
{
…
}

finally
{
if (r != null)
((IDisposable)r).Dispose();

}

So make sure you know that you need to dispose of certain resources.  In fairness, these can be hard issues to detect, particularly in an inter-procedural context, but good regimented use of code reviews, coupled with a good static analyzer should get you well on your way.