Michail the friendly, programming vampire.

Instead of fomenting dissent (that barely exists) in a brazen attempt to boost readership, I’m changing tactics to look at ways in which testing and development are complementary, beyond their common goal of releasing quality software products.

What can I say? After my previous post, How developers drive testers nuts–let’s count the ways, I’m clearly getting less edgy.

I approached our newest addition to the Klocwork development team, Michail Greshishchev. While he’s a new full-timer, Greshishchev is not a new face around here.

The recent Carleton University engineering graduate did two co-op terms here–one in professional services and the other in testing.

So I asked Greshishchev how his stint in testing affected his development. Here’s exactly what he said:

1. Writing short, efficient unit tests comes naturally after dealing with mammoth testing frameworks. Most of the code I write are tests – the techniques and skills I’ve learned in testing are fully applicable to development.
2. Developers have no idea how to execute a test in our automated test system (I don’t blame them–the test machine is a large, well-oiled beast distributed over dozens of operating environments). Having the knowledge to run test team tests on developer builds means I don’t need to wait for nightly build test results to address issues. More importantly, I can add my own tests to the test team’s automated test system.
3. It’s common for a developer to request more information about a tester’s problem report. My experience with the test team allows me to access the information on test machines myself, saving time for everyone.
4. The test report pages actually make sense. This allows me to investigate test failures in the nightly build before a tester comes to my desk to tell me I broke something.

His experience as part of the test team has been win-win for him and his colleagues. Testing and development sound like allies, don’t they? Well, as much as werewolves and vampires can be allies, I suppose. And he was such a nice guy too, but the change is upon him.

In my last post, I took us back in time to an era of bad fashion, questionable music, legendary television shows, and source code analysis tools that were made specifically for software developers. It was the 1970s. In this post, I fast forward to just after the turn of the century to discuss the next evolution of static analysis tools.

The Early 21^st Century

Not long after we first viewed hairy-footed Hobbits on the silver screen, and the sham that was affectionately known as Y2K, a new generation of source code analysis tools emerged to cure the errors of the first-generation tools.

These new tools looked beyond the syntactical analysis of previous tools, and instead provided inter-procedural and data-flow analysis. Low hanging fruit was definitely not the target for these tools.

These new techniques were serious–finding complex defects that could impact code quality and security, and they did that while ensuring that the “noise” (i.e. false positive rate) was greatly reduced compared to the first-generation tools. In addition to local defects, they were now identifying resource management issues, security vulnerabilities, concurrency issues, and so on. These were serious defects that,  if left undetected and unfixed, had the potential for massive problems to the code stream.

In order to perform this much deeper analysis, a fundamental change in the analysis techniques had to occur. These engines needed an unfiltered view of the entire code stream, and so they became tightly integrated with the integration build process.

Umm, Houston, we have a problem. If the analysis takes place at integration build time, then that means the analysis is no longer being initiated by the developers. Source code analysis tools became centralized and moved into a more downstream process such as part of a code audit function.

Developers were now being told they created bugs well after they actually checked in the code. They had already moved onto something entirely different, so now bringing them these day-old, or week-old defects was certainly not the most productive use of their time. It is well documented that the earlier you find defects in your code, the more cost effective it is to fix them, so you can clearly see the problems with these second-generation tools.

If only there was a way to bring these second-generation analysis capabilities to the developer desktop. More about that in my next entry.

Our marketing people here at Klocwork like to see me racking up frequent flyer miles and expending CO2 at roadshows, conferences and tradeshows. Whenever I’m out speaking, I always like to gauge audience familiarity with Static Code Analysis.

I’m happy to say that SCA knowledge has definitely increased over the years, but it is still not up to levels enjoyed by unit testing or integration testing.

What I plan to do over the next three weeks is to provide you with a history lesson on how Static Code Analysis has evolved over the past few decades (yes, it has been around that long!). The three different eras I will be addressing are The Early Years, The Early 21^st Century, and  The Present Day.

The Early Years

As I mentioned earlier, Static Code Analysis has actually been around since the time of bell bottoms, disco music, and Space Invaders (check out the Space Invaders link)–yes, the good ole 1970s. Who out there has heard of Lint (and no, I’m not talking about the fluff coming from your old bell bottoms pockets)?

Lint was one of these first-generation SCA tools introduced in the late 70s. These tools targeted low hanging fruit in C code, such as missing or extra semi-colons, missing curlicues, potentially dangerous implicit casts, and so on.

These tools were closely integrated with the compile and link process, and so this seemed like the best time to show its errors and warnings, while the developer was actually “in process” and fixing problems found by the compiler. Since these tools delivered its warnings at compile time, it quickly became a tool that was adopted and owned by the developers themselves.

Life was good. Well, until the bugs that were being found were deemed to be relatively trivial or completely erroneous (the dreaded false positive). The problem was that these tools were only able to see one file at a time, but for accurate static analysis there is a strong need to know everything that’s going on within the entire stream.

Without that vision of the entire stream, no matter how sophisticated the analysis tools are, they will make incorrect assumptions most of the time. Because of these inaccuracies, these first-generation tools never gained the widespread acceptance of software developers.

Next up will be a look at static analysis tools during a time when “Whassssuuuupp” became a household term.

The two sides of testing team lead Jonathan Patchell.

At daily standup meetings, they eye each other from opposite sides of the room. Sitting on the same side of the cubicle wall is unthinkable.

They’re united only by their desire to produce quality software products and their appreciation for coffee and energy drinks. What’s good to one side can be anathema to the other when it comes to code.

I’m talking, of course, about testing and development teams. In the interests of generating more comments improving dialogue between two very important functions in a software organization, our marketing director asked me to interview our testing team lead, Jonathan Patchell, about the ways in which developers drive his team nuts.

Patchell, a computer systems engineer, has been with Klocwork for five years and a team lead for two. He struck a fairly conciliatory tone for this interview, which sorta ruins the adversarial approach, but don’t let his diplomacy fool you. I’ve seen him suffering as the release date approaches and his demeanour changes completely.

Here are Patchell’s top dev peeves:

In a previous post I discussed the process where we practice dogfooding.  This is the process of using Klocwork on Klocwork (KonK).  We started this program several years back with the hopes that we would learn some valuable lessons about usability, performance and anything else that would give us an edge.  The truth is that KonK has consistently allowed us to test our design assumptions early by allowing our own developers to use Klocwork as part of their development.

One of the unexpected results was inadvertently uncovering data that further validated for us the importance of bugs found by a static analysis tool. We correlated testing-found issues and KonK (static analysis) issues assigned to each developer. The result as this graph shows (note: graph is using a logarithmic scale and is a few years old), is there was a very high relationship between the two. Those familiar with bugs found by static analysis tools know that they are often quite different in nature from a bug found by testing where you’re testing requirements, yet developers with a high count of testing-found issues also had a high KonK count. In some cases they pointed to the same problem, in others an indication of the overall bad to the bone problems in some of the new components.

Since we already eat our own dogfood, we’re already believers in the use of static analysis, but this kind of data is a great example of the benefit of finding issues early using static analysis

Testing can have many goals: to assess quality, to assess conformance to specification, to help managers decide whether or not to ship, etc. These all affect a test team’s approach.  Very often, it seems that “Finding Important Bugs Early” is the tester’s primary goal during a development cycle in an agile environment. It’s the ideal: find a quality impacting issue as quickly as possible after it was introduced so that the code is fresh in developer’s minds. “Finding Important Bugs Early” is a goal shared by countless test teams.

It struck me as odd, then, that James Whittaker of JW on Test in his excellent “The Future of Software Testing” series (Part 4), stated that in the future, we need to “close the gap between when a bug is created and when that same bug is detected.” In my opinion, why wait? There are plenty of things we can do to close that gap right now!

The most efficient tactics I’ve seen are the following: early involvement of testers in the development process, good unit testing, code inspection, static code analysis, continuous integration, and effective use of stubs.

Early involvement of testers is not a new concept; it’s already a part of good test process practices. For example, it is a key area of TPI (Test Process Improvement) known as the “moment of involvement”. As a bonus, I’ve always found that testers make great requirement appraisers; not only can they help assess that a requirement is verifiable, but also that it is complete, consistent and unambiguous.  Keeping testers in the loop is especially important in an agile setting where it can help make the documentation barely sufficient, as recommended by this type of development process.

We all know that unit tests are your “first line of defence” for testing executable code. To the developers to whom this task most often falls, I encourage you to ask your test team for help should the need arise. A good test team member has a way of seeing testing from a unique perspective. Use this hidden resource! You can pay the favour back when it’s time for them to use a new automation framework. It’s worth it: a good unit test suite can find many problems early that can be hard to diagnose later downstream.

Code inspection is another early detection tool. Used effectively, it can find logic and design problems at the source, not as a symptom (i.e. when system testing will stumble upon the issue). Moreover, there is no prerequisite of compilation to start it! The same is true for static code analysis. We all know the benefits so apply early, apply often, and save a lot of trouble later on.

For environments that are not of that kind where you “throw the product over the wall” to be tested, continuous integration is necessary. Otherwise, your process may be hindering the test team’s effort to Find Bugs Early. Should your testers be validating last week’s code? Last month’s? If they are, they have no chance of finding your most immediate problems. Don’t forget about bug interactions. It frequently happens that a bug is hidden behind another bug; fix one and possibly expose a slew of others. I heard a question asked once that truly captures the idea: “Why is it that when a bug dies, all the others come to its funeral?” You won’t know who’s in attendance if you’re not testing the latest code.

You may recall that I mentioned the effective use of stubs at the beginning of this post. I hope it piqued your curiosity. It’s something I learned by observing effective testers over the years. When a widget comes off the conveyor belt, they don’t wait for the thingamajig it screws into to start testing. The thingamajig may not be ready for another week! They instead whip up a mock-up of the thingamajig that works somewhat like the real thingamajig and try out their widget, pronto! Very often, the combination doesn’t work as expected and they must ask themselves: are their assumptions about the thingamajig wrong or is it the widget itself? It’s a good thing they didn’t wait until next week to discover this potential problem! This type of experiment enables early integration testing: the component didn’t work correctly in its intended environment. This prompts an investigation into the root cause: is it a bug in the component or a poorly defined interface between two components? Beware however, software stubs can fall into the same trap as a test oracle: you can make it as complex as the real thing if you’re not careful.

These are but a few of the tools for Finding Important Bugs Early. There are many other testing approaches that are not addressed here (so don’t throw away your regression tests!) This post focussed on the “early” part of the goal. What about tactics for the “important” part? Good question! I might be coerced into a follow up post one day…

Here’s Gwyn chatting about general software development challenges, in particular the whole goal of “in-phase defect containment” – i.e. identifying and correcting defects in the same development phase they’re created. Near the end of the video, there’s a short discussion on how this objective fits in an Agile context. With Agile’s focus on the frequent delivery of working software, in-phase containment becomes even more important, even though it’s more often associated with more formal methodologies such as CMMI and Six Sigma.