99 posts
« Previous 1 / 2 / 3 / 4 / 5 Next »10...Last »

Archive for the ‘General Coding’ Category


Golden rules of AST checker development

Posted by Patti Murphy   January 24th, 2012

In my previous post, It’s time to create a custom checker…, we looked at the considerations involved in deciding which checker to create–AST or path?

In this post, we’re going to use a custom checker to enforce an internal coding standard that extends the default set of checkers in our source code analysis tool.

To do this, I’ve called upon Steve Howard, our head of Partner Support in Europe, to get us started with an AST checker to accomplish our goal.

Steve has coached many customers through the checker creation process. In his experience, the appeal of custom checkers lies in their ability to enforce naming conventions and code constructions across organizations.

The standard we want to enforce is the use of a compound statement block rather than single statements as the body of a for loop. An AST checker is the way to go because detection depends solely on the syntax of the code itself and not runtime behavior.

See the example below:

Incorrect: Correct:
for( i – 0; i < 10; i++ )
doSomething( );
for( i – 0; i < 10; i++ ) {
doSomething();
}

To flag this violation, we need to instruct the checker to find all instances of for loop nodes that contain a Statement node as an immediate descendant.

A tool that shows you a visual representation of the AST for the test case is quite helpful in the checker creation process. Here at Klocwork, we use Checker Studio to:

  • browse the AST structure of test cases,
  • identify nodes of interest, and
  • test XPath-like expressions that identify node types, qualifiers, conditions and variables to traverse the AST and flag the defect.

Note: If we wanted to enforce the compound statement rule in all loops, then we’d need to have one pattern (created using the XPath-like expression) for each possible kind, such as while loops and do while loops.

Armed with the test case, Checker Studio, and a syntax guide, Steve identified the following expression that flags the infraction:

// ForStmt [not (Stmt::CompoundStmt)]

Here’s how the test case and expression appear in Checker Studio:

Golden rules

Based on his experience, Steve has a number of golden rules that get you from idea to defect detection faster:

  • Start simple: Use a simple test case that contains the defect you want to detect and work with one simple pattern at a time. Add more complexity as you go along
  • Start rough and refine later: Don’t worry about false positives at first. In some cases it may even be easier to search for  instances that are OK and then negate the rule at the end
  • Divide and conquer: With a more complex checker, work separately on each aspect of the defect you want to detect and then bring it all together at the end for testing in Checker Studio
  • Watch your levels: Make the highlighting as relevant as possible for the issue you’re trying to find. For example, “// ClassType [MemberDecls[*]::MemberDecl]” will highlight classes that match, whereas “// ClassType/MemberDecls[*]::MemberDecl”  will highlight class members that match. The rule is the same, but the focus is different
  • Weed out false negatives: Add negative examples (good code) to check for false negatives

For more information about our custom AST checkers, watch our Checker Studio video.


It’s been a while since our last post, but we’ve been busy…

Posted by Gwyn Fisher   January 17th, 2012

So it’s that time again, when our team finally gets to sleep for a day or two and get their lives back in order. What am I talking about?

Answer: Klocwork Insight 9.5 releases today and boy are we ever happy to see it go live — we’re “out there Jerry” and yes, of course we’re loving every minute of it.

This has been a long release, taking significant research and development to bring to fruition, all the while continuing to release more traditional shipments as we went, but finally culminating in a new, game-changing technology for source code analysis. On-the-fly, as-you-type, instant-like-for-reals, call it what you like… full-on, in-depth C/C++ analysis performed as the developer enters their code, using the “squiggly line” usability metaphor created by spell checkers. It’s one of those “why would you do it any other way” moments and we’re happy to be unique.

Not to be outdone, our web tools team has done amazing stuff with a complete redesign of our Review and Inspect tools, showcasing an awesome look and feel that leverages the toys that come with HTML5 and, amongst many other new capabilities, brings drag/drop pivot report design to the web for on-the-fly metrics and trending analysis that managers and development leads will just eat up.

To our customers, our partners and our friends in the industry, we’d like to say Welcome to Insight 9.5, hope you enjoy it.


Klocwork Insight Logo


It’s time to create a custom checker, but what kind?

Posted by Patti Murphy   November 15th, 2011

You’ve been using source code analysis on your integration build or your desktop, or (ideally) both. And then there’s “a situation”.

The situation

Either you:

  • Noticed a false negative you want detected, or
  • Need a way to enforce a corporate coding standard, such as the requirement for the use of  a compound statement block rather than single statements as the body of a loop.

Now what?

Time to create a custom checker, that’s what. But what kind of checker?

Source code analysis involves two families of checkers, those that involve:

  • Abstract Syntax Tree (AST) validation, and
  • Code path analysis.

An AST provides a tree-based structural representation of the source code. An AST checker allows you to pinpoint problematic syntax using XPath or XPath-derived grammar to define the problem you’re looking for. AST checkers (our version is called Klockwork AST checkers, or KAST for short) don’t require program execution to run; they detect defects right away on source code.

Code path analysis, on the other hand, targets defects related to value tracking at program execution time. Instead of style violations, you’d use a path checker to answer questions such as:

  • Is this newly-created object released before all aliases to it are removed from scope?
  • Is this data object ever range-checked before being passed to an OS function?
  • Is this string checked for special characters before being submitted as an SQL query?

To create a path checker, you don’t need to know how data is tracked by the checker. What you do need to know are the function types and values you want to track for the analysis starting point and the analysis end point where the defect (or event) is recognized and reported.

Which checker when?

Create an AST checker when the problem you want to detect:

  • is a local defect
  • does not involve program execution
  • has to do with the way the program was written
  • does not involve tracking a value
  • does not involve a path

Create a path checker when the problem you want to detect:

  • involves tracking a value
  • has a starting point (where the analysis starts) and end point (where the defect is detected)
  • involves program execution

Stay tuned for the next post in this series on best practices for AST checker creation.

For more information, see Writing custom checkers with Klocwork Extensibility or check out our member discussions in the C/C++ custom checkers forum.

–With files from CTO Gwyn Fisher


Security Issues with Apple iOS?

Posted by Todd Landry   November 8th, 2011

As a pretty avid Apple user (2 iPhones, 1 iPad2, iMac, iPod Touch, MacBook Pro, etc.), and the fact that I work in the business of software quality and security, I must admit that this article caught my attention. The article outlines how a well-known security researcher, who focuses on Apple, has found a software flaw in the iPhone and iPad, which could allow hackers to build malicious apps.What makes this even more scary is that the Apple Store may not catch these malicious apps.

To add another twist to this story, the researcher in question has been ejected from participating in Apple’s developer programs. Read about that here.

Are we now getting to the point where hackers are going to start trying more aggressively to exploit Apple products? In a survey done in 2010, over 50% of respondents thought Windows was either “very” or “extremely” vulnerable compared to only 20% for Apple. I wonder if that has changed? More importantly, do I need to start worrying about my daughter downloading the Archie comic app from the App Store?


What’s the Right Iteration Length?

Posted by Todd Landry   November 1st, 2011

The question of “what’s the right iteration length” may not be as interesting as any of the questions found here (gum really doesn’t stay in you for 7 years. Who knew?), but it is a common question from organizations moving to agile development. You can certainly get a lot of different opinions on this from a Google search, but since you’re reading this now, I’ll give you mine, based on personal experience.

A number of years ago, one of the projects I was PM on decided to try out Scrum. I had attended some Product Owner training, and our soon-to-be Scrum Master had been on some training as well, but we were very green and decided to approach things with a “let’s see what works best for us” mentality. At the time, we thought the best way for us to get immersed and efficient with Scrum was lots of repetitions. We went with 1-week iterations, thinking that by having a rapid and regular cycle of sprint planning meetings, demo meetings, retrospective meetings, etc. we would learn more quickly the “proper” way of doing development with Scrum.

We certainly did learn a lot during our first 3 or 4 sprints, mainly that having this regular weekly cycle of meetings was just too much overhead, and the actual amount of value produced at the end of each sprint was too little. Next on our list, the 2-week sprint.

The 2-week sprints worked great for us, and we saw the differences from the 1-week sprints almost immediately. We were producing what we thought was a good amount of value from each sprint, but with a better and more balanced level of overhead. We hit our groove and established a good cadence with these 2-week sprints, and from the looks of the burn-down chart, we were becoming a more efficient team with every sprint.

The team definitely was cruising and enjoying the pace, but the holiday season snuck up on us and we thought that it might make sense to make some adjustments to deal with the vacation time various team members would be taking.

After collecting everyone’s vacation schedule, we were able to determine a start and finish date for our “holiday sprint” that would essentially start when everyone was still in the office, and finish when everyone returned from their vacation. Call it either luck or good management, but we had planned a 4-week sprint. I won’t go through all the gory details, but let’s just say that upon reflection, the 4-week iteration just felt wrong.

The initial planning session felt harder to estimate the amount of work we could do. The cadence we developed didn’t show itself, and it really felt like we never gained any momentum during the 4 weeks. Now I’m sure that the whole holiday season thing played a role in this, but it was our least effective iteration ever, and by a lot. We never tried the 4-week iteration again.

The bottom line is that all teams are different and need to go with the iteration length that feels right for them. For us, the 2-week one was best.

For the record, I have always wondered if the 7-year rule for chewing gum was true. Glad to hear it isn’t.


Is Pure Agile Always an Option?

Posted by Todd Landry   October 4th, 2011

Over the past few years I’ve talked to a number of customers in the embedded software and medical devices industries, and I continue to see a significant number of these organizations either moving to, or planning on moving to agile development processes.

With all of the inherent challenges for agile in these organizations such as standards/regulatory compliance, hardware changes and integration, security issues, etc. I must say that I’m a little shocked that customers are moving away from their current processes towards something like agile. Add to this the fact that the Agile Manifesto specifically states “Working software over comprehensive documentation” and it doesn’t exactly sound like agile is a great fit for embedded systems in general, let alone medical device.

Now, don’t get me wrong, I am a huge proponent of agile, and I certainly realize that there are many pros for moving to agile, and these have been well documented, but I have to wonder just how agile are these specific industries going?  I would bet that most (all?) of these organizations are adopting some of the key fundamentals of agile, but to say they are going “all in” would be a bit of a stretch.


Even industries heavy on process (because of regulatory requirements) are taking the leap into agile. How agile are they?

Looking at the manifesto a little closer, some of the principles are fairly generic and feel more like common sense than anything revolutionary, so they probably apply to any industry. There are a few principles however that are fairly easy to imagine in these industries, such as:

  • getting all stakeholders involved in defining requirements (Principle #4), or
  • embracing more face-to-face conversations (Principle #6), and
  • simplicity, or minimizing the amount of work not done (Principle #10).

But do people really think that Principles #1 (early and often delivery of software), and #2 (welcome changing requirements) really apply to the embedded or medical devices industries? Personally I don’t see it.

So what do you think? Are the embedded software or medical devices industries capable of going full out Agile?


Secure Coding eLearning Resource

Posted by Brendan Harrison   September 15th, 2011

One of the common challenges we hear from customers regarding their software security assurance programs is developer education. Sure, there are many great tools out there that can help with security, but when it comes down to it, if you’re going to truly build a culture of secure software (and not just audit your system now and then), your development team needs to be well versed on key security concepts, defensive coding principles, common attack vectors, not to mention the ins and outs of specific coding vulnerabilities like buffer overflows.

Secure Coding for C/C++ Course

Well, we agree. That’s why we’ve partnered with our friends at Security Innovation to make some of their developer eLearning courses available for free on the new, revamped Klocwork University. I encourage you to check out the Secure Coding for C/C++ course – it’s approx 60 minutes in length, features interactive material, and is a great introductory course into many of the key concepts required to build secure software. We also have a course on Microsoft’s Secure SDL and the OWASP Top 10. Check it out!


Klocwork University consolidates learning resources into a single roster

Posted by Patti Murphy   September 7th, 2011

Klocwork Developer Network presents Klocwork University, which consolidates all our online learning resources onto a single page.

Klocwork University is your one stop for self-paced online learning and how-tos about:

  • Setting up and using our static analysis tools on your desktop or integration build
  • The latest trends in software security
  • Agile coding practices and how they intersect with static analysis
  • Klocwork product overviews

At Klocwork University you’ll see helpful descriptions of:

  • In-house and partner-generated e-learning courses
  • Video how-tos
  • Webinars

After you browse our offerings on the Klocwork University page, click your selection and access your resource. If you’re not already logged in to the Klocwork Developer Network, you’ll be prompted to log in or register to use these free resources.

This change pulls the course content descriptions from behind the login wall to provide a searchable list for members and non-members alike.

At Klocwork University, you get the information up front and you can schedule your pub breaks when and where you want. Join today. There’s no free beer though.


Squeezing max from the ‘try/finally’ blocks

Posted by Mikhail Ksenzov   August 23rd, 2011

I often hear that closing resources properly is way too verbose in Java, especially considering that resource freeing methods such as ‘close()’ are often throwing some type of an exception. However, if you handle resources properly it might turn out to be less of a burden than one might think. Let’s start with the following snippet, where I use an SQL driver to retrieve the list of “codes” matching the given “id”:

09 List<String> requestCodes(String dbUrl, String id) {
10   List<String> result = new ArrayList<String>();
11   try {
12     Connection conn = DriverManager.getConnection(dbUrl);
13     PreparedStatement stmt = conn.prepareStatement("SELECT * FROM customers WHERE id = ?");
14     stmt.setString(1, id);
15     ResultSet rs = stmt.executeQuery();
16     while (rs.next()) {
17       result.add(rs.getString("code"));
18     }
19   } catch (SQLException e) {
20      e.printStackTrace();
21   }
22   return result;
23 }

The problem with the code above is that it allocates SQL server resources but fails to properly release them. More specifically:

  • Line 12: SQL connection ‘conn’ is not closed on exit.
  • Line 13: SQL object ‘stmt’ is not closed on exit.
  • Line 16: SQL object ‘rs’ is not closed on exit.

The next snippet illustrates how one can fix the defects listed above:

09 List<String> requestCodes(String dbUrl, String id) {
10   List<String> result = new ArrayList<String>();
11   Connection conn = null;
12   PreparedStatement stmt = null;
13   ResultSet rs = null;
14   try {
15     conn = DriverManager.getConnection(dbUrl);
16     stmt = conn.prepareStatement("SELECT * FROM customers WHERE id = ?");
17     stmt.setString(1, id);
18     rs = stmt.executeQuery();
19     while (rs.next()) {
20       result.add(rs.getString("code"));
21     }
22   } catch (SQLException e) {
23     e.printStackTrace();
24   } finally {
25     if (rs != null) {
26       try {
27         rs.close();          // close() throws an exception...
28       } catch (SQLException e) {
29         e.printStackTrace(); // ...have to catch it to free 'stmt'
30       }
31     }
32     if (stmt != null) {
33       try {
34         stmt.close();        // again, close() throws an exception...
35       } catch (SQLException ignore) {
36         e.printStackTrace(); // ...have to catch it to free 'conn'
37       }
38     }
39     if (conn != null) {
30       try {
41         conn.close();
42       } catch (SQLException ignore) {
43         e.printStackTrace();
44       }
45     }
46   }
47   return result;
48 }

The code above is correct but extremely verbose. However, it can be improved without sacrifices in semantics…

Tip #1: It is better to allocate a resource before the ‘try/finally’ block, not inside it. Let’s start with the following code:

Connection conn = null;
try {
  conn = DriverManager.getConnection(dbUrl);
  // use conn
} finally {
  if (conn != null) {
    conn.close();
  }
}

can be rewritten as:

Connection conn = DriverManager.getConnection(dbUrl);
try {
  // use conn
} finally {
  conn.close();
}

Tip #2: Use nested ‘try/finally’ blocks if you allocate a sequence of resources. Let’s start with a snippet:

try {
  Connection conn = DriverManager.getConnection(dbUrl);
  PreparedStatement stmt = conn.prepareStatement("SELECT * FROM customers WHERE id = ?");
  try {
    // use conn
    // use stmt
  } finally {
    try {
      conn.close();
    } catch (SQLException e) {
      e.printStackTrace();
    }
    try {
       stmt.close();
    } catch (SQLException e) {
      e.printStackTrace();
    }
  }
} catch (SQLException e) {
  e.printStackTrace();
}

How many problems have you noticed in the snippet above? I found three:

  • Allocation of resource ‘stmt’ can throw an exception before we enter the outer ‘try/catch/finally’. If happens ‘conn’ will never be freed.
  • We duplicate code for the SQLException handling. We were lucky to have only one line of code replicated, but it in general cases exception handling can be a bit more involved that we see here…
  • The order of resource allocation does not match the order of deallocation: here the order of deallocation should be reversed to be correct.

The only robust way to handle resource allocation/deallocation and to address the issues listed above is to use nested try/finally blocks:

try {
  Connection conn = DriverManager.getConnection(dbUrl);
  try {
    // use conn
    PreparedStatement stmt = conn.prepareStatement("SELECT * FROM customers WHERE id = ?");
    try {
      // use stmt
    } finally {
      stmt.close();
    }
  } finally {
    conn.close();
  }
} catch (SQLException e) {
  e.printStackTrace();
}

Let’s apply tips #1 and #2 to our original method and fix the resource leaks on lines 12, 13, 16:

09 List<String> requestCodes(String dbUrl, String id) {
10   List<String> result = new ArrayList<String>();
11   try {
12     Connection conn = DriverManager.getConnection(dbUrl);
13     try {
14       PreparedStatement stmt = conn.prepareStatement("SELECT * FROM customers WHERE id = ?");
15       try {
16         stmt.setString(1, id);
17         ResultSet rs = stmt.executeQuery();
18         try {
19           while (rs.next()) {
20             result.add(rs.getString("code"));
21           }
22         } finally {
23           rs.close();
24         }
25       } finally {
26          stmt.close();
27       }
28     } finally {
29       conn.close();
30     }
31   } catch (SQLException e) {
32     e.printStackTrace();
33   }
34   return result;
35 }

This is way shorter than the original solution!

Tip #3: If after applying tip #2 you feel that all your code drifted way too close to the right page margin it means that you probably have too much nested ‘try/finally’ blocks and that is time to check if you actually want to have all the resources allocated at the same time. Chances are that you do not need them all; otherwise use the Extract Method refactoring pattern to move out some of the resource access logic.

Tip #4: Know specific behavior of resources you are dealing with. While tips #1 – #3 provide a robust and compact approach for dealing with resource allocation/deallocation in general, in certain cases you can make code even more compact. In our example: Statement.close() closes its current ResultSet object if one exists. Likewise Connections.close() releases JDBC resources. It means that if you deal specifically with JDBC it would be sufficient to close the ‘parent’ resource to be sure that all ‘subresources’ will be properly released:

09 List<String> requestCodes(String dbUrl, String id) {
10   List<String> result = new ArrayList<String>();
11   try {
12     Connection conn = DriverManager.getConnection(dbUrl);
13     try {
14       PreparedStatement stmt = conn.prepareStatement("SELECT * FROM customers WHERE id = ?");
15       stmt.setString(1, id);
16       ResultSet rs = stmt.executeQuery();
17       while (rs.next()) {
18         result.add(rs.getString("code"));
19       }
20     } finally {
21       conn.close();
22     }
23   } catch (SQLException e) {
24     e.printStackTrace();
25   }
26   return result;
27 }

He crossed the line–testing to development

Posted by Patti Murphy   July 12th, 2011

Michail the friendly, programming vampire.

Instead of fomenting dissent (that barely exists) in a brazen attempt to boost readership, I’m changing tactics to look at ways in which testing and development are complementary, beyond their common goal of releasing quality software products.

What can I say? After my previous post, How developers drive testers nuts–let’s count the ways, I’m clearly getting less edgy.

I approached our newest addition to the Klocwork development team, Michail Greshishchev. While he’s a new full-timer, Greshishchev is not a new face around here.

The recent Carleton University engineering graduate did two co-op terms here–one in professional services and the other in testing.

So I asked Greshishchev how his stint in testing affected his development. Here’s exactly what he said:

  1. Writing short, efficient unit tests comes naturally after dealing with mammoth testing frameworks. Most of the code I write are tests – the techniques and skills I’ve learned in testing are fully applicable to development.
  2. Developers have no idea how to execute a test in our automated test system (I don’t blame them–the test machine is a large, well-oiled beast distributed over dozens of operating environments). Having the knowledge to run test team tests on developer builds means I don’t need to wait for nightly build test results to address issues. More importantly, I can add my own tests to the test team’s automated test system.
  3. It’s common for a developer to request more information about a tester’s problem report. My experience with the test team allows me to access the information on test machines myself, saving time for everyone.
  4. The test report pages actually make sense. This allows me to investigate test failures in the nightly build before a tester comes to my desk to tell me I broke something.

His experience as part of the test team has been win-win for him and his colleagues. Testing and development sound like allies, don’t they? Well, as much as werewolves and vampires can be allies, I suppose. And he was such a nice guy too, but the change is upon him.