One of the common challenges we hear from customers regarding their software security assurance programs is developer education. Sure, there are many great tools out there that can help with security, but when it comes down to it, if you’re going to truly build a culture of secure software (and not just audit your system now and then), your development team needs to be well versed on key security concepts, defensive coding principles, common attack vectors, not to mention the ins and outs of specific coding vulnerabilities like buffer overflows.

Well, we agree. That’s why we’ve partnered with our friends at Security Innovation to make some of their developer eLearning courses available for free on the new, revamped Klocwork University. I encourage you to check out the Secure Coding for C/C++ course – it’s approx 60 minutes in length, features interactive material, and is a great introductory course into many of the key concepts required to build secure software. We also have a course on Microsoft’s Secure SDL and the OWASP Top 10. Check it out!

The Code Integrity folks have developed a lot of best practices on deploying static analysis and have compiled many of them in a solid whitepaper. They include a Top 10 list of what they call “The Well Traveled Path to Success”. Below is their (somewhat paraphrased in spots) list.

1. Determine who cares. Who has a vested interest that bugs actually get fixed. How much do they care?

2. Get an expert to tune the solution for your codebase. Static analysis tuning will maximize defect finding while minimizing false positives.
3. If possible, pilot with a small group to gain early successes.
4. Appoint the proper roles, particularly management sponsor, administrator, defect triagers, fixers and verifiers.
5. Set up the proper process, incentives and consequences. Integrate the SCA tool into your environment. Automate and simplify as much as possible.
6. Get a team to handpick good, high-priority defects for the team rather than have them sift through potential false positives.
7. Set up a central resource website that includes simplified documentation, policies, procedures, roles, reports, etc.
8. Set up various reports like the daily dashboard, top ten list and the “wall of shame”. Make it public. Do a little bit of marketing.
9. Train and mentor the team providing guidance, support and discipline. Either in-person or static analysis e-learning courses work.
10. Determine success criteria and measure it. Provide status updates often, work on a source code analysis ROI model that works for your organization.

I agree with the general thrust of most of these, but some might be overkill depending on the size of your deployment. My other quibble is that many of the recommendations presume a centralized defect triage model where you’d have a central group of code reviewers sifting though bug reports.

That’s a common deployment model, but we’re seeing more people choose to just provide the tool to their developers via desktop static analysis. With the possible exception of your backlog, this will eliminate (or greatly reduce) the need for a central code review team that stares at bugs all day long. Regardless, they’re all good considerations to at least, well… consider.

With the launch of the Klocwork Developer Network, we’re making a deliberate and concerted effort to make many of these kinds of deployment resources freely available to our customers. I’ve included links where appropriate.

We’ve talked at length before regarding software security assurance and the role static analysis can play in ensuring code is written securely. We’ve got a bunch of great resources for anyone looking to dive into this particular aspect of software security:

• Summary of various secure coding standards, including links to specific checkers supported by Klocwork
• Free secure coding e-learning courses, including an intro to Microsoft’s secure development lifecycle
• A ‘buyer’s guide’ to selecting a static analysis tool as part of a secure coding program authored by a major payment software company

To build on this, next month our CTO Gwyn Fisher and the CTO of Security Innovation, Jason Taylor will be hosting a talk that expands the discussion beyond secure coding strategies alone. Jason will be talking at length on how to build a threat model for software, in particular embedded software. Gwyn will then walk through how customers should be building their software with this threat model in mind – everything from code reviews to static analysis and testing strategies. I urge you to register for the webinar and check it out – there will be lots of good information being discussed.

Static Analysis Tools Not Created Equal

Yes, it’s true (!) and as anyone in this space knows there is a huge difference between static analysis tools, their level of sophistication, and their approach to developer adoption. Gary McGraw & John Steven from Cigital describe their views on this topic including ‘5 pitfalls’ that customers should avoid when evaluating tools. These pitfalls mostly amount to the fact that analysis results across different tools, code bases, and tool operators can make results vary significantly, so be aware of this fact when conducting your benchmarking. Their overall recommendation:

“The upshot? Use your own code instead of a pre-fab evaluation suite. You probably have the makings of a good set of tests within your own organization’s application base….”

I agree with this and can honestly say we rarely, if ever, run into evaluations where customers exclusively use pre-fab test suites instead of their own code for many of the reasons outlined in their article. So, I’d say the market is (and has been for some time) embracing this recommendation wholeheartedly. So, beyond this recommendation, what else should customers consider when evaluating these tools? Here are a couple other significant areas to consider where you’ll find that, yes, all tools aren’t created equal.

• Environment support. In particular in the embedded software space, considerations such as integration with your build environments, compiler support, ability to work with multiple software branches, are all crucial considerations for a successful deployment. Not all tools have good support in these areas, but these capabilities can often make or break a deployment.
• Developer adoption. This is everything frankly, and a big part of achieving developer adoption is the quality of the analysis issues raised in the article. Obviously, a tool that generates accurate, useful results will get you well on your way to strong developer adoption, but that’s not everything. How are the defects described to developers, including the trace info? Do developers want to run their own desktop static analysis rather than fetching results periodically from the integration build? If so, how smart is the vendor’s desktop analysis?

So basically, picking a tool boils down to assessing: quality & flexibility of the analysis, support for your dev environment (not just the one you’re using in the eval!), and thinking ahead to developer adoption issues. Assess these three areas thoroughly and you’ll end-up picking the tool that’s right for your needs.

Nice blog post from Walter Bright over at Dr. Dobbs on the Patterns of Bugs. He ties together bug patterns, recommended process changes, and the resulting productivity payoff from making these improvements. He recommends a bunch of process changes, including static analysis, code reviews, and coding standards, then goes on to review examples of different bug patterns. A few can be detected with static analysis (coding mistakes as written) but many are errors with the code as intended (something static analysis doesn’t check… that’s what testing is for). His main recommendation seems to be that bugs can often be pattern based, so once a bug is identified, take steps to remove that pattern from your code through process or tool changes.

Once the pattern is identified, then look for changes in process that will permanently eliminate that bug. Eliminate enough of the bug patterns, and you should enjoy a substantial increase in productivity.

His experience is that, over time this kind of systematic approach to fixing bugs makes developers better.

I’ve noticed in my decades of writing programs that I just don’t make the kinds of mistakes I used to. Apparently I’ve unconsciously evolved coping strategies to avoid them. Identifying and building such strategies into the process means everyone can benefit from that experience.

We certainly see that in the area of static analysis bug detection; lots of customers report that their developers make fewer mistakes, defect injection rates go down, and overall productivity is improved.

Ok… bad pun but question still stands. We wanted to try and answer that question so we worked with the team at VDC Research to try and quantify some of these questions. You can download their full report on the multicore and multiprocessor landscape, but here’s one piece of data that I thought might be interesting. Basically, heterogeneous processor architectures are growing quickly and the number of projects using simple processor architectures is diminishing fast.

Data from VDC Research showing trend in multiprocessor development

Really, backs-up what we all instinctively know and understand but nice to see some empirical evidence to add to the mix of customer feedback, social media chatter, and articles we’ve been seeing on the topic. We also tried to collect a bunch of other resources on this topic in one spot with a previous endian blog post we published.

Of course, our focus is on the code analysis tooling available to address these issues. If you’re doing a port, the prospect of manually (or semi-manually) crawling through thousands of instances where you need to modify code to support a different endian format really sucks. Some automation would certainly be nice. Our CTO Gwyn Fisher will be talking in more detail on this topic next week. He has a presentation on improving software reliability in multicore and multiprocessor architectures. Check it out.

We get many questions from medical devices customers on how they should validate the use of Klocwork’s static analysis tools for the FDA. I suspect the situation would be similar for most vendors of software development tools. As we’ve done before, we thought it would be a good idea to ask Bruce Swope from SterlingTech Software to clarify this whole topic for us.

[Brendan] First, what is tool validation?
[Bruce] Tool validation is the act of demonstrating that a tool will consistently produce expected results.

[Brendan] How can a medical device company know whether they should validate a tool or not?
[Bruce] From 21CFR820.75, “Where the results of a process cannot be fully verified by subsequent inspection and test, the process shall be validated with a high degree of assurance and approved according to established procedures”. For example, if you are using a tool to perform work and you do not plan on using any other method to verify that the work was done properly then the tool will need validation. Please note that you must validate the tool for your intended use, not the entire tool.

FDA Tool Validation Checklist

[Brendan] Ok, let’s take a specific example. What would validating a static analysis tool involve? 
[Bruce] Here’s a basic list of what needs to be completed.

a) Develop a Tool Validation Plan. This should include the test environment and methods to be used
b) Develop a set of requirements that the tool is intended to meet
c) Develop a test protocol to verify that the requirements have been met
d) Develop a traceability matrix and verify that all requirements have been tested
e) Execute the test protocol
f) Write a test report summarizing the results

[Brendan] Do most companies do this themselves or can they outsource this activity? 
[Bruce] If the company has the internal trained resources available then they can certainly do the tool validation themselves. Many companies lack the staffing, or expertise, to perform validation of a software tool. It is common for these companies to outsource the documentation and testing activities to a firm like SterlingTech. We’ve completed this process numerous times before and it’s a good way to reduce the cost and effort around the validation process.

[Brendan] Thanks Bruce.

Complexity of Medical Device Software Continues to Grow

IEC 62304 is becoming a hot topic amongst medical device software professionals. We asked Bruce Swope, VP Engineering at SterlingTech Software for his views on this standard and what it all means for medical device companies. Bruce has extensive experience in medical device software development and he is an expert in leading Class III medical software products to commercial release. His depth of experience also spans the development of enterprise solutions, security applications, internal applications, and process control systems. He has been an early adopter of quality practices including ISO 9000 processes, Common Criteria Certification and Capability Maturity Model implementation.

[Brendan] Thanks Bruce.

As readers know, we’ve been talking about code reviews pretty regularly here and elsewhere over the past few months. To continue that discussion, here’s a question we run into often: are in-person code reviews as the primary way to communicate, by definition a bad thing?

Here’s some more data from the Forrester Consulting study commissioned by Klocwork that shows the majority of respondents still conduct in-person reviews… elsewhere in the survey only 36% of respondents indicated that they worked on a centralized team with everyone in one location. So that means, if 60% still conduct in-person reviews, they’re likely excluding valuable contributors to the review.

Data that shows majority still conduct in-person code reviews

Is this practice just being done because “that’s the way it is” or are there good reasons for in-person meetings being the primary way to review code? I could see the odd in-person meeting being necessary for a variety of reasons but given how distributed teams are these days and the variety of tools available to effectively review code remotely, it doesn’t seem that efficient.

There’s a general philosophy gaining more prominence around meeting reduction, whether in software development or elsewhere. We’re seeing many organizations question why their code review process needs to be in-person when it excludes people who aren’t co-located and generally takes up too much of people’s time. What are you seeing?

It’s often taken as read that developers think code reviews are just a pain in the behind. Maybe that sentiment is true when a developer’s sitting amongst his/her peers and getting interrogated on the quality of their code, but some of the data from a Forrester Consulting study commissioned by Klocwork seems to contradict that a bit. The survey asked software development professionals a whole bunch of questions related to code reviews (some of which we’ve referenced before) and here are two interesting data points that suggest developers see real benefits from code reviews.

So 79% of respondents indicate that, yes, code reviews have been effective at reducing the number of bugs found later in the development cycle. Furthermore, 43% state that code reviews have caused a fundamentally positive shift in their project’s direction. Cool.

Of course, in other parts of the survey, respondents complain about aspects of code review, in particular how time consuming and difficult they can be to implement consistently. Nonetheless, the data indicates that when organizations put their heads down and make them part of their development process, real benefits will be realized. So, the challenge is making them part of the process – of course we advocate a tools-based approach, making them more lightweight, and combining automation into your software verification strategy so that manual reviews aren’t the only technique being used to find implementation errors.

This data line-up with what you’re seeing within your organization?