We get many questions from medical devices customers on how they should validate the use of Klocwork’s static analysis tools for the FDA. I suspect the situation would be similar for most vendors of software development tools. As we’ve done before, we thought it would be a good idea to ask Bruce Swope from SterlingTech Software to clarify this whole topic for us.

[Brendan] First, what is tool validation?
[Bruce] Tool validation is the act of demonstrating that a tool will consistently produce expected results.

[Brendan] How can a medical device company know whether they should validate a tool or not?
[Bruce] From 21CFR820.75, “Where the results of a process cannot be fully verified by subsequent inspection and test, the process shall be validated with a high degree of assurance and approved according to established procedures”. For example, if you are using a tool to perform work and you do not plan on using any other method to verify that the work was done properly then the tool will need validation. Please note that you must validate the tool for your intended use, not the entire tool.

FDA Tool Validation Checklist

[Brendan] Ok, let’s take a specific example. What would validating a static analysis tool involve? 
[Bruce] Here’s a basic list of what needs to be completed.

a) Develop a Tool Validation Plan. This should include the test environment and methods to be used
b) Develop a set of requirements that the tool is intended to meet
c) Develop a test protocol to verify that the requirements have been met
d) Develop a traceability matrix and verify that all requirements have been tested
e) Execute the test protocol
f) Write a test report summarizing the results

[Brendan] Do most companies do this themselves or can they outsource this activity? 
[Bruce] If the company has the internal trained resources available then they can certainly do the tool validation themselves. Many companies lack the staffing, or expertise, to perform validation of a software tool. It is common for these companies to outsource the documentation and testing activities to a firm like SterlingTech. We’ve completed this process numerous times before and it’s a good way to reduce the cost and effort around the validation process.

[Brendan] Thanks Bruce.

Complexity of Medical Device Software Continues to Grow

IEC 62304 is becoming a hot topic amongst medical device software professionals. We asked Bruce Swope, VP Engineering at SterlingTech Software for his views on this standard and what it all means for medical device companies. Bruce has extensive experience in medical device software development and he is an expert in leading Class III medical software products to commercial release. His depth of experience also spans the development of enterprise solutions, security applications, internal applications, and process control systems. He has been an early adopter of quality practices including ISO 9000 processes, Common Criteria Certification and Capability Maturity Model implementation.

[Brendan] Thanks Bruce.

With the recent story that the iPad has inherent security vulnerabilities, I thought it might be an appropriate time to delve into the world of software security guidelines…but I must warn you, this blog will contain an abnormal amount of acronyms, and may not be suitable for all audiences.

When talking about software security guidelines, there are really 5 or 6 organizations that are leading the charge, and they include:

-          OWASP

-          SANS Institute

-          MITRE

-          PCI Security Standards Council

-          SEI

Let’s first look at OWASP. OWASP stands for Open Web Application Security Project, which is a not-for-profit charitable organization that is focused on improving the security of application software. They are probably best known for their Top 10 lists from 2004, 2007, and most recently 2010.

Next is the SANS Institute. SANS of course is a FLA that stands for SysAdmin, Audit, Networking, Security. The SANS Institute claims to be the most trusted source for computer security training, certification and research, and have been developing and releasing their Top 20 annually for the past 7 years or so.

The MITRE Corporation is a not-for-profit organization that was founded in the late 50’s, and has over 7,000 very smart dudes (65% have Masters or PhDs). MITRE has come up with their own security guideline as well, that is the CWE (Common Weakness Enumeration) and it provides a common language of discourse for discussing, finding and dealing with the causes of software security vulnerabilities as they are found in code, design, or system architecture. The CWE lists over 800 programming errors, design errors, and architectural errors that can lead to exploitable vulnerabilities. Interestingly, MITRE and SANS decided to collaborate to come up with the CWE Top 25, yet another “Top” list they have been putting together for the last couple of years.

The PCI Security Standards Council was founded by American Express, Discover Financial Services, JCB International, MasterCard Worldwide, and Visa, Inc. and is an open global forum for the ongoing development, enhancement, storage, dissemination and implementation of security standards for account data protection. The PCI SSC has come up with the PCI DSS, “a multifaceted security standard that includes requirements for security management, policies, procedures, network architecture, software design and other critical protective measures. This comprehensive standard is intended to help organizations proactively protect customer account data”.

Finally, there is the SEI (the Software Engineering Institute, which is a federally funded R&D center at CMU, aka Carnegie Mellon University). The SEI is home to CERT which was established in 1988 to address internet security problems and to find ways to reduce the number and impact of security breaches. CERT focuses on protection, detection, and response to attacks on networked computer systems. Surprisingly enough, CERT is not actually an acronym.

Neither PCI nor CERT has received the memo yet that in order to be cool, you have to have a “Top X” list…perhaps next year?

Now, not to be left out of the fun, the NCSD (National Cyber Security Division) of the DHS (Department of Homeland Security) has their own strategic initiative called BSI (Build Security In). The NCSD obviously wants to cover pretty much all the bases since, in addition to their own BSI, they also sponsor pretty much all of the other guidelines.

I would be remiss if I didn’t at least acknowledge a few other notables with respect to software security guidelines, and to make it more interesting, I will only provide the acronym. I challenge you to come up with the full name. So, a few others involved in security guidelines are NIST (who run a project called SAMATE, and also run an event called SATE, which BTW is also sponsored by DHS NCSD), WASC, and finally STIG. For fun, I’ll throw in CVE, even though it is not a guideline, but more of a dictionary or list that was put together by MITRE, and shockingly is sponsored by DHS NCSD. I’m starting to think that DHS wants to be everyone’s BFF.

Hopefully you’ve learned a little more about the alphabet soup of security guidelines out there. If you’re scratching your head thinking WTF, you’re probably not alone…

Previously I posted the value of using coding standards, specifically MISRA C and MISRA C++.  This time I wanted to go through some general experiences we had with some of the checkers, specifically the ones that seem to throw a lot of violated rules, to the point that on some code bases MISRA flagged more than one error per LOC!

There are still tons of great rules you can apply even if you don’t make an embedded product.  But as I said before, it doesn’t make sense to turn on all the MISRA rules.  After running through many code bases and looking at the value of MISRA we certainly noticed a trend with a few culprits.  Here are a few examples that we found to be noisy with non-MISRA compliant code.

MISRA C 6.3 and MISRA C++ 3-9-2:

MISRA has the distinction of “required” rules and “advisory” rules.  This is an “advisory” rule.  Essentially it wants you to avoid using types like char, int, short, long etc. and to use specific length typedefs instead.  Obviously, many code bases use the basic types, so be prepared for many issues.

MISRA C 14.9 and MISRA C++ 6-4-1:

This is a “required” rule.  This rule  is really about making sure you have braces for your if/else keywords.  Good practice to have but how many of us really do this?

MISRA C 12.13 and MISRA C++ 5-2-10:

This is an “advisory” rule.  The rule doesn’t want you to mix increment and decrement operators into an expression.  This makes sense because it can be pretty confusing to read.  But in our experiments this seems to be something that many developers do.

MISRA C 17.4 and MISRA C++ 5-0-15:

This is a “required” rule.  The rule only wants to allow you to use array indexing for pointer arithmetic.  Everything else is non-compliant.

MISRA C 14.7 and MISRA C++ 6-6-5:

This is a “required” rule and another control flow example.  A function can only have one point of exit at the end of a function.  I can understand this but as you know, that is not reality.

MISRA C 13.2:

This is an “advisory” rule.  It states that tests against zero should be made explicit.  In other words: if (x != 0) is the proper way, not if (x).  The exception to this is if the operand is Boolean.  I don’t know about you, but you can crown me the super wiener on this? I never make it explicit.

So if you plan to pick up MISRA on your existing project beware of these rules.  I’d like to hear if you do any of those things in your code base.

What is MISRA? More Irrelevant Software Requirements Again…uh no but certainly the sentiment of many developers.  MISRA (Motor Industry Software Reliability Association) is a coding standard, which first released MISRA C in 1998 and has since been revised.  Obviously, this came out of the automotive sector with a clear focus on helping software systems to be more reliable and maintainable.

MISRA has since grown.  Now you see more and more industries adopting these standards.   In 2008, MISRA released the C++ equivalent standard.  So the obvious question is, should I apply this to my software source code base?  In short, yes.  Maybe you don’t need ALL of the MISRA rules, but there are certainly some that are worth your time.

Examples range from the very simple checker rules sets that mostly fall into the coding style category:

MISRA C rule 19.1:  All #include directives in a file shall only be preceded by other preprocessor directives or comments.

MISRA C rule 2.2: You should only use /*…*/ style comments.

Down to more important issues (I hope everyone has the tools in place to find this):

MISRA C rule 9.1: Unintialized variables.

It really depends what you’re trying to accomplish.  Certain rules such as 9.1 are very important, while others can help you with portability or just serve to make your code more maintainable and less complex going forward.

Word of warning, taking an existing software code base and apply the entire MISRA rule set is guaranteed to throw many errors.  Take the zlib project, obviously not MISRA compliant, but to illustrate the impact of turning on all the MISRA rules, will throw over 5000+ issues.  That is an issue for every line of code…yeah that’s useful.  So be warned, don’t try to boil the ocean.  Start iteratively, even if that means only one rule at a time.

Just came across this post about the 5 code metrics you need to watch.  I thought it was worth mentioning as I just blogged about this below (including something similar a while back).  These are interesting metrics and more high level, but certainly important.  I like labeling duplicated code as something important.  I think we often forget how much we reuse code and have the same mistakes in many places.

In a previous post, I talked about different types of metrics and how they can apply.  These were mostly focused on a system level and provided insight into system complexity from  release to release or build after build.  Now, let’s take a look at some other metrics and how they can be used effectively with the developers.

There are literally hundreds of metrics you can measure, so what is really important?  That’s a tough question because it depends on your organization.  Perhaps you have some coding standards in place or simply want to measure some basic functionality to help developers improve code quality.

Here are some of the more common metrics I’ve seen used in practice:
•    Lines of code
•    Number of functions/methods
•    Commented code
•    Nested includes
•    Number of compiler directives

Most of the metrics listed above are about maintainability.  The more lines of code you have or the more of functions/methods, the more you have to maintain for an individual file.  If these values are high, then perhaps it deserves a new file or new function.  The opposite end is commented code, where you obviously want more code commented.  Nested includes are a great way to alert the developer that adding a new include has all of a sudden added a long compile tool chain, potentially increasing your compile time and size of your system.  Tracking compiler directives is something, I think, is really useful–especially with platform specific code.  Compiler directives are a great way to give you visibility on how much you are using conditional compilation and hopefully, gives you a good understanding of the scope if a new platform is added.

Thresholds are the way to use any of these metrics.  For example, create a threshold that would throw an error if the number of functions per file is greater than 100.  Or,  first give a warning to developers when the number of functions exceeds 75 per file and then, if it goes over 100 print an error.  But what values should you use?  You can use the standard deviation to figure this out or use coding standard guidelines.  For example, many standards say that a method should not reach over 1000 lines or code, or the maximum level of include nesting should not exceed 8.

If you find that these metrics are not enough, start combining them.  Any good tool worth its  salt will let you customize these core metrics, so you can start creating your own calculations.  One example is using the lines of code metric with the commented code metric to create a calculation on the percent of comments.  We are all guilty of not documenting our code.  Use percentage of comments to make sure everyone knows when they are under the norm,  so if you have less than 10% of your file commented you get an error and less than 20% you get a warning.

Showing developers potential areas of concern will help them to be smarter at what they do and spend more time focusing on what they are good at–innovating.

Part II – Joy is the word…

OK, so Grease is really the word, but it didn’t fit my theme, gimme a break… Anyway, back on topic, since Joy of code review – part one of this series was published last year we’ve seen our new code review product in action in a variety of customer and prospect situations, and much like the eponymous hair product in the musical mentioned above, what we thought of as an interesting twist on an existing paradigm has turned into a bit of a barn burner. I refer, in this case, to the notion of what constitutes a code review if you remove the formalism of the invite from the process.

Consider what I’ll call, for the sake of being what marketers insist on terming “edgy” (for no really good reason as far as I can make out), old fashioned code reviews. You know the type, we talk about how we really should do more of them all the time. Check in your code, mail out a bunch of invites, mail some more when those get declined, gather around a table, project your code and wait for the insults to come rolling in.

You want to try that again, Mr. Coding Specialist...?

On the down side of these things are all the obvious problems… People don’t like getting reviewed, and unless you have a particularly unpleasant architect, the reviewer is no happier about being in the room than the person on the sharp end. Factor in the time, the annoyance of the arrangements, the opportunity cost of yanking the architect away from whatever they were previously doing, and you’ve got a really expensive, not very productive, but very important from a pointy-haired-manager-perspective process.

It’s really the classical no-win situation. Your manager requires it to be done. You hate it, and you know everybody else in the room hates it too. It’s like a giant dose of spinach to a five year old – doesn’t matter how good it is for you, you’d rather scream and sit in the naughty chair all day than let that stuff past your lips.

So when we were thinking about changing the approach to code review, it seemed obvious to us that whilst code review itself is valuable, the means by which it gets accomplished is fundamentally broken. Factor in peoples’ unthinking delight when confronted with anything social and what the heck, we figured, let’s turn the whole thing on its head. Instead of going top-down into a software organization and helping the manager enforce something unpleasant in an all new and collaborative-y, enterprise-y way, how about reaching out and encouraging bottom-up engagement through a model that people are comfortable with anyway, namely formless (a.k.a. social) communities.

Who’s the most obvious person to review the code of a good developer, after all? It might be their architect, but the chances of a good developer making a blunder of the architectural type (or any kind of dumb error) is probably reasonably low. Not saying it doesn’t happen, but we pay people at that level a good amount of money on the understanding that they produce decent code, so why then treat them like kids? Instead, if the code produced by that guy is made available for anybody to review, quite literally, then rather than getting the architect grumpy because he’d rather be thinking about the next huge money maker than what this guy happened to have done mostly right but nit-pickingly-wrong in this one situation, you get other team members taking part who have (in most cases) more useful input to impart anyway.

Instead of feedback of the “so… rather than using that particular transitive constructor, I’ve found that explicitly instantiating a new object and then initializing only what I need saves me, on average, 3 cycles a day” type, you might get the “hey, I was hacking on that a while back… might want to filter that data, cuz Bob’s front end passes in all kinds of crap… just saying” type instead – your choice, but personally I’d rather hear an hour’s worth of the latter than a moment’s worth of the former…

So who is at the review turns out to be much more important than whether it’s held, given some arbitrary set of “holding” conditions. But of course this comes with its own set of challenges, notably how do you know when you’re done if there’s no formal “meeting” to review your code (and to insult you, have we mentioned that part?).

In fact, it’s much like how the transition from waterfall to Agile was accompanied by many a gnashing of management gums and misplaced wails of “but how will I know if it’s going to be done on time?” But hey, that didn’t work out so bad, did it? People got used to time boxing, to changing requirement sets, to not waiting until it was arbitrarily “finished” and instead shipping it so as to gather feedback faster.

In my next post I’ll look at this new world order from the top down and examine the benefits to encouraging (rather than imposing) a social code review paradigm, and how it can make those metrics we know you care about look better than ever before.

How do you measure your software?  There are simple metrics that help with quality, such as keeping track of the number of bugs or security vulnerabilities in your system.  Trending these metrics is a no-brainer. When trending is in place, action can be taken because everyone knows 6 security vulnerabilities is worse than 5.  But what about other types of software metrics (and there are many)?  Have you ever heard of a maintainability metric? Halstead program volume? McCabe cyclomatic complexity?  Coupling/Cohesion?  The question becomes what do you do with these metrics and are they valuable?

Choosing a metric will really depend on what you’re after.  A good reason for measuring your code is to get predictable quality.   If you don’t have a metric in mind, the easiest place to start is with McCabe’s cyclomatic complexity metric.  I’ve seen many software organizations implement this as a good measure to help predict system “complexity”.  In other words, to help them understand where they may need to refactor or redesign their code.  McCabe cyclomatic complexity uses a measure of the linearly independent paths in the source code and is measured on functions or methods.

McCabe’s Cyclomatic complexity uses values to define what is complex.  Something greater than 20 is considered very complex.  You should think about re-writing that function because it is getting out of control.  Since the inception of McCabe’s Cyclomatic complexity metric,  several other variations have appeared, including Extended Cyclomatic Complexity and Plain Cyclomatic Complexity.  Back to the question, with so many metrics, which ones do you use and are they valuable?

No one can answer that question. In fact, software metrics is quite ambiguous.  It is hard to find anyone who says,’Thou shall use metric “x” because it will help you improve quality by “y” amount.’  The value “x” and “y” just don’t exist (although many have tried to put some data together).  Even more ambiguous are the values that may be defined with these metrics.  Don’t get caught up with these values; they are really arbitrary.  I’ve run into organizations where the majority of their code was deemed “very complex”.  Does this mean they should redesign their entire code base?  Certainly not. These numbers will vary depending on what you’re building.  So be careful if you use the “recommended” values for any metric.

Instead of focusing on the value of your next metric, what you really should be doing is trending that metric.  Find out if that value went up or down.  Up bad; down good.   Taking it one step further (if you really have a “thing” for the values), you could start by finding the standard deviation of your metrics.  In other words, find the average value of any metric, say complexity, plus the standard deviation.  Now, you can keep track of that value knowing that if you go outside your bounds of deviation, then you may want to look.

Software metrics certainly have their place and can help give some predictability on your system.  In another post, I’ll talk about how you can take some low level metrics for the developers and give them insight into the software system.

In my previous blog post, we talked about the value of compiler warnings and reasons to have source code analysis. Now, I’d like to get into the value of coding standards and touch on how you can fit this altogether.

Coding standards are a set of rules or guidelines usually created as part of an industry. The goal is simple, provide guidelines, so you can create better code and increase your code quality. Probably the most common coding standard I run into is called MISRA C. This is a standard created for C code in 1998 and revised in 2004. A new standard from MISRA was released in 2008 for C++ code. MISRA was developed for the Motor Industry but has since been adopted by many other industries. Other coding standards such as Joint Strike Fighter are focused on other industries, such as the aerospace world.  And there are more generic types of guidelines, such as the Power of 10, which contains more high level practices.

Either way, these standards cover everything from simply “thou shall comment code” to more specific coding no-no’s. So, how do you apply these to your process?

The advantage of these coding standards is that compliance is something you can quickly scan for using source code analysis. Any source code analysis tool worth its salt incorporates these standards into their issue checkers.

Implementing a solution for developers is key to this process. After developers check to ensure there are no compiler errors (or warnings!) they can run another process (or integrate into your existing process) using source code analysis techniques to find infractions with various coding standards in the code.

Remember that compiler warnings can be very helpful, so use them. Don’t be surprised when your source code analysis vendor asks if you are using your compiler warnings on your next checker feature request. Once you have cleaned up your compiler warnings and you want to take the next step to improve code quality, there are many good coding standards that will bring up the quality of your code. Use source code analysis tools to help you automate this process and you will guarantee a better report card.