With the recent story that the iPad has inherent security vulnerabilities, I thought it might be an appropriate time to delve into the world of software security guidelines…but I must warn you, this blog will contain an abnormal amount of acronyms, and may not be suitable for all audiences.

When talking about software security guidelines, there are really 5 or 6 organizations that are leading the charge, and they include:

-          OWASP

-          SANS Institute

-          MITRE

-          PCI Security Standards Council

-          SEI

Let’s first look at OWASP. OWASP stands for Open Web Application Security Project, which is a not-for-profit charitable organization that is focused on improving the security of application software. They are probably best known for their Top 10 lists from 2004, 2007, and most recently 2010.

Next is the SANS Institute. SANS of course is a FLA that stands for SysAdmin, Audit, Networking, Security. The SANS Institute claims to be the most trusted source for computer security training, certification and research, and have been developing and releasing their Top 20 annually for the past 7 years or so.

The MITRE Corporation is a not-for-profit organization that was founded in the late 50’s, and has over 7,000 very smart dudes (65% have Masters or PhDs). MITRE has come up with their own security guideline as well, that is the CWE (Common Weakness Enumeration) and it provides a common language of discourse for discussing, finding and dealing with the causes of software security vulnerabilities as they are found in code, design, or system architecture. The CWE lists over 800 programming errors, design errors, and architectural errors that can lead to exploitable vulnerabilities. Interestingly, MITRE and SANS decided to collaborate to come up with the CWE Top 25, yet another “Top” list they have been putting together for the last couple of years.

The PCI Security Standards Council was founded by American Express, Discover Financial Services, JCB International, MasterCard Worldwide, and Visa, Inc. and is an open global forum for the ongoing development, enhancement, storage, dissemination and implementation of security standards for account data protection. The PCI SSC has come up with the PCI DSS, “a multifaceted security standard that includes requirements for security management, policies, procedures, network architecture, software design and other critical protective measures. This comprehensive standard is intended to help organizations proactively protect customer account data”.

Finally, there is the SEI (the Software Engineering Institute, which is a federally funded R&D center at CMU, aka Carnegie Mellon University). The SEI is home to CERT which was established in 1988 to address internet security problems and to find ways to reduce the number and impact of security breaches. CERT focuses on protection, detection, and response to attacks on networked computer systems. Surprisingly enough, CERT is not actually an acronym.

Neither PCI nor CERT has received the memo yet that in order to be cool, you have to have a “Top X” list…perhaps next year?

Now, not to be left out of the fun, the NCSD (National Cyber Security Division) of the DHS (Department of Homeland Security) has their own strategic initiative called BSI (Build Security In). The NCSD obviously wants to cover pretty much all the bases since, in addition to their own BSI, they also sponsor pretty much all of the other guidelines.

I would be remiss if I didn’t at least acknowledge a few other notables with respect to software security guidelines, and to make it more interesting, I will only provide the acronym. I challenge you to come up with the full name. So, a few others involved in security guidelines are NIST (who run a project called SAMATE, and also run an event called SATE, which BTW is also sponsored by DHS NCSD), WASC, and finally STIG. For fun, I’ll throw in CVE, even though it is not a guideline, but more of a dictionary or list that was put together by MITRE, and shockingly is sponsored by DHS NCSD. I’m starting to think that DHS wants to be everyone’s BFF.

Hopefully you’ve learned a little more about the alphabet soup of security guidelines out there. If you’re scratching your head thinking WTF, you’re probably not alone…

Previously I posted the value of using coding standards, specifically MISRA C and MISRA C++.  This time I wanted to go through some general experiences we had with some of the checkers, specifically the ones that seem to throw a lot of violated rules, to the point that on some code bases MISRA flagged more than one error per LOC!

There are still tons of great rules you can apply even if you don’t make an embedded product.  But as I said before, it doesn’t make sense to turn on all the MISRA rules.  After running through many code bases and looking at the value of MISRA we certainly noticed a trend with a few culprits.  Here are a few examples that we found to be noisy with non-MISRA compliant code.

MISRA C 6.3 and MISRA C++ 3-9-2:

MISRA has the distinction of “required” rules and “advisory” rules.  This is an “advisory” rule.  Essentially it wants you to avoid using types like char, int, short, long etc. and to use specific length typedefs instead.  Obviously, many code bases use the basic types, so be prepared for many issues.

MISRA C 14.9 and MISRA C++ 6-4-1:

This is a “required” rule.  This rule  is really about making sure you have braces for your if/else keywords.  Good practice to have but how many of us really do this?

MISRA C 12.13 and MISRA C++ 5-2-10:

This is an “advisory” rule.  The rule doesn’t want you to mix increment and decrement operators into an expression.  This makes sense because it can be pretty confusing to read.  But in our experiments this seems to be something that many developers do.

MISRA C 17.4 and MISRA C++ 5-0-15:

This is a “required” rule.  The rule only wants to allow you to use array indexing for pointer arithmetic.  Everything else is non-compliant.

MISRA C 14.7 and MISRA C++ 6-6-5:

This is a “required” rule and another control flow example.  A function can only have one point of exit at the end of a function.  I can understand this but as you know, that is not reality.

MISRA C 13.2:

This is an “advisory” rule.  It states that tests against zero should be made explicit.  In other words: if (x != 0) is the proper way, not if (x).  The exception to this is if the operand is Boolean.  I don’t know about you, but you can crown me the super wiener on this? I never make it explicit.

So if you plan to pick up MISRA on your existing project beware of these rules.  I’d like to hear if you do any of those things in your code base.

What is MISRA? More Irrelevant Software Requirements Again…uh no but certainly the sentiment of many developers.  MISRA (Motor Industry Software Reliability Association) is a coding standard, which first released MISRA C in 1998 and has since been revised.  Obviously, this came out of the automotive sector with a clear focus on helping software systems to be more reliable and maintainable.

MISRA has since grown.  Now you see more and more industries adopting these standards.   In 2008, MISRA released the C++ equivalent standard.  So the obvious question is, should I apply this to my software source code base?  In short, yes.  Maybe you don’t need ALL of the MISRA rules, but there are certainly some that are worth your time.

Examples range from the very simple checker rules sets that mostly fall into the coding style category:

MISRA C rule 19.1:  All #include directives in a file shall only be preceded by other preprocessor directives or comments.

MISRA C rule 2.2: You should only use /*…*/ style comments.

Down to more important issues (I hope everyone has the tools in place to find this):

MISRA C rule 9.1: Unintialized variables.

It really depends what you’re trying to accomplish.  Certain rules such as 9.1 are very important, while others can help you with portability or just serve to make your code more maintainable and less complex going forward.

Word of warning, taking an existing software code base and apply the entire MISRA rule set is guaranteed to throw many errors.  Take the zlib project, obviously not MISRA compliant, but to illustrate the impact of turning on all the MISRA rules, will throw over 5000+ issues.  That is an issue for every line of code…yeah that’s useful.  So be warned, don’t try to boil the ocean.  Start iteratively, even if that means only one rule at a time.

Just came across this post about the 5 code metrics you need to watch.  I thought it was worth mentioning as I just blogged about this below (including something similar a while back).  These are interesting metrics and more high level, but certainly important.  I like labeling duplicated code as something important.  I think we often forget how much we reuse code and have the same mistakes in many places.

In a previous post, I talked about different types of metrics and how they can apply.  These were mostly focused on a system level and provided insight into system complexity from  release to release or build after build.  Now, let’s take a look at some other metrics and how they can be used effectively with the developers.

There are literally hundreds of metrics you can measure, so what is really important?  That’s a tough question because it depends on your organization.  Perhaps you have some coding standards in place or simply want to measure some basic functionality to help developers improve code quality.

Here are some of the more common metrics I’ve seen used in practice:
•    Lines of code
•    Number of functions/methods
•    Commented code
•    Nested includes
•    Number of compiler directives

Most of the metrics listed above are about maintainability.  The more lines of code you have or the more of functions/methods, the more you have to maintain for an individual file.  If these values are high, then perhaps it deserves a new file or new function.  The opposite end is commented code, where you obviously want more code commented.  Nested includes are a great way to alert the developer that adding a new include has all of a sudden added a long compile tool chain, potentially increasing your compile time and size of your system.  Tracking compiler directives is something, I think, is really useful–especially with platform specific code.  Compiler directives are a great way to give you visibility on how much you are using conditional compilation and hopefully, gives you a good understanding of the scope if a new platform is added.

Thresholds are the way to use any of these metrics.  For example, create a threshold that would throw an error if the number of functions per file is greater than 100.  Or,  first give a warning to developers when the number of functions exceeds 75 per file and then, if it goes over 100 print an error.  But what values should you use?  You can use the standard deviation to figure this out or use coding standard guidelines.  For example, many standards say that a method should not reach over 1000 lines or code, or the maximum level of include nesting should not exceed 8.

If you find that these metrics are not enough, start combining them.  Any good tool worth its  salt will let you customize these core metrics, so you can start creating your own calculations.  One example is using the lines of code metric with the commented code metric to create a calculation on the percent of comments.  We are all guilty of not documenting our code.  Use percentage of comments to make sure everyone knows when they are under the norm,  so if you have less than 10% of your file commented you get an error and less than 20% you get a warning.

Showing developers potential areas of concern will help them to be smarter at what they do and spend more time focusing on what they are good at–innovating.

Part II – Joy is the word…

OK, so Grease is really the word, but it didn’t fit my theme, gimme a break… Anyway, back on topic, since Joy of code review – part one of this series was published last year we’ve seen our new code review product in action in a variety of customer and prospect situations, and much like the eponymous hair product in the musical mentioned above, what we thought of as an interesting twist on an existing paradigm has turned into a bit of a barn burner. I refer, in this case, to the notion of what constitutes a code review if you remove the formalism of the invite from the process.

Consider what I’ll call, for the sake of being what marketers insist on terming “edgy” (for no really good reason as far as I can make out), old fashioned code reviews. You know the type, we talk about how we really should do more of them all the time. Check in your code, mail out a bunch of invites, mail some more when those get declined, gather around a table, project your code and wait for the insults to come rolling in.

You want to try that again, Mr. Coding Specialist...?

On the down side of these things are all the obvious problems… People don’t like getting reviewed, and unless you have a particularly unpleasant architect, the reviewer is no happier about being in the room than the person on the sharp end. Factor in the time, the annoyance of the arrangements, the opportunity cost of yanking the architect away from whatever they were previously doing, and you’ve got a really expensive, not very productive, but very important from a pointy-haired-manager-perspective process.

It’s really the classical no-win situation. Your manager requires it to be done. You hate it, and you know everybody else in the room hates it too. It’s like a giant dose of spinach to a five year old – doesn’t matter how good it is for you, you’d rather scream and sit in the naughty chair all day than let that stuff past your lips.

So when we were thinking about changing the approach to code review, it seemed obvious to us that whilst code review itself is valuable, the means by which it gets accomplished is fundamentally broken. Factor in peoples’ unthinking delight when confronted with anything social and what the heck, we figured, let’s turn the whole thing on its head. Instead of going top-down into a software organization and helping the manager enforce something unpleasant in an all new and collaborative-y, enterprise-y way, how about reaching out and encouraging bottom-up engagement through a model that people are comfortable with anyway, namely formless (a.k.a. social) communities.

Who’s the most obvious person to review the code of a good developer, after all? It might be their architect, but the chances of a good developer making a blunder of the architectural type (or any kind of dumb error) is probably reasonably low. Not saying it doesn’t happen, but we pay people at that level a good amount of money on the understanding that they produce decent code, so why then treat them like kids? Instead, if the code produced by that guy is made available for anybody to review, quite literally, then rather than getting the architect grumpy because he’d rather be thinking about the next huge money maker than what this guy happened to have done mostly right but nit-pickingly-wrong in this one situation, you get other team members taking part who have (in most cases) more useful input to impart anyway.

Instead of feedback of the “so… rather than using that particular transitive constructor, I’ve found that explicitly instantiating a new object and then initializing only what I need saves me, on average, 3 cycles a day” type, you might get the “hey, I was hacking on that a while back… might want to filter that data, cuz Bob’s front end passes in all kinds of crap… just saying” type instead – your choice, but personally I’d rather hear an hour’s worth of the latter than a moment’s worth of the former…

So who is at the review turns out to be much more important than whether it’s held, given some arbitrary set of “holding” conditions. But of course this comes with its own set of challenges, notably how do you know when you’re done if there’s no formal “meeting” to review your code (and to insult you, have we mentioned that part?).

In fact, it’s much like how the transition from waterfall to Agile was accompanied by many a gnashing of management gums and misplaced wails of “but how will I know if it’s going to be done on time?” But hey, that didn’t work out so bad, did it? People got used to time boxing, to changing requirement sets, to not waiting until it was arbitrarily “finished” and instead shipping it so as to gather feedback faster.

In my next post I’ll look at this new world order from the top down and examine the benefits to encouraging (rather than imposing) a social code review paradigm, and how it can make those metrics we know you care about look better than ever before.

How do you measure your software?  There are simple metrics that help with quality, such as keeping track of the number of bugs or security vulnerabilities in your system.  Trending these metrics is a no-brainer. When trending is in place, action can be taken because everyone knows 6 security vulnerabilities is worse than 5.  But what about other types of software metrics (and there are many)?  Have you ever heard of a maintainability metric? Halstead program volume? McCabe cyclomatic complexity?  Coupling/Cohesion?  The question becomes what do you do with these metrics and are they valuable?

Choosing a metric will really depend on what you’re after.  A good reason for measuring your code is to get predictable quality.   If you don’t have a metric in mind, the easiest place to start is with McCabe’s cyclomatic complexity metric.  I’ve seen many software organizations implement this as a good measure to help predict system “complexity”.  In other words, to help them understand where they may need to refactor or redesign their code.  McCabe cyclomatic complexity uses a measure of the linearly independent paths in the source code and is measured on functions or methods.

McCabe’s Cyclomatic complexity uses values to define what is complex.  Something greater than 20 is considered very complex.  You should think about re-writing that function because it is getting out of control.  Since the inception of McCabe’s Cyclomatic complexity metric,  several other variations have appeared, including Extended Cyclomatic Complexity and Plain Cyclomatic Complexity.  Back to the question, with so many metrics, which ones do you use and are they valuable?

No one can answer that question. In fact, software metrics is quite ambiguous.  It is hard to find anyone who says,’Thou shall use metric “x” because it will help you improve quality by “y” amount.’  The value “x” and “y” just don’t exist (although many have tried to put some data together).  Even more ambiguous are the values that may be defined with these metrics.  Don’t get caught up with these values; they are really arbitrary.  I’ve run into organizations where the majority of their code was deemed “very complex”.  Does this mean they should redesign their entire code base?  Certainly not. These numbers will vary depending on what you’re building.  So be careful if you use the “recommended” values for any metric.

Instead of focusing on the value of your next metric, what you really should be doing is trending that metric.  Find out if that value went up or down.  Up bad; down good.   Taking it one step further (if you really have a “thing” for the values), you could start by finding the standard deviation of your metrics.  In other words, find the average value of any metric, say complexity, plus the standard deviation.  Now, you can keep track of that value knowing that if you go outside your bounds of deviation, then you may want to look.

Software metrics certainly have their place and can help give some predictability on your system.  In another post, I’ll talk about how you can take some low level metrics for the developers and give them insight into the software system.

In my previous blog post, we talked about the value of compiler warnings and reasons to have source code analysis. Now, I’d like to get into the value of coding standards and touch on how you can fit this altogether.

Coding standards are a set of rules or guidelines usually created as part of an industry. The goal is simple, provide guidelines, so you can create better code and increase your code quality. Probably the most common coding standard I run into is called MISRA C. This is a standard created for C code in 1998 and revised in 2004. A new standard from MISRA was released in 2008 for C++ code. MISRA was developed for the Motor Industry but has since been adopted by many other industries. Other coding standards such as Joint Strike Fighter are focused on other industries, such as the aerospace world.  And there are more generic types of guidelines, such as the Power of 10, which contains more high level practices.

Either way, these standards cover everything from simply “thou shall comment code” to more specific coding no-no’s. So, how do you apply these to your process?

The advantage of these coding standards is that compliance is something you can quickly scan for using source code analysis. Any source code analysis tool worth its salt incorporates these standards into their issue checkers.

Implementing a solution for developers is key to this process. After developers check to ensure there are no compiler errors (or warnings!) they can run another process (or integrate into your existing process) using source code analysis techniques to find infractions with various coding standards in the code.

Remember that compiler warnings can be very helpful, so use them. Don’t be surprised when your source code analysis vendor asks if you are using your compiler warnings on your next checker feature request. Once you have cleaned up your compiler warnings and you want to take the next step to improve code quality, there are many good coding standards that will bring up the quality of your code. Use source code analysis tools to help you automate this process and you will guarantee a better report card.

I had a chance to speak with Bruce Swope, the VP of Engineering at SterlingTech, an ISO13485 Registered full-service medical device software organization offering software development and validation services. SterlingTech has developed software for an array of medical products including implantable devices as well as external support and monitoring equipment. Their team has worked on Class I, II, and III devices that resulted in successful FDA 510(k)s, PMAs, and CE submissions. 

Bruce has extensive experience in medical device software development and he is an expert in leading Class III medical software products to commercial release. His depth of experience also spans the development of enterprise solutions, security applications, internal applications, and process control systems. He has been an early adopter of quality practices including ISO 9000 processes, Common Criteria Certification and Capability Maturity Model implementation.

I wanted to talk to him more about the challenges developing software in an FDA-regulated context and what all this means to medical device software development teams.

[Brendan] Given your experience working with a variety of medical device companies, what do you see as the biggest business challenge they face?

[Bruce] The biggest challenge is developing a medical product in a cost effective manner that meets FDA and international regulatory regulations.  Most companies have very limited resources available and have boards or investors that are not used to the rigors of regulated development.  This often leads to a gap between investor expectations and the reality of getting the product ready for market.

[Brendan] What about technology challenges?

[Bruce] The hardware platforms that the systems are developed for are very expensive in time and money to update once fielded.  Often, the hardware is impossible to update without dramatic impact to the patient such as surgery.  This creates a need for software developers to find creative ways to extend the life of the hardware by introducing new functionality without updating the hardware. This can often cause the software to become much more complex than planned.

Further, device manufacturers must balance the expectations of customers against the rigor and security required with making a medical product.  Consumers are very accustomed to seeing feature rich devices reach the palm of their hand and wonder why their heart pump can’t double as a PDA or MP3 player or why they can’t plug their device into the internet to download new alarm tones.   

[Brendan] What’s the most common problem your firm is hired to solve?

[Bruce] Many of our customers are looking for an organization that has experience in working with a given technology to create a product that will be approved by the FDA and international regulatory authorities. They are looking for someone that has the experience to deliver a quality product and a complete design history file without wasted effort or significant delays.

[Brendan] In your experience, do most medical device companies have a clear understanding of the regulatory environment or is there still confusion in the market?

[Bruce] Many of our customers are early stage companies that are looking for us to provide the knowledge of the regulatory environment.  Other clients may have an understanding of some aspects of the regulatory environment such as mechanical or electrical but need assistance with the software aspects.

Unless companies invest in dedicated regulatory resources early on and get the FDA or notified body involved sooner rather than later, there will always be confusion and opportunity to misdirect effort.

[Brendan] Any common misconceptions related to compliance issues you can share?

[Bruce] Companies have come to us with a misunderstanding of the impact “level of the concern” will have on the development process for their proposed device.  Companies will often put in place a Quality System that is overly burdensome on the software development process. 

The result of these mistakes is often that either too much or too little is done to develop the software.  Either outcome is damaging.  In the case where too much is done, extra cost is incurred and the project completion and entry to the market is delayed.  In the case where too little is done, a rejected submission could result leading to further cost and delays. 

[Brendan] What’s the #1 recommendation you give to clients as it relates to the intersection of compliance issues and software development?

[Bruce] Make sure that your company has a good solid Quality System as it applies to software development. Do not put a Quality System into place that you can not follow. This is the cause of most audit problems. Use automated tools in your process to allow your developers to focus on the creative parts of the software development.  Keep things as simple as possible.  Drive out risk early.

[Brendan] Where can people go to get more information? Any good online resources out there?

[Bruce] For an executive overview for determining whether a new device is a medical device or for ideas on how to use a static code analysis tool in medical device development, we have a library of whitepapers people can download.

[Brendan] Thanks!

My first day at the SWA forum was actually the 3^rd day at the conference, and from all accounts it has been a very productive and relevant first 2 days. Today was no different as it was kicked off with a panel discussion on the Evolution of Software Assurance Processes, and included speakers from Lockheed Martin, Waters Edge LLC, SEI/CERT, and SafeCode. I thought it was an entertaining discussion from a group definitely passionate about the topic. Something seemed missing though as I came out of it hoping for something more…Some good questions rounded out the first session.

Next was my turn to be on stage. I was speaking as part of the “Understanding Technology Stakeholders: Their Progress and Challenges” panel which was made up of John Giligan (The Giligan Group), Djenana Campara (KDM), Bruce Weimer (US Army), and Sean Barnum (Cigital)…and myself. It was an interesting mix of speakers representing various sectors of the software assurance community including assurance ‘consulting’ stakeholders, assurance ‘standards’ stakeholders, assurance ‘consumer’ stakeholders, and assurance ‘tool’ vendor stakeholders. My basic message was that the DHS Forum had done a great job of communicating their message to the assurance community (including a large number of our customers), but fundamentally flawed in a number of other ways.  Unfortunately, the panel part went long, so the Q&A with the Plenary was shortened. The feedback I received was all positive, and that it was refreshing that we didn’t sugar-coat our thoughts.

As I mentioned earlier, there just seems to be something missing from the sessions I’m attending. Perhaps it is too much talk, and not enough action…not sure yet. Hopefully the next two days will leave me with a more positive feeling on this.

I speak again on Friday when I share my experiences and observations on the Static Analysis Tool Exposition 2009. I guess it will be another ‘refreshing’ session…