Just came across this post about the 5 code metrics you need to watch.  I thought it was worth mentioning as I just blogged about this below (including something similar a while back).  These are interesting metrics and more high level, but certainly important.  I like labeling duplicated code as something important.  I think we often forget how much we reuse code and have the same mistakes in many places.

In a previous post, I talked about different types of metrics and how they can apply.  These were mostly focused on a system level and provided insight into system complexity from  release to release or build after build.  Now, let’s take a look at some other metrics and how they can be used effectively with the developers.

There are literally hundreds of metrics you can measure, so what is really important?  That’s a tough question because it depends on your organization.  Perhaps you have some coding standards in place or simply want to measure some basic functionality to help developers improve code quality.

Here are some of the more common metrics I’ve seen used in practice:
•    Lines of code
•    Number of functions/methods
•    Commented code
•    Nested includes
•    Number of compiler directives

Most of the metrics listed above are about maintainability.  The more lines of code you have or the more of functions/methods, the more you have to maintain for an individual file.  If these values are high, then perhaps it deserves a new file or new function.  The opposite end is commented code, where you obviously want more code commented.  Nested includes are a great way to alert the developer that adding a new include has all of a sudden added a long compile tool chain, potentially increasing your compile time and size of your system.  Tracking compiler directives is something, I think, is really useful–especially with platform specific code.  Compiler directives are a great way to give you visibility on how much you are using conditional compilation and hopefully, gives you a good understanding of the scope if a new platform is added.

Thresholds are the way to use any of these metrics.  For example, create a threshold that would throw an error if the number of functions per file is greater than 100.  Or,  first give a warning to developers when the number of functions exceeds 75 per file and then, if it goes over 100 print an error.  But what values should you use?  You can use the standard deviation to figure this out or use coding standard guidelines.  For example, many standards say that a method should not reach over 1000 lines or code, or the maximum level of include nesting should not exceed 8.

If you find that these metrics are not enough, start combining them.  Any good tool worth its  salt will let you customize these core metrics, so you can start creating your own calculations.  One example is using the lines of code metric with the commented code metric to create a calculation on the percent of comments.  We are all guilty of not documenting our code.  Use percentage of comments to make sure everyone knows when they are under the norm,  so if you have less than 10% of your file commented you get an error and less than 20% you get a warning.

Showing developers potential areas of concern will help them to be smarter at what they do and spend more time focusing on what they are good at–innovating.

Part II – Joy is the word…

OK, so Grease is really the word, but it didn’t fit my theme, gimme a break… Anyway, back on topic, since Joy of code review – part one of this series was published last year we’ve seen our new code review product in action in a variety of customer and prospect situations, and much like the eponymous hair product in the musical mentioned above, what we thought of as an interesting twist on an existing paradigm has turned into a bit of a barn burner. I refer, in this case, to the notion of what constitutes a code review if you remove the formalism of the invite from the process.

Consider what I’ll call, for the sake of being what marketers insist on terming “edgy” (for no really good reason as far as I can make out), old fashioned code reviews. You know the type, we talk about how we really should do more of them all the time. Check in your code, mail out a bunch of invites, mail some more when those get declined, gather around a table, project your code and wait for the insults to come rolling in.

You want to try that again, Mr. Coding Specialist...?

On the down side of these things are all the obvious problems… People don’t like getting reviewed, and unless you have a particularly unpleasant architect, the reviewer is no happier about being in the room than the person on the sharp end. Factor in the time, the annoyance of the arrangements, the opportunity cost of yanking the architect away from whatever they were previously doing, and you’ve got a really expensive, not very productive, but very important from a pointy-haired-manager-perspective process.

It’s really the classical no-win situation. Your manager requires it to be done. You hate it, and you know everybody else in the room hates it too. It’s like a giant dose of spinach to a five year old – doesn’t matter how good it is for you, you’d rather scream and sit in the naughty chair all day than let that stuff past your lips.

So when we were thinking about changing the approach to code review, it seemed obvious to us that whilst code review itself is valuable, the means by which it gets accomplished is fundamentally broken. Factor in peoples’ unthinking delight when confronted with anything social and what the heck, we figured, let’s turn the whole thing on its head. Instead of going top-down into a software organization and helping the manager enforce something unpleasant in an all new and collaborative-y, enterprise-y way, how about reaching out and encouraging bottom-up engagement through a model that people are comfortable with anyway, namely formless (a.k.a. social) communities.

Who’s the most obvious person to review the code of a good developer, after all? It might be their architect, but the chances of a good developer making a blunder of the architectural type (or any kind of dumb error) is probably reasonably low. Not saying it doesn’t happen, but we pay people at that level a good amount of money on the understanding that they produce decent code, so why then treat them like kids? Instead, if the code produced by that guy is made available for anybody to review, quite literally, then rather than getting the architect grumpy because he’d rather be thinking about the next huge money maker than what this guy happened to have done mostly right but nit-pickingly-wrong in this one situation, you get other team members taking part who have (in most cases) more useful input to impart anyway.

Instead of feedback of the “so… rather than using that particular transitive constructor, I’ve found that explicitly instantiating a new object and then initializing only what I need saves me, on average, 3 cycles a day” type, you might get the “hey, I was hacking on that a while back… might want to filter that data, cuz Bob’s front end passes in all kinds of crap… just saying” type instead – your choice, but personally I’d rather hear an hour’s worth of the latter than a moment’s worth of the former…

So who is at the review turns out to be much more important than whether it’s held, given some arbitrary set of “holding” conditions. But of course this comes with its own set of challenges, notably how do you know when you’re done if there’s no formal “meeting” to review your code (and to insult you, have we mentioned that part?).

In fact, it’s much like how the transition from waterfall to Agile was accompanied by many a gnashing of management gums and misplaced wails of “but how will I know if it’s going to be done on time?” But hey, that didn’t work out so bad, did it? People got used to time boxing, to changing requirement sets, to not waiting until it was arbitrarily “finished” and instead shipping it so as to gather feedback faster.

In my next post I’ll look at this new world order from the top down and examine the benefits to encouraging (rather than imposing) a social code review paradigm, and how it can make those metrics we know you care about look better than ever before.

How do you measure your software?  There are simple metrics that help with quality, such as keeping track of the number of bugs or security vulnerabilities in your system.  Trending these metrics is a no-brainer. When trending is in place, action can be taken because everyone knows 6 security vulnerabilities is worse than 5.  But what about other types of software metrics (and there are many)?  Have you ever heard of a maintainability metric? Halstead program volume? McCabe cyclomatic complexity?  Coupling/Cohesion?  The question becomes what do you do with these metrics and are they valuable?

Choosing a metric will really depend on what you’re after.  A good reason for measuring your code is to get predictable quality.   If you don’t have a metric in mind, the easiest place to start is with McCabe’s cyclomatic complexity metric.  I’ve seen many software organizations implement this as a good measure to help predict system “complexity”.  In other words, to help them understand where they may need to refactor or redesign their code.  McCabe cyclomatic complexity uses a measure of the linearly independent paths in the source code and is measured on functions or methods.

McCabe’s Cyclomatic complexity uses values to define what is complex.  Something greater than 20 is considered very complex.  You should think about re-writing that function because it is getting out of control.  Since the inception of McCabe’s Cyclomatic complexity metric,  several other variations have appeared, including Extended Cyclomatic Complexity and Plain Cyclomatic Complexity.  Back to the question, with so many metrics, which ones do you use and are they valuable?

No one can answer that question. In fact, software metrics is quite ambiguous.  It is hard to find anyone who says,’Thou shall use metric “x” because it will help you improve quality by “y” amount.’  The value “x” and “y” just don’t exist (although many have tried to put some data together).  Even more ambiguous are the values that may be defined with these metrics.  Don’t get caught up with these values; they are really arbitrary.  I’ve run into organizations where the majority of their code was deemed “very complex”.  Does this mean they should redesign their entire code base?  Certainly not. These numbers will vary depending on what you’re building.  So be careful if you use the “recommended” values for any metric.

Instead of focusing on the value of your next metric, what you really should be doing is trending that metric.  Find out if that value went up or down.  Up bad; down good.   Taking it one step further (if you really have a “thing” for the values), you could start by finding the standard deviation of your metrics.  In other words, find the average value of any metric, say complexity, plus the standard deviation.  Now, you can keep track of that value knowing that if you go outside your bounds of deviation, then you may want to look.

Software metrics certainly have their place and can help give some predictability on your system.  In another post, I’ll talk about how you can take some low level metrics for the developers and give them insight into the software system.

In my previous blog post, we talked about the value of compiler warnings and reasons to have source code analysis. Now, I’d like to get into the value of coding standards and touch on how you can fit this altogether.

Coding standards are a set of rules or guidelines usually created as part of an industry. The goal is simple, provide guidelines, so you can create better code and increase your code quality. Probably the most common coding standard I run into is called MISRA C. This is a standard created for C code in 1998 and revised in 2004. A new standard from MISRA was released in 2008 for C++ code. MISRA was developed for the Motor Industry but has since been adopted by many other industries. Other coding standards such as Joint Strike Fighter are focused on other industries, such as the aerospace world.  And there are more generic types of guidelines, such as the Power of 10, which contains more high level practices.

Either way, these standards cover everything from simply “thou shall comment code” to more specific coding no-no’s. So, how do you apply these to your process?

The advantage of these coding standards is that compliance is something you can quickly scan for using source code analysis. Any source code analysis tool worth its salt incorporates these standards into their issue checkers.

Implementing a solution for developers is key to this process. After developers check to ensure there are no compiler errors (or warnings!) they can run another process (or integrate into your existing process) using source code analysis techniques to find infractions with various coding standards in the code.

Remember that compiler warnings can be very helpful, so use them. Don’t be surprised when your source code analysis vendor asks if you are using your compiler warnings on your next checker feature request. Once you have cleaned up your compiler warnings and you want to take the next step to improve code quality, there are many good coding standards that will bring up the quality of your code. Use source code analysis tools to help you automate this process and you will guarantee a better report card.

I had a chance to speak with Bruce Swope, the VP of Engineering at SterlingTech, an ISO13485 Registered full-service medical device software organization offering software development and validation services. SterlingTech has developed software for an array of medical products including implantable devices as well as external support and monitoring equipment. Their team has worked on Class I, II, and III devices that resulted in successful FDA 510(k)s, PMAs, and CE submissions. 

Bruce has extensive experience in medical device software development and he is an expert in leading Class III medical software products to commercial release. His depth of experience also spans the development of enterprise solutions, security applications, internal applications, and process control systems. He has been an early adopter of quality practices including ISO 9000 processes, Common Criteria Certification and Capability Maturity Model implementation.

I wanted to talk to him more about the challenges developing software in an FDA-regulated context and what all this means to medical device software development teams.

[Brendan] Given your experience working with a variety of medical device companies, what do you see as the biggest business challenge they face?

[Bruce] The biggest challenge is developing a medical product in a cost effective manner that meets FDA and international regulatory regulations.  Most companies have very limited resources available and have boards or investors that are not used to the rigors of regulated development.  This often leads to a gap between investor expectations and the reality of getting the product ready for market.

[Brendan] What about technology challenges?

[Bruce] The hardware platforms that the systems are developed for are very expensive in time and money to update once fielded.  Often, the hardware is impossible to update without dramatic impact to the patient such as surgery.  This creates a need for software developers to find creative ways to extend the life of the hardware by introducing new functionality without updating the hardware. This can often cause the software to become much more complex than planned.

Further, device manufacturers must balance the expectations of customers against the rigor and security required with making a medical product.  Consumers are very accustomed to seeing feature rich devices reach the palm of their hand and wonder why their heart pump can’t double as a PDA or MP3 player or why they can’t plug their device into the internet to download new alarm tones.   

[Brendan] What’s the most common problem your firm is hired to solve?

[Bruce] Many of our customers are looking for an organization that has experience in working with a given technology to create a product that will be approved by the FDA and international regulatory authorities. They are looking for someone that has the experience to deliver a quality product and a complete design history file without wasted effort or significant delays.

[Brendan] In your experience, do most medical device companies have a clear understanding of the regulatory environment or is there still confusion in the market?

[Bruce] Many of our customers are early stage companies that are looking for us to provide the knowledge of the regulatory environment.  Other clients may have an understanding of some aspects of the regulatory environment such as mechanical or electrical but need assistance with the software aspects.

Unless companies invest in dedicated regulatory resources early on and get the FDA or notified body involved sooner rather than later, there will always be confusion and opportunity to misdirect effort.

[Brendan] Any common misconceptions related to compliance issues you can share?

[Bruce] Companies have come to us with a misunderstanding of the impact “level of the concern” will have on the development process for their proposed device.  Companies will often put in place a Quality System that is overly burdensome on the software development process. 

The result of these mistakes is often that either too much or too little is done to develop the software.  Either outcome is damaging.  In the case where too much is done, extra cost is incurred and the project completion and entry to the market is delayed.  In the case where too little is done, a rejected submission could result leading to further cost and delays. 

[Brendan] What’s the #1 recommendation you give to clients as it relates to the intersection of compliance issues and software development?

[Bruce] Make sure that your company has a good solid Quality System as it applies to software development. Do not put a Quality System into place that you can not follow. This is the cause of most audit problems. Use automated tools in your process to allow your developers to focus on the creative parts of the software development.  Keep things as simple as possible.  Drive out risk early.

[Brendan] Where can people go to get more information? Any good online resources out there?

[Bruce] For an executive overview for determining whether a new device is a medical device or for ideas on how to use a static code analysis tool in medical device development, we have a library of whitepapers people can download.

[Brendan] Thanks!

My first day at the SWA forum was actually the 3^rd day at the conference, and from all accounts it has been a very productive and relevant first 2 days. Today was no different as it was kicked off with a panel discussion on the Evolution of Software Assurance Processes, and included speakers from Lockheed Martin, Waters Edge LLC, SEI/CERT, and SafeCode. I thought it was an entertaining discussion from a group definitely passionate about the topic. Something seemed missing though as I came out of it hoping for something more…Some good questions rounded out the first session.

Next was my turn to be on stage. I was speaking as part of the “Understanding Technology Stakeholders: Their Progress and Challenges” panel which was made up of John Giligan (The Giligan Group), Djenana Campara (KDM), Bruce Weimer (US Army), and Sean Barnum (Cigital)…and myself. It was an interesting mix of speakers representing various sectors of the software assurance community including assurance ‘consulting’ stakeholders, assurance ‘standards’ stakeholders, assurance ‘consumer’ stakeholders, and assurance ‘tool’ vendor stakeholders. My basic message was that the DHS Forum had done a great job of communicating their message to the assurance community (including a large number of our customers), but fundamentally flawed in a number of other ways.  Unfortunately, the panel part went long, so the Q&A with the Plenary was shortened. The feedback I received was all positive, and that it was refreshing that we didn’t sugar-coat our thoughts.

As I mentioned earlier, there just seems to be something missing from the sessions I’m attending. Perhaps it is too much talk, and not enough action…not sure yet. Hopefully the next two days will leave me with a more positive feeling on this.

I speak again on Friday when I share my experiences and observations on the Static Analysis Tool Exposition 2009. I guess it will be another ‘refreshing’ session…

In the first blog series, we discussed the value of compiler warnings and wondered why a static analysis tool would have similar error checking features. In this installment, we want to dive deeper into this question by reviewing errors that can be found by compilers, why they matter, and what limitations compilers have in this area.

Let’s take an example of the “implicit int” rule:

int foo() {
   const x = 0;
   return x;
}

This is a situation where failure to specify a type results in this compiler warning from (gcc v.3.4.4) or Microsoft cl (v.14):

gcc -Wall -c main.c
main.c: In function `foo':
main.c:2: warning: type defaults to `int' in declaration of `x'

cl -c -Wall main.c
main.c(2) : warning C4431: missing type specifier - int assumed. Note: C no longer
supports default-int

You can’t rely on the standard C/C++ implementations to support the implicit int anymore and these compilers alert you to that.  I do have to say, I’ve never seen anyone do this in practice, but it’s nice to know it’s there.

Let’s look at another example:

void foo() {
   if (sizeof(char) < 2)  // defect - the condition is constant
   {
      /* ... */
   }
}

The issue above is that the condition is constant.  See the C99 standard for details on this (section 6.6).  If we run the cl compiler we get:

cl -c -Wall main2.c
main2.c(2) : warning C4127: conditional expression is constant

Here, the cl compiler finds the issue, gcc does not (well, at least my version).   Okay, interesting let’s take a look at a C++ example:

class A
{
   public:
   // non-virtual destructor
   ~A();
   virtual void f1();
};

With this example, if you run either gcc or cl you get the same thing:

gcc -Wall -c main3a.cpp
main3a.cpp:2: warning: `class A' has virtual functions but non-virtual destructor

cl -c -Wall main3a.cpp
main3a.cpp(7) : warning C4265: 'A' : class has virtual functions, but destructor is
not virtual instances of this class may not be destructed correctly

According to the output from both compilers, we made a boneheaded mistake and forgot to assign the destructor as virtual.  Let’s go one step further and define a new method:

void deleteA(A *a) {
   delete a;
}

This method adds a new level of complexity.  When an object of a class derived from the given one is deleted through a pointer to the given class, the destructor of the derived class is not executed, and members of the derived class are not disposed of properly.  In this case, you will not get any warnings from any compiler.  The difference here is that compilers only work within the context of the file/function.  In this case, you are out of luck with compilers, but luckily source code analysis excels in this.

So, the message here is that compiler warnings are quite useful, but they do have their limitations.  Not all compilers report the same things consistently, nor do they cover analysis beyond a single function or file.  Still, make sure you run the compiler warnings, then implement static source code analysis as part of your process to go deeper and find some more complex issues in your code.

For the next blog of this series I’ll cover coding standards and where they fit in your code quality process.

Main topic of today is using Agile in an FDA regulated medical device context. Sounds like an impossibility I know, but the folks from Agiletek and Abbott presented a very interesting case study on how they did it. They started off by presenting “the way it used to work”, highlighting an older product development cycle from the 1990s that had very strictly defined dev phases, including a 10-12 week integration cycle – yikes! When they decided to implement Agile on a more recent project they broke up their 3-5 year dev cycle in 6 week iterations. Here were the biggest barriers they found to achieving this:

• Documentation – they tackled this topic upfront. There is a perception that the FDA wants truckloads of docs from medical device manufacturers. The reality is, according to the presenters, that’s not the case… the FDA wants “enough” documentation to demonstrate your process (“least burdensome” in FDA-speak). The biggest area is of course documenting requirements which they did through a Capability Matrix.
• Requirements – this required a big culture shift. They talked about past projects with 14 month requirements definition phases… which still didn’t capture everything! Now, they realize it’s a myth that all the req’ts can be defined upfront, and as the gentleman from Abbott stated: “Your requirements are final when the product is retired from market.”
• Software verification and validation (V&V) – they emphasized a risk-based approach. Run code inspections and reviews on the most critical areas of code. Keep your requirements focused and high-level so testers are testing the important stuff.

Anyway, here are the results they found by modernizing their development with Agile: higher visibility, lower costs (estimated schedule and team size reduction of 20-30%), higher quality product (availability of working software allows for continuous V&V), and overall the project had a steady pace to it rather than mad integration scrambles or backend V&V chaos.

The one big aspect of Agile they weren’t able to implement is the customer feedback component. This is mainly due to the limitations med device companies have around “pre-marketing” their product.

All in all, a very interesting case study. Be interested to hear where anyone else has seen this done in a highly regulated environment.  Signing off from Agile 2009… be sure to follow us on Twitter:

Brendan Harrison
Todd Landry
Alen Zukich

Good start to the Agile 2009 show… seems bigger than last year and well organized so far.

Interesting, we’re seeing lots of people from safety-critical shops – medical devices, telecom, military & aerospace, etc. Anecdotally seems like a big change from last year. Todd attended an Agile in safety critical talk that was a good general overview of why Agile is better than traditional development methods for these shops, but lacked specifics on how, why, etc. There’s an FDA & Agile presentation later in the week which will present an actual medical device case study – that should offer more specifics on how to reconcile the process-heavy, documentation-centric approach to a typical regulated environment with Agile.

We also attended Effective Code Reviews in Agile Teams.  Good overview from the Spartez team.  Originally thought this might be vendor specific (Mark’s blog also touched on this) as Spartez does lots of work for Atlassian Crucible but overall it was a good overview even though all the demos were with Crucible.  Despite that, they kept it vendor neutral and described the challenges of adopting code reviews and the keys to success. They reviewed a bunch of common code review misconceptions and pitfalls.  Namely that people see this as a frantic bug hunt and that it will find all bugs in your code.  Also many developers see this as a “big brother” tool where metrics including every comment will be tracked. For managers, they warned not to expect a clear picture that code reviews have saved you x amount of time.  Well that would be nice. ;)  In terms of how this fits in an Agile context there is definitely one clear message, everyone can join/review/invite/modify a code review.  It’s all about the team (of course!) and about learning, not blaming. They see code reviews not as a tool-centric activity but basically a way to enhance communication or emphasize individuals and interactions, to use an Agile Manifesto term.

Be sure to follow us throughout the day on Twitter:

Brendan Harrison
Todd Landry
Alen Zukich

The exhibit hall was also quite busy, lots of good discussions with attendees.