As readers know, we’ve been talking about code reviews pretty regularly here and elsewhere over the past few months. To continue that discussion, here’s a question we run into often: are in-person code reviews as the primary way to communicate, by definition a bad thing?

Here’s some more data from the Forrester Consulting study commissioned by Klocwork that shows the majority of respondents still conduct in-person reviews… elsewhere in the survey only 36% of respondents indicated that they worked on a centralized team with everyone in one location. So that means, if 60% still conduct in-person reviews, they’re likely excluding valuable contributors to the review.

Data that shows majority still conduct in-person code reviews

Is this practice just being done because “that’s the way it is” or are there good reasons for in-person meetings being the primary way to review code? I could see the odd in-person meeting being necessary for a variety of reasons but given how distributed teams are these days and the variety of tools available to effectively review code remotely, it doesn’t seem that efficient.

There’s a general philosophy gaining more prominence around meeting reduction, whether in software development or elsewhere. We’re seeing many organizations question why their code review process needs to be in-person when it excludes people who aren’t co-located and generally takes up too much of people’s time. What are you seeing?

It’s often taken as read that developers think code reviews are just a pain in the behind. Maybe that sentiment is true when a developer’s sitting amongst his/her peers and getting interrogated on the quality of their code, but some of the data from a Forrester Consulting study commissioned by Klocwork seems to contradict that a bit. The survey asked software development professionals a whole bunch of questions related to code reviews (some of which we’ve referenced before) and here are two interesting data points that suggest developers see real benefits from code reviews.

So 79% of respondents indicate that, yes, code reviews have been effective at reducing the number of bugs found later in the development cycle. Furthermore, 43% state that code reviews have caused a fundamentally positive shift in their project’s direction. Cool.

Of course, in other parts of the survey, respondents complain about aspects of code review, in particular how time consuming and difficult they can be to implement consistently. Nonetheless, the data indicates that when organizations put their heads down and make them part of their development process, real benefits will be realized. So, the challenge is making them part of the process – of course we advocate a tools-based approach, making them more lightweight, and combining automation into your software verification strategy so that manual reviews aren’t the only technique being used to find implementation errors.

This data line-up with what you’re seeing within your organization?

Our partners at Code Integrity have a good blog that touches on many of the benefits and barriers to static analysis within a development organization. They have an interesting post on “False false positives” – a great phrase that captures one of the key challenges in developer adoption of the technology.

While increased sophistication means that static analysis tools can catch more problems with a higher degree of accuracy, the burden increases on the reviewer of the results to interpret them correctly. If you were grep’ing through some code for something you can quickly review (and dismiss) many of the results because you understand what your “analysis” is doing. With static source code analysis, this is much less apparent.

We see many engineers look at a complex bug report and not take the necessary time to understand the problem and fix it. This is mostly because they don’t understand what the static analysis tool is doing and how deep it is analyzing the code. The result is a real bug being marked as a false positive – or a “false false positive” if you will. These bugs then disappear off the queue never to be seen again – a lost opportunity.

One of their key recommendations to overcoming this barrier is using training and joint review of results to educate developers on why the tool is flagging a potential error, what the mitigation options are, etc. Code Integrity has a bunch of deployment and training services to help customers with these types of deployment hurdles.

In our experience, all developers need is one ‘aha’ moment where the tool finds a nasty, subtle bug that would be hard to find using any other method. Once that happens, the developer is a convert. I would also say the burden isn’t just on training, but the tool vendors as well. We all have to continue making the usability of the tool such that developers should be able to instantly recognize why the tool is flagging the error and give the developer all the info they need to recognize the bug and take the appropriate action.

The importance of code reviews has already been well covered by lots of smart people like Jack Ganssle and Jason Cohen. Recently, the subject has become more important around here, so we want to offer our take. In particular, we’re looking at the best way(s) to incorporate code reviews into an overall software verification strategy and how automated tools (such as static analysis, no shock there) can help unleash the benefits of peer code review. More on that angle another time, first the bigger picture.

Klocwork recently commissioned a survey conducted by Forrester research on this whole topic and the results are pretty interesting. While there’s a whole bunch of data that can’t be covered in a single blog post, a general theme we found is that developers see the value of code reviews, they’re often mandatory, but the process itself seems to be ad-hoc and quite ‘behind the times’. Here’s an example of what I mean:

Code Reviews - Mandatory but Ad-Hoc

So, code reviews are mandatory but you can kinda invite whoever you want to review the code. Shouldn’t who reviews the code be pretty important? (Hint: Yes)

We’re gonna keep talking about different aspects of this important development milestone, so stay tuned and we’d be interested to hear anything you have to say on the topic.

We try not to plug Klocwork on this blog too much and keep the discussions around general software development issues, topics, and trends (with an obvious bias towards static analysis and software verification since those are our areas of expertise).

So, today I’ll go ahead and break that rule and encourage you to visit our new online resource library that includes videos, webinars, whitepapers, analyst reports, etc. All the assets are organized by category and tagged with their subject/content so everything is easy to find. I encourage you to check it out… here’s an example of the kinds of demos we’ve put up there so people can quick see what our tools are all about.

I had a chance to speak with Bruce Swope, the VP of Engineering at SterlingTech, an ISO13485 Registered full-service medical device software organization offering software development and validation services. SterlingTech has developed software for an array of medical products including implantable devices as well as external support and monitoring equipment. Their team has worked on Class I, II, and III devices that resulted in successful FDA 510(k)s, PMAs, and CE submissions. 

Bruce has extensive experience in medical device software development and he is an expert in leading Class III medical software products to commercial release. His depth of experience also spans the development of enterprise solutions, security applications, internal applications, and process control systems. He has been an early adopter of quality practices including ISO 9000 processes, Common Criteria Certification and Capability Maturity Model implementation.

I wanted to talk to him more about the challenges developing software in an FDA-regulated context and what all this means to medical device software development teams.

[Brendan] Given your experience working with a variety of medical device companies, what do you see as the biggest business challenge they face?

[Bruce] The biggest challenge is developing a medical product in a cost effective manner that meets FDA and international regulatory regulations.  Most companies have very limited resources available and have boards or investors that are not used to the rigors of regulated development.  This often leads to a gap between investor expectations and the reality of getting the product ready for market.

[Brendan] What about technology challenges?

[Bruce] The hardware platforms that the systems are developed for are very expensive in time and money to update once fielded.  Often, the hardware is impossible to update without dramatic impact to the patient such as surgery.  This creates a need for software developers to find creative ways to extend the life of the hardware by introducing new functionality without updating the hardware. This can often cause the software to become much more complex than planned.

Further, device manufacturers must balance the expectations of customers against the rigor and security required with making a medical product.  Consumers are very accustomed to seeing feature rich devices reach the palm of their hand and wonder why their heart pump can’t double as a PDA or MP3 player or why they can’t plug their device into the internet to download new alarm tones.   

[Brendan] What’s the most common problem your firm is hired to solve?

[Bruce] Many of our customers are looking for an organization that has experience in working with a given technology to create a product that will be approved by the FDA and international regulatory authorities. They are looking for someone that has the experience to deliver a quality product and a complete design history file without wasted effort or significant delays.

[Brendan] In your experience, do most medical device companies have a clear understanding of the regulatory environment or is there still confusion in the market?

[Bruce] Many of our customers are early stage companies that are looking for us to provide the knowledge of the regulatory environment.  Other clients may have an understanding of some aspects of the regulatory environment such as mechanical or electrical but need assistance with the software aspects.

Unless companies invest in dedicated regulatory resources early on and get the FDA or notified body involved sooner rather than later, there will always be confusion and opportunity to misdirect effort.

[Brendan] Any common misconceptions related to compliance issues you can share?

[Bruce] Companies have come to us with a misunderstanding of the impact “level of the concern” will have on the development process for their proposed device.  Companies will often put in place a Quality System that is overly burdensome on the software development process. 

The result of these mistakes is often that either too much or too little is done to develop the software.  Either outcome is damaging.  In the case where too much is done, extra cost is incurred and the project completion and entry to the market is delayed.  In the case where too little is done, a rejected submission could result leading to further cost and delays. 

[Brendan] What’s the #1 recommendation you give to clients as it relates to the intersection of compliance issues and software development?

[Bruce] Make sure that your company has a good solid Quality System as it applies to software development. Do not put a Quality System into place that you can not follow. This is the cause of most audit problems. Use automated tools in your process to allow your developers to focus on the creative parts of the software development.  Keep things as simple as possible.  Drive out risk early.

[Brendan] Where can people go to get more information? Any good online resources out there?

[Bruce] For an executive overview for determining whether a new device is a medical device or for ideas on how to use a static code analysis tool in medical device development, we have a library of whitepapers people can download.

[Brendan] Thanks!

Good first day at ESC Boston2009. Gwyn and Alen presented a well attended talk on using source code analysis (SCA) to improve developer productivity. Key takeaways from the presentation:

• How SCA will impact your development velocity
• Quick history on SCA – talked about lint and the general evolution of the technology
• How the information generated by static code analysis can be used to solve a variety of development challenge
• Demo of where SCA fits from a code review, refactoring and bug detection standpoint

Interesting change from past presentations where most people now understand the basics of the technology… no need to spend too much time talking about its history and technology building blocks.

The presentation was recorded so we’ll load the video up at a later date for everyone’s viewing pleasure :)

Succinctly communicating what Klocwork does and how it helps improve productivity during code reviews, integration builds, and of course for developers is always a challenge. We’ve tried to capture this visually with this simple SDLC image, and of course we always talk about the importance of finding bugs early.

Then somebody else comes along and makes the point in a way you never thought of… here’s a funny that’s been going around for some time, but I figured it’s worth passing along to the Kloctalk readers.

Main topic of today is using Agile in an FDA regulated medical device context. Sounds like an impossibility I know, but the folks from Agiletek and Abbott presented a very interesting case study on how they did it. They started off by presenting “the way it used to work”, highlighting an older product development cycle from the 1990s that had very strictly defined dev phases, including a 10-12 week integration cycle – yikes! When they decided to implement Agile on a more recent project they broke up their 3-5 year dev cycle in 6 week iterations. Here were the biggest barriers they found to achieving this:

• Documentation – they tackled this topic upfront. There is a perception that the FDA wants truckloads of docs from medical device manufacturers. The reality is, according to the presenters, that’s not the case… the FDA wants “enough” documentation to demonstrate your process (“least burdensome” in FDA-speak). The biggest area is of course documenting requirements which they did through a Capability Matrix.
• Requirements – this required a big culture shift. They talked about past projects with 14 month requirements definition phases… which still didn’t capture everything! Now, they realize it’s a myth that all the req’ts can be defined upfront, and as the gentleman from Abbott stated: “Your requirements are final when the product is retired from market.”
• Software verification and validation (V&V) – they emphasized a risk-based approach. Run code inspections and reviews on the most critical areas of code. Keep your requirements focused and high-level so testers are testing the important stuff.

Anyway, here are the results they found by modernizing their development with Agile: higher visibility, lower costs (estimated schedule and team size reduction of 20-30%), higher quality product (availability of working software allows for continuous V&V), and overall the project had a steady pace to it rather than mad integration scrambles or backend V&V chaos.

The one big aspect of Agile they weren’t able to implement is the customer feedback component. This is mainly due to the limitations med device companies have around “pre-marketing” their product.

All in all, a very interesting case study. Be interested to hear where anyone else has seen this done in a highly regulated environment.  Signing off from Agile 2009… be sure to follow us on Twitter:

Brendan Harrison
Todd Landry
Alen Zukich

Off to Agile 2009 next week in Chicago where Klocwork will be both attending and exhibiting at the conference. We’ll blog throughout the week to keep people updated and let you know the latest. There are a few sessions in particular that we’ll be sure to report on and let readers know anything useful we learned (or not):

• Adopting Agile in an FDA Regulated Environment
• Java and Ruby Tools for Code Quality
• Effective Code Reviews in Agile Teams
• Zen and the Art of Software Quality
• Scrum and CMMI – from Good to Great

Be sure to check back often!