We try not to plug Klocwork on this blog too much and keep the discussions around general software development issues, topics, and trends (with an obvious bias towards static analysis and software verification since those are our areas of expertise).

So, today I’ll go ahead and break that rule and encourage you to visit our new online resource library that includes videos, webinars, whitepapers, analyst reports, etc. All the assets are organized by category and tagged with their subject/content so everything is easy to find. I encourage you to check it out… here’s an example of the kinds of demos we’ve put up there so people can quick see what our tools are all about.

I had a chance to speak with Bruce Swope, the VP of Engineering at SterlingTech, an ISO13485 Registered full-service medical device software organization offering software development and validation services. SterlingTech has developed software for an array of medical products including implantable devices as well as external support and monitoring equipment. Their team has worked on Class I, II, and III devices that resulted in successful FDA 510(k)s, PMAs, and CE submissions. 

Bruce has extensive experience in medical device software development and he is an expert in leading Class III medical software products to commercial release. His depth of experience also spans the development of enterprise solutions, security applications, internal applications, and process control systems. He has been an early adopter of quality practices including ISO 9000 processes, Common Criteria Certification and Capability Maturity Model implementation.

I wanted to talk to him more about the challenges developing software in an FDA-regulated context and what all this means to medical device software development teams.

[Brendan] Given your experience working with a variety of medical device companies, what do you see as the biggest business challenge they face?

[Bruce] The biggest challenge is developing a medical product in a cost effective manner that meets FDA and international regulatory regulations.  Most companies have very limited resources available and have boards or investors that are not used to the rigors of regulated development.  This often leads to a gap between investor expectations and the reality of getting the product ready for market.

[Brendan] What about technology challenges?

[Bruce] The hardware platforms that the systems are developed for are very expensive in time and money to update once fielded.  Often, the hardware is impossible to update without dramatic impact to the patient such as surgery.  This creates a need for software developers to find creative ways to extend the life of the hardware by introducing new functionality without updating the hardware. This can often cause the software to become much more complex than planned.

Further, device manufacturers must balance the expectations of customers against the rigor and security required with making a medical product.  Consumers are very accustomed to seeing feature rich devices reach the palm of their hand and wonder why their heart pump can’t double as a PDA or MP3 player or why they can’t plug their device into the internet to download new alarm tones.   

[Brendan] What’s the most common problem your firm is hired to solve?

[Bruce] Many of our customers are looking for an organization that has experience in working with a given technology to create a product that will be approved by the FDA and international regulatory authorities. They are looking for someone that has the experience to deliver a quality product and a complete design history file without wasted effort or significant delays.

[Brendan] In your experience, do most medical device companies have a clear understanding of the regulatory environment or is there still confusion in the market?

[Bruce] Many of our customers are early stage companies that are looking for us to provide the knowledge of the regulatory environment.  Other clients may have an understanding of some aspects of the regulatory environment such as mechanical or electrical but need assistance with the software aspects.

Unless companies invest in dedicated regulatory resources early on and get the FDA or notified body involved sooner rather than later, there will always be confusion and opportunity to misdirect effort.

[Brendan] Any common misconceptions related to compliance issues you can share?

[Bruce] Companies have come to us with a misunderstanding of the impact “level of the concern” will have on the development process for their proposed device.  Companies will often put in place a Quality System that is overly burdensome on the software development process. 

The result of these mistakes is often that either too much or too little is done to develop the software.  Either outcome is damaging.  In the case where too much is done, extra cost is incurred and the project completion and entry to the market is delayed.  In the case where too little is done, a rejected submission could result leading to further cost and delays. 

[Brendan] What’s the #1 recommendation you give to clients as it relates to the intersection of compliance issues and software development?

[Bruce] Make sure that your company has a good solid Quality System as it applies to software development. Do not put a Quality System into place that you can not follow. This is the cause of most audit problems. Use automated tools in your process to allow your developers to focus on the creative parts of the software development.  Keep things as simple as possible.  Drive out risk early.

[Brendan] Where can people go to get more information? Any good online resources out there?

[Bruce] For an executive overview for determining whether a new device is a medical device or for ideas on how to use a static code analysis tool in medical device development, we have a library of whitepapers people can download.

[Brendan] Thanks!

Good first day at ESC Boston2009. Gwyn and Alen presented a well attended talk on using source code analysis (SCA) to improve developer productivity. Key takeaways from the presentation:

• How SCA will impact your development velocity
• Quick history on SCA – talked about lint and the general evolution of the technology
• How the information generated by static code analysis can be used to solve a variety of development challenge
• Demo of where SCA fits from a code review, refactoring and bug detection standpoint

Interesting change from past presentations where most people now understand the basics of the technology… no need to spend too much time talking about its history and technology building blocks.

The presentation was recorded so we’ll load the video up at a later date for everyone’s viewing pleasure :)

Succinctly communicating what Klocwork does and how it helps improve productivity during code reviews, integration builds, and of course for developers is always a challenge. We’ve tried to capture this visually with this simple SDLC image, and of course we always talk about the importance of finding bugs early.

Then somebody else comes along and makes the point in a way you never thought of… here’s a funny that’s been going around for some time, but I figured it’s worth passing along to the Kloctalk readers.

Main topic of today is using Agile in an FDA regulated medical device context. Sounds like an impossibility I know, but the folks from Agiletek and Abbott presented a very interesting case study on how they did it. They started off by presenting “the way it used to work”, highlighting an older product development cycle from the 1990s that had very strictly defined dev phases, including a 10-12 week integration cycle – yikes! When they decided to implement Agile on a more recent project they broke up their 3-5 year dev cycle in 6 week iterations. Here were the biggest barriers they found to achieving this:

• Documentation – they tackled this topic upfront. There is a perception that the FDA wants truckloads of docs from medical device manufacturers. The reality is, according to the presenters, that’s not the case… the FDA wants “enough” documentation to demonstrate your process (“least burdensome” in FDA-speak). The biggest area is of course documenting requirements which they did through a Capability Matrix.
• Requirements – this required a big culture shift. They talked about past projects with 14 month requirements definition phases… which still didn’t capture everything! Now, they realize it’s a myth that all the req’ts can be defined upfront, and as the gentleman from Abbott stated: “Your requirements are final when the product is retired from market.”
• Software verification and validation (V&V) – they emphasized a risk-based approach. Run code inspections and reviews on the most critical areas of code. Keep your requirements focused and high-level so testers are testing the important stuff.

Anyway, here are the results they found by modernizing their development with Agile: higher visibility, lower costs (estimated schedule and team size reduction of 20-30%), higher quality product (availability of working software allows for continuous V&V), and overall the project had a steady pace to it rather than mad integration scrambles or backend V&V chaos.

The one big aspect of Agile they weren’t able to implement is the customer feedback component. This is mainly due to the limitations med device companies have around “pre-marketing” their product.

All in all, a very interesting case study. Be interested to hear where anyone else has seen this done in a highly regulated environment.  Signing off from Agile 2009… be sure to follow us on Twitter:

Brendan Harrison
Todd Landry
Alen Zukich

Off to Agile 2009 next week in Chicago where Klocwork will be both attending and exhibiting at the conference. We’ll blog throughout the week to keep people updated and let you know the latest. There are a few sessions in particular that we’ll be sure to report on and let readers know anything useful we learned (or not):

• Adopting Agile in an FDA Regulated Environment
• Java and Ruby Tools for Code Quality
• Effective Code Reviews in Agile Teams
• Zen and the Art of Software Quality
• Scrum and CMMI – from Good to Great

Be sure to check back often!

Here’s a short blog review of Klocwork Solo by Jeanne Boyarsky. Klocwork Solo is a downloadable Eclipse plug-in for Java. Aside from a few installation hiccups, it’ s a good review with kudos for the range of checkers we provide in the default configuration.

Try it out yourself and let us know what you think.

Interesting paper and presentation (pdf) from Emmanuel Chenu at Thales Avionics that describes how they’re using several Agile concepts as part of their safety-critical avionics software projects. With the exception of pair programming, my read is that much of this is mapping activities that have been done in a safety-critical environment (e.g. test driven development) to several Agile principles, rather than the introduction of concepts that are foreign to safety-critical development. The other one that probably hasn’t been done in most safety-critical shops is continuous integration, but I’d argue that CI (or at least a “build early and often” philosophy), has transcended Agile and is just becoming “the way things are done”, regardless of whether you’re a “Big A Agile”, agile, or iterative development shop.

Either way, it’s interesting how even the most heavy, formal, process-driven development teams are looking at aspects of Agile they can embrace to make their development more flexible, responsive, while still producing highly reliable software. Of course, as he notes, there’s obviously a limit to how “Agile” an avionics development team can really become given the level of formal documentation required through all aspects of a DO-178B project. I’m pretty sure if you ever submitted this kind of documentation to a certification authority they’d probably not accept it:

Today, I had a chance to connect with Connie Beane, the Director of Certification and Safety Critical Software for ENEA Embedded Technology, Inc. Connie has a deep background in safety-critical avionics systems development as a Federal Aviation Administration Designated Engineering Representative (DER) with authority for design assurance level A systems, software and complex electronic hardware. Her additional experience includes 12 years with the FAA in the Transport Airplane Directorate as a Project Officer, Federal representative and Secretary of the RTCA committee SC-180, which produced DO-254, as well as 8 years at Boeing as a Lead Engineer in Boeing Commercial and Boeing Aerospace Divisions.

I wanted to talk to her more about avionics software development, specifically give Kloctalk readers some additional background on the DO-178B guidance that’s used in avionics software development.

[Brendan]:  Can you describe the short version of the history behind this guidance, its purpose, and use in the industry?
[Connie]: In the early 1990’s Boeing was developing the 777 aircraft, which included very software intensive systems.  As a result, industry began using software development and verification tools as a means of developing software more efficiently. The FAA saw this trend as a concern since many of these tools were being used with very little human oversight. When DO-178B was published in 1992, a section on Tool Qualification was included.

[Brendan]:  Is DO-178B guidance only used by teams developing software for commercial aircraft or is it used elsewhere?
[Connie]: DO-178 was developed for commercial aircraft, however, today DO-178B is being used for much more.  It is being used for commercial systems which aid aircraft such as ground based navigation and communication systems. Interestingly, it’s also now being used in military applications both airborne and ground based.

[Brendan]: What’s the most common misunderstanding people have about DO-178B?
[Connie]: That it involves much more than good engineering practices.  DO-178B was developed by government and industry.  They incorporated good engineering practices and some additional activities to ensure safe, reliable software.

[Brendan]: What are a few of the core software development best practices that are required by DO-178B?
[Connie]: Quality Assurance audits, requirements traceability, robustness testing, standards for design, coding and verification.

[Brendan]:  Sounds like a pretty rigorous approach to software development, which of course is a good thing given its safety-critical nature. What kind of cost burden does DO-178B add to a typical project?
[Connie]: That depends on whether the software being developed is performing highly critical functions or not.  For highly critical software, the cost of developing to DO-178B could double the average development cost.

[Brendan]:  Switching gears a bit to your role at ENEA. Out of the full range of DO-178B service offerings you provide, what’s the most common problem you’re brought in to solve?
[Connie]:  Probably the most common issue is a project that is behind schedule and needs to be certified to meet a customer deadline.

[Brendan]:  What’s the biggest change you’ve seen in aircraft systems development during your career?
[Connie]:  Growth in use of software and complex electronic hardware to develop highly integrated avionics systems.

[Brendan]: What do you think the future holds for this guidance? Do you see it evolving in a particular direction that’s different from where it is now?
[Connie]: There is an international committee currently working on the next version of DO-178. This committee is addressing issues such a object-oriented design, model based development, and further defining qualification of software tools.  This committee is also developing a companion guideline to DO-178 which will be applied to ground based software.

[Brendan]: Any final thoughts?
[Connie]:  Companies resist embracing DO-178.  However, once a company has the processes and procedures in place to ensure compliance to DO-178, their software development process is easily understood, repeatable, and maintainable.  This translates to software which is reliable, understandable, traceable and maintainable.

Here’s Gwyn chatting about general software development challenges, in particular the whole goal of “in-phase defect containment” – i.e. identifying and correcting defects in the same development phase they’re created. Near the end of the video, there’s a short discussion on how this objective fits in an Agile context. With Agile’s focus on the frequent delivery of working software, in-phase containment becomes even more important, even though it’s more often associated with more formal methodologies such as CMMI and Six Sigma.