Michail the friendly, programming vampire.

Instead of fomenting dissent (that barely exists) in a brazen attempt to boost readership, I’m changing tactics to look at ways in which testing and development are complementary, beyond their common goal of releasing quality software products.

What can I say? After my previous post, How developers drive testers nuts–let’s count the ways, I’m clearly getting less edgy.

I approached our newest addition to the Klocwork development team, Michail Greshishchev. While he’s a new full-timer, Greshishchev is not a new face around here.

The recent Carleton University engineering graduate did two co-op terms here–one in professional services and the other in testing.

So I asked Greshishchev how his stint in testing affected his development. Here’s exactly what he said:

1. Writing short, efficient unit tests comes naturally after dealing with mammoth testing frameworks. Most of the code I write are tests – the techniques and skills I’ve learned in testing are fully applicable to development.
2. Developers have no idea how to execute a test in our automated test system (I don’t blame them–the test machine is a large, well-oiled beast distributed over dozens of operating environments). Having the knowledge to run test team tests on developer builds means I don’t need to wait for nightly build test results to address issues. More importantly, I can add my own tests to the test team’s automated test system.
3. It’s common for a developer to request more information about a tester’s problem report. My experience with the test team allows me to access the information on test machines myself, saving time for everyone.
4. The test report pages actually make sense. This allows me to investigate test failures in the nightly build before a tester comes to my desk to tell me I broke something.

His experience as part of the test team has been win-win for him and his colleagues. Testing and development sound like allies, don’t they? Well, as much as werewolves and vampires can be allies, I suppose. And he was such a nice guy too, but the change is upon him.

The two sides of testing team lead Jonathan Patchell.

At daily standup meetings, they eye each other from opposite sides of the room. Sitting on the same side of the cubicle wall is unthinkable.

They’re united only by their desire to produce quality software products and their appreciation for coffee and energy drinks. What’s good to one side can be anathema to the other when it comes to code.

I’m talking, of course, about testing and development teams. In the interests of generating more comments improving dialogue between two very important functions in a software organization, our marketing director asked me to interview our testing team lead, Jonathan Patchell, about the ways in which developers drive his team nuts.

Patchell, a computer systems engineer, has been with Klocwork for five years and a team lead for two. He struck a fairly conciliatory tone for this interview, which sorta ruins the adversarial approach, but don’t let his diplomacy fool you. I’ve seen him suffering as the release date approaches and his demeanour changes completely.

Here are Patchell’s top dev peeves:

I had a chance to speak with Bruce Swope, the VP of Engineering at SterlingTech, an ISO13485 Registered full-service medical device software organization offering software development and validation services. SterlingTech has developed software for an array of medical products including implantable devices as well as external support and monitoring equipment. Their team has worked on Class I, II, and III devices that resulted in successful FDA 510(k)s, PMAs, and CE submissions.

Bruce has extensive experience in medical device software development and he is an expert in leading Class III medical software products to commercial release. His depth of experience also spans the development of enterprise solutions, security applications, internal applications, and process control systems. He has been an early adopter of quality practices including ISO 9000 processes, Common Criteria Certification and Capability Maturity Model implementation.

I wanted to talk to him more about the challenges developing software in an FDA-regulated context and what all this means to medical device software development teams.

[Brendan] Given your experience working with a variety of medical device companies, what do you see as the biggest business challenge they face?

[Bruce] The biggest challenge is developing a medical product in a cost effective manner that meets FDA and international regulatory regulations.  Most companies have very limited resources available and have boards or investors that are not used to the rigors of regulated development.  This often leads to a gap between investor expectations and the reality of getting the product ready for market.

[Brendan] What about technology challenges?

[Bruce] The hardware platforms that the systems are developed for are very expensive in time and money to update once fielded.  Often, the hardware is impossible to update without dramatic impact to the patient such as surgery.  This creates a need for software developers to find creative ways to extend the life of the hardware by introducing new functionality without updating the hardware. This can often cause the software to become much more complex than planned.

Further, device manufacturers must balance the expectations of customers against the rigor and security required with making a medical product.  Consumers are very accustomed to seeing feature rich devices reach the palm of their hand and wonder why their heart pump can’t double as a PDA or MP3 player or why they can’t plug their device into the internet to download new alarm tones.

[Brendan] What’s the most common problem your firm is hired to solve?

[Bruce] Many of our customers are looking for an organization that has experience in working with a given technology to create a product that will be approved by the FDA and international regulatory authorities. They are looking for someone that has the experience to deliver a quality product and a complete design history file without wasted effort or significant delays.

[Brendan] In your experience, do most medical device companies have a clear understanding of the regulatory environment or is there still confusion in the market?

[Bruce] Many of our customers are early stage companies that are looking for us to provide the knowledge of the regulatory environment.  Other clients may have an understanding of some aspects of the regulatory environment such as mechanical or electrical but need assistance with the software aspects.

Unless companies invest in dedicated regulatory resources early on and get the FDA or notified body involved sooner rather than later, there will always be confusion and opportunity to misdirect effort.

[Brendan] Any common misconceptions related to compliance issues you can share?

[Bruce] Companies have come to us with a misunderstanding of the impact “level of the concern” will have on the development process for their proposed device.  Companies will often put in place a Quality System that is overly burdensome on the software development process.

The result of these mistakes is often that either too much or too little is done to develop the software.  Either outcome is damaging.  In the case where too much is done, extra cost is incurred and the project completion and entry to the market is delayed.  In the case where too little is done, a rejected submission could result leading to further cost and delays.

[Brendan] What’s the #1 recommendation you give to clients as it relates to the intersection of compliance issues and software development?

[Bruce] Make sure that your company has a good solid Quality System as it applies to software development. Do not put a Quality System into place that you can not follow. This is the cause of most audit problems. Use automated tools in your process to allow your developers to focus on the creative parts of the software development.  Keep things as simple as possible.  Drive out risk early.

[Brendan] Where can people go to get more information? Any good online resources out there?

[Bruce] For an executive overview for determining whether a new device is a medical device or for ideas on how to use a static code analysis tool in medical device development, we have a library of whitepapers people can download.

[Brendan] Thanks!

We are starting to see a large amount of Android phones such as the Droid and Xperia X10 (see a review here) and the (soon-to-be-released) first Google phone, Nexus One. With this, expect the number of apps to increase significantly.

Droid vs. iPhone

So with the increased number of apps, do these developers have the right tools to find and fix bugs? Take a look at the leader of phone applications-iPhone. There have been several posts (here and here) that recommend using the Clang static analyzer. Apple has taken it one step further, apparently rejecting iPhone apps that access private APIs. But Clang won’t help you with Java apps.

So what do the Android developers have? Android is just Java, so there are lots of tools, right? Certainly there are static analysis tools, profilers, unit testing tools and many more. But are these tools really taking into account the Android specifics?

Let’s take an example of a resource leak. Resources such as streams, connections and graphic objects must be explicitly closed; otherwise, you run the risk of throwing exceptions depending on the open resource.




For example:

1 static final String propertyFile = "my_config.ini";
2
3 static String getProperyFromConfigFile(String name)throws IOException {
4    Properties prop = new Properties();
5    FileInputStream st = new FileInputStream(propertyFile);
6    prop.load(st);
7    return prop.getProperty(name);
8 }

Here, a resource leak should be identified since line 5 opens up a FileInputStream, but is never closed before exiting the method. Now, this is all well and good and valuable to be found in any Android specific code, but what happens if I’m using built-in classes from the Android SDK?

For example:

1 public boolean onKeyDown(final int keyCode, final KeyEvent event) {
2    if (keyCode == KeyEvent.KEYCODE_DPAD_CENTER) {
3          final MediaPlayer player = MediaPlayer.create(this, ringtoneUri);
4          player.start();
5    }
6    return super.onKeyDown(keyCode, event);
7 }

Here, you have a situation where a MediaPlayer resource is created at line 3, but never closed on exit. Without the knowledge that MediaPlayer is a resource that should be closed, you will miss this type of issue. This extends to many resources and different issues. You can also have Android-specific null pointer exceptions and use of free issues.

Let me know if you’re doing Android development. I want to hear what you are doing to find these kinds of bugs.

Next week I’m heading out to the Software Assurance Forum (use SOF96945 for the conference code) in Washington D.C. (well, actually Arlington, Virginia, but D.C. sounds more glamorous). If you’re not familiar with what the SWA is, in a nutshell, its key objective is to encourage software developers to raise overall software quality and security from the start, rather than relying on applying patches to systems after vulnerabilities are discovered.

Anyways, while I’m there, I’ll be taking part in 2 speaking opportunities. The first will be as part of a 6 person panel discussion entitled “Understanding Technology Stakeholders: Their Progress and Challenges” (10:30 – 12:00 on Wednesday). The panel is made up of stakeholders from varying disciplines such as industry, academia, standards, and government. A good well rounded panel should provide for an interesting and entertaining hour and a half.

My second session (Friday at 2:20) will see me fly solo as I discuss our (Klocwork’s) experiences and observations as they relate to SATE. I’m not given much time, so I’ll be revving up the motor mouth to make sure I get our points across. I have a sneaking suspicion I just *may* go a little OT.

So, is anyone out there also going to this event? If so, drop me a line either by email (todd.landry@klocwork.com), or Twitter (@todd_landry) and perhaps we can get together to chat. Look for my next blog next Thursday, as I will recap the panel discussion and the other sessions I attend at this event.

Truepath Analysis

Ever been faced with that glassy-eyed expression, the look of unthinking, unwholesome fear when some long, incomprehensible word escapes your geeky mouth and upsets the maiden aunts around the once-a-year, wear-your-best-tie, try-not-to-die-before-the-cake’s-all-gone tea table? OK, so this paper won’t help you in that situation whatsoever, but if you replace your maiden aunts with a bunch of your best geek friends, and replace the tea with a sturdy helping of Dew, knowing how a real whole program analysis solution works might just conceivably come in handy. Some day. “Dude, I was totally stoked when I read this thing, trust me it’s ahh-some.” Maybe.

Anyway, in the best traditions of self-serving corporate PR blogs everywhere, I give you… drum roll please… Whole Program Analysis, the Klocwork Way.

Enjoy.

Today, I had a chance to connect with Connie Beane, the Director of Certification and Safety Critical Software for ENEA Embedded Technology, Inc. Connie has a deep background in safety-critical avionics systems development as a Federal Aviation Administration Designated Engineering Representative (DER) with authority for design assurance level A systems, software and complex electronic hardware. Her additional experience includes 12 years with the FAA in the Transport Airplane Directorate as a Project Officer, Federal representative and Secretary of the RTCA committee SC-180, which produced DO-254, as well as 8 years at Boeing as a Lead Engineer in Boeing Commercial and Boeing Aerospace Divisions.

I wanted to talk to her more about avionics software development, specifically give Kloctalk readers some additional background on the DO-178B guidance that’s used in avionics software development.

[Brendan]:  Can you describe the short version of the history behind this guidance, its purpose, and use in the industry?
[Connie]: In the early 1990’s Boeing was developing the 777 aircraft, which included very software intensive systems.  As a result, industry began using software development and verification tools as a means of developing software more efficiently. The FAA saw this trend as a concern since many of these tools were being used with very little human oversight. When DO-178B was published in 1992, a section on Tool Qualification was included.

[Brendan]:  Is DO-178B guidance only used by teams developing software for commercial aircraft or is it used elsewhere?
[Connie]: DO-178 was developed for commercial aircraft, however, today DO-178B is being used for much more.  It is being used for commercial systems which aid aircraft such as ground based navigation and communication systems. Interestingly, it’s also now being used in military applications both airborne and ground based.

[Brendan]: What’s the most common misunderstanding people have about DO-178B?
[Connie]: That it involves much more than good engineering practices.  DO-178B was developed by government and industry.  They incorporated good engineering practices and some additional activities to ensure safe, reliable software.

[Brendan]: What are a few of the core software development best practices that are required by DO-178B?
[Connie]: Quality Assurance audits, requirements traceability, robustness testing, standards for design, coding and verification.

[Brendan]:  Sounds like a pretty rigorous approach to software development, which of course is a good thing given its safety-critical nature. What kind of cost burden does DO-178B add to a typical project?
[Connie]: That depends on whether the software being developed is performing highly critical functions or not.  For highly critical software, the cost of developing to DO-178B could double the average development cost.

[Brendan]:  Switching gears a bit to your role at ENEA. Out of the full range of DO-178B service offerings you provide, what’s the most common problem you’re brought in to solve?
[Connie]:  Probably the most common issue is a project that is behind schedule and needs to be certified to meet a customer deadline.

[Brendan]:  What’s the biggest change you’ve seen in aircraft systems development during your career?
[Connie]:  Growth in use of software and complex electronic hardware to develop highly integrated avionics systems.

[Brendan]: What do you think the future holds for this guidance? Do you see it evolving in a particular direction that’s different from where it is now?
[Connie]: There is an international committee currently working on the next version of DO-178. This committee is addressing issues such a object-oriented design, model based development, and further defining qualification of software tools.  This committee is also developing a companion guideline to DO-178 which will be applied to ground based software.

[Brendan]: Any final thoughts?
[Connie]:  Companies resist embracing DO-178.  However, once a company has the processes and procedures in place to ensure compliance to DO-178, their software development process is easily understood, repeatable, and maintainable.  This translates to software which is reliable, understandable, traceable and maintainable.

I thought for today’s topic I would delve into something that many organizations have to confront when moving to Agile that is, how to structure their iterations. Many organizations will find that iterations work better if development, testing, and documentation are all done within the same iteration period. Other organizations prefer to offset the testing and/or documentation ½ or a full iteration period after the development is complete. Having lived through both, I thought it might be useful to briefly list the good, and not so good of iteration offsetting.

First the good:

* Testing and documentation are not given features that have been completed at 5 pm on the last day of an iteration, and then expected to have their work done by the following Monday.
* Many dev features are totally complete prior to documentation or testing, so the doc and test teams can start their iteration running, rather than crawling.
* The documentation team is provided with some insight as to the “big picture” of a feature (and not just a standalone story in an iteration) and can plan their documentation for those features with that information in mind. This could help reduce rework on their part.

Now the not so good:

* Development can be drawn back into a previous iteration for bug fixes.
* If demo’ing iterations is a standard part of your process, it could contain a number of bugs since it hasn’t been through any formal testing yet.
* Communication is fragmented…some teams talking about Iteration x, some about Iteration x-1.
* Possible confusion, and more difficult to manage since there are multiple iterations on the go.

Is this list complete? Probably not…but hopefully it will give you something to think about if you’re either starting Agile development, or are struggling with your current implementation. For many projects, the negatives outweigh the positives on this list. As a Product Manager, I’ve seen that many situations where offsets as described above just don’t work. If that’s the case for you, then one possible approach that I’ve seen work is to implement a code freeze 3 days prior to the end of an iteration, and the developers then become part of the testing team.

My personal preference having worked in both scenarios, is to have dev, test, and documentation all occur within the same iteration. Now, my reason is somewhat selfish…when an iteration finishes, I want to show what we’ve done with some level of comfort of the quality and then be able to close the book on that iteration at that point. If that is the approach you prefer, here is a good blog about how to set up such an iteration. Of course, as with most things Agile, there’s never a “right” answer; it’s about finding the right iteration methodology that works for your team’s culture and project goals.

Testing can have many goals: to assess quality, to assess conformance to specification, to help managers decide whether or not to ship, etc. These all affect a test team’s approach.  Very often, it seems that “Finding Important Bugs Early” is the tester’s primary goal during a development cycle in an agile environment. It’s the ideal: find a quality impacting issue as quickly as possible after it was introduced so that the code is fresh in developer’s minds. “Finding Important Bugs Early” is a goal shared by countless test teams.

It struck me as odd, then, that James Whittaker of JW on Test in his excellent “The Future of Software Testing” series (Part 4), stated that in the future, we need to “close the gap between when a bug is created and when that same bug is detected.” In my opinion, why wait? There are plenty of things we can do to close that gap right now!

The most efficient tactics I’ve seen are the following: early involvement of testers in the development process, good unit testing, code inspection, static code analysis, continuous integration, and effective use of stubs.

Early involvement of testers is not a new concept; it’s already a part of good test process practices. For example, it is a key area of TPI (Test Process Improvement) known as the “moment of involvement”. As a bonus, I’ve always found that testers make great requirement appraisers; not only can they help assess that a requirement is verifiable, but also that it is complete, consistent and unambiguous.  Keeping testers in the loop is especially important in an agile setting where it can help make the documentation barely sufficient, as recommended by this type of development process.

We all know that unit tests are your “first line of defence” for testing executable code. To the developers to whom this task most often falls, I encourage you to ask your test team for help should the need arise. A good test team member has a way of seeing testing from a unique perspective. Use this hidden resource! You can pay the favour back when it’s time for them to use a new automation framework. It’s worth it: a good unit test suite can find many problems early that can be hard to diagnose later downstream.

Code inspection is another early detection tool. Used effectively, it can find logic and design problems at the source, not as a symptom (i.e. when system testing will stumble upon the issue). Moreover, there is no prerequisite of compilation to start it! The same is true for static code analysis. We all know the benefits so apply early, apply often, and save a lot of trouble later on.

For environments that are not of that kind where you “throw the product over the wall” to be tested, continuous integration is necessary. Otherwise, your process may be hindering the test team’s effort to Find Bugs Early. Should your testers be validating last week’s code? Last month’s? If they are, they have no chance of finding your most immediate problems. Don’t forget about bug interactions. It frequently happens that a bug is hidden behind another bug; fix one and possibly expose a slew of others. I heard a question asked once that truly captures the idea: “Why is it that when a bug dies, all the others come to its funeral?” You won’t know who’s in attendance if you’re not testing the latest code.

You may recall that I mentioned the effective use of stubs at the beginning of this post. I hope it piqued your curiosity. It’s something I learned by observing effective testers over the years. When a widget comes off the conveyor belt, they don’t wait for the thingamajig it screws into to start testing. The thingamajig may not be ready for another week! They instead whip up a mock-up of the thingamajig that works somewhat like the real thingamajig and try out their widget, pronto! Very often, the combination doesn’t work as expected and they must ask themselves: are their assumptions about the thingamajig wrong or is it the widget itself? It’s a good thing they didn’t wait until next week to discover this potential problem! This type of experiment enables early integration testing: the component didn’t work correctly in its intended environment. This prompts an investigation into the root cause: is it a bug in the component or a poorly defined interface between two components? Beware however, software stubs can fall into the same trap as a test oracle: you can make it as complex as the real thing if you’re not careful.

These are but a few of the tools for Finding Important Bugs Early. There are many other testing approaches that are not addressed here (so don’t throw away your regression tests!) This post focussed on the “early” part of the goal. What about tactics for the “important” part? Good question! I might be coerced into a follow up post one day…