Deadlock: frog vs. snake
Photo: David Maitland / National Geographic

Continuing the discussion about the embedded community moving to muticore/hetrogeneous hardware from watch out here comes multicore, embedded software development teams have historically been shielded from mulitcore issues. This is due to the specialized functionality of many embedded application classes and the inherent serialized nature of the C language.^[1]

Muticore is an ambiguous term for software developers and one they don’t really use; software developers think in terms of threads/processes and concurrency, not how many cores or processors are available on the target. Concurrency is not a new topic either as Mark Smotherman captured in a history of multithreading, it has been a subject in computer science since its early beginnings in the 1950s.

What has changed is the rapidly increasing use of multicore technologies for embedded devices. One of the prominent software challenges that moving to multicore execution exposes is latent deadlocking bugs as true parallel execution comes into play, instead of a single core’s task scheduling/context switching techniques.

As an example, consider the following code snippet, which has been paraphrased from a deadlock discovered in a real-world open source multithreaded project.

Can you spot the deadlock?

lock_t lock1, lock2;
int refCount = 0;

void enter() {
   reserve_lock(lock1);
      if( refCount == 0 )
         reserve_lock(lock2);
       release_lock(lock1);
   refCount++;
}

void leave() {
   reserve_lock(lock1);
   refCount--;
   if( refCount == 0 )
      release_lock(lock2);
   release_lock(lock1);
}

To see the answer and understand the conditions that lead to the deadlock, download Klocwork’s whitepaper on Developing software for multicore.

A little about the picture in this post.  I found it when searching for pictures of deadlocks.  The photographer, David Maitland, titled his image “Deadlock” and describes it as a continuing struggle between a Morelet’s tree frog and cat-eyed snake. After three hours in the high-stakes cage match, Maitland said there was no clear winner.

VDC Research, “Next Generation Embedded Hardware Architectures: Driving Onset of Project Delays, Costs Overruns, and Software Development Challenges”, September 2010.

Endianness refers to the ordering of bytes into memory. As many of you are aware, computers do ordering differently. You can have the representation of Big-endian or Little-endian.

Essentially Big-endian stores data big-end first, meaning the first byte is the biggest and Little-endian stores data little-end first, meaning the first byte is smallest.

Because all machines are different and write data either as big or little-endian, a computer could read this data incorrectly.  If you are not prepared ahead of time for heterogeneous processor architectures, then you might be in for a world of hurt. This article describes the problem in detail and also provides solutions when you’re starting off.

But what if you’re not prepared?  This can be a costly and complicated problem when developing on large systems with multiple processors or during a porting effort.  You need to be able to verify the developers provide data interaction with the target processor in the proper endian format.  Here’s a good post that talks about this issue in more details, and references a bunch of good third party articles, some of which outline how source code analysis can help; the basic idea being that source code analysis can flag instances where data is being transmitted to or from the target without being transformed.

We’ve just introduced some capabilities in this area which are described in this white paper authored by our CTO Gwyn Fisher. Check it out and let us know what you think.

As promised….

In my last blog, I promised a joke about recruiters.  And here it is…the classic recruiting joke.

A recruiter was hit by a bus on his way to work one day, despite the excellent medical care he received, he unfortunately died on the operating table.

According to the After Life Standard Operating Procedure, the recruiter was told he had to visit both Heaven and Hell before making his choice of where he would like to remain.  He told them there was no need to visit Hell, he would just go to Heaven…however, SOPs being what they are (eerily similar to HR policies, in fact), he was required to visit both Heaven and Hell.

First up was Heaven, he went through the Pearly Gates and was greeted with peace and quiet, lots of lovely clouds, gentle, soothing music, angels floating around and it was all very serene and very nice.

Then he went down to Hell.  Much to his surprise, he was greeted by the Devil, who appeared to be a very friendly, fun kinda guy.  He could tell great jokes, made the recruiter laugh and then he took him on a tour of Hell.  It was nothing like he imagined…yes, it was hot, but people were standing around in groups, laughing and appearing to have a very good time.  He was treated to a fabulous BBQ buffet, full of delicious meats cooked over the fires of Hell.  After his buffet, he was entertained by a very talented band called the Hounds of Hell, and he thoroughly enjoyed himself.

He was brought before St. Peter after his tours, and he was asked how his day was and was told he could now make his decision on whether he wanted to spend his afterlife in Heaven or Hell.  After much hemming and hawing, he decided that based on his tours, he would prefer to go to Hell.  St. Peter asked if he was sure, because that really was a surprising decision and it was final, the recruiter said, yes, I am sure and he was sent back down to Hell.

When he arrived back in Hell, he was once again greeted by the Devil…but this time the Devil grabbed him, attached chains to him, and gave him a shovel.  He pushed him into a fiery pit with the rest of the people in Hell who were also all chained and toiling away with picks and shovels.  The recruiter looked up at the Devil and said, “What happened?  This is not how Hell looked when I was on my tour”  The Devil gave him an evil grin and said “Before we were recruiting you, now you are considered an employee”.

Processing Architecture for embedded devices in the current projects and expected in next two years

Although multicore and multiprocessor technologies have started to proliferate in the embedded market, smartphone manufacturers are in the midst of a rapid shift to multicores due the market transistion from business users to consumers. It’s becoming evident that in order for handset manufactures to differentiated their products, a feature war has commenced: wireless connectivity, video/flash/audio playback and even video creation/editing/conferencing have become are market drivers.

CPU designers have already responded to this need, just last week ARM announced its next generation of Cortex^™-A15 MPCore^™ design and Intel entered the smartphone cpu market earlier this year with the new Atom Z6 core.

For perspective, in a short few years mobile handset industry has had to move from relatively simple devices supporting cellular communication and user-interface tasks to multicore/multi-chip designs in order to grow features and thus market share. Until the arrival of power-aware multicore and heterogeneous chipsets, cellular devices have been limited by computational horsepower, power envelopes and time-to-market issues.

In any market with tremendous competition, time-to-market is crucial and one of the fastest methods to add features within the tight power budget is to add additional task-specialized silicon either as heterogeneous cores or chips from third-party suppliers that can be put to sleep or turned off entirely when not required.

Feature growth has in turn placed new burdens on mobile software development teams. Leaving the comfortable environment of a single core means addressing potential threading/concurrency problems and endian ordering issues when integer data is transmitted off-core.

Klocwork commissioned VDC Research to quantify the business impact of moving to multicore/multiprocessor environments and the results of this growing complexity is stark: multicore and multiprocessor software projects are

• 4.5X more expensive,
• have 25% longer schedules,
• and require almost 3x as many software engineers.^[1]

Embedded and especially smartphone development teams need to be gearing up for the feature arms race to come and should be looking at methods to mitigate theses impacts by implementing productivity tools like static analysis and automated code reviews sooner than later.

[1] VDC Research, “Next Generation Embedded Hardware Architectures: Driving Onset of Project Delays, Costs Overruns, and Software Development Challenges”, September 2010.

There have been a number of stories about internet dating gone bad lately.  One regularly hears stories about how people misrepresent themselves on dating websites; use old pictures, or even someone else’s pictures to lure unsuspecting love interests in.

These type of experiences are not limited to those seeking love…it also happens to those seeking a job, or those seeking an employee.  Technology is a wonderful thing but should never be used in place of the good old face to face meeting that must happen in the recruiting process (and in the love match process) .  I made a rookie mistake way back at the start of my career that reinforced this for me.   I was working for a high tech company that shall remain nameless, as I was not the only idiotic one in this story and I must protect those who are not as forthcoming about their idiocy.  We were looking for a developer, and the search was not going well.  Finally, we came across a resume from an individual, and it looked quite impressive.  As the individual lived in a different city, we set up a phone call for the interview.  It went well enough for us to want to have a technical interview with the individual, so in the interests of saving time and money, we also set that up by phone.  I am not sure I would have been able to tell there was anything amiss if I had been on that call, but in hindsight, I should have at least placed the call, or sat through a few minutes of it.  The technical interview went very well.  Then… we made our big error in judgement, we extended an offer without having laid eyes on the person, the offer included relocation costs.

Within 2 hours of the person starting, the hiring manager came to me in a panic.  Apparently, she was quite certain that this was not the same person she had interviewed over the phone…this became blatantly obvious as time went on.   We are still  not entirely sure what happened but we suspect the candidate had someone else do the technical interview for him…and since it was over the phone, we had no way of knowing.  Needless to say, the new employee did not stay an employee for very long and I learned a very valuable lesson.

This goes both ways…it is worth the trip to check out an employer face to face.  I have no doubt that candidates have been wooed by companies that were far less than they appeared and had a face to face meeting at the company’s site been conducted, lots of relevant and valuable information would have been gathered.  Do your due diligence in looking for a job, don’t accept what a recruiter says at face value, there are some wonderful jokes about recruiters and I will share one in my next blog.

We get many questions from medical devices customers on how they should validate the use of Klocwork’s static analysis tools for the FDA. I suspect the situation would be similar for most vendors of software development tools. As we’ve done before, we thought it would be a good idea to ask Bruce Swope from SterlingTech Software to clarify this whole topic for us.

[Brendan] First, what is tool validation?
[Bruce] Tool validation is the act of demonstrating that a tool will consistently produce expected results.

[Brendan] How can a medical device company know whether they should validate a tool or not?
[Bruce] From 21CFR820.75, “Where the results of a process cannot be fully verified by subsequent inspection and test, the process shall be validated with a high degree of assurance and approved according to established procedures”. For example, if you are using a tool to perform work and you do not plan on using any other method to verify that the work was done properly then the tool will need validation. Please note that you must validate the tool for your intended use, not the entire tool.

FDA Tool Validation Checklist

[Brendan] Ok, let’s take a specific example. What would validating a static analysis tool involve? 
[Bruce] Here’s a basic list of what needs to be completed.

a) Develop a Tool Validation Plan. This should include the test environment and methods to be used
b) Develop a set of requirements that the tool is intended to meet
c) Develop a test protocol to verify that the requirements have been met
d) Develop a traceability matrix and verify that all requirements have been tested
e) Execute the test protocol
f) Write a test report summarizing the results

[Brendan] Do most companies do this themselves or can they outsource this activity? 
[Bruce] If the company has the internal trained resources available then they can certainly do the tool validation themselves. Many companies lack the staffing, or expertise, to perform validation of a software tool. It is common for these companies to outsource the documentation and testing activities to a firm like SterlingTech. We’ve completed this process numerous times before and it’s a good way to reduce the cost and effort around the validation process.

[Brendan] Thanks Bruce.

Complexity of Medical Device Software Continues to Grow

IEC 62304 is becoming a hot topic amongst medical device software professionals. We asked Bruce Swope, VP Engineering at SterlingTech Software for his views on this standard and what it all means for medical device companies. Bruce has extensive experience in medical device software development and he is an expert in leading Class III medical software products to commercial release. His depth of experience also spans the development of enterprise solutions, security applications, internal applications, and process control systems. He has been an early adopter of quality practices including ISO 9000 processes, Common Criteria Certification and Capability Maturity Model implementation.

[Brendan] Thanks Bruce.