Deadlock: frog vs. snake
Photo: David Maitland / National Geographic

Continuing the discussion about the embedded community moving to muticore/hetrogeneous hardware from watch out here comes multicore, embedded software development teams have historically been shielded from mulitcore issues. This is due to the specialized functionality of many embedded application classes and the inherent serialized nature of the C language.^[1]

Muticore is an ambiguous term for software developers and one they don’t really use; software developers think in terms of threads/processes and concurrency, not how many cores or processors are available on the target. Concurrency is not a new topic either as Mark Smotherman captured in a history of multithreading, it has been a subject in computer science since its early beginnings in the 1950s.

What has changed is the rapidly increasing use of multicore technologies for embedded devices. One of the prominent software challenges that moving to multicore execution exposes is latent deadlocking bugs as true parallel execution comes into play, instead of a single core’s task scheduling/context switching techniques.

As an example, consider the following code snippet, which has been paraphrased from a deadlock discovered in a real-world open source multithreaded project.

Can you spot the deadlock?

lock_t lock1, lock2;
int refCount = 0;

void enter() {
   reserve_lock(lock1);
      if( refCount == 0 )
         reserve_lock(lock2);
       release_lock(lock1);
   refCount++;
}

void leave() {
   reserve_lock(lock1);
   refCount--;
   if( refCount == 0 )
      release_lock(lock2);
   release_lock(lock1);
}

To see the answer and understand the conditions that lead to the deadlock, download Klocwork’s whitepaper on Developing software for multicore.

A little about the picture in this post.  I found it when searching for pictures of deadlocks.  The photographer, David Maitland, titled his image “Deadlock” and describes it as a continuing struggle between a Morelet’s tree frog and cat-eyed snake. After three hours in the high-stakes cage match, Maitland said there was no clear winner.

VDC Research, “Next Generation Embedded Hardware Architectures: Driving Onset of Project Delays, Costs Overruns, and Software Development Challenges”, September 2010.

Processing Architecture for embedded devices in the current projects and expected in next two years

Although multicore and multiprocessor technologies have started to proliferate in the embedded market, smartphone manufacturers are in the midst of a rapid shift to multicores due the market transistion from business users to consumers. It’s becoming evident that in order for handset manufactures to differentiated their products, a feature war has commenced: wireless connectivity, video/flash/audio playback and even video creation/editing/conferencing have become are market drivers.

CPU designers have already responded to this need, just last week ARM announced its next generation of Cortex^™-A15 MPCore^™ design and Intel entered the smartphone cpu market earlier this year with the new Atom Z6 core.

For perspective, in a short few years mobile handset industry has had to move from relatively simple devices supporting cellular communication and user-interface tasks to multicore/multi-chip designs in order to grow features and thus market share. Until the arrival of power-aware multicore and heterogeneous chipsets, cellular devices have been limited by computational horsepower, power envelopes and time-to-market issues.

In any market with tremendous competition, time-to-market is crucial and one of the fastest methods to add features within the tight power budget is to add additional task-specialized silicon either as heterogeneous cores or chips from third-party suppliers that can be put to sleep or turned off entirely when not required.

Feature growth has in turn placed new burdens on mobile software development teams. Leaving the comfortable environment of a single core means addressing potential threading/concurrency problems and endian ordering issues when integer data is transmitted off-core.

Klocwork commissioned VDC Research to quantify the business impact of moving to multicore/multiprocessor environments and the results of this growing complexity is stark: multicore and multiprocessor software projects are

• 4.5X more expensive,
• have 25% longer schedules,
• and require almost 3x as many software engineers.^[1]

Embedded and especially smartphone development teams need to be gearing up for the feature arms race to come and should be looking at methods to mitigate theses impacts by implementing productivity tools like static analysis and automated code reviews sooner than later.

[1] VDC Research, “Next Generation Embedded Hardware Architectures: Driving Onset of Project Delays, Costs Overruns, and Software Development Challenges”, September 2010.

Refuctoring describes the process of making your code unmaintainable by anybody but yourself.  I love some of the examples of Refuctoring such as “Pig Latin”, “Treasure Hunt” and my personal favorite “Stating the Bleeding Obvious”:

For example:

//initialize a to 1
int a=1;

Not that I’d ever do that (pause while I go clean up some code). Ahem, right anyways I thought I’d throw out some other refuctorings:

1.    “Catch me if you can” – Use so many goto statements that it will make anyone’s head spin.  Especially when you start adding backward goto’s.  Take a look at the CVS source code,  they have some nice (nasty) examples.

2.    “Giant tar pit of hell” – This is hard to blame one single developer as it really encompasses many developers getting together to create one big cohesive piece of crap.  You know you have a problem when you run out of printer toner trying to print these.

"Giant tar pit of hell"

3.    “WTF” – Using names that no one will know…ever.  If you create a bankaccount object, instead of calling it ‘x’, here’s a wild idea, why not call it ‘bankaccount’.  Let’s face it we are all guilty of this.

Now if you really want to be special (I don’t mean in a good way), try combining #1 and #3:

int afunction()
{

there:
   ...
   if(something)
   {
      goto here;
   }
   if(somethingelse)
   {
      goto there;
   }
here:
   ...
   if(somethingelseagain)
   {
   goto end;
   }
end:
...
return;
}

I have to admit, I’m not winning any awards either. It is certainly time to get on the refactoring bandwagon.

• The U.S. Army’s Future Combat System is estimated at 60 million lines of code (MLOC)
• The software that runs the Boeing 787 is almost 7 MLOC, triple that of the 777
• GM says future cars will have >100 MLOC (that sounds high, but hey, <insert GM joke here>)

So, yes there’s a lot of code out there, it’s growing, and it’s getting more complex. It’s tough to put these numbers into perspective… Jack Ganssle has a clever column that does just that:

• A million lines of code printed out would be 18,000 pages
• A million lines of code will typically have 100,000 bugs pre-test
• A million lines of code costs $20m to $40m

I won’t try to compete with Jack on the stats front since that column speaks for itself, and it’s not just the mission/safety critical systems that are trying to deal with the challenges of more code, more complexity, and less time (as in time-to-market). The consumer market is no different. Guess how much code is in the Xbox HD DVD Player? Not the whole system, just the player? 4.7 MLOC. Heck, you could fly an airplane with all that code (hopefully most of it can also be re-used on a non-obsolete disc format). Just as a point of comparison to today’s gaming systems, ask yourself how many software engineers worked on this “game system”:

Yes, I had to throw Digital Derby under the bus didn’t I? Not because I don’t have fond memories of playing this game that was about the size of carry-on luggage. In fact, I played it until the poor little conveyor belt just stopped working and I put up with the, ahem, limitations in the technology. I was actually quite patient (more patient than I am now with technology) and gave my little Digital Derby second and third chances to work properly.

More code, more complexity, impatient customers, and very little margin for error – these are the trends that are driving tool and process automation across the board in the SDLC.