Previously I posted the value of using coding standards, specifically MISRA C and MISRA C++.  This time I wanted to go through some general experiences we had with some of the checkers, specifically the ones that seem to throw a lot of violated rules, to the point that on some code bases MISRA flagged more than one error per LOC!

There are still tons of great rules you can apply even if you don’t make an embedded product.  But as I said before, it doesn’t make sense to turn on all the MISRA rules.  After running through many code bases and looking at the value of MISRA we certainly noticed a trend with a few culprits.  Here are a few examples that we found to be noisy with non-MISRA compliant code.

MISRA C 6.3 and MISRA C++ 3-9-2:

MISRA has the distinction of “required” rules and “advisory” rules.  This is an “advisory” rule.  Essentially it wants you to avoid using types like char, int, short, long etc. and to use specific length typedefs instead.  Obviously, many code bases use the basic types, so be prepared for many issues.

MISRA C 14.9 and MISRA C++ 6-4-1:

This is a “required” rule.  This rule  is really about making sure you have braces for your if/else keywords.  Good practice to have but how many of us really do this?

MISRA C 12.13 and MISRA C++ 5-2-10:

This is an “advisory” rule.  The rule doesn’t want you to mix increment and decrement operators into an expression.  This makes sense because it can be pretty confusing to read.  But in our experiments this seems to be something that many developers do.

MISRA C 17.4 and MISRA C++ 5-0-15:

This is a “required” rule.  The rule only wants to allow you to use array indexing for pointer arithmetic.  Everything else is non-compliant.

MISRA C 14.7 and MISRA C++ 6-6-5:

This is a “required” rule and another control flow example.  A function can only have one point of exit at the end of a function.  I can understand this but as you know, that is not reality.

MISRA C 13.2:

This is an “advisory” rule.  It states that tests against zero should be made explicit.  In other words: if (x != 0) is the proper way, not if (x).  The exception to this is if the operand is Boolean.  I don’t know about you, but you can crown me the super wiener on this? I never make it explicit.

So if you plan to pick up MISRA on your existing project beware of these rules.  I’d like to hear if you do any of those things in your code base.

In a previous post I discussed the process where we practice dogfooding.  This is the process of using Klocwork on Klocwork (KonK).  We started this program several years back with the hopes that we would learn some valuable lessons about usability, performance and anything else that would give us an edge.  The truth is that KonK has consistently allowed us to test our design assumptions early by allowing our own developers to use Klocwork as part of their development.

One of the unexpected results was inadvertently uncovering data that further validated for us the importance of bugs found by a static analysis tool. We correlated testing-found issues and KonK (static analysis) issues assigned to each developer. The result as this graph shows (note: graph is using a logarithmic scale and is a few years old), is there was a very high relationship between the two. Those familiar with bugs found by static analysis tools know that they are often quite different in nature from a bug found by testing where you’re testing requirements, yet developers with a high count of testing-found issues also had a high KonK count. In some cases they pointed to the same problem, in others an indication of the overall bad to the bone problems in some of the new components.

Since we already eat our own dogfood, we’re already believers in the use of static analysis, but this kind of data is a great example of the benefit of finding issues early using static analysis

When I was documenting a new refactoring plugin for Vim, I checked out the Vim web site, and came across this blasphemy:

Vim isn’t an editor designed to hold its users’ hands. It is a tool, the use of which must be learned.

Later, someone sent me this beauty, from Elitist Jerks:

Stop being lazy and read.

Are users lazy? Do they expect hand-holding? Do they expect one button and no manual?

Or is this more true to life?

In the end, it probably comes down to this: Make tools usable. Then technical communicators can spend more time creating truly helpful help, and less time explaining a bad UI.

If our goal as communicators is not just great help, but a great product, Agile makes more sense than ever.  If we want our usability suggestions to be implemented, we need to get developers’ attention while they’re working on that feature. Find the rough edges that would require a lot of splainy-splainy, request a change, and then rejoice that we don’t need to explain what’s now obvious. Sometimes it’s hard for me to decide what’s a rough edge. Would my audience of developers find this as confusing as I do? But I’m learning to trust my gut.

At the same time, though I appreciate the benefits of Agile, I’ve started to worry less about the help being “done” at the end of an iteration; instead, I want to make sure I understand a feature well enough before the end of the iteration to suggest design changes and know what help will be required.

I just got back from a visit to the Valley and had an awesome week in San Jose/San Fran.  I even had time to play a bit of the tourist this time (I ran the Golden Gate bridge/Presidio).  All that was fun, but what I always enjoy is the conversations we had with customers and prospects at this year’s ESC SJ 2010 conference.

It is always interesting listening to their successes and teasing out the trending topics and new issues that matter to development teams.  Here are the top three themes that caught my ear this year:

1. The economic rebound is well underway, with growth and optimism from every quarter.  It may be too early to see results on the balance sheets, but the positive attitude is back.
2. Embedded developers are searching for enterprise-class developer productivity tools, like static analysis, for even tiny codebases (less than 40 kLOC).
3. By far, the most-often raised topic in one-on-one conversations was coding standards, with MISRA C and C++ as the favorite.  MISRA’s time has definitely arrived for the embedded community.

So all in all, a great time, and looking forward to next year.

Dogfooding?  Is this the process of making sure Rover is well fed?  Maybe it’s a movement of people eating dog food?  Or maybe Rover IS the dinner (cue animal activists).  No, dogfooding is coined from the saying “Eating one’s own dog food”.

So what on earth am I talking about.  Well first I’m breaking a golden rule here when it comes to blogging which is talking about your company (I don’t know how I’ll sleep tonight).  Klocwork does eat its own dog food.  We call this KonK – Klocwork on Klocwork.

So why is this important?  Ultimately we make a software product that we sell to other companies that make software.  So who better to experience first hand what we are designing.  By using KonK and updating it frequently it gives us immediate feedback on usability and scalability (our code base is quite large).  Plus being in the business of bug detection it helps us sort out the value of the quality of those bugs.  As the product manager I’m not using it day to day like the developers, so they are the ones that bring any kind of deficiencies to the design front and center.  Hopefully I can talk a little about the benefits and conclusions we have made by using KonK in a later post.

One thing that has helped me with dogfooding is requirements capture.  Being in product management obviously the clear objective is to work closely with customers to define requirements and your product direction.  Those requirements don’t necessarily paint the picture as much as you would expect or hope.  Now that I can play with the intended design on our own code base it paints a clear picture of what may be missing or what may just be plain ugly.

Just got back from our 5^th annual Klocwork Customer Advisory Board, graciously hosted in hot and sunny Phoenix, Arizona by one of our top customers. These events are now running like, umm, clockwork, as we have come up with a winning recipe that mixes a nice combination of Klocwork delivered material and customer delivered material over the course of 2 days. We had a great mix of ‘seasoned veterans’ and new members to this year’s CAB which worked out extremely well. We decided to add a keynote speaker to kick our meetings off, which by all accounts was very well received. I think that will now be a permanent part of our meetings going forward.

While we were inside a meeting room for the better part of 2 days, we did have some time to experience the great outdoors in Phoenix.  The afterhours ‘event’ this year was riding around the Tonto National Forest…at night, without headlights…in Hummers. Not your garden-variety Hummers either, but bad-ass Hummers that seem unstoppable. Throw in some night-vision goggles, scorpions, cacti, and tarantulas and you’ve got yourself one heck of an outing. If you’re ever in the Phoenix area, you have to check out Desert Storm Hummer Tours.

Every year I come back from CAB wondering how we will top the last one, and it is no different this year. This CAB has set the bar very high, but I’m sure we will find a way to exceed it.

Our partners at Code Integrity have a good blog that touches on many of the benefits and barriers to static analysis within a development organization. They have an interesting post on “False false positives” – a great phrase that captures one of the key challenges in developer adoption of the technology.

While increased sophistication means that static analysis tools can catch more problems with a higher degree of accuracy, the burden increases on the reviewer of the results to interpret them correctly. If you were grep’ing through some code for something you can quickly review (and dismiss) many of the results because you understand what your “analysis” is doing. With static source code analysis, this is much less apparent.

We see many engineers look at a complex bug report and not take the necessary time to understand the problem and fix it. This is mostly because they don’t understand what the static analysis tool is doing and how deep it is analyzing the code. The result is a real bug being marked as a false positive – or a “false false positive” if you will. These bugs then disappear off the queue never to be seen again – a lost opportunity.

One of their key recommendations to overcoming this barrier is using training and joint review of results to educate developers on why the tool is flagging a potential error, what the mitigation options are, etc. Code Integrity has a bunch of deployment and training services to help customers with these types of deployment hurdles.

In our experience, all developers need is one ‘aha’ moment where the tool finds a nasty, subtle bug that would be hard to find using any other method. Once that happens, the developer is a convert. I would also say the burden isn’t just on training, but the tool vendors as well. We all have to continue making the usability of the tool such that developers should be able to instantly recognize why the tool is flagging the error and give the developer all the info they need to recognize the bug and take the appropriate action.

There has been a start to bring the concepts of  lean manufacturing  into agile development. Recently, Mike Cottmeyer in How to Build a Large Agile Organization proposes that Agile on its own is not enough for a large organization.  In his view, Agile falls short and needs to be supplemented by additional methodologies like Lean or Kanban when coordinating outside the development team.

If adoption of Agile is impeded by its very nature in large organizations and Kanban is the proposed answer, then the Agile solution is insufficient. Agile needs to expand its scope to be relevant and useful for non-developers as well as across development teams.

To understand how Lean applies to Agile development, I’m going to take a short detour though history.

Mapping manufacturing principles to software development is an interesting cross-pollination of ideas. Discrete manufacturing is quite different from application development, but that doesn’t mean the software industry can’t learn a thing or two from a different sector.

Lean was born out of a need to re-invent the manufacturing industry, which had not really evolved since the inventions of Henry Ford and the production line. From Ford’s time to the post second world war period, most manufacturing was very good at making enormous quantities of the same product, regardless of the demand. Ford’s famous quote about color clearly exemplified the thinking of the day: “Any customer can have a car painted any colour that he wants so long as it is black”. In other words, Ford’s production line was optimized for manufacturing, not profit, and turned out to be quite inflexible when market conditions changed.

In the 1950s, Sakichi Toyoda made a revolutionary leap forward with two principles:

1. Pull vs. push – at any point in the production process, the trigger to start work on a production unit is governed by its upstream neighbor.  As an example, I do not start my work on a product unit until the guy following me says he will be able to receive it.
2. Efficient manufacturing depends on the management of three key inefficiencies: overburden (muri), inconsistency (mura), and eliminating waste (muda).

Together, these elements formed the underlying principles that Sakichi spearheaded into what is now known as The Toyota Production System (TPS). The TPS has subsequently been used as the the basis for Western derivatives such as Just-In-Time, value-stream mapping, Six Sigma and Lean, to name a few.

So what does this have to do with Agile and large organizations?

There are well-documented cases where agile alone was not enough, and that’s where Lean/TPS can add value.  For the most part though, the application of Lean principles has been limited to just one part: Kanban.

The TPS Kanban methodology has two aspects. First,  a Kanban card is attached to every unit under production and carries contextual information (metadata) about the tasks that need to be performed on that unit  and second, task readiness and data are used to trigger an specific action (work).

Over the past decade, the Agile methodology has been used successfully within  development teams, usually sized between  8 and 15 people. Agile’s benefits and values for this type of environment have been well articulated by many others (including on this blog), but most Agile adopters may not have realized the close mapping to Lean/TPS.

• Muri (overburden) – overproduction – in an Agile context, this is usually expressed as over-planning
• Mura (inconsistency) - elimination of bugs at the earliest stages, resulting in more  stable and reliable iterations
• Muda (waste) – close interaction with the customer to absorb change and prevent wasted iterations
• Kaizen (continuous improvement) – refactoring, unit testing, system integration

Secondly and more importantly for large teams, the TPS/Lean idea of pull vs. push is key. But there are other aspects of Lean/TPS that would benefit software development, Kanban being an important one but not the only one.

In an Agile context, Kanban is usually expressed as a board or wall with movable index cards to visualize units of customer value and work flow. This is where I think the rails have come off Agile/Kanban compared to the original TPS philosophy.  Kanban is just one gear in the whole TPS methodology.  Its an integral part but no more important than the other parts.  To function optimally, the TPS/Lean requires all the piece to be implemented not just one.

The other aspects of TPS/Lean are:

• Andon (signage, early warning)  - literally means paper lantern and is used to call attention to a problem in the process.  For Agile, it should be express as how do you measure your team’s progress and convey that information to the whole organization.
• Jidoka (autonomation) – automation with human intelligence.  The efficient use of tools like static analysis and continuous build to aid in development.
• Poka-yoke (fail-safing) – not just exception handling, but actual prevention of faults and counter-measure strategies to prevent the fault from reoccurring.

These other parts of the TPS were not born because people like more processes and rules; they came out of need, something the agile methodology has yet to realize it requires.

Part IV – Joy is in the eye of the beholder

In preceding posts on this topic, I’ve outlined the continuing shift from in-person, physical interactions as being the defining notion of both social and business contexts, towards virtual interactions and marketplaces, and the fact that in all aspects except the most personal the latter can fulfill everything expected of the former. But what does all this have to do with engendering a vibrant and successful code review practice within a development organization? On the face of it, nothing much. Code review, you could determine, tends to happen within organizations that enforce it, so all we need to do is to pass a rule requiring code review before shipment and we’re gold, right? I mean, what could go wrong?

Unfortunately the reality isn’t so clear cut. Most organizations worth their salt have a “requirement” for code review to be performed. At least on the important bits (a definition I particularly like, particularly when it comes to motivating programmers working on the “unimportant” bits). Or the hard bits (likewise). One awesome process description I saw in action recently called for the architects in the team to review each others’ code, but everybody else got a free ride – because obviously the code that our highest paid, most talented developers produce is the stuff we’re really worried about being wrong, amirite?

Despite such requirements, whether they make sense or not, the rarity of code review is out there for all to see. In fact, we recently sponsored the analysts at Forrester to produce a survey of code review practice in various different development environments (embedded, ISV, IT, etc.) and found that although most developers appear to believe that they live in organizations that do code review, most also claim that it doesn’t happen consistently for some reason or other.

Let’s take the most prevalent excuse claimed: too busy, got other stuff to do. Why is this claimed so uniformly? Certainly developers are busy people – we pay them a lot and expect a lot for that remuneration, after all, so sure they’ve got other stuff to do. But if we put this in another context, perhaps outside of the development process, say visiting Grandma, I’m sure there’s a parallel to be drawn. After all, we should definitely go visit Grandma more often. She’s probably not going to be around that long. She’s a nice lady, and she cooks cute little cupcakes when we do go there. So why not do it more often? Too busy, got other stuff to do. And, of course, it’s really annoying to drop whatever you are legitimately doing (whether development or otherwise) and go visit the old lady with the bad habit of talking about you as if you weren’t in the room.

As a manager, therefore, your task in ensuring code review has a much lower friction profile to it, is to remove what I think I’ll try to trademark as The Grandma Effect. If I have to stop what I’m doing, put on my Sunday best, and sit listening to stories from three decades past, I’m going to find all kinds of reasons not to. But if instead (to stretch the analogy to what I’m sure is the breaking point), Grandma were to learn how to play a wtf-pwning Mutilate-spec’d Rogue, then my interactions with her become much more palatable and manageable. Replace the trip to Grandma’s house with the annoyance of setting up a formal code review and you’ll get the picture.

There’s a tipping point here, and it revolves around the place and relevance of social tooling within your development team. You, your peers and your reports are currently interacting with friends over MSN, Twitter, Facebook, you name it. They’re booking dates, arranging weekend schedules, and getting the latest news from Reddit. You name it, they’re doing it and it’s all coming to them through a few pretty simple to leverage mechanisms.

Replace their social and common knowledge-gathering activities with knowledge leveraged within the code base, and it’s pretty easy to see how you could graft what has the potential to be a very annoying activity, e.g. code review, onto a natural way of conducting business. Instead of creating an entire workflow around the invite, simply inform interested parties about commits and let them decide whether to review or not. Instead of insisting on top-down imposition, encourage a bottom-up adoption simply through ubiquitous availability of information.

Nobody forces anybody to use a service like Reddit, after all. It exists and thrives because of the community that finds value in its presence. Personally I interact with it through RSS as I find that the most natural way of learning what it’s got to say. Lots of different feeds of information for the different types of news, all presented through a common aggregation mechanism that feels natural, that works well, and that I don’t have to think about.

So, if commits that my team members are making, or commits that others are making to a component for which I feel either moral or actual responsibility are available through that same mechanism, I’m going to take advantage of the tools to review those commits and to make my presence felt.

No formal review.

No formal sign-off.

But also a guarantee of way more participation, and what’s more, a broad reach around the typical chain-of-command style code reviews that we know and hate, instead engaging atypical contributors, not to mention the legion of lurkers just out to learn more. And isn’t that what it’s all about at the end of the day?

In summary: don’t require the architect, but appreciate their presence. And instead of bringing the people to the code, bring the code to the people.

What is MISRA? More Irrelevant Software Requirements Again…uh no but certainly the sentiment of many developers.  MISRA (Motor Industry Software Reliability Association) is a coding standard, which first released MISRA C in 1998 and has since been revised.  Obviously, this came out of the automotive sector with a clear focus on helping software systems to be more reliable and maintainable.

MISRA has since grown.  Now you see more and more industries adopting these standards.   In 2008, MISRA released the C++ equivalent standard.  So the obvious question is, should I apply this to my software source code base?  In short, yes.  Maybe you don’t need ALL of the MISRA rules, but there are certainly some that are worth your time.

Examples range from the very simple checker rules sets that mostly fall into the coding style category:

MISRA C rule 19.1:  All #include directives in a file shall only be preceded by other preprocessor directives or comments.

MISRA C rule 2.2: You should only use /*…*/ style comments.

Down to more important issues (I hope everyone has the tools in place to find this):

MISRA C rule 9.1: Unintialized variables.

It really depends what you’re trying to accomplish.  Certain rules such as 9.1 are very important, while others can help you with portability or just serve to make your code more maintainable and less complex going forward.

Word of warning, taking an existing software code base and apply the entire MISRA rule set is guaranteed to throw many errors.  Take the zlib project, obviously not MISRA compliant, but to illustrate the impact of turning on all the MISRA rules, will throw over 5000+ issues.  That is an issue for every line of code…yeah that’s useful.  So be warned, don’t try to boil the ocean.  Start iteratively, even if that means only one rule at a time.