Dogfooding?  Is this the process of making sure Rover is well fed?  Maybe it’s a movement of people eating dog food?  Or maybe Rover IS the dinner (cue animal activists).  No, dogfooding is coined from the saying “Eating one’s own dog food”.

So what on earth am I talking about.  Well first I’m breaking a golden rule here when it comes to blogging which is talking about your company (I don’t know how I’ll sleep tonight).  Klocwork does eat its own dog food.  We call this KonK – Klocwork on Klocwork.

So why is this important?  Ultimately we make a software product that we sell to other companies that make software.  So who better to experience first hand what we are designing.  By using KonK and updating it frequently it gives us immediate feedback on usability and scalability (our code base is quite large).  Plus being in the business of bug detection it helps us sort out the value of the quality of those bugs.  As the product manager I’m not using it day to day like the developers, so they are the ones that bring any kind of deficiencies to the design front and center.  Hopefully I can talk a little about the benefits and conclusions we have made by using KonK in a later post.

One thing that has helped me with dogfooding is requirements capture.  Being in product management obviously the clear objective is to work closely with customers to define requirements and your product direction.  Those requirements don’t necessarily paint the picture as much as you would expect or hope.  Now that I can play with the intended design on our own code base it paints a clear picture of what may be missing or what may just be plain ugly.

Just got back from our 5^th annual Klocwork Customer Advisory Board, graciously hosted in hot and sunny Phoenix, Arizona by one of our top customers. These events are now running like, umm, clockwork, as we have come up with a winning recipe that mixes a nice combination of Klocwork delivered material and customer delivered material over the course of 2 days. We had a great mix of ‘seasoned veterans’ and new members to this year’s CAB which worked out extremely well. We decided to add a keynote speaker to kick our meetings off, which by all accounts was very well received. I think that will now be a permanent part of our meetings going forward.

While we were inside a meeting room for the better part of 2 days, we did have some time to experience the great outdoors in Phoenix.  The afterhours ‘event’ this year was riding around the Tonto National Forest…at night, without headlights…in Hummers. Not your garden-variety Hummers either, but bad-ass Hummers that seem unstoppable. Throw in some night-vision goggles, scorpions, cacti, and tarantulas and you’ve got yourself one heck of an outing. If you’re ever in the Phoenix area, you have to check out Desert Storm Hummer Tours.

Every year I come back from CAB wondering how we will top the last one, and it is no different this year. This CAB has set the bar very high, but I’m sure we will find a way to exceed it.

Our partners at Code Integrity have a good blog that touches on many of the benefits and barriers to static analysis within a development organization. They have an interesting post on “False false positives” – a great phrase that captures one of the key challenges in developer adoption of the technology.

While increased sophistication means that static analysis tools can catch more problems with a higher degree of accuracy, the burden increases on the reviewer of the results to interpret them correctly. If you were grep’ing through some code for something you can quickly review (and dismiss) many of the results because you understand what your “analysis” is doing. With static source code analysis, this is much less apparent.

We see many engineers look at a complex bug report and not take the necessary time to understand the problem and fix it. This is mostly because they don’t understand what the static analysis tool is doing and how deep it is analyzing the code. The result is a real bug being marked as a false positive – or a “false false positive” if you will. These bugs then disappear off the queue never to be seen again – a lost opportunity.

One of their key recommendations to overcoming this barrier is using training and joint review of results to educate developers on why the tool is flagging a potential error, what the mitigation options are, etc. Code Integrity has a bunch of deployment and training services to help customers with these types of deployment hurdles.

In our experience, all developers need is one ‘aha’ moment where the tool finds a nasty, subtle bug that would be hard to find using any other method. Once that happens, the developer is a convert. I would also say the burden isn’t just on training, but the tool vendors as well. We all have to continue making the usability of the tool such that developers should be able to instantly recognize why the tool is flagging the error and give the developer all the info they need to recognize the bug and take the appropriate action.

There has been a start to bring the concepts of  lean manufacturing  into agile development. Recently, Mike Cottmeyer in How to Build a Large Agile Organization proposes that Agile on its own is not enough for a large organization.  In his view, Agile falls short and needs to be supplemented by additional methodologies like Lean or Kanban when coordinating outside the development team.

If adoption of Agile is impeded by its very nature in large organizations and Kanban is the proposed answer, then the Agile solution is insufficient. Agile needs to expand its scope to be relevant and useful for non-developers as well as across development teams.

To understand how Lean applies to Agile development, I’m going to take a short detour though history.

Mapping manufacturing principles to software development is an interesting cross-pollination of ideas. Discrete manufacturing is quite different from application development, but that doesn’t mean the software industry can’t learn a thing or two from a different sector.

Lean was born out of a need to re-invent the manufacturing industry, which had not really evolved since the inventions of Henry Ford and the production line. From Ford’s time to the post second world war period, most manufacturing was very good at making enormous quantities of the same product, regardless of the demand. Ford’s famous quote about color clearly exemplified the thinking of the day: “Any customer can have a car painted any colour that he wants so long as it is black”. In other words, Ford’s production line was optimized for manufacturing, not profit, and turned out to be quite inflexible when market conditions changed.

In the 1950s, Sakichi Toyoda made a revolutionary leap forward with two principles:

1. Pull vs. push – at any point in the production process, the trigger to start work on a production unit is governed by its upstream neighbor.  As an example, I do not start my work on a product unit until the guy following me says he will be able to receive it.
2. Efficient manufacturing depends on the management of three key inefficiencies: overburden (muri), inconsistency (mura), and eliminating waste (muda).

Together, these elements formed the underlying principles that Sakichi spearheaded into what is now known as The Toyota Production System (TPS). The TPS has subsequently been used as the the basis for Western derivatives such as Just-In-Time, value-stream mapping, Six Sigma and Lean, to name a few.

So what does this have to do with Agile and large organizations?

There are well-documented cases where agile alone was not enough, and that’s where Lean/TPS can add value.  For the most part though, the application of Lean principles has been limited to just one part: Kanban.

The TPS Kanban methodology has two aspects. First,  a Kanban card is attached to every unit under production and carries contextual information (metadata) about the tasks that need to be performed on that unit  and second, task readiness and data are used to trigger an specific action (work).

Over the past decade, the Agile methodology has been used successfully within  development teams, usually sized between  8 and 15 people. Agile’s benefits and values for this type of environment have been well articulated by many others (including on this blog), but most Agile adopters may not have realized the close mapping to Lean/TPS.

• Muri (overburden) – overproduction – in an Agile context, this is usually expressed as over-planning
• Mura (inconsistency) - elimination of bugs at the earliest stages, resulting in more  stable and reliable iterations
• Muda (waste) – close interaction with the customer to absorb change and prevent wasted iterations
• Kaizen (continuous improvement) – refactoring, unit testing, system integration

Secondly and more importantly for large teams, the TPS/Lean idea of pull vs. push is key. But there are other aspects of Lean/TPS that would benefit software development, Kanban being an important one but not the only one.

In an Agile context, Kanban is usually expressed as a board or wall with movable index cards to visualize units of customer value and work flow. This is where I think the rails have come off Agile/Kanban compared to the original TPS philosophy.  Kanban is just one gear in the whole TPS methodology.  Its an integral part but no more important than the other parts.  To function optimally, the TPS/Lean requires all the piece to be implemented not just one.

The other aspects of TPS/Lean are:

• Andon (signage, early warning)  - literally means paper lantern and is used to call attention to a problem in the process.  For Agile, it should be express as how do you measure your team’s progress and convey that information to the whole organization.
• Jidoka (autonomation) – automation with human intelligence.  The efficient use of tools like static analysis and continuous build to aid in development.
• Poka-yoke (fail-safing) – not just exception handling, but actual prevention of faults and counter-measure strategies to prevent the fault from reoccurring.

These other parts of the TPS were not born because people like more processes and rules; they came out of need, something the agile methodology has yet to realize it requires.

Part IV – Joy is in the eye of the beholder

In preceding posts on this topic, I’ve outlined the continuing shift from in-person, physical interactions as being the defining notion of both social and business contexts, towards virtual interactions and marketplaces, and the fact that in all aspects except the most personal the latter can fulfill everything expected of the former. But what does all this have to do with engendering a vibrant and successful code review practice within a development organization? On the face of it, nothing much. Code review, you could determine, tends to happen within organizations that enforce it, so all we need to do is to pass a rule requiring code review before shipment and we’re gold, right? I mean, what could go wrong?

Unfortunately the reality isn’t so clear cut. Most organizations worth their salt have a “requirement” for code review to be performed. At least on the important bits (a definition I particularly like, particularly when it comes to motivating programmers working on the “unimportant” bits). Or the hard bits (likewise). One awesome process description I saw in action recently called for the architects in the team to review each others’ code, but everybody else got a free ride – because obviously the code that our highest paid, most talented developers produce is the stuff we’re really worried about being wrong, amirite?

Despite such requirements, whether they make sense or not, the rarity of code review is out there for all to see. In fact, we recently sponsored the analysts at Forrester to produce a survey of code review practice in various different development environments (embedded, ISV, IT, etc.) and found that although most developers appear to believe that they live in organizations that do code review, most also claim that it doesn’t happen consistently for some reason or other.

Let’s take the most prevalent excuse claimed: too busy, got other stuff to do. Why is this claimed so uniformly? Certainly developers are busy people – we pay them a lot and expect a lot for that remuneration, after all, so sure they’ve got other stuff to do. But if we put this in another context, perhaps outside of the development process, say visiting Grandma, I’m sure there’s a parallel to be drawn. After all, we should definitely go visit Grandma more often. She’s probably not going to be around that long. She’s a nice lady, and she cooks cute little cupcakes when we do go there. So why not do it more often? Too busy, got other stuff to do. And, of course, it’s really annoying to drop whatever you are legitimately doing (whether development or otherwise) and go visit the old lady with the bad habit of talking about you as if you weren’t in the room.

As a manager, therefore, your task in ensuring code review has a much lower friction profile to it, is to remove what I think I’ll try to trademark as The Grandma Effect. If I have to stop what I’m doing, put on my Sunday best, and sit listening to stories from three decades past, I’m going to find all kinds of reasons not to. But if instead (to stretch the analogy to what I’m sure is the breaking point), Grandma were to learn how to play a wtf-pwning Mutilate-spec’d Rogue, then my interactions with her become much more palatable and manageable. Replace the trip to Grandma’s house with the annoyance of setting up a formal code review and you’ll get the picture.

There’s a tipping point here, and it revolves around the place and relevance of social tooling within your development team. You, your peers and your reports are currently interacting with friends over MSN, Twitter, Facebook, you name it. They’re booking dates, arranging weekend schedules, and getting the latest news from Reddit. You name it, they’re doing it and it’s all coming to them through a few pretty simple to leverage mechanisms.

Replace their social and common knowledge-gathering activities with knowledge leveraged within the code base, and it’s pretty easy to see how you could graft what has the potential to be a very annoying activity, e.g. code review, onto a natural way of conducting business. Instead of creating an entire workflow around the invite, simply inform interested parties about commits and let them decide whether to review or not. Instead of insisting on top-down imposition, encourage a bottom-up adoption simply through ubiquitous availability of information.

Nobody forces anybody to use a service like Reddit, after all. It exists and thrives because of the community that finds value in its presence. Personally I interact with it through RSS as I find that the most natural way of learning what it’s got to say. Lots of different feeds of information for the different types of news, all presented through a common aggregation mechanism that feels natural, that works well, and that I don’t have to think about.

So, if commits that my team members are making, or commits that others are making to a component for which I feel either moral or actual responsibility are available through that same mechanism, I’m going to take advantage of the tools to review those commits and to make my presence felt.

No formal review.

No formal sign-off.

But also a guarantee of way more participation, and what’s more, a broad reach around the typical chain-of-command style code reviews that we know and hate, instead engaging atypical contributors, not to mention the legion of lurkers just out to learn more. And isn’t that what it’s all about at the end of the day?

In summary: don’t require the architect, but appreciate their presence. And instead of bringing the people to the code, bring the code to the people.