Our dilemma: How do we remove the barriers to knowledge about Klocwork’s toolset and developer best practices for creating high-quality code?

The answer: Klocwork Developer Network–a new online portal designed for learning, sharing and discussing all things source code analysis. We have had a lot of fun and a few sleepless nights as we assembled industry knowledge, online forums, computer-based training, best practices from industry experts, and lots of reference and learning resources.

A significant portion of the content on the Developer Network is open for public consumption. By registering and logging in, you get additional videos, demos, CBT and more.

We have a lot of fresh content to add to the site in the upcoming weeks and months, and we want to hear from you about what you would like to see. Why not register now at developer.klocwork.com? Then tell other Klocwork users about the portal too.

Visit Klocwork’s Developer Network at developer.klocwork.com.

Already a my.klocwork.com user? Access the Klocwork Developer Network using your existing my.klocwork.com login. (But note that my.klocwork.com remains the place to go for support tickets and for FTP access to the latest software releases.)

While I may not be the most active Twitter-er in the world, the one thing I have noticed is that there is an awful lot of activity around the term “code review” lately. Since code reviews have become a widely used practice, I thought I would share one of my experiences about code reviews with you, from a product manager perspective.

In my first Agile team, many years ago, it was tabled (in our retrospective meeting after a couple of Sprints) that code reviews should be added to our definition of “Done”.  Let’s just say my initial response was less than enthusiastic… but why was that?  Well, in my opinion (perhaps uneducated on this topic), doing code reviews seems to add more to the time it takes to finish stories, so that means less stories are getting done per iteration, which potentially means longer release times, or releases with less functionality than hoped for. This is not something a Product Manager is usually receptive to. After some debate, we put it to a vote where the “yays” defeated the “nays” by a fairly healthy margin (okay, it missed being unanimous by one vote).  So we updated our “Done” criteria and moved into our next Sprint.

Our next couple of sprints went off similar to our earlier sprints, I didn’t really notice any differences. We seemed to have about the same number of stories being started and completed, and I for one was mildly surprised that we were able to maintain the same velocity, even with the extra process of doing code reviews for each story. Curious, I decided to talk to one of the more senior developers about what was going on. He walked me over to our Scrum board and asked me if anything looked different. Nothing jumped out at me initially, until he pointed out that the number of ‘bug’ cards (the dreaded red cards) were significantly less than in those early iterations. He proceeded to tell me that the code reviews were playing a major role in this. Developers were finding things early and fixing them before passing the code onto the testers, leaving the testers to focus on testing the actual features …crazy, I know.

It really appeared as though the code reviews were producing better code, without actually slowing down the development process. My opinions of code reviews did a complete 180…now they were helping to contribute to better quality code that I could show our customers, without having to sacrifice anything in the way of release delays or velocity degradation. I had become a believer!

 I think I have something to Twitter about now…

Most code reviews are done in-person, 60%  according to data  from a Forrester Consulting study commissioned by Klocwork.  So how do you accommodate remote sites, out-of-office employees  or off-shore development shops?

Most software developer teams will face some form of remote development challenge during their careers or product cycles.  As demonstrated from the data above, the breakdown of remote need is as follows:

• 76% use some form of outsourcing,
• 64% have some developers  located outside of the main campus,
• 40% of reviews are conducted with remote participants.

You can’t let development come to a grinding halt simply because a critical team member is not physically available at the scheduled time or location.  For most organizations, code reviews need to be performed and employee travel is not the solution for cost and timing reasons.  This has driven the adoption of lightweight review processes and new tools that support it.

Klocwork built a code review tool for this express purpose.  Other ones exist like Code Collaborator and the open source Review Board .  How do you support your remote code reviews?  Email?  Wiki? Or a purpose-built tool like one of the ones mentioned?

There has been lots of discussion on this blog (and others for that matter) on the importance of early defect detection, refactoring, and code reviews, but what does it all mean to a team of developers trying to maximize their velocity in a 2 week iteration? Based on a number of studies, and some real-world customer feedback  we have put together the following ROI…but note that this ROI is not measured in dollars, but rather in hours saved, because a development team can more easily relate to a 20 hour time savings per iteration rather than a break even point of 14.5 months. A few assumptions first…the team is made up of 10 developers, working on 5 stories (each story creates about 300 LOC) every 2 week iteration. Also, we used internal estimates for the refactoring time savings since we couldn’t find any 3^rd party data on refactoring ROI. . If you have anything more concrete, I’d love to hear about it.

From this table (which has been a regular slide in our Agile in Action roadshow series) we see that tools can help, in this example just over 40 hours/iteration, which if you break that down further works out to about 1/2 day per developer every 2 weeks. Now that is an ROI that an agile development team can relate to…

It’s often taken as read that developers think code reviews are just a pain in the behind. Maybe that sentiment is true when a developer’s sitting amongst his/her peers and getting interrogated on the quality of their code, but some of the data from a Forrester Consulting study commissioned by Klocwork seems to contradict that a bit. The survey asked software development professionals a whole bunch of questions related to code reviews (some of which we’ve referenced before) and here are two interesting data points that suggest developers see real benefits from code reviews.

So 79% of respondents indicate that, yes, code reviews have been effective at reducing the number of bugs found later in the development cycle. Furthermore, 43% state that code reviews have caused a fundamentally positive shift in their project’s direction. Cool.

Of course, in other parts of the survey, respondents complain about aspects of code review, in particular how time consuming and difficult they can be to implement consistently. Nonetheless, the data indicates that when organizations put their heads down and make them part of their development process, real benefits will be realized. So, the challenge is making them part of the process – of course we advocate a tools-based approach, making them more lightweight, and combining automation into your software verification strategy so that manual reviews aren’t the only technique being used to find implementation errors.

This data line-up with what you’re seeing within your organization?

Just returned from my second stint on the Agile in Action roadshow with our friends from Electric Cloud, Perforce, and VersionOne, this time visiting the cities of Toronto, Philadelphia and Chicago. Rather than going into minute detail (and the fact it is a Friday afternoon before a long weekend), I thought I would share a few random observations from this trip:

• Organizations (and individuals) are begging for as much information and guidance as they can get on Agile and tools for Agile, and are willing to give up a days in the office and brave horrific traffic to get it
• Teams that are 6 to 9 months practicing Agile think they’re novices, but in reality are seasoned veterans and have lived through most of the nightmares newer teams are currently facing
• Toronto cab drivers have a random-number generator for their “flat-rate” fares from the airport
• The majority of our audience would rank low to medium on both their knowledge and their adoption of Agile…they all want to go Agile, they just don’t know where to start (or if they were started, how they could improve things)
• Window seats suck, but not as much as middle seats
• Developers do code reviews, but don’t like doing them…
• …but you could always count of the one guy in the audience who claimed to like them…obviously someone’s living in denial
• And finally, if you are in 3 different hotels in 3 nights, keep the sleeve your room key comes in on you at all times…I guarantee you’ll forget your room number at least once during the trip.

What is the ROI to the company for developer tools?  This has always been and continues to be a struggle in any development organization. Developer productivity to date has been very poorly measured and studied. Programming has always been a creative task bound within the constraints of  a framework.  Many people have tried to measure direct individual developer productivity with less than convincing results:

Bugs per dev, bugs per team, bug regression rates, bug trend lines, comparative .NET Framework book sales rates,  # of [static analysis] violations per kloc, etc, etc.. the list is really endless…

So when it comes to assessing the value of development tools, what are the motivational and productivity drivers?  During my time as a development manager, my focus was always on reducing roadblocks and mindless repetitive tasks.  When you are well compensating developers to use their grey matter, you don’t want their efforts to be wasted on the simple stuff.

Focusing on a bottom-up approach, I base the tool choices on the marginal cost for adding a developer.  After getting the basics out of the way, including corporate overhead, dev box, standard  software load and a complier, what’s next?

Well, there are a plethora of vendors all screaming at you to buy their product, but let’s take a step back and look at what makes a developer produce better code.

Here is my list in order and why.  It starts at the developer desktop and depends on frequency of use and then spreads out to the rest of the development team:

• Source control (with nightly backups) – Even for a development team of one, don’t leave home without it.  The benefits of source control really can’t be overstated: revision control, easy diffs, build integration, product branching  and maintenance, etc.

• static analysis – like grammar checking for Office but so much deeper. It has been proven time and time again that getting rid of the minor flaws and style inconsistencies right when the developer is working and the code is fresh in their brain is the most productive.  Waiting 2 hours, a  day and for some even a week costs a context switch.  When you add in the regression analysis for the maintainability, this tool just shines and pays for itself quickly

• Issue tracking – some might debate that this tool’s placement should be above static analysis, but as I said,  my focus here is on the developer’s desktop and not team or management needs.

• refactoring – only sightly better than find/search/replace - not essential but when used frequently and consistently can be used for good productivity benefits .  It’s one of the small tools that get you through the day.

• code review – similar to static analysis but now we have widened our view off of the desktop and include the team to build better code. No one person can know everything and extra pairs of eyes on the code is only going to make it better.

• unit testing – anything to take away the drudgery of building simple test cases that completely cover the class/api interfaces and maintaining with the refactoring that will happen over time.  You might ask why unit testing is lower on the list. Simple. You can’t test code that has not been written yet.

• dynamic/performance profiling and input injection – your customer will like you for this one.  Over time, most applications grow, add new features and become sluggish pigs.  Version performance testing and making critical judgments in trading off performance for value is one of the  keys to repeat customers.

There you have it.  A list of tools for developers (note I said developers not PMs or teams) that I consider essential to have on each desktop to get them through their day.

Part IV – Joy is in the eye of the beholder

In preceding posts on this topic, I’ve outlined the continuing shift from in-person, physical interactions as being the defining notion of both social and business contexts, towards virtual interactions and marketplaces, and the fact that in all aspects except the most personal the latter can fulfill everything expected of the former. But what does all this have to do with engendering a vibrant and successful code review practice within a development organization? On the face of it, nothing much. Code review, you could determine, tends to happen within organizations that enforce it, so all we need to do is to pass a rule requiring code review before shipment and we’re gold, right? I mean, what could go wrong?

Unfortunately the reality isn’t so clear cut. Most organizations worth their salt have a “requirement” for code review to be performed. At least on the important bits (a definition I particularly like, particularly when it comes to motivating programmers working on the “unimportant” bits). Or the hard bits (likewise). One awesome process description I saw in action recently called for the architects in the team to review each others’ code, but everybody else got a free ride – because obviously the code that our highest paid, most talented developers produce is the stuff we’re really worried about being wrong, amirite?

Despite such requirements, whether they make sense or not, the rarity of code review is out there for all to see. In fact, we recently sponsored the analysts at Forrester to produce a survey of code review practice in various different development environments (embedded, ISV, IT, etc.) and found that although most developers appear to believe that they live in organizations that do code review, most also claim that it doesn’t happen consistently for some reason or other.

Let’s take the most prevalent excuse claimed: too busy, got other stuff to do. Why is this claimed so uniformly? Certainly developers are busy people – we pay them a lot and expect a lot for that remuneration, after all, so sure they’ve got other stuff to do. But if we put this in another context, perhaps outside of the development process, say visiting Grandma, I’m sure there’s a parallel to be drawn. After all, we should definitely go visit Grandma more often. She’s probably not going to be around that long. She’s a nice lady, and she cooks cute little cupcakes when we do go there. So why not do it more often? Too busy, got other stuff to do. And, of course, it’s really annoying to drop whatever you are legitimately doing (whether development or otherwise) and go visit the old lady with the bad habit of talking about you as if you weren’t in the room.

As a manager, therefore, your task in ensuring code review has a much lower friction profile to it, is to remove what I think I’ll try to trademark as The Grandma Effect. If I have to stop what I’m doing, put on my Sunday best, and sit listening to stories from three decades past, I’m going to find all kinds of reasons not to. But if instead (to stretch the analogy to what I’m sure is the breaking point), Grandma were to learn how to play a wtf-pwning Mutilate-spec’d Rogue, then my interactions with her become much more palatable and manageable. Replace the trip to Grandma’s house with the annoyance of setting up a formal code review and you’ll get the picture.

There’s a tipping point here, and it revolves around the place and relevance of social tooling within your development team. You, your peers and your reports are currently interacting with friends over MSN, Twitter, Facebook, you name it. They’re booking dates, arranging weekend schedules, and getting the latest news from Reddit. You name it, they’re doing it and it’s all coming to them through a few pretty simple to leverage mechanisms.

Replace their social and common knowledge-gathering activities with knowledge leveraged within the code base, and it’s pretty easy to see how you could graft what has the potential to be a very annoying activity, e.g. code review, onto a natural way of conducting business. Instead of creating an entire workflow around the invite, simply inform interested parties about commits and let them decide whether to review or not. Instead of insisting on top-down imposition, encourage a bottom-up adoption simply through ubiquitous availability of information.

Nobody forces anybody to use a service like Reddit, after all. It exists and thrives because of the community that finds value in its presence. Personally I interact with it through RSS as I find that the most natural way of learning what it’s got to say. Lots of different feeds of information for the different types of news, all presented through a common aggregation mechanism that feels natural, that works well, and that I don’t have to think about.

So, if commits that my team members are making, or commits that others are making to a component for which I feel either moral or actual responsibility are available through that same mechanism, I’m going to take advantage of the tools to review those commits and to make my presence felt.

No formal review.

No formal sign-off.

But also a guarantee of way more participation, and what’s more, a broad reach around the typical chain-of-command style code reviews that we know and hate, instead engaging atypical contributors, not to mention the legion of lurkers just out to learn more. And isn’t that what it’s all about at the end of the day?

In summary: don’t require the architect, but appreciate their presence. And instead of bringing the people to the code, bring the code to the people.

The importance of code reviews has already been well covered by lots of smart people like Jack Ganssle and Jason Cohen. Recently, the subject has become more important around here, so we want to offer our take. In particular, we’re looking at the best way(s) to incorporate code reviews into an overall software verification strategy and how automated tools (such as static analysis, no shock there) can help unleash the benefits of peer code review. More on that angle another time, first the bigger picture.

Klocwork recently commissioned a survey conducted by Forrester research on this whole topic and the results are pretty interesting. While there’s a whole bunch of data that can’t be covered in a single blog post, a general theme we found is that developers see the value of code reviews, they’re often mandatory, but the process itself seems to be ad-hoc and quite ‘behind the times’. Here’s an example of what I mean:

Code Reviews - Mandatory but Ad-Hoc

So, code reviews are mandatory but you can kinda invite whoever you want to review the code. Shouldn’t who reviews the code be pretty important? (Hint: Yes)

We’re gonna keep talking about different aspects of this important development milestone, so stay tuned and we’d be interested to hear anything you have to say on the topic.

Part III – Joy is All Around Us

When you think of a social activity, what do you think of? Perhaps a rave? Or maybe a quiet bridge foursome is more your style? Or even a Matrix-style meet-and-greet complete with latex and contortionists? Ahem…

Or maybe you’ve finally let go of this old-world requirement to actually be in the presence of an individual to enjoy a social encounter with them, and instead have embraced the reality of the 21^st century, that society and social interactions no longer require physical presence, and instead surround us every day, at every minute, as long as we (virtually) get out there and find them. Speaking as a long-time online gamer, I have a circle of folks I consider friends, with whom I talk most evenings, with whom I’ve spent quality time learning and beating goal-based activities, yet none of whom I’ve ever met. And whilst their reaction to some family tragedy on my part may result in no more than a weak “dude, that blows…” on some forum or other, in every other aspect of social interplay, they fulfill exactly the same role as those few- and far-between actual, you know, friends that each of us cling to throughout life.

According to a study on the topic conducted earlier this decade, friendship is becoming something of a luxury for the average American adult. Rather than expanding our circle of friends as travel has become more reachable for the masses, we’ve instead decreased that circle from an average of 3 to just above 1. So are we all just becoming obnoxious, introverted, “bah humbug!” Ebenezer Scrooge wannabes? Perhaps, and certainly that’s the trite response to the statistics for people in search of a quick buzzword or appliance to blame.

But perhaps instead of this reflecting a net diminution of our quality of life, we’re simply replacing much of what was considered necessary in previous generations (beer with the boys, poker night, ice fishing trips, whatever floats your boat) with a more constant, more consistent, but at the same time more arms length notion of friendship and social interaction. Though different, it fulfills everything we need in terms of communication and support, but leaves us free to concentrate on our family lives, or personal hobbies, or whatever else makes us happy to be, well, us.

Friendship when we want it, on our terms, and only then.

One potential projection of all of this can be found in the ongoing trending of the social nexus of life, business and relationships towards the online marketplaces that have sprung up around activity-, or focus-based requirements (I referred to this in my first post on this topic, drawing the correlation between Facebook and dating, LinkedIn and prospecting, etc.).

Find a marketplace, find a life (or maybe, a Second Life) – and frankly, is that really any different from the actual bricks-and-mortar reality of the rat-infested, smelly locales of the distant past (minus, you know, the scary crone shouting on the street corner, and the propensity for picking up the Black Death at a moment’s notice…)?

Indeed, my Chief Architect likes to describe an attendee at a recent conference as saying something like, “But what should we do about all these old people who can only e-mail or even worse need to use the phone? I mean, how am I supposed to communicate with somebody who doesn’t have a Facebook account, or doesn’t keep up with Twitter?” Note that this wasn’t a casual conversation over a beer, but rather a key point in a presentation (presumably to a room full of people with the requisite qualifications to be able to laugh affably at such an observation).

Whether we like it or not, whether we can personally deal with our relationships migrating into the ether, that’s where they’re headed, at double-quick time. So are you the guy with a red flag making sure that cars only drive at the same speed as horses, or are you busy building a Formula 1 car in your back yard?

And actually, perhaps more importantly, whether you’re either of these, you’d better believe your staff are busy climbing onboard with everything the new paradigm has to offer, so do you really want to be left playing catch up?

At a recent customer meeting I was surprised to hear that this highly compartmentalized, classified installation was putting a social media strategy in place (they termed it “our space”) to embrace what was happening anyway, and obviously to attempt to contain it within the security mechanisms required by their business. If they can do it, with all the restrictions and fenced-off classified strictures they have to deal with, why can’t we all?

Code review, you say? Social code review, more like. The current means of accomplishing the goal is fundamentally broken and will never scale, just like the requirement to only befriend people you could physically reach out and touch. The paradigm is changing, time to keep up…

And now in a deferential nod to the awesome Douglas Adams, this trilogy of posts on code review as a social activity will be continued in part IV, coming to a blog near you soon.