Over the last few weeks, we’ve been working on cleaning up our product backlog, and the one thing that I always find to be the most challenging is the prioritization. I’ve worked on a number of different products over the years, with a number of different teams, and have used a number of different methods to prioritize a backlog, and I thought I would take a few minutes to share them with you now.

1)    Feature rating – This is the method where Excel really shines, because you get to create a cool matrix consisting of your features and a number of categories that are important (such as revenue generation, importance to existing customers, competitive differentiation, and so on). You then put a rating (usually 1-5, with 5 being the most important) in each of these categories for each feature…add them up, and the feature with the highest total becomes “most important feature #1”. Now, you can make this even more interesting by giving a weight to each category, to show which categories are the most important. So, for example if revenue generation is the most important factor in your company, then give that category a .5. Give your remaining categories a weighting, ensuring that the total weighting equals 1. This method can give very clear results, but can tend to be biased (I really like this feature, so I will give it a higher rating than this feature I don’t really like…)

2)    Baseline prioritization – In this method you pick a feature that you feel should be in a release, then compare all other features to this one. If any of the features are more important than that initial feature, then they become a higher priority, and conversely, those that are less important become a lower priority. I have found this method okay, but difficult to get a really good prioritized list…yes you get a list of the most important features, but not really fully prioritized list saying feature 1 is the most important, followed by feature 2, etc.

3)    Gut feel – This method is certainly the least scientific of the bunch, but it still relies on instinct, and can work well if you have a really good grasp of the market and your customer base. In the times I’ve done this, I’ve also tried to take the concepts of method 1 above, and apply that… so for example, Feature A has 6 different customers crying for this, will definitely generate more revenue, and further differentiates our product from our competition…sounds like our top feature for this release! This is the method that I find the most effective because you get to apply a number of concepts from a number of different methods.

There are a number of other methods out there that I haven’t used, so I would be interested to hear what experiences you have had with them, or with the ones I’ve listed above.

Testing can have many goals: to assess quality, to assess conformance to specification, to help managers decide whether or not to ship, etc. These all affect a test team’s approach.  Very often, it seems that “Finding Important Bugs Early” is the tester’s primary goal during a development cycle in an agile environment. It’s the ideal: find a quality impacting issue as quickly as possible after it was introduced so that the code is fresh in developer’s minds. “Finding Important Bugs Early” is a goal shared by countless test teams.

It struck me as odd, then, that James Whittaker of JW on Test in his excellent “The Future of Software Testing” series (Part 4), stated that in the future, we need to “close the gap between when a bug is created and when that same bug is detected.” In my opinion, why wait? There are plenty of things we can do to close that gap right now!

The most efficient tactics I’ve seen are the following: early involvement of testers in the development process, good unit testing, code inspection, static code analysis, continuous integration, and effective use of stubs.

Early involvement of testers is not a new concept; it’s already a part of good test process practices. For example, it is a key area of TPI (Test Process Improvement) known as the “moment of involvement”. As a bonus, I’ve always found that testers make great requirement appraisers; not only can they help assess that a requirement is verifiable, but also that it is complete, consistent and unambiguous.  Keeping testers in the loop is especially important in an agile setting where it can help make the documentation barely sufficient, as recommended by this type of development process.

We all know that unit tests are your “first line of defence” for testing executable code. To the developers to whom this task most often falls, I encourage you to ask your test team for help should the need arise. A good test team member has a way of seeing testing from a unique perspective. Use this hidden resource! You can pay the favour back when it’s time for them to use a new automation framework. It’s worth it: a good unit test suite can find many problems early that can be hard to diagnose later downstream.

Code inspection is another early detection tool. Used effectively, it can find logic and design problems at the source, not as a symptom (i.e. when system testing will stumble upon the issue). Moreover, there is no prerequisite of compilation to start it! The same is true for static code analysis. We all know the benefits so apply early, apply often, and save a lot of trouble later on.

For environments that are not of that kind where you “throw the product over the wall” to be tested, continuous integration is necessary. Otherwise, your process may be hindering the test team’s effort to Find Bugs Early. Should your testers be validating last week’s code? Last month’s? If they are, they have no chance of finding your most immediate problems. Don’t forget about bug interactions. It frequently happens that a bug is hidden behind another bug; fix one and possibly expose a slew of others. I heard a question asked once that truly captures the idea: “Why is it that when a bug dies, all the others come to its funeral?” You won’t know who’s in attendance if you’re not testing the latest code.

You may recall that I mentioned the effective use of stubs at the beginning of this post. I hope it piqued your curiosity. It’s something I learned by observing effective testers over the years. When a widget comes off the conveyor belt, they don’t wait for the thingamajig it screws into to start testing. The thingamajig may not be ready for another week! They instead whip up a mock-up of the thingamajig that works somewhat like the real thingamajig and try out their widget, pronto! Very often, the combination doesn’t work as expected and they must ask themselves: are their assumptions about the thingamajig wrong or is it the widget itself? It’s a good thing they didn’t wait until next week to discover this potential problem! This type of experiment enables early integration testing: the component didn’t work correctly in its intended environment. This prompts an investigation into the root cause: is it a bug in the component or a poorly defined interface between two components? Beware however, software stubs can fall into the same trap as a test oracle: you can make it as complex as the real thing if you’re not careful.

These are but a few of the tools for Finding Important Bugs Early. There are many other testing approaches that are not addressed here (so don’t throw away your regression tests!) This post focussed on the “early” part of the goal. What about tactics for the “important” part? Good question! I might be coerced into a follow up post one day…