Review: Source-Code Assessment Tools Kill Bugs Dead
Source-code assessment tools identify security holes resulting from shoddy development. Check out our examination of this burgeoning market and read our analysis of 12 commercial and open-source products.
By Jeff Forristal
Infosec practitioners worth their Red Bull know that perfect security is an ideal worth striving for but extremely difficult to achieve. No application of any size and complexity can be perfect in its first implementation; bugs will be present, and some will affect security.
Some software vendors seem to operate on the premise, If bugs are a fact of life, why expend time and resources to find and remove them before an application is released? Why not just issue patches and updates as holes are discovered? Seasoned software makers know that fixing bugs after the general release is more costly than finding and removing bugs during development (check out Applied Software Measurement: Assuring Productivity and Quality, by Capers Jones [McGraw-Hill, 1991] for supporting evidence).
If the financial bottom line isn't reason enough, how about this: More and more attackers are reverse-engineering application security patches to uncover vulnerabilities. It is far quicker and easier to recover technical details based on what is fixed by a security patch than it is to discover a vulnerability--Sabre Security even offers a movie showing how reverse-engineering is used in its BinDiff product to analyze the vulnerability fixed by Microsoft's MS05-025 patch. And the still-present gap between the time a vendor releases a patch and the time the patch is widely deployed means attackers have enough time for exploitation. The concept of 0-day (unknown/unpublished) vulnerabilities is becoming moot when 1-day (recently patched) vulnerabilities are just as lethal and far more research-efficient. Of course, this phenomenon is brought to us courtesy of vendors that follow the "release now, patch later" security cycle.
We understand secure application design isn't easy. A high level of programmer education and awareness are required, as are continuous security assessments and process evaluations. Depending on humans for this analysis work is risky--people make mistakes and don't scale well to handle large, complex applications; therefore an element of automation is practically a necessity. But the source-code security assessment market has been growing, with a slew of vendors offering products that support different programming languages and take different analytic approaches. We set out to determine how mature the market is, how the various tools work and how good a job they do. Which features matter, and how do the free and open-source options compare with the commercial products? This primer and the corresponding product writeups are based on our market analysis and examination of 12 source-code assessment products that focus on the security ramifications and remediation of discovered defects. (We didn't include general defect-discovery products that don't identify a defect's security implications.) We examined Compuware DevPartner SecurityChecker, Fortify Software Source Code Analysis Suite, Klocwork K7 Development Suite, Microsoft Visual Studio 2005 Team Edition, Ounce Labs Prexis, Parasoft Jtest, Secure Software CodeAssure, SPI Dynamics DevInspect & SecureObjects, as well as the open-source Cigital ITS4, Flawfinder, Secure Software RATS (Rough Auditing Tool for Security) and Splint 3.1.1 in our Portland, Ore., Neohapsis partner lab. Our features chart summarizes the offerings.
Notable vendors missing from our list include Coverity and Application Security. To let us evaluate its product, Coverity sought to impose legal restrictions that could have conflicted with us publishing our impressions of its product in this article. Application Security never responded to our e-mail messages or phone calls. We didn't consider other open-source projects, including CQual, MOPS, BOON, PScan and FindBugs, because of their niche focus or the nonproduction/beta quality of the current version.
Old Idea, New Offerings
The concept of using an automated tool to find defects in an application's source code is nothing new. Many products discover general application defects and programming policy violations. Even the notion of discovering security-specific defects isn't unique, as security defects are a specialized subset of general defects. Purists would argue that most application defects are security defects--anything that can cause an application to crash or consume an excess of resources affects availability, and thus can be vaguely classified as a denial-of-service attack.
We weren't surprised then to see some vendors taking general defect discovery products and repurposing them with a security spin. A perfect example is Parasoft's Jtest, a Java defect-discovery and -testing tool that offers a subset of security-specific analysis rules. On the other end of the spectrum are commercial products designed to tackle the challenges of security analysis. Products from Ounce Labs and Fortify Software, among others, exist for the sole purpose of performing application security assessments.
The origins of these products also vary widely. Coverity's Prevent was spawned from Stanford University's xgcc/Metal research, and Splint hails from the University of Virginia. Microsoft gained Prefast's predecessor through its 1999 acquisition of Intrinsa, and Klocwork was spun out of Nortel Networks. The collective driving motivator is the desire to discover, quantify and remove security risks in an application at as early a stage as possible.
It's not just developers who can benefit from source-code security assessments. All the tools we examined assume (exclusively, in some cases) a developer audience--which makes perfect sense because developers are writing the insecure code--but QA managers and engineering VPs who oversee the quality aspect of the application-development process will find these tools provide valuable insight. Project leads and development liaisons can analyze code created by outsource development firms to judge their overall attentiveness to and understanding of security. Global security teams and CSOs can discover what general security problems are likely to exist before deploying a software package. Application security auditors can use these tools to find security bugs faster and more effectively. Even tech-savvy legal departments can leverage security analyzers to determine the risk and liability presented by a collection of source code.
Software Development Lifecycle
The SDLC (software development lifecycle) has been a hot topic for years, but it seems to gain new momentum with every "breakthrough" process to address quality and security issues. For example, frequent discoveries of security defects and public concerns about Microsoft's software security quality led the company to enact a "security push" in 2002; it redefined its development processes to better account for security in the SDLC. The general goal of such a security push is twofold: Ingrain security principles and risk reduction procedures into daily SDLC processes, and stop treating security as an after-the-fact (as in, after-the-application-is-done-being-developed) concern. In fact, Microsoft's security push was so successful, it's been pushed right out to customers. Microsoft's latest Visual Studio offering (Visual Studio 2005 Team Edition) features security analysis tools embedded into the IDE and build process, with the hope of making security-defect discovery and remediation ubiquitous in the life of a developer.
The obvious sweet spot of SDLC integration is embedded security analysis in the IDE on a developer's desktop. Common IDEs supported include Microsoft Visual Studio and the open-source Eclipse, as well as several commercial products, such as WebSphere Studio Application Developer and Rational Developer suite, that use Eclipse under the hood. Adding source-code security-assessment capabilities to the list of available developer tools should encourage developers to analyze source-code additions continually. This practice lets developers discover flaws immediately, prompting fixes before buggy code is propagated to the central repository. Visual Studio 2005 can even prevent developers from checking their code into the central repository if a local security analysis discovers defects; a piece of source code isn't considered finished and usable until a security analyzer gives it a passing grade.
But not all source-code analyzers destined for the developer workstation are integrated into an existing IDE. Secure Software and Parasoft offer standalone versions of their products, meaning they throw in a free, full installation of Eclipse to host their Eclipse plug-ins. Fortify offers a nondevelopment interface for auditors. Ounce Labs' Prexis doesn't actually integrate into IDEs; rather, it offers completely proprietary GUIs. The open-source RATS, ITS4, FlawFinder and Splint eschew GUIs in favor of CLIs.
A lack of IDE integration means developers must invoke a separate tool each time they want to analyze their code. It also means analysis findings don't get repopulated with the native defect management/task list found in most IDEs; developers are left with the overhead of moving analysis findings back into the IDE.
Centralized build systems offer an attractive alternative to running source-code analyses on individual workstations. The concept is simple: Embed source-code analysis into the central build process, and have it report to a central location that can be accessed by everyone. This makes the analysis process transparent to developers, centralizes findings and requires fewer software deployments. However, it also means vulnerable code is committed to the central repository before it is discovered. Nearly all the products we appraised can be integrated into central build processes. This is most often implemented by a CLI for direct analysis. The products from Fortify, Klocwork, Microsoft, Ounce Labs, Parasoft and Secure Software offer central reporting components, commonly implemented as an HTTP portal of some sort.
Under the Hood
We found the sheer number of technical approaches to performing source-code security assessments a bit overwhelming. We got even more confused when we looked at the names vendors give their analysis methods. Ounce Labs' Prexis uses the company's "patent-pending contextual analysis process." Fortify's Source Code Analysis Suite touts a four-element tag team, including data flow, control flow, semantic and configuration analyses. Compuware's DevPartner SecurityChecker uses a three-phase approach consisting of compile-time, run-time and integrity analyses. Splint uses an annotation-assisted static analysis engine. Many simply define their analysis as "static." Some just don't say, occasionally citing trade secret and intellectual property concerns.
We smell marketing. Some subtle technical wizardry does confer slight advantages to a few analysis types, but overall, we consider these details irrelevant for the average user. Still, to understand the main points of what vendors are selling and how the products differ, it helps to be familiar with some of the more common terms and concepts, many of which come from compiler design and formal language theory.
The most generic and encompassing analysis type is called "static" or "compile-time." Static analysis can mean slightly different things depending on whom you ask, but everyone seems to agree on one basic principle: A static analysis occurs against a nonrunning application. It's an as-is analysis of the application's source code or binaries (often referred to as "object code") without executing the application. Once you start executing an application to interact with and analyze it, you fall into the other broad category, run-time analysis. Unit testing and real-time debugging are common examples of run-time analyses. With a little bit of generalization, most analysis types fall into the static or run-time categories.
You can perform the crudest type of static analysis using basic word-search techniques. If you're concerned about insecure use of the printf() function in your C application, say, take your favorite text editor and find all occurrences of the word printf among your source-code files. This process is often referred to as grepping, after the venerable Unix file-search command grep.
Although this approach seems easy and effective, it isn't. Generic text searching is not aware of programming language syntax, for example, and thus will report every occurrence of the target search term in the application, whether it's used as a function name, in a variable name or mentioned in a comment. Even worse, you're likely to find all uses of sprintf, vsprintf, fprintf and more because they contain printf.
After wasting time figuring out which occurrences are actual uses of the function printf(), you then must determine manually if the uses make your application vulnerable--most functions with known security problems can be used legitimately and safely, so their mere presence doesn't indicate a vulnerability. The person doing the analysis must know how the function is vulnerable in the first place and often must trace program execution up to that point to see what state the application is in.
Obviously, basic searching for known vulnerabilities doesn't scale well and requires significant security and programming chops. To truly analyze a program's source code, the analyzer must understand and follow the target programming language's syntax. The bare minimum is to read the source code word for word and determine the comments, function calls and variable names. This process encompasses multiple abstract stages--compiler geeks call them lexical, syntax and semantic analysis. The key stage here is semantic analysis, which many vendors capitalize on to some degree in their products. If the analyzer understands the semantics of the source code, it can directly search encountered function calls for a particular exact function name, such as printf. Semantic analysis allows the tool to discard irrelevant occurrences of printf found in comments, variable names and similar function names and focus specifically on where printf() is in actual use and how it's being called. Thus, semantic analysis can be thought of as a context-aware "smart search."
ITS4, RATS and Flawfinder use this technique. They employ semantic analysis to compare all encountered function call names in the source code against an internal database list of known vulnerable function names. If an encountered function call is found to be on the list, an alert is generated. Thereafter, the shortcomings of semantic analysis are identical to simple word searching--namely, you still need to verify the function use and determine the risk manually. ITS4, RATS and Flawfinder do attempt to minimize false-positive alerts by performing checks to determine whether a function's parameters are constant, and thus safe, at compile time. However, the output from RATS and others still requires significant manual verification. The best you'll get from semantic analysis is, "This could be vulnerable under certain conditions" for every occurrence of a given function name, whether you meet those conditions or not. Although this leads to a very high false-positive rate, it rarely delivers a false negative.
Another limitation of common semantic analysis is the association of risk to specific function names. Security defects aren't always caused by the use of a function. It's quite possible for a C/C++ application, for example, to be vulnerable to a buffer overflow using a simple "for" loop and an incremental memory pointer to write beyond the bounds of an allocated memory buffer.
Still, despite its shortcomings, semantic analysis is better than nothing. Many commercial products use semantic analysis, specifically the comparison of function names against a list of known risky names, in combination with more thorough analysis approaches. We already mentioned how Fortify's Source Code Analysis Suite includes semantic analysis as one of its four analysis types. Many commercial products fall back to semantic analysis when their other results come up inconclusive.
Beyond semantic analysis, products start getting into data flow and control flow analysis--the tracking of data and execution paths through an application, respectively. Most of the commercial products we examined implement these two types of analysis in varying ways, with varying degrees of success. The better an analyzer can determine data and control flow, the fewer false positives you'll have. Some products, like Fortify's Source Code Analysis Suite and Compuware's DevPartner SecurityChecker, also include configuration analysis; this typically means the analyzer will parse and account for application deployment/environment settings found in common configuration files, such as web.config for ASP.net or web.xml for J2EE.
Fully understanding what an application is intending to do can be tricky, yet this is what these products are attempting. Microsoft's Prefast and Splint offer mechanisms to help "guide" the analyzer during its attempts to understand the application's source code. By adding particular comment annotations throughout your source code, the analysis engine can better understand the assumptions of the logic and thus report on violations. Although annotation-assisted static analysis can offer a more thorough investigation of your application, you'll incur initial overhead because developers must add appropriate annotations to the source code.
C/C++ macros are another snag when it comes to understanding an application's source code. ITS4, RATS and Flawfinder don't process macros, which can greatly affect their discovery abilities. All commercial C/C++-capable products process macros as a standard preprocessor would, but that causes a new problem: Different macro definitions could lead to different code paths. Therefore, it may be necessary to re-analyze the application with different macro definitions to ensure full analysis coverage.
Putting static analysis methods aside for a second, it's worth making a quick mention of run-time analysis methods. Compuware's DevPartner SecurityChecker, Parasoft's Jtest and SPI Dynamics' DevInspect & SecureObjects include run-time analysis in their products. Jtest uses automatic unit test generation and execution to flush out execution path issues and to validate that logic blocks perform as expected. Unit testing is a concept that's long been part of the SDLC and actively used for both regression testing and quality assurance.
DevInspect and DevPartner SecurityChecker also include a form of analysis that's virtually identical in spirit to a remote Web vulnerability scanner. This form of analysis has a few names: SecurityChecker calls it an "integrity analysis," while "fuzzing" is commonly heard in technical circles. The process is simple: The engine emulates actual attacks by submitting a long list of predetermined malicious or malformed data to the application, and the application's response is monitored to determine if the input had any adverse effects. Because the analyzer is external to the application and monitoring the application's response from an outside perspective, this type of analysis can fail to detect what's really happening within the belly of the app. For example, SecurityChecker tries to discover SQL tampering vulnerabilities in this fashion, yet a quick look into the source code may show, with much more conclusiveness, whether any input data is unsafely used in the SQL query.
Even worse, an application could fall prey to a SQL tampering attack, but proper exception handling and recovery techniques could hide any abnormal behavior from external perception. This whole approach could seem paradoxical: If the analyzer has full access to the application's innards--the source code--why does it ignore the code and try to determine what the application is doing from an outsider's perspective? SPI Dynamics says it believes attack emulation results in a more comprehensive and accurate view of the application's security, with fewer false positives, than what a straight source-code analysis could provide.
Be warned that many vendors will try to dazzle you with technical tidbits, lingo and implementation details regarding how their products perform various types of analyses. It's common to hear about intermediary language compilation, heuristic weighting and policy-driven analysis. Ultimately, the terms don't give any conclusive indicator of the overall effectiveness of the product, any more than knowing a certain automobile has specific premium types of spark plugs, brakes and tires gives you a conclusive idea of the final MPG rating. Whether it's a source-code analyzer or a car, you should take a test drive to really determine how well it works for you. Don't rely on a list of factors that may have more marketing than technical relevance.
Must-Have Features
We used our roundup of source-code analysis products to come up with a list of features worth looking for.
First, check whether the product supports the programming language(s) you support or may consider migrating to. Flawfinder is not a good choice for Java applications, and Jtest won't work on your ASP.Net code.
The purpose of these tools is to find security defects, so it makes sense to make bug-finding ability the second-biggest decision factor. The success of a source-code assessment is measured both in the thoroughness of the analysis (maximal number of actual defects) and the accuracy of those findings (minimal number of false positives). If you're knowledgeable on the root causes of security vulnerabilities and have time to manually trace and verify items marked as potentially vulnerable, you may tolerate a product that exhibits a higher false-positive rate. Otherwise, look for a tool with fewer false positives to keep from having to validate findings manually--something easier said than done.
Many of these products boast internal knowledge bases that provide supporting vulnerability description and remediation documentation, all instantly accessible and cross-referenced to discovered findings. Unless you have a thorough understanding of possible vulnerabilities within an application, you'll depend on this information to understand what the problem is, and how to fix it. Ideally, the product will provide description and remediation information specific for the exact context of the discovered vulnerability.
We've mentioned the importance of IDE integration. If you plan to have your developers incorporate security analysis into their daily routines, look for products that integrate well with your current development platforms. The tighter and closer the analysis tool is integrated into the IDE, the faster and more efficiently developers can wield it.
False-positive findings are a fact of life, much like the bugs you're looking to find. Therefore, it's important that the product have a finding-suppression mechanism to prevent the errant finding from reoccurring once you've verified it as a non-issue/false positive. Suppression should be robust enough to survive changes in the surrounding code; a naïve suppression engine may simply rely on line numbers, which change rapidly as you add or remove code during development.
Many of the tools offer ways to define additional analysis rules. This extensibility can be a crucial feature if you wish to tailor the analysis to your application. Creating your own rules lets you leverage the analysis engine to enforce internal coding policies, for example, while tweaking existing rules may help reduce false positives--or catch false negatives. You'll often need to add rules to help the analysis engine understand third-party APIs/libraries that your code may depend on. Each product has its own ways to add rules, ranging from a simple addition to an XML file (RATS) to having to create a C/C++ library utilizing the analyzer's API (Klocwork's K7). Unfortunately, not all provide a way to play with the rules; Visual Studio PreFast and Secure Software's CodeAssure rules are hard-coded modifications to the compiler, and as such, don't allow any tweaking.
If you have a large team of developers and multiple managers who need access to security-assessment findings, you'll want a centralized reporting component. Half the products have portals that provide basic dashboards, trending and overview reporting. A few add functionality, including more comprehensive reporting and full report analysis and finding verification/suppression. Determine how many people need access to data relating to the security assessment, and in what form they need that data, to decide which centralized portal features best suit your needs.
It's important to mention the issue of full development environments, or lack thereof. Some products require a development environment capable of compiling/building the application prior to performing an analysis. This includes access to all third-party/external libraries the application depends on as well as all necessary build tools. This requirement can prove problematic for security auditors, global security teams and anyone else attempting to assess source code outside of its original development environment. If you intend to use these products anywhere other than the developer's desktop or central build process, consider how much of the original build environment the product needs to perform an assessment.
Ultimately, feature priority will be determined by your role and how you intend to use the product. Developers are going to care more about IDE integration and supplemental vulnerability documentation/advice. Creators of development policy and implementation standards will look for analysis engine extensibility as a way to enforce their policies. CSOs, CTOs, QA managers and engineering VPs will make the most of dashboards, security defect trending and the myriad reports offered by centralized portals. Security auditors will likely prefer a simple analysis interface that doesn't require a full development environment.
The Essentials
Insecure coding is at the root of many a breach--you might even say it's the source of our security woes. The need to develop applications with security in mind and to remove security defects from applications before they are released is becoming greater. Not doing so means costs, both financial and intangible--what's customer confidence worth? Moreover, attackers are now reverse-engineering patches to discover vulnerabilities, then seeking out unpatched instances. Time to call in the source-code detectives.
Product Category: Source-code security assessment products
Market Data: Enterprises that don't include security as a criterion when building or buying software will incur system downtime caused by vulnerability exploits. How much? The average 5 percent of downtime in 2004 will grow to 15 percent of downtime in 2008, Gartner says.
Products Examined: Compuware DevPartner SecurityChecker, Fortify Software Source Code Analysis Suite, Klocwork K7 Development Suite, Microsoft Visual Studio 2005 Team Edition, Ounce Labs Prexis, Parasoft Jtest, Secure Software CodeAssure, SPI Dynamics DevInspect & SecureObjects as well as the open-source Cigital ITS4, FlawFinder, Secure Software RATS and Splint.
Products Not examined: Coverity and Application Security Binary Code Analyzers
Nearly every corporation uses a closed-source application somewhere in its enterprise. Closed source means the user company doesn't have access to the application's original source code and, therefore, cannot run a source-code security scan to gain insight into the types of vulnerabilities and risks the application harbors.
Fortunately, all is not lost: An emerging class of product analyzes compiled application executables/binaries. However, many products analyze the intermediate application binaries produced by Microsoft .Net and Java compilers. Whether these constitute machine-executable binary code is a bit of a debate, as the intermediate binaries typically are translated machine representations of the original code and often can be turned back into the original code (or a close approximate) with little effort.
LogicLibrary's LogiScan appliance is based on the BugScan product acquired from HBGary. We've used it and can report the analysis process is straightforward: You upload application binaries to the appliance using your Web browser, wait a bit while the analysis runs, then view the resulting HTML report of findings.
There's also the BugScam open-source project on SourceForge, whose name implies the opinion the authors have for that other binary-analysis product. BugScam is a set of plug-ins for DataRescue's Interactive Disassembler (IDA), and thus requires a working familiarity with IDA, reverse engineering and assembly code analysis. The core author of BugScam has joined Sabre Security, which plans on releasing a commercial binary analysis tool named BinAudit. Sabre Security representatives told us BinAudit is not yet in a releasable state and has no estimated delivery date.
We also stumbled on Security Innovation's BEAST (Binary Executable Analysis-Security Tester) right before press time. It seems to fill the bill for binary security-analysis products, but we're basing that purely on what we found on the company's Web site.
Product Analysis
Cigital ITS4 1.1.1 ITS4
Cigital ITS4 1.1.1 ITS4, (which is an acronym for "It's The Software, Stupid! Security Scanner") is one of the three commonly referenced open-source security code analysis tools (the other two being FlawFinder and RATS). ITS4 statically analyzes C and C++ code for a predefined list of common insecure function names. The interface is exclusively command line, letting it run on practically any OS. ITS4 offers a large set of command-line configuration options, which are useful for suppressing specific named findings or focusing on only a certain list of functions, thus tailoring your analysis to your application.
ITS4 is largely based on semantic analysis, which caused it to produce an above-average number of false positives. We were a bit dismayed to see ITS4 give us alerts about our use of printf() with a compile-time constant string (a definite false positive and a fundamental example of semantic analysis shortcomings). Vulnerability descriptions and remediation advice were never longer than a single sentence each, and the remediation advice often left us cringing--especially the pearls of wisdom, "Don't use it," and "Be careful."
ITS4 is open source and free for use except where your use of ITS4 within a commercial service offering would directly compete with Cigital's service offerings.
ITS4 1.1.1, free, with some use restrictions. Cigital.
Compuware DevPartner SecurityChecker 1.1
SecurityChecker comes as a Visual Studio 2003 plug-in for analyzing ASP.Net applications built on the .Net 1.1 framework. SecurityChecker integrates compile-time, run-time and integrity analyzers for a three-pronged approach to finding security defects. The interface is impressive and the tool easy to use, overall. However, we were a little disappointed that commonly exploited vulnerability types, including SQL tampering, cross-site scripting and command injection, were found only through integrity analysis and not through compile-time or run-time analysis. Integrity analysis requires you to deploy a fully functional version of your ASP.Net application--including all back-end database data and communication tiers--before the tool can thoroughly inspect it for some notable problems. Despite this caveat, Compuware believes this to be the best overall method for fully discovering and confirming these attack vectors.
SecurityChecker can generate a technical findings report, which we found a bit verbose but still surprisingly readable. We also found the supporting vulnerability documentation and remediation advice above average compared with the other products.
DevPartner SecurityChecker 1.1, $12,000 per concurrent user. Compuware Corp.
D. Wheeler's FlawFinder 1.26
FlawFinder is a Python script implementing static semantic checking for C and C++ applications. Because it's written in Python, it can be used anywhere a Python interpreter exists, which is nearly all platforms (many Unix platforms come with Python these days; Windows users will need to install one of the various free packages).
FlawFinder enables extra reference checks by default, which can cause a few more false positives at the expense of possibly catching some extra vulnerabilities. This is contrary to RATS and ITS4, where you have to explicitly use a command-line option to enable the same types of checks ("-r" and "-p," respectively). You can disable the checks in FlawFinder with the "-F" option.
FlawFinder correctly ignored printf calls with compile-time constant strings, which is indicative of smart checking that goes slightly above a generic semantic analysis. FlawFinder also can save the results to a "hitlist" file and use that file to display only the differences between saved and current analysis results. FlawFinder specifically understands gettext()-based internationalization, which helps further reduce false positives.
Remediation advice given in the output is on par with RATS and ITS4 ... namely, minimalist and highly technical. Also similar to RATS and ITS4 was the ability to suppress findings by adding an "ignore" comment to the code.
FlawFinder 1.26, Free, GPL. Developed by David Wheeler.
Product Analysis
Fortify Software Source Code Analysis Suite 3.1.2
Fortify Source Code Analysis (SCA) is a multicomponent suite comprising an analysis engine, various IDE plug-ins, an audit workbench, an analysis rules editor and a central Web reporting portal. The Fortify SCA Engine can process C, C++, C#, Java, JSP, PLSQL (Oracle) and TSQL (Microsoft) source code. SCA Engine is the main workhorse, performing all of the analysis work over a command-line tool and generating output in human-readable text form, which was terse, highly technical and somewhat cryptic at times, or in a proprietary format, dubbed FVDL (Fortify Vulnerability Description Language).
The command-line aspect of the SCA Engine is architected to drop into build processes with minimal effort. The Visual Studio plug-in GUI is extremely sparse, but it provides enough functionality to get the job done (fortunately the plug-in is completely revamped for the next version). Vulnerability explanation and remediation information was below average, particularly because all the advice was generalized among a vulnerability type; our insecure use of wcstombs(), strcpy() and gets()-- three functions serving widely different purposes but all suffering from a potential buffer overflow--received the same general, "What is a buffer overflow?" description and remediation advice that did not make any recommendations for our particular functions. Still, Fortify performed far above average in the bug-finding category, dodging most of our false-positive tests to boot.
Non-developers who wish to work with analysis results can use Audit WorkBench, a minimal Eclipse interface tailored for handling FVDL files generated from the SCA Engine. Audit WorkBench let us generate a basic findings overview report; for more comprehensive reporting and trending we had to import our FVDL files into the SSM. SSM provides the standard fare of dashboards, trending, reporting and policy monitoring.
Source Code Analysis Suite 3.1.2, SCA Team Suite: $16,800 annual/$29,670 perpetual, includes SCA Engine, Audit Workbench and Rules Builder for 10 developers. SCA Premium Suite: $62,000 annual/$110,400 perpetual, includes SSM and Team Suite for 50 developers. Fortify Software.
Klocwork K7 Development Suite
We were almost overwhelmed by the number of tools Klocwork supplies in its K7 Development Suite. Typically you'd start off with the desktop clients, which come in IDE plug-in and standalone versions; clients perform general security assessments and provide results for immediate use. For central builds, you would use the Klocwork Management Center (KMC) to navigate the analysis project creation and configuration process. We could launch assessments via the KMC or through build process integration. To view analysis results, we had to access the Project Central Web portal. Project Central offers detailed reporting, metrics trending and architectural analysis capabilities to help understand and manage code. A large number of CLI utilities provide the ability to integrate seamlessly into build processes, import existing project definitions, even analyze the stack overflow potential of recursive functions. We should note that we looked at the Development Suite, which has all the bells and whistles. There is a slimmer Security + Defects suite that lacks architecture analysis, metrics measurement and policy development tools.
The K7 Development Suite includes inSight Architect, an impressive tool giving full graphical visualization of our entire application architecture. We could perform logic flow charting, visual impact analysis and dependency cross-referencing with the goal of discovering and analyzing architecture problems. This lets security analysis go well beyond individual defects and see problems with the application's global design.
The K7 analysis engine is extensible via C/C++ APIs; you must implement and compile a K7 library extension just as you would any other shared application library. Needless to say, this is much more involved than the simple GUIs or text-file editing afforded by other products.
The K7 analysis engine fared average against our source code test cases; however, despite missing a few things, there was not a single false positive among any of its findings--something no other tool could claim in our tests. No false positives plus extra architecture analysis capabilities lead us to an above average analysis rating in the end.
K7 Development Suite, $3,995 per user. Klocwork. www.klocwork.com
Microsoft Visual Studio 2005
Team Edition Microsoft's newest Visual Studio offering features two separate source code security assessment tools. Microsoft embedded its previously unavailable PreFast C/C++ analyzer and a new, more powerful version of FxCop (a .Net security analyzer) into Visual Studio 2005 Team Edition.
Enabling either analyzer is as simple as checking an option box under the project build properties; from that point on Visual Studio performs a security analysis during every project (re)build and reports findings via the native IDE task list. PreFast also supports an annotation system for tailoring the analysis and enforcing execution assumptions and policies. PreFast's output resembles compiler warning and error messages, which could be hard to understand and often lacked decent supporting information.
FxCop, on the other hand, is much more configurable--we could selectively enable or disable certain checks--and offers excellent supporting vulnerability description and remediation information. Both PreFast and FxCop analysis findings can be exported to the Team Foundation Server, which offers a number of defect-tracking-related reports to be generated.
Visual Studio 2005 Team Edition, $5,469 per user for Developer Edition; $2,799 for optional TFS. Microsoft Corp.
Product Analysis
Ounce Labs Prexis 3.2
Ounce Labs' Prexis is another multicomponent suite, similar to Klocwork and Fortify. The main Prexis/Engine hosts the bulk of the analysis capability and can be directly used to launch assessments. Prexis/Pro is a standalone interface to run assessments and view findings.
Ounce Labs does not offer any official IDE plug-ins for the Prexis components; rather, Prexis/Pro can be configured to invoke a source-code editor of your choice for viewing individual findings. The product's vulnerability discussion and remediation advice far exceeded our expectations, and the SmartTrace graphical visualization of vulnerable data flow was an added perk. Prexis has above-average support for adding new analysis rules and defining validation/encoding routines to further reduce false positives. Prexis/Insight is the centralized dashboard and reporting component, which offered the common lineup of reports and application portfolio visualizations.
Prexis 3.2, department installations for 50 developers and 2 managers start at $100,000. Ounce Labs.
Parasoft JTest 7.0 JTest
has been called a one-stop shop for all things Java-testing related, and that's pretty much true. JTest is officially sold as an "automated Java unit testing and coding standard analysis" product, focusing on reliability, functionality, security, performance and maintainability. We found JTest a great example of a general defect analysis product with some security-specific features, but security is not an exclusive focus.
JTest's library of analysis rules covers a very broad range of topics; only 8 percent are specifically designated as security-related. Because security defects can manifest under the guise of general defects, we hand picked through all of JTest's rules and found that about 20 percent were useful for security-assessment purposes.
Overall, JTest fared average on its security-bug-finding ability, despite its non-security-specific focus. JTest can propagate analysis suppression information directly to other team members through a Team Configuration Manager (TCM) server, and all reporting is done through the Group Reporting System (GRS). We particularly liked the fact that JTest shipped with a number of common Java libraries at the ready, which made it easier to satisfy common application dependencies without having to manually hunt them down. JTest also offers a wizard to help automate the creation and import of source code into an Eclipse project.
We found that the graphical rule builder took a bit of getting used to before we could start producing good results, and the supporting vulnerability and remediation information was below average, largely due to incomplete or missing entries.
Note that the security analysis, rule builder and CLI functionalities are all separately licensed features, and TCM and GRS servers are separate software packages. Parasoft's $50,000 price tag for JTest does include a brief application evaluation by Parasoft to create and tailor a security policy specific to a customer's needs. The JTest configuration is then delivered with custom policy enforcement rules loaded.
JTest 7.0, $50,000 for typical deployment, including JTest Server Edition (with CLI, Security Rules and Rules Builder features), TCM and custom policy evaluation. Parasoft.
Secure Software CodeAssure 2.0
Secure Software offers CodeAssure WorkBench for analyzing C, C++ and Java code, as well as CodeAssure Management Center for the typical lineup of dashboard and reporting functions. We looked at version 2.0, though 3.0 will be available by press time.
WorkBench comes as an Eclipse plug-in, which can be installed in an existing Eclipse environment or use a standalone Eclipse copy. You'll need to wrangle your project into Eclipse and make sure it can compile before CodeAssure can perform any analysis. CodeAssure offers a basic ability to integrate into central build processes, but it's not as robust or polished as some of the other products.
CodeAssure was the only product we looked at that provided purely informational findings, such as network socket binding and file creation, to aid security auditors in navigating to potential problem areas. Of course, this assumes auditors who will use the tool as an aid to their manual assessment, rather than expecting it to do all the work.
We found the vulnerability information and remediation adviser slightly above average; the information is directly sourced from Secure Software's comprehensive CLASP reference material, which does a great job at explaining why the vulnerabilities are a problem, but sometimes is a bit generic on remediation advice.
Version 3.0 is going to have quite a few changes. First is the availability of three different flavors of CodeAssure, for solo users, small teams and large enterprises. The solo-user version will be Java-only and is planned to have a yearly subscription cost of less than $500. CodeAssure Teams will introduce new collaboration services to share findings and suppression information among all team members. The enterprise flavor will offer additional rule extensibility and customization, more policy compliance features, and a specific central build integration package.
CodeAssure 2.0, $48,000 for 10 developers. Secure Software.
Product Analysis
Secure Software RATS 2.1The Rough Auditing Tool for Security, or RATS, is the most comprehensive open-source security analysis tool available, supporting five different programming languages: C, C++, Perl, PHP and Python. And yes, you read that right: Secure Software offers RATS for free while selling its CodeAssure product. RATS and CodeAssure are miles apart in nearly every aspect and implemented in entirely different manners.
We found the XML database format used by RATS extremely simple to modify for inclusion of custom function declarations. RATS uses a general semantic analysis approach, and as such, suffers from all the semantic analysis downsides. RATS offers output of findings in text, HTML and XML. We found the supporting descriptions and advice minimal and below average. Like FlawFinder and ITS4, RATS can suppress findings by including an "ignore" comment within your code.
RATS 2.1, free, GPL. Secure Software.
SPI Dynamics DevInspect & SecureObjects 1.5.0
DevInspect and SecureObjects are two separate products for addressing security vulnerability discovery and remediation in ASP.Net applications, though they are designed to closely work together--so much so, they are offered only as a bundle.
DevInspect uses run-time application testing/fuzzing techniques to remotely discover security vulnerabilities, similar to Compuware SecurityChecker's integrity analysis phase. In fact, DevInspect appears to be a tailored, repackaged Visual Studio plug-in version of SPI's flagship Web application scanner, WebInspect. As such, DevInspect operates identically to a remote Web application vulnerability scanner and does not perform local analysis of the application's source code. SPI did tell us that future versions will leverage source-code access to better direct DevInspect's run-time analysis.
SecureObjects, on the other hand, does work on a source-code level but not in the same manner as the other products we looked at. SecureObjects analyzes ASP.Net Web application code to find all potential data input points, and then determines if you are properly validating the input data. If you're lacking validation, SecureObjects can generate appropriate validation code automatically, based on predefined data-type templates such as credit-card number, user login name or street address, and insert it into your source code. SecureObjects can also insert violation logging routines, to better enable your application to track suspect input.
While all the other source code analysis products attempted to discover (and presumably aid in remediating) security defects in our application, SecureObjects instead took the route of preventing malicious data from reaching vulnerable areas in the first place. Because many security defects are triggered by specific malicious data, this is a novel concept-assuming the data format templates are correctly chosen. The pros and cons are very similar in nature to a Web application firewall--namely, a properly configured one can filter incoming data and render it benign against most vulnerabilities ... but one slip of the configuration and you're left exposed.
DevInspect & SecureObjects 1.5.0, $3,090 for both. SPI Dynamics.
Splint 3.1.1 Splint
Splint 3.1.1 Splint is a "lightweight" static analysis tool for C, based in spirit on the historical "lint" C source-code checking tool. Splint's primary capabilities revolve around its annotation engine; however, Splint offers some basic security analysis of non-annotated code as well. We found Splint's engine more advanced than any of the other open-source tools we looked at, but the engine hasn't been harnessed and leveraged for intensive security-specific analysis. Until that happens, we believe Splint is not for the faint-of-heart--it can generate a lot of highly technical warnings about subtle code nuances. Security-specific stuff can be buried in those nuances, but it may take a bit of interpretation to really extrapolate because most of it is not framed in a security light--the findings are presented as general defects amid other, non-security general defects. And, Splint's lack of C++ support is limiting, especially when it comes to traditional Windows application code-bases, though Splint is not shy about its Unix/POSIX bias.
Overall, we feel Splint has some great untapped power, but serves only a niche audience. We would be really interested to see someone take the Splint code-base and combine it with the RATS or FlawFinder engine. At that point, we might have an open-source tool that can to rival the analysis engines of commercial products.
Splint 3.1.1, free, GPL.
http://www.secureenterprisemag.com/article/printableArticle.jhtml?articleId=174402221
Contacts:Nick Allen
Director of Marketing
Klocwork, Inc.
603-433-2025
Michael Parker
Lois Paul & Partners
781-782-5714