Static analysis for Ruby/Python

on Jun 29, 09 • by Denis Sidorov • with 13 Comments

As a developer of static analysis tool for mainstream statically-typed languages, like C++ and Java, I was wondering for quite a while about how well static analysis applies to dynamically-typed languages, like Ruby and Python. And recently, I have come across this interesting project on GitHub: Reek – Code...

Home » Static Analysis » Static analysis for Ruby/Python

As a developer of static analysis tool for mainstream statically-typed languages, like C++ and Java, I was wondering for quite a while about how well static analysis applies to dynamically-typed languages, like Ruby and Python. And recently, I have come across this interesting project on GitHub: Reek – Code smell detector for Ruby. Well, I suppose that’s just a fancy way to name a static analysis tool.

What can Reek detect? It does not do heavyweight data/control flow analysis, so the list is not very exciting:

Interestingly, despite the poor set of features, compared to modern C++/Java static analysis solutions, Reek gets positive feedback from Ruby community. Some take it to the extreme – integrate Reek into the build and handle code smells as failing tests, and that of course breaks the build every time a new code smell is detected. So, is Ruby community starving for real static-analysis tool?..

This question forced me to run a quick research on what’s the current state of static analysis tools for dynamic languages. I was able to find the following tools for Ruby and Python:

Python tools appear to be pretty conservative about program analysis and try to “compensate” the dynamic nature of the language.

  • PyChecker and pylint – These two tools aim to provide the same kind of warnings/errors, a compiler for statically-typed language would report automatically. For example: too few/many arguments in a method call, unused variables, using return value of a method that does not actually return any value, etc. One of the checkers for pylint, in fact tries to “simulate” static type system by using type inference, and detect missing members and functions, and this, in turn, might involve some elements of data flow analysis. Beside that, there is a bunch of metrics-based rules that can be checked automatically – you can put a limit on number of methods in a class, number of variables in a function, size/complexity of a function etc.

Tools for Ruby introduce the same kind of checks, but from slightly different angle. Rather than adopting the lint metaphor (a complementary tool, finding bugs/errors that compiler does not detect), the tools are positioned to find design flaws.

  • Reek – See above.
  • Flay – Checks code bases for duplicates. Aims to enforce the DRY design principle.
  • Flog – A tool to spot the most complex functions in your code. Metric-based.
  • Roodi – Supports a bunch of simple syntax-based and metric-bases checks. For example: assignment in condition, case missing else, max method/class/module line count, method/class/module name check.

Having looked through these projects, I’ve come to find two things:

First – There are some tools that do static analysis for Ruby/Python, but they are quite simple and don’t do (or don’t try to do?) any advanced heavyweight analysis.

Second – What Ruby/Python developers want to be automatically found in their code, is quite different from what their C/C++ fellows expect from a static analysis tool. And that is because the languages and programming cultures are quite different. For example, in Ruby there is no such thing as:

  • null pointer dereference – nil is a first class object
  • array bounds violation – array would return nil when index is out of bounds
  • uninitialized variables – all variables are automatically initialized with nil
  • memory leaks – garbage collection takes care of that

But they do have errors in their programs, don’t they? Yes, but Ruby/Python developers rely on tests pretty heavily. In fact, some claim, that tests are the only right way to deal with bugs in your program. This way a tool for automatic error detection might even be considered harmful – just because it can be used as an excuse for not writing tests.

So, what kind of job is left for a static analysis tool? Well, detect design flaws, a.k.a. “code smells”. In other words – automatically find subjects for refactoring (a change that does not affect program functionality). This way static analysis tool fits naturally into Red/Green/Refactor cycle.

Another possible area where static analysis based error detection can prove useful is security vulnerabilities – it is pretty hard to spot all corner cases up-front (or through exploratory testing), just because security is a complex domain and requires good amount of knowledge and expertise.

Related Posts

13 Responses to Static analysis for Ruby/Python

  1. Igor Bronshteyn says:

    I can’t tell something regarding Ruby, but as for Python I could imagine 3 of 4 defects you mentioned:
    * null pointer dereference – OK, let’s call it not NPD, but None-dereference
    * array bounds violation – there’s an exception if your list index is out of range
    * memory leaks – rare, but possible, if you have reference cycles (http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks).

    But I guess that all of these defects are quite rare, and one must take a start from a true-to-life defects, not just apply an approach to statically-typed languages here.

  2. Nick says:

    Check out “Pex Microsoft”

    It just shows what is going to come down the line when it comes to analysis tools.

    Personally, the next big thing is going to be design by contract. DBC opens up software for lots of things. Interestingly it reduces the need for unit tests. After all, if the code is being tested for all cases you come across, rather than the handfull that are in unit tests, you are going to get better code as a result.

    Combine that with Pex where it uses the debugging library to trace tests, and then uses this analysis to find cases that will test other paths through your code, and you are running at a level of testing far above that which can be achieved by hand

  3. Mark says:

    1. Memory leaks still happen in Ruby. Objects that contain references to them are not GC’d.
    2. Joe Damato has been doing some interesting things with libdl:
    http://timetobleed.com/extending-ltrace-to-make-your-rubypythonperlphp-apps-faster/
    http://timetobleed.com/memprof-a-ruby-level-memory-profiler/
    His blog is worth a good read regarding this topic

  4. For python, you might want to include Clone Digger (http://clonedigger.sourceforge.net/).

  5. Michael Peters says:

    Just to make your overview of dynamic languages more complete, Perl has Perl::Critic (http://search.cpan.org/perldoc?Perl::Critic) which is similar to Reek.

  6. Astor says:

    C has been around for so long and very prone to many issues so unique tools were created to help those issues. While python and ruby make programming fun and exciting for many more people than C. It’s still imperative that C has it’s suit of tools that make it bearable for the large extent of it’s programmers. Yes people like programming in C

  7. I would love to have a tool that could do more heavy-weight analysis on Python code. For example, suppose you call foo() 100 times in your application, and check the return value 99 times. It would be nice if there was a tool that could flag you to take a look at that one call where you didn’t check the return value.

    I am not even too concerned about things like eval or calling functions in other clever ways that would not be caught by static analyzers. If a tool caught the normal cases that would still be a big win.

  8. It’s not just that Ruby developers want different things, but that almost everything a typical static analysis tool does in C/C++ is next to impossible to check for in Ruby without first making a lot of assumptions which may or may not hold.

    Doing a “proper” static analysis tool + type inference in Ruby isn’t *impossible* but it would require the user to make a lot of promises to the checker about what you do (or don’t do).

    For example, in the presence of a single eval() that takes a user provided string in a Ruby program, you *can’t* statically determine what type an object has, or what method a class has, or how many arguments (and of what type) a method takes. Effectively the world becomes almost entirely unknown past that point. The way around that is for the developer to effectively say “yeah, but we only ever feed this eval() code that is restricted to these types of effects: A, B, C”

    The same is the case for a lot of things, such as runtime “require” of files based on a string that may not be possible to statically determine.

    Handling that kind of user annotation adds a *lot* of complexity. You can of course provide reasonable defaults. E.g. it’s reasonable 99.9% of the time to assume nobody will add instance variables to FixNum’s, or overload the “+” method of String to do something other than concatenation, even in the presence of eval().

    But there’s a big grey area of things that may not be common, but that happens. Getting a Ruby-tool like this as comprehensive as one for C is a *huge* task.

  9. Paddy3118 says:

    Hi,
    You should make the point that there can be very little that a static, compile-time checker can do except suggest problems, as the code can be changed dramatically at run time. objects are typed rather than variable names; containers such as lists and dicts can have values of more than one type; duck typing; …., and when you get right down to it, you need the tests anyway.

    It would be great if you could break into a running program and do some type analysis for dynamic languages.

    - Paddy.

  10. Denis Sidorov says:

    @Kevin: Yes, most of the Ruby projects doing program analysis (including Reek) are pretty young. I guess one of the reasons is that ParseTree API was introduced not very long ago.

    And, I’m certainly not trying to judge :) I’d like see if Ruby can break yet another stereotype and re-define program analysis (e.g. the way it redefined Web Frameworks). It is particularly interesting how Ruby error detection tools would approach the infamous “false positive” problem – an inherent weakness of all static analysis tools, when a real error/flaw could sometimes be buried under a pile of falsely reported issues.

    BTW, have you tried Reek on some large Ruby codebases, like Rails? How many code smells were found? Got any interesting ideas on how to triage the detected smells?

  11. Hi, Thanks for the brief objective look at Reek! Please bear in mind that Reek is very much a work in progress (look at its Issues page on github and you’ll see I have many more code smells and other checks still to add). As you suspect, it is also intended as an experiment, to see just how far we can go without static analysis in Ruby.

    But I think the main thing to note is that a number of such tools have sprung up for Ruby in the last year, possibly indicating a lot about the maturing of the Ruby community.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Scroll to top