Samstag, 25. August 2012

Groovy 2.0 Performance compared to Java

Also published on java.dzone.com, see link.

End of July 2012 Groovy 2.0 was released with support for static type checking and some performance improvements through the use of JDK7 invokedynamic and type inference as a result of type information now available through static typing.

I was interested in seeing some estimate as to how significant the performance improvements in Groovy 2.0 have turned out and how Groovy 2.0 would now compare to Java in terms of performance. In case the performance gap meanwhile had become minor or at least acceptable it would certainly be time to take a serious look at Groovy. Groovy is ready for production for a long time. Let's see whether it can compare with Java in terms of performance.

The only performance measurement I could find on the Internet was this little benchmark measurment on jlabgroovy. The measurement only consists of calculating Fibonacci numbers with and without the @CompileStatic annotation. That's it. Certainly not very meaningful to get an overall impression, but I was only interested in obtaining some rough estimate how Groovy now compares to Java concerning performance.

Java performance measurement included

Alas, no measurement was included in this little benchmark how much time Java takes to calculate Fibonacci numbers. So I "ported" the Groovy code to Java (here it is) and repeated the measurements. All measurements were done on an Intel Core2 Duo CPU E8400 3.00 GHz using JDK7u6 running on Windows 7 with Service Pack 1. I used Eclipse Juno with the Groovy plugin using the Groovy compiler version 2.0.0.xx-20120703-1400-e42-RELEASE. These are the figures I obtained without having a warm-up phase:


Groovy 2.0
without @CompileStatic
Groovy/Java
performance
factor
Groovy 2.0
with @CompileStatic
Groovy/Java
performance
factor
Kotlin
0.1.2580
Java
static ternary 4352ms 4.7 926ms 1.0 1005ms 924ms
static if 4267ms 4.7 911ms 0.9 1828ms 917ms
instance ternary 4577ms 2.7 1681ms 1.8 994ms 917ms
instance if 4592ms 2.9 1604ms 1.7 1611ms 969ms

I also did measurements with a warm-up phase of various length with the conclusion that there is no benefit for neither language with @CompileStatic or without. Since the Fibonacci algorithm is that recursive the warm-up phase seems to be "included" for any Fibonacci number that is not very small.

We can see that the performance improvements due to static typing has made quite a difference. This little comparison does not make up for an only little ambitious performance comparison. But to me the impression that static typing in Groovy in conjunction with type inference has led to significat performance improvements, the same way as with Groovy++, has become very strong. With @CompileStatic the performance of Groovy is about 1-2 times slower than Java and without about 3-5 times slower. Unhappily, the measurements for "instance ternary" and "instance if" are the slowest. Unless we want to create master pieces in programming with static functions, the measurements for "static ternary" and "static if" are not that relevant for most of the code with the ambition to be object-oriented (based on instances).

Conclusion

The times where Groovy was somewhat 10-20 times slower than Java (see benchmark table almost at the end of this article) are definitely over whether @CompileStatic is used or not. This means to me that Groovy is ready for applications where performance has to be somewhat comparable to Java. Earlier, Groovy (or Ruby, Closure, etc.) could only serve as a plus on your CV, because of the performance impediment (at least here in Europe).

New JVM kid on the block: Kotlin

I added the figures for Kotlin as well (here is the code). Kotlin is a relatively new statically typed JVM-based Java-compatible programming language. Kotlin is more concise than Java by supporting variable type inference, higher-order functions (closures), extension functions, mixins and first-class delegation, etc. Contrary to Groovy, it is more geared towards Scala, but also integrates well with Java. Kotlin is still under development and not officially released, yet. So the figures have to be taken with caution as the guys at JetBrains are still working on the code optimization (see KT-2687). Ideally, Kotlin should be as fast as Java (see this post). The measurements were done with the current "official" release 0.1.2580.

And what about future performance improvements?

At the time when JDK1.3 was the most recent JDK I still earned my pay with Smalltalk development. At that time the performance of VisualWorks Smalltalk (now Cincom Smalltalk) and IBM VA for Smalltalk (now owned by Instantiations) was very well comparable to Java. And Smalltalk is a dynamically typed language like pre-Groovy 2.0 and Ruby, where the compiler cannot make use of type inference to do optimizations. Because of this, it always appeared strange to me that Groovy, Ruby and other JVM-based dynamic languages had such a big performance penalty compared to Java when Smalltalk had not. Well, coming to think about it: Hot Spot runtime optimization in Java was taken from Smalltalk, anyway (see this article). Nothing beats the arrogance of a Smalltalk developer, even not a Mac enthusiast... Seems like creating a JVM-based language is easier than optimizing it's byte code. From that point of view I think that there is still room for Groovy performance improvements beyond @CompileStatic.

Kommentare:

  1. You might want to remove all the above spam comments ;)
    BTW, thanks for the comparison!

    AntwortenLöschen
  2. According to the docs, you shouldn't need to compile with @ComplieStatic on Java 7. Did you follow these instructions: Enabling invoke dynamic support
    Using the "indy" JARs is not enough, however, to compile your Groovy code so that it leverages the "invoke dynamic" support. For that, you’ll have to use the --indy flag when using the "groovyc" compiler or the "groovy" command. This also means that even if you’re using the indy JARs, you can still target JDK 5 or 6 for compilation.

    http://www.infoq.com/articles/new-groovy-20

    AntwortenLöschen
    Antworten
    1. Hello Scott, thanks for your comment. I mentioned in the first sentence that Groovy 2.0 has some support for invokedynamic, but I did not do any perorfmance measurement that included invokedynamic. This is because >full< support for invokedynamic was announced by the Groovy team for Groovy 2.1, which is meanwhile released but wasn't at the time I wrote this article. I'm working on adapting Robert Hundt's benchmark for Google Go (see https://days2011.scala-lang.org/sites/days2011/files/ws3-1-Hundt.pdf) for Groovy 2.1 to make measurements that include invokdynamic. I got a performance increase of roughly 6% when using Groovy with the INDY jars to leak some information. I will publish the whole thing in some few weeks to come.

      Löschen
  3. i assume you did not run groovy code as dynamic script. Rather you compied the code with "groovyc" and then the time given here is only the running time of already compiled groovy class (because compilation time can be significant).
    This "groovyc" compilation time is included or not(groovy code run as script or not), is not clear in the article?

    AntwortenLöschen
    Antworten
    1. The code was compiled with "groovyc" before it was run. The measurements do not include compilation times.

      Löschen