The Adobe RIA crowd have been causing some echos in the chamber lately over Bubblemark, an animation benchmark originally released over a year ago. Personally, I have no real skin in the game, but a game it is. Some Flash developers were curious about variations in the results when run on different browsers, platforms and in the AIR runtime. It makes for some interesting reading and I came away with a better understanding the variations, browser limitations and timer mechanisms.
What caused me to gush milk out my nose was reading that this benchmark appears to be some kind of de facto standard for RIAs. No way! Someone is going to have to explain, speaking slowly and using big letters, to me how rendering an bunch of bouncing balls and calculating a framerate from the bouncing balls is a benchmark for RIAs. I guess if your RIA is an animation of bouncing balls (or non-balls too, I guess), this test has you covered. Thankfully, I’m not the only one. The Flash guys have more than a few gripes with the test. Sean Christmann sums up my biggest gripe:
The test just moves balls around! This is my biggest beef with the benchmark because it only tests one simple aspect of the rendering engine in these technologies, which is bitmap translation. How do bitmaps moving around the screen tell you anything about the capabilities of the respective technologies? Do the JavaFX guys really think optimizing this usecase will make their technology relevant? The only thing Bubblemark will tell you is which runtimes might best handle bitmap particle emitters….thats about it.
Instead, I’ll offer that the Bubblemark test has value in that many different technologies have implemented the test. Therefore, it provides some level of benchmark for how different technologies can be optimized to implement the test. Of course, I could be way off base here and the RIA of which they speak is actually “Really Interesting Animations.” If so, whoops, my bad.
The test has lots of issues, no doubt about it, but there’s a huge lack of independant benchmarks across all of the different technologies. Putting faith in any one benchmark is very, very silly but if you want a general idea, BubbleMark is the only place to go for now.