Zippity & Test Harness Updates

A few weeks ago I posted about Zippity, a way for anyone to help collect performance data for Firefox Mobile. Data has been coming in and so has some feedback. I added some features to the Test Harness add-on to make it easier to use and collect some new types of data. Here’s a list of changes:

• Added a proper in-content page for starting tests, and checking out previous result.
• Added support for measuring startup time, as well as, SunSpider and V8 JS benchmarks.
• Added basic support for desktop Firefox (thanks to Dietrich Ayala.)
• Added a page timeout for pageload tests. Sometimes pages just get stuck – either a bug in Fennec or a mystery of the ether.
• Added a simple goodput measure for pageload tests. We time how long it takes to download a 250KB test file from Zippity. It’s a simple metric we can use to categorize the pageload results.

All tests are preconfigured, you just need to tap the test you want to run and the add-on does the rest. Here’s a basic rundown of how each test works, so you have an idea of what to expect:

• Pageload: Opens a new tab and starts loading a predefined list of pages. We load the list of pages 3 times and take the average.
• Startup: Fennec will restart itself five (5) times, remembering the startup time for each restart and then taking the average. Startup time is defined as the time from launching the process to the end of loading the initial page.
• SunSpider and V8: Each subtest of the benchmark is a web page, loaded and executed. The page reports the time needed to complete the subtest. The final result is the total of all subtests.

I also updated the Zippity server to support the new data types and added some new features:

• Added graphs for viewing startup time, SunSpider and V8 results.
• New mobile pages built using jQuery Mobile. Now you can install the add-on and view the graphs from your mobile device. Still needs some tweaks, but it’s a start.
• Added the goodput measure to the graph tooltips. Will only appear for newly collected data that has the metric.

The SunSpider and V8 tests are taken from the Mozilla Talos test suite. The Test Harness add-on can load and time the tests just like the Talos system can, so I didn’t need to change a thing about the tests. Yes, that means I can add other Talos based page tests too, such as the DHTML and SVG tests.

I addition to some new Talos tests, I was thinking of adding some monitoring to Test Harness. Dietrich has an add-on to collect some UI responsiveness measures. I think we could add something like that to Zippity. Taras Glek has ideas for monitoring memory usage and posting the results once a day. That also sounds like a good use of Zippity.

If you use Zippity / Test Harness and have ideas, comments or found bugs – let me know. Comment here or join Mozilla IRC and look for “mfinkle” in #mobile (or several other channels).

Updated: I fixed a problem with the Pageload test manifest preference. It was pointing a 404 so the pageload tests wouldn’t work. The add-on is fixed now. Thanks AaronMT.