Zippity – Using the Crowd to Collect Performance Data

If you’re serious about improving your application’s performance, you need to collect data. At Mozilla, we use Talos. The Mobile team has been using Talos to track performance on Maemo (using N810 and N900) devices for a few years and recently started tracking Android (using Tegras) as well. Here are a few of the metrics we track:

  • Startup (Ts)
  • Pageload (Tp)
  • HTML Rendering (Tdhtml)
  • Sunspider (Tss)

We build the application every time code is checked into the source repository. After each build, we run these performance tests and watch for changes. Regressions happen and need to be fixed as soon as possible.

One limitation of the current Talos system is the hardware variety. For Firefox Mobile, we run tests on two (2) types of hardware: N900 (Maemo) and NVidia Tegra (Android). One problem is that this doesn’t map to the real world very well. Don’t get me wrong – the current system minimizes measurement noise and we can see regressions fairly easily. However, without testing on a variety of devices, we just don’t get a good picture of how Firefox Mobile runs on your device.

Hello Zippity!

One idea that has talked about is to get the community involved in testing performance. If we could leverage many different people, using many different phones, we might get a better picture of how Firefox Mobile performs in the real world.

I created a simple add-on (Test Harness) that can install in any Firefox 4 for Mobile release and can run a series of tests. Then it posts the result to a public data server (Zippity) and anyone can view the results. When a test result is posted to Zippity, we capture a few bits of information:

  • Test Type: Pageload, startup – whatever the test might be tracking.
  • Test Manifest: What was the actual sequence performed? What series of pages did you load? This is important if we want to try to minimize noise in the results. If you want to only compare test runs that used the same input data, it can help reduce noise and make trends easier to see.
  • User Key: In case you want to only see results submitted by a particular user. The user key is completely optional. It can be used to help filter extraneous results.
  • Device Metadata: Information like device type, OS, OS version. Again, helpful for filtering.

I have been running Firefox Mobile nightly builds on several Android devices using a static set of webpages, hosted from a local webserver. Very controlled, to minimize noise and try to see small changes over time. You can see the results here.

Anyone can run the same set of live, real world pages using the add-on. Because of differences in devices, networking latency and changes to the actual website content – the results won’t be the same. That’s OK! The real world results are just as meaningful and provide just as much valuable insight.

Currently, the Test Harness add-on only sends pageload (Tp) data to Zippity, but plans are underway to extend the test types. If you want to play along, just install the Test Harness add-on. It is already configured to run the pre-defined pageset and submit the results to Zippity. Just push the button:

Zippity is still evolving and doesn’t have a lot of features. The source code should be in my Mercurial user repo soon. I’ll post more about Zippity and the Test Harness add-on as things improve.

Firefox 4 for Mobile Beta 5 – Performance Improvements

We just released Firefox 4 for Mobile (Beta 5) and it’s packed with some noticeable improvements. The primary objectives of Beta 5 were improvements in general performance, reducing memory usage and improving rendering while loading and panning the web content. Here are some results to show the progress that’s been made.

Startup

We improved application startup speed. We have a system that reports startup time metrics once a day when the application does a check with AMO (addons.mozilla.org). Using this data, we can see the metrics improving from Firefox Mobile Beta 4 to Firefox Mobile Beta 5.

The “First Paint” metric is arguably the most important of the three. That’s when the user should start to see the application appear – not the splash screen, the actual application. Based on the Feb 23rd data, Beta 5 has a nearly 25% improved startup time. The earlier Beta 5 data (Feb 18th) was taken from internal testing and doesn’t represent a large enough population of users and devices. The Feb 23rd data is based almost 2200 data pings and is more comparable to the Beta 4 data, also from Feb 23rd.

Updated: Used actual Beta 5 data and not Beta 5 pre-release data.

Page Loading

We improved page loading performance. We have test suites that measure how long it takes to load a series of static copies of popular websites from a localhost webserver. The test uses static copies, built using wget and some tweaks, to remove the variability of live websites changing over time and network latency issues.

The data shows the exact same test being run on nightly builds of Firefox Mobile, using 3 different Android devices. The SCH-I800 is a Samsung Galaxy Tab. We can see a roughly 21% improvement from Beta 4, beginning of chart, to Beta 5, at the end.

JavaScript

The Mozilla JavaScript team keeps pushing the limits and mobile users get the benefits. We ran some benchmarks on some Android devices using the stock webkit-based browser, Firefox Mobile Beta 3, Firefox Mobile Beta 4 and Firefox Mobile Beta 5.

Go install Firefox 4 Beta 5 for Mobile. Hopefully, you’ll notice the speed improvements too!