If you’re serious about improving your application’s performance, you need to collect data. At Mozilla, we use Talos. The Mobile team has been using Talos to track performance on Maemo (using N810 and N900) devices for a few years and recently started tracking Android (using Tegras) as well. Here are a few of the metrics we track:
- Startup (Ts)
- Pageload (Tp)
- HTML Rendering (Tdhtml)
- Sunspider (Tss)
We build the application every time code is checked into the source repository. After each build, we run these performance tests and watch for changes. Regressions happen and need to be fixed as soon as possible.
One limitation of the current Talos system is the hardware variety. For Firefox Mobile, we run tests on two (2) types of hardware: N900 (Maemo) and NVidia Tegra (Android). One problem is that this doesn’t map to the real world very well. Don’t get me wrong – the current system minimizes measurement noise and we can see regressions fairly easily. However, without testing on a variety of devices, we just don’t get a good picture of how Firefox Mobile runs on your device.
Hello Zippity!
One idea that has talked about is to get the community involved in testing performance. If we could leverage many different people, using many different phones, we might get a better picture of how Firefox Mobile performs in the real world.
I created a simple add-on (Test Harness) that can install in any Firefox 4 for Mobile release and can run a series of tests. Then it posts the result to a public data server (Zippity) and anyone can view the results. When a test result is posted to Zippity, we capture a few bits of information:
- Test Type: Pageload, startup – whatever the test might be tracking.
- Test Manifest: What was the actual sequence performed? What series of pages did you load? This is important if we want to try to minimize noise in the results. If you want to only compare test runs that used the same input data, it can help reduce noise and make trends easier to see.
- User Key: In case you want to only see results submitted by a particular user. The user key is completely optional. It can be used to help filter extraneous results.
- Device Metadata: Information like device type, OS, OS version. Again, helpful for filtering.
I have been running Firefox Mobile nightly builds on several Android devices using a static set of webpages, hosted from a local webserver. Very controlled, to minimize noise and try to see small changes over time. You can see the results here.
Anyone can run the same set of live, real world pages using the add-on. Because of differences in devices, networking latency and changes to the actual website content – the results won’t be the same. That’s OK! The real world results are just as meaningful and provide just as much valuable insight.
Currently, the Test Harness add-on only sends pageload (Tp) data to Zippity, but plans are underway to extend the test types. If you want to play along, just install the Test Harness add-on. It is already configured to run the pre-defined pageset and submit the results to Zippity. Just push the button:
Zippity is still evolving and doesn’t have a lot of features. The source code should be in my Mercurial user repo soon. I’ll post more about Zippity and the Test Harness add-on as things improve.
Hum, how long does it take to complete a run ? it’s been running for a little while here and I think one of the cnn pages locked the tests.
One nice feature would be to have a countdown somehow …
This is fantastic, Mark. Can’t wait to get it working on the desktop Firefox too 😉
Maybe a bug: My test got stuck on a CNN page that said “not available in your region”. Is the test checking for specific page content?
@Dietrich – The add-on won’t check for specific content, but tries to load a specific page. I added a page timeout to the newest add-on, which should help get past hung pages. The pageset the add-on uses by default is here:
http://people.mozilla.com/~mfinkle/zippity/pageload-live-tp.json
Entering “test harness” in “Get Add-ons” in Fennec/4.0b5 gives no results, you have to visit your zippity.appserver.com/addon “Test Harness” link and allow the install. Then, in Firefox Mobile’s Add-ons panel, the [Restart] “Restart required” button doesn’t actually restart Firefox, it just exits it. (On stock HTC Evo Android 2.2.)
But now Test Harness running away through a bunch of sites, very nice! I have set a very short inactivity timeout for my phone, so early on in the tests my phone turned off. I don’t know if that will make my results invalid, if so you should mention turning this off (on my phone Settings > Display > Screen timeout > Never turn off). After I powered back on and entered my PIN the tests continued. I periodically dragged a page slightly to defeat the timeout.
Upon completion I OK’d the [Post results to server?] dialog, but then I got a confusing Android dialog inviting me to e-mail, just like your screenshots. I assume this is a separate optional step, if so Test Harness should have a separate dialog [E-mail results to someone as well?] and rephrase the web page’s “Once the process is finished, the average startup time is posted to Zippity. A detailed report of individual startup times can be shared.” to something like “When testing completes, Test Harness offers to post the average startup time to the Zippity server, and then you can e-mail a detailed report of individual startup times to a loved one.”
Where are the tooltips for touch interfaces? 🙁 A month from now I’ll have forgotten what that Test Harness icon in the sidebar is for 🙂 Perhaps all the Firefox Mobile icons should say what they do on press-and-hold, in the absence of any other press-and-hold behavior.