Zippity & Test Harness Updates

A few weeks ago I posted about Zippity, a way for anyone to help collect performance data for Firefox Mobile. Data has been coming in and so has some feedback. I added some features to the Test Harness add-on to make it easier to use and collect some new types of data. Here’s a list of changes:

  • Added a proper in-content page for starting tests, and checking out previous result.
  • Added support for measuring startup time, as well as, SunSpider and V8 JS benchmarks.
  • Added basic support for desktop Firefox (thanks to Dietrich Ayala.)
  • Added a page timeout for pageload tests. Sometimes pages just get stuck – either a bug in Fennec or a mystery of the ether.
  • Added a simple goodput measure for pageload tests. We time how long it takes to download a 250KB test file from Zippity. It’s a simple metric we can use to categorize the pageload results.

All tests are preconfigured, you just need to tap the test you want to run and the add-on does the rest. Here’s a basic rundown of how each test works, so you have an idea of what to expect:

  • Pageload: Opens a new tab and starts loading a predefined list of pages. We load the list of pages 3 times and take the average.
  • Startup: Fennec will restart itself five (5) times, remembering the startup time for each restart and then taking the average. Startup time is defined as the time from launching the process to the end of loading the initial page.
  • SunSpider and V8: Each subtest of the benchmark is a web page, loaded and executed. The page reports the time needed to complete the subtest. The final result is the total of all subtests.


I also updated the Zippity server to support the new data types and added some new features:

  • Added graphs for viewing startup time, SunSpider and V8 results.
  • New mobile pages built using jQuery Mobile. Now you can install the add-on and view the graphs from your mobile device. Still needs some tweaks, but it’s a start.
  • Added the goodput measure to the graph tooltips. Will only appear for newly collected data that has the metric.

The SunSpider and V8 tests are taken from the Mozilla Talos test suite. The Test Harness add-on can load and time the tests just like the Talos system can, so I didn’t need to change a thing about the tests. Yes, that means I can add other Talos based page tests too, such as the DHTML and SVG tests.

I addition to some new Talos tests, I was thinking of adding some monitoring to Test Harness. Dietrich has an add-on to collect some UI responsiveness measures. I think we could add something like that to Zippity. Taras Glek has ideas for monitoring memory usage and posting the results once a day. That also sounds like a good use of Zippity.

If you use Zippity / Test Harness and have ideas, comments or found bugs – let me know. Comment here or join Mozilla IRC and look for “mfinkle” in #mobile (or several other channels).

Updated: I fixed a problem with the Pageload test manifest preference. It was pointing a 404 so the pageload tests wouldn’t work. The add-on is fixed now. Thanks AaronMT.

Comments (4)

Zippity – Using the Crowd to Collect Performance Data

If you’re serious about improving your application’s performance, you need to collect data. At Mozilla, we use Talos. The Mobile team has been using Talos to track performance on Maemo (using N810 and N900) devices for a few years and recently started tracking Android (using Tegras) as well. Here are a few of the metrics we track:

  • Startup (Ts)
  • Pageload (Tp)
  • HTML Rendering (Tdhtml)
  • Sunspider (Tss)

We build the application every time code is checked into the source repository. After each build, we run these performance tests and watch for changes. Regressions happen and need to be fixed as soon as possible.

One limitation of the current Talos system is the hardware variety. For Firefox Mobile, we run tests on two (2) types of hardware: N900 (Maemo) and NVidia Tegra (Android). One problem is that this doesn’t map to the real world very well. Don’t get me wrong – the current system minimizes measurement noise and we can see regressions fairly easily. However, without testing on a variety of devices, we just don’t get a good picture of how Firefox Mobile runs on your device.

Hello Zippity!

One idea that has talked about is to get the community involved in testing performance. If we could leverage many different people, using many different phones, we might get a better picture of how Firefox Mobile performs in the real world.

I created a simple add-on (Test Harness) that can install in any Firefox 4 for Mobile release and can run a series of tests. Then it posts the result to a public data server (Zippity) and anyone can view the results. When a test result is posted to Zippity, we capture a few bits of information:

  • Test Type: Pageload, startup – whatever the test might be tracking.
  • Test Manifest: What was the actual sequence performed? What series of pages did you load? This is important if we want to try to minimize noise in the results. If you want to only compare test runs that used the same input data, it can help reduce noise and make trends easier to see.
  • User Key: In case you want to only see results submitted by a particular user. The user key is completely optional. It can be used to help filter extraneous results.
  • Device Metadata: Information like device type, OS, OS version. Again, helpful for filtering.

I have been running Firefox Mobile nightly builds on several Android devices using a static set of webpages, hosted from a local webserver. Very controlled, to minimize noise and try to see small changes over time. You can see the results here.

Anyone can run the same set of live, real world pages using the add-on. Because of differences in devices, networking latency and changes to the actual website content – the results won’t be the same. That’s OK! The real world results are just as meaningful and provide just as much valuable insight.

Currently, the Test Harness add-on only sends pageload (Tp) data to Zippity, but plans are underway to extend the test types. If you want to play along, just install the Test Harness add-on. It is already configured to run the pre-defined pageset and submit the results to Zippity. Just push the button:

Zippity is still evolving and doesn’t have a lot of features. The source code should be in my Mercurial user repo soon. I’ll post more about Zippity and the Test Harness add-on as things improve.

Comments (6)

Firefox 4 for Mobile Beta 5 – Performance Improvements

We just released Firefox 4 for Mobile (Beta 5) and it’s packed with some noticeable improvements. The primary objectives of Beta 5 were improvements in general performance, reducing memory usage and improving rendering while loading and panning the web content. Here are some results to show the progress that’s been made.

Startup

We improved application startup speed. We have a system that reports startup time metrics once a day when the application does a check with AMO (addons.mozilla.org). Using this data, we can see the metrics improving from Firefox Mobile Beta 4 to Firefox Mobile Beta 5.

The “First Paint” metric is arguably the most important of the three. That’s when the user should start to see the application appear – not the splash screen, the actual application. Based on the Feb 23rd data, Beta 5 has a nearly 25% improved startup time. The earlier Beta 5 data (Feb 18th) was taken from internal testing and doesn’t represent a large enough population of users and devices. The Feb 23rd data is based almost 2200 data pings and is more comparable to the Beta 4 data, also from Feb 23rd.

Updated: Used actual Beta 5 data and not Beta 5 pre-release data.

Page Loading

We improved page loading performance. We have test suites that measure how long it takes to load a series of static copies of popular websites from a localhost webserver. The test uses static copies, built using wget and some tweaks, to remove the variability of live websites changing over time and network latency issues.

The data shows the exact same test being run on nightly builds of Firefox Mobile, using 3 different Android devices. The SCH-I800 is a Samsung Galaxy Tab. We can see a roughly 21% improvement from Beta 4, beginning of chart, to Beta 5, at the end.

JavaScript

The Mozilla JavaScript team keeps pushing the limits and mobile users get the benefits. We ran some benchmarks on some Android devices using the stock webkit-based browser, Firefox Mobile Beta 3, Firefox Mobile Beta 4 and Firefox Mobile Beta 5.

Go install Firefox 4 Beta 5 for Mobile. Hopefully, you’ll notice the speed improvements too!

Comments (13)

Restartless Add-ons – Default Preferences

I saw a neat technique in Bug 564675 for creating default preferences in bootstrapped add-ons. Edward Lee posted a link to some code he uses for a bootstrapped add-on. Essentailly, the default preference branch is writable, but not permanent. This is a good thing for bootstrapped add-ons. The add-on should write it’s default preferences every time it loads and the preferences disappear when the session ends. The code looks like:


const PREF_BRANCH = "extensions.myaddon.";
const PREFS = {
  someIntPref: 1,
  someStringPref: "some text value"
};

function setDefaultPrefs() {
  let branch = Services.prefs.getDefaultBranch(PREF_BRANCH);
  for (let [key, val] in Iterator(PREFS)) {
    switch (typeof val) {
      case "boolean":
        branch.setBoolPref(key, val);
        break;
      case "number":
        branch.setIntPref(key, val);
        break;
      case "string":
        branch.setCharPref(key, val);
        break;
    }
  }
}

function startup(aData, aReason) {
  // Always set the default prefs as they disappear on restart
  setDefaultPrefs();
  ...
}

I haven’t tested to see if there is a clean way to remove these preferences when a bootstrapped add-on is disabled or uninstalled. The default preferences will likely still hang around until a shutdown occurs.

We’re starting to amass a sizable collection of code snippets for bootstrapped add-ons. I think a new section in the MDC Code Snippets area might be needed.

Comments (4)

Restartless Add-ons – More Resources

After playing around with bootstrap (restartless) add-ons a bit more, I came to the point where I needed some extra resources in the add-on. For images, I could have stuck with the data: URI approach I mentioned previously, but I also wanted to add an options UI for my add-on. Turns out it was pretty simple. Thanks to Vlad for getting the ball rolling.

The trick I am using now is to programmatically add a resource: alias for my add-on. A resource: alias creates a handy way for you to create URIs pointing to resources, like scripts and images, in your add-on bundle. It’s pretty simple to add a resource: URI pointing to the add-on’s root folder or XPI bundle. Here is the code snippet taken from my bootstrap.js file:


function startup(aData, aReason) {
  let resource = Services.io.getProtocolHandler("resource").QueryInterface(Ci.nsIResProtocolHandler);
  let alias = Services.io.newFileURI(aData.installPath);
  if (!aData.installPath.isDirectory())
    alias = Services.io.newURI("jar:" + alias.spec + "!/", null, null);
  resource.setSubstitution("myaddonpackage", alias);
  ...
}

function shutdown(aData, aReason) {
  if (aReason == APP_SHUTDOWN) return;

  let resource = Services.io.getProtocolHandler("resource").QueryInterface(Ci.nsIResProtocolHandler);
  resource.setSubstitution("myaddonpackage", null);
  ...

The isDirectory check is needed because it is possible for add-ons to be installed in folders or XPI bundles, which require the jar: URI syntax. The code adds a resource alias, after which you can use URIs like this:


resource://myaddonpackage/images/someimage.png
resource://myaddonpackage/content/somescript.js

In these examples, images and content are folders inside your add-on bundle. Now you can add URI-based resources to your install.rdf like this:


  ...
  <em:iconURL>resource://myaddonpackage/content/icon.png</em:iconURL>
  <em:optionsURL>resource://myaddonpackage/content/options.xul</em:optionsURL>
  ...

Yes, the em:optionsURL works just fine for providing an options UI in your add-on. I hope this tip gives you more help converting your traditional add-on to a restartless add-on.

Updated: I added the removal of the resource: alias on shutdown based on feedback from Nils in comments. Check out his link to some great bootstrap add-on information too. The part about stale caches after an add-on update presents a painful problem. Waldimir has also previously posted about the limitations of bootstrapped add-ons.

Comments (16)

Add-on Develoeprs – AMO is Firefox Mobile Beta 4 Ready

Just an update for add-on developers who are targeting Firefox Mobile. AMO (addons.mozilla.org) is ready for you to bump MaxVersion to “4.0b4″

A quick look at AMO shows a pretty nice list of add-ons for Firefox Mobile. I spotted a few restartless add-ons too. Very cool!

Comments off

Firefox Mobile – Managing Profiles

Last week we were talking about the need for private browsing, or something like it, in Firefox Mobile. Even though you might not share your phone with other people, you might share a tablet – especially the “family tablet”, sitting there on the coffee table. Private browsing is an obtrusive system, at least as implemented in Mozilla, and it’s doubtful we could add it for Firefox Mobile in time for the upcoming release. Also, private browsing doesn’t really satisfy the sharing use case for tablets.

Firefox has the concept of profiles, which could be useful for sharing. From the Firefox Support site:

Firefox saves your personal information such as bookmarks, passwords, and user preferences in a set of files called your profile, which is stored in a separate location from the Firefox program files. You can have multiple Firefox profiles, each containing a separate set of user information. The Profile Manager allows you to create, remove, rename, and switch profiles.

The built-in Profile Manager, used in desktop Firefox, isn’t really a good fit for mobile, so I made a simple, restartless add-on for managing profiles in Firefox Mobile, called Mobile Profiles. From the Preferences panel, you can add, remove and switch profiles.

I’m thinking about adding support for password protecting profiles and clearing private data when closing Firefox. Let me know what you think.

Development Details

I found the restartless, bootstrapped system I have been playing with worked well for this add-on. The nsIToolkitProfileService handles most of the heavy lifting. The UI is a little more complicated than my original trivial example, but I still only have two (2) files in my XPI. View the source here.

Without XUL overlays, I had to add the UI using DOM operations (createElement and setAttribute). Not a big deal, but it got tedious, so I made a little helper function:


function $x(aDocument, aElement, aAttributes) {
  if (typeof(aElement) == "string")
    aElement = aDocument.createElement(aElement);
  for (attribute in aAttributes)
    aElement.setAttribute(attribute, aAttributes[attribute]);
  return aElement;
}

Using this helper means adding elements via DOM is a little less syntax:


let itemManage = $x(window.document, "setting", {
  "id": "prefs-mobileprofiles-profiles",
  "title": "User Profiles",
  "desc": "Add, remove or switch profiles",
  "class": "setting-group",
  "type": "control"
});

I could probably make the helper function be a little more useful by adding some features you’d find in HTML DOM builders, but it’s good enough for now. I’ll probably add a chrome.manifest so I can support normal localization using properties files.

Comments (3)

Bootstrap Jones – Adventures in Restartless Add-ons

We recently landed some code in Firefox Mobile to better handle restartless (bootstrapped) add-ons. Turns out, Dietrich made a non-Jetpack add-on using the SDK and tried unsuccessfully to install it into Firefox Mobile. In the process of getting everything working, I decided to make a simple restartless add-on we could use as a testcase. It’s a copy of Dietrich’s original add-on, without any SDK usage. I’ve not taken the SDK plunge yet. The experience was interesting, educational and slightly enjoyable. The process of testing the install / uninstall and enable / disable mechanism without a restart left me as giddy as a schoolboy.

I’ve decided to approach any future add-on development using the bootstrapped system. If I find a reason why it’s not possible, them I’ll switch back to the traditional XUL overlay approach. Below is the code from the testcase I created. There are only two (2) files! install.rdf and bootstrap.js. That’s it!

The install.rdf has the basic stuff along with the em:bootstrap flag that turns on the magic. My bootstrap.js file has the basic bootstrap methods, but only two, startup and shutdown, needed to be implemented. I added some code to handle adding and removing my add-on UI on demand. All I do is implement the loadIntoWindow and unloadFromWindow methods. Pretty easy, especially for lightweight add-ons.

Notice how I am using a data: URI for a button image, so I don’t need to package the file and add a chrome.manifest.

Just to be clear, I am not using Jetpack or the SDK. Firefox Mobile doesn’t really support Jetpack yet. This is just a simple bootstrapped, restartless add-on. I’m in the process of using this system for a bigger add-on, so I might need to make some adjustments. I’ll let you know how it goes.

install.rdf


<?xml version="1.0"?>
<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:em="http://www.mozilla.org/2004/em-rdf#">
    <Description about="urn:mozilla:install-manifest">
        <em:id>bootstrapdemo@starkravingfinkle.org</em:id>
        <em:type>2</em:type>
        <em:name>Bootstrap Jones</em:name>
        <em:version>1.0</em:version>
        <em:bootstrap>true</em:bootstrap>
        <em:creator>Mark 'Bootstrap' Finkle</em:creator>
        <em:targetApplication>
            <Description>
                <em:id>{a23983c0-fd0e-11dc-95ff-0800200c9a66}</em:id>
                <em:minVersion>4.0b4pre</em:minVersion>
                <em:maxVersion>4.*</em:maxVersion>
            </Description>
        </em:targetApplication>
    </Description>
</RDF>

bootstrap.js


var Cc = Components.classes;
var Ci = Components.interfaces;

function loadIntoWindow(window) {
  if (!window) return;

  // Get the anchor for "overlaying" but make sure the UI is loaded
  let forward = window.document.getElementById("tool-forward");
  if (!forward) return;

  // Place the new button after the last button in the top set
  let anchor = forward.nextSibling;

  let button = window.document.createElement("toolbarbutton");
  button.setAttribute("id", "tool-homebutton");
  button.setAttribute("label", "Home");
  button.setAttribute("class", "button-control");
  button.style.listStyleImage = "url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAAo0lEQVR42u3VMQrCMBSA4Q7uOnsNB4%2FR0cM6ZPYyyRRIGhAzly5CNQrfD49Alhe%2BJdMkSZIk6d1ijMdSyryelNJp710hhMPWrpzzeRjAsiyX1tpzPf3%2B%2BgnsrV211hsAAAAAAAAAAAAAAAAAAAAAAACA0QDfHAAAAPweQH%2FUo5%2F3PafvCn4BAAAAAAAAAAAAAAAAAAAAAAAAAADGAkiSJOmPewEpGDS2TaImnAAAAABJRU5ErkJggg%3D%3D)";

  button.addEventListener("click", function() {
    window.Browser.loadURI("about:home");
  }, false);

  anchor.parentNode.insertBefore(button, anchor);
}

function unloadFromWindow(window) {
  if (!window) return;
  let button = window.document.getElementById("tool-homebutton");
  if (button)
    button.parentNode.removeChild(button);
}

/*
 bootstrap.js API
*/

var windowListener = {
  onOpenWindow: function(aWindow) {
    // Wait for the window to finish loading
    let domWindow = aWindow.QueryInterface(Ci.nsIInterfaceRequestor).getInterface(Ci.nsIDOMWindowInternal || Ci.nsIDOMWindow);
    domWindow.addEventListener("load", function() {
      domWindow.removeEventListener("load", arguments.callee, false);
      loadIntoWindow(domWindow);
    }, false);
  },
  onCloseWindow: function(aWindow) { },
  onWindowTitleChange: function(aWindow, aTitle) { }
};

function startup(aData, aReason) {
  let wm = Cc["@mozilla.org/appshell/window-mediator;1"].getService(Ci.nsIWindowMediator);

  // Load into any existing windows
  let enumerator = wm.getEnumerator("navigator:browser");
  while (enumerator.hasMoreElements()) {
    let win = enumerator.getNext().QueryInterface(Ci.nsIDOMWindow);
    loadIntoWindow(win);
  }

  // Load into any new windows
  wm.addListener(windowListener);
}

function shutdown(aData, aReason) {
  // When the application is shutting down we normally don't have to clean up any UI changes
  if (aReason == APP_SHUTDOWN) return;

  let wm = Cc["@mozilla.org/appshell/window-mediator;1"].getService(Ci.nsIWindowMediator);

  // Stop watching for new windows
  wm.removeListener(windowListener);

  // Unload from any existing windows
  let enumerator = wm.getEnumerator("navigator:browser");
  while (enumerator.hasMoreElements()) {
    let win = enumerator.getNext().QueryInterface(Ci.nsIDOMWindow);
    unloadFromWindow(win);
  }
}

function install(aData, aReason) { }

function uninstall(aData, aReason) { }

Comments (12)

Firefox Mobile – Setting Your Own Start Page

Firefox Mobile comes with a handy start page, but it might not be your favorite. If you have your own favorite page, you can set it as the start page. First, navigate to the web page you want to use. Then, go to Preferences, tap the Start page button and pick Use Current Page. That’s it!

If you do keep the built-in Firefox Start page – check out Start Page Plus and make your start page cooler.

Comments (1)

Firefox Mobile – A Few Add-ons

In addition to working on Firefox Mobile, I like to tinker around with building add-ons too. Not surprising I guess, since I seem to go on and on about how to build them.

Anyway, I have a few add-ons that you might find useful – either installed in Firefox Mobile or as examples you can rip apart and use to build your own add-ons.

Mobile Tools

Mobile Tools is a collection of a few utilities that might be useful when debugging mobile webpages. You’ll find commands for viewing page source and basic page info. Both open in new tabs. Page source shows the web page’s raw HTML content. You can zoom and pan through the source just like any other web page. Page info shows some simple bits of information about the web page:

Mobile Tools also supports window.console a little better than the built-in native support, including:

  • String formatting in logging methods (%s, %d, %i, %f)
  • dir and dirxml
  • Simplified trace
  • group and groupEnd
  • time and timeEnd

Start Page Plus

The built-in start page, Firefox Start, is pretty, loads fast and has some nice features. We can always add more! Start Page Plus adds a few sections to the standard Firefox Start page:

  • Favorites: A list of bookmarks you want displayed on the start page. Just tag a bookmark with “start” and it appears on the start page.
  • Local Search: Quickly find various types of places near you with a simple tap. Includes food, hotels, gas, movie theaters and your own location – shown on Google Maps. This feature does use the built-in geolocation feature.
  • Local Twitter: Shows tweets happening near you. You don’t need to be logged into Twitter. The stream is based on your location.

Use the add-on options to show and hide sections as well as move them around the page. You can also set the zone used to pull in tweets from Twitter.

Comments (1)

« Previous Page« Previous entries « Previous Page · Next Page » Next entries »Next Page »