Archive for Data

Zippity Update – Telemetry

I updated Zippity, our crowd sourcing data collection system, to add support for memory and CPU data telemetry. What is memory and CPU data telemetry you ask? Basically, we send data about the current Firefox Mobile memory and CPU usage patterns to Zippity. There are two ways to send the data:

  • Manually push a button: This might be a handy way to report metrics when you think Firefox Mobile is running slow on your phone.
  • Silently on idle once a day: During an idle moment, Zippity scans the metrics. This gives us data on the Firefox Mobile resting state. Hopefully we don’t see large CPU usage. Just install the Zippity add-on, and you start sending data – no additional work for you!

Why send this data you ask? We care a lot of the performance of Firefox Mobile, and we want to better understand how Firefox Mobile is running on your phone. We run a lot of tests at Mozilla, but these tests sometimes don’t compare well to the real world. For example, I have no performance issues running Firefox Mobile on my Nexus One, but we get lots of feedback from users that the performance is too slow on their phones. We need to run tests and collect data from Firefox Mobile running on your phone.

What kind of data is sent you ask? Firefox Mobile has a built-in memory reporting system and Zippity will enumerate that system to get detailed memory usage data. We also use the Linux /proc system to get information about the CPU usage. We only check the Firefox main and child processes. We don’t log any information about other processes. Thanks to Brad Lassey and Taras Glek for helping me get the data collected. Here is an example of the JSON formatted data sent to Zippity:

{
    "product": "Fennec 6.0a1",
    "os": "Android",
    "buildid": "20110416004121",
    "type": "metrics",
    "date": 1303152081000,
    "testkey": "telemetry-v1",
    "appid": "{a23983c0-fd0e-11dc-95ff-0800200c9a66}",
    "device": "Nexus One",
    "userkey": "",
    "osversion": "REL (10)",
    "data": {
        "reason": "manual",
        "malloc/allocated": 32377108,
        "malloc/mapped": 34603008,
        "malloc/committed": 34603008,
        "malloc/dirty": 77824,
        "js/gc-heap": 5242880,
        "js/string-data": 514630,
        "js/mjit-code": 88500,
        "storage/sqlite/pagecache": 4536536,
        "storage/sqlite/other": 1050816,
        "storage/permissions.sqlite/LookAside_Used": 29,
        "storage/permissions.sqlite/Cache_Used": 99208,
        "storage/permissions.sqlite/Schema_Used": 1272,
        "storage/permissions.sqlite/Stmt_Used": 5744,
        "storage/extensions.sqlite/LookAside_Used": 438,
        "storage/extensions.sqlite/Cache_Used": 428248,
        "storage/extensions.sqlite/Schema_Used": 6928,
        "storage/extensions.sqlite/Stmt_Used": 119024,
        "gfx/surface/image": 1088248,
        "images/chrome/used/raw": 0,
        "images/chrome/used/uncompressed": 1025124,
        "images/chrome/unused/raw": 0,
        "images/chrome/unused/uncompressed": 3648,
        "images/content/used/raw": 0,
        "images/content/used/uncompressed": 2144,
        "images/content/unused/raw": 4168,
        "images/content/unused/uncompressed": 8288,
        "layout/all": 563338,
        "storage/places.sqlite/LookAside_Used": 143,
        "storage/places.sqlite/Cache_Used": 428488,
        "storage/places.sqlite/Schema_Used": 11704,
        "storage/places.sqlite/Stmt_Used": 44152,
        "content/canvas/2d_pixel_bytes": 81120,
        "storage/cookies.sqlite/LookAside_Used": 14,
        "storage/cookies.sqlite/Cache_Used": 165000,
        "storage/cookies.sqlite/Schema_Used": 1816,
        "storage/cookies.sqlite/Stmt_Used": 0,
        "storage/formhistory.sqlite/LookAside_Used": 14,
        "storage/formhistory.sqlite/Cache_Used": 197920,
        "storage/formhistory.sqlite/Schema_Used": 1656,
        "storage/formhistory.sqlite/Stmt_Used": 0,
        "storage/addons.sqlite/LookAside_Used": 152,
        "storage/addons.sqlite/Cache_Used": 296616,
        "storage/addons.sqlite/Schema_Used": 4280,
        "storage/addons.sqlite/Stmt_Used": 29152,
        "storage/search.sqlite/LookAside_Used": 20,
        "storage/search.sqlite/Cache_Used": 99192,
        "storage/search.sqlite/Schema_Used": 1216,
        "storage/search.sqlite/Stmt_Used": 1840,
        "shmem/allocated": 0,
        "shmem/mapped": 0,
        "storage/index.sqlite/LookAside_Used": 148,
        "storage/index.sqlite/Cache_Used": 296656,
        "storage/index.sqlite/Schema_Used": 3032,
        "storage/index.sqlite/Stmt_Used": 51704,
        "uptime": 159848,
        "process": {
            "parent": {
                "user": 13,
                "system": 2,
                "rss": 77456
            },
            "child": {
                "user": 22,
                "system": 2,
                "rss": 25132
            },
            "cpu": 57
        }
    }
}

You can now access Metrics charts on Zippity to view the data. Trying to find ways to visualize the data isn’t easy. There is a lot of data. The charts are simple for now. The data point tooltip shows more detailed information.

Visit Zippity from your phone and install the add-on today. Thanks for using Zippity to help us collect data to make Firefox Mobile better.

Comments (3)

Firefox 3 – Offline App Demo

Firefox 3′s offline capabilities have been getting some attention lately: Chris Double’s post on porting Zimbra to use offline shows-off the potential. Robert O’Callahan and John Resig give details on the different pieces of the capabilities, namely: Offline Cache, Offline Events and DOMStorage.

A Simple Demo – Task Helper

I put together a simple offline application sample to help illustrate how the features can work. Call it – explanation through code. The application is a simple task list system. The current functionality includes:

  • Add tasks – Enter text in field and press ‘Add’
  • Complete tasks – Mark completed tasks and press ‘Complete’
  • Remove tasks – Mark tasks and press ‘Remove’
  • Data is stored as JSON and XHR is used to interact with server (PHP)

taskhelper.png

Nothing fancy. However, the application is “offline-aware”, meaning:

  • Application resources (HTML & JS) are tagged as offline cacheable
  • Before interacting with server, online status is checked. If online, use XHR. If offline, use DOMStorage
  • Online/Offline status is monitored using events. When the application switches form offline to online, data is resync’ed with the server

There is really nothing fancy about the offline stuff either, but putting it all together does make for a neat application. Using the latest Firefox 3 alpha, you should be able to:

  • Use the application while online.
  • Go offline using the “Work Offline” menu. The event log should show that the app went offline
  • Continue to use the application while offline
  • Go online using the “Work Offline” menu. The event log should show that the app is back online and has updated the server with any offline changes

If you were using a version of Firefox 3 with the offline cache patch (bug 367447) you would be able to a little more:

  • Use the application while online.
  • Go offline using the “Work Offline” menu. The event log should show that the app went offline
  • Continue to use the application while offline
  • Restart browser
  • Switch to offline (or have no network connection)
  • Enter URL to app (or use a bookmark)
  • Start using app with data from last session
  • Switching to online will update the server with offline data

Currently, the offline events only fire when offline is toggled via the “Work Offline” menu. Getting the events to fire by watching the network connection is the goal and would make this much more unobtrusive for the user.

Here is the source to the task helper application: taskhelper-src.zip

Comments (7)

I am My Data, My Data is Me

I couldn’t help but get caught up in the recent threads about data lock-in. I have become quite a “data-snob” recently which, I’m finding, is quite abnormal for a developer. Developers are all about the code. On one side you have the Mark Pilgrim & Jon Gruber (& Mark’s reply) discussing Apple’s data lock-in problems. On the other side you have Stewart Butterfield’s not-so-positive comments about allowing Flickr to export to Zooomr or Google’s new online Photo Ablums.

Lots of people are making the connection that these different threads have much in common: People use software to store important data; The data belongs to the people, not the software. This is not about Mac vs. Ubuntu or Flickr vs. Google. It’s about users feeling secure that their data won’t rot away in some application’s proprietary file format.

Is it any wonder that people are more at ease storing photographs in a shoebox than on a computer? It’s easy to take the photos out of the shoebox and move them to a photo album, or to a picture frame. How easy it is to move your photos and metadata to newer software? Or your letters, or your finanical information, or your tax returns?

Things are getting better. Microformats, Atom, and a desire for applications (specifically web applications) to interoperate is raising the bar. Even Microsoft Office finally has a relatively open format with full fidelity. Full fidelity is important or else you’ll lose data when you try to convert to another format. Software developers need to realize that the data will live longer than the software and there should be a simple, easy way to access the data without the software. Until then, your data may not be safe.

Comments (1)