Random Management: Unblocking Technical Leadership

I’m an Engineering Manager, previously a Senior Developer. I have done a lot of coding. I still try to do a little coding every now and then. Because of my past, I could be oppressive to senior developers on my teams. When making decisions, I found myself providing both the management viewpoint and the technical viewpoint. This usually means I was keeping a perfectly qualified technical person from participating at a higher level of responsibility. This creates an unhealthy technical organization with limited career growth opportunities.

As a manager with a technical background, I found it difficult to separate the two roles, but admitting there was a problem was a good first step. Over the last few years, I have been trying to get better at creating more room for technical people to grow on my teams. It seems to be more about focusing on outcomes for them to target, finding opportunities for them to tackle, listening to what they are telling me, and generally staying out of the way.

Another thing to keep in mind, it’s not just an issue with management. The technical growth track is a lot like a ladder: Keep developers climbing or everyone can get stalled. We need to make sure Senior Developers are working on suitable challenges or they end up taking work away from Junior Developers.

I mentioned this previously, but it’s important to create a path for technical leadership. With that in mind, I’m really happy about the recently announced Firefox Technical Architects Group. Creating challenges for our technical leadership, and roles with more responsibility and visibility. I’m also interested to see if we get more developers climbing the ladder.

Random Thoughts on Management

I have ended up managing people at the last three places I’ve worked, over the last 18 years. I can honestly say that only in the last few years have I really started to embrace the job of managing. Here’s a collection of thoughts and observations:

Growth: Ideas and Opinions and Failures

Expose your team to new ideas and help them create their own voice. When people get bored or feel they aren’t growing, they’ll look elsewhere. Give people time to explore new concepts, while trying to keep results and outcomes relevant to the project.

Opinions are not bad. A team without opinions is bad. Encourage people to develop opinions about everything. Encourage them to evolve their opinions as they gain new experiences.

“Good judgement comes from experience, and experience comes from bad judgement” – Frederick P. Brooks

Create an environment where differing viewpoints are welcomed, so people can learn multiple ways to approach a problem.

Failures are not bad. Failing means trying, and you want people who try to accomplish work that might be a little beyond their current reach. It’s how they grow. Your job is keeping the failures small, so they can learn from the failure, but not jeopardize the project.

Creating Paths: Technical versus Management

It’s important to have an opinion about the ways a management track is different than a technical track. Create a path for managers. Create a different path for technical leaders.

Management tracks have highly visible promotion paths. Organization structure changes, company-wide emails, and being included in more meetings and decision making. Technical track promotions are harder to notice if you don’t also increase the person’s responsibilities and decision making role.

Moving up either track means more responsibility and more accountability. Find ways to delegate decision making to leaders on the team. Make those leaders accountable for outcomes.

Train your engineers to be successful managers. There is a tradition in software development to use the most senior engineer to fill openings in management. This is wrong. Look for people that have a proclivity for working with people. Give those people management-like challenges and opportunities. Once they (and you) are confident in taking on management, promote them.

Snowflakes: Engineers are Different

Engineers, even great ones, have strengthens and weaknesses. As a manager, you need to learn these for each person on your team. People can be very strong at starting new projects, building something from nothing. Others can be great at finishing, making sure the work is ready to release. Some excel at user-facing code, others love writing back-end services. Leverage your team’s strengthens to efficiently ship products.

“A 1:1 is your chance to perform weekly preventive maintenance while also understanding the health of your team” – Michael Lopp (rands)

The better you know your team, the less likely you will create bored, passionless drones. Don’t treat engineers as fungible, swapable resources. Set them, and the team, up for success. Keep people engaged and passionate about the work.

Further Reading

The Role of a Senior Developer
On Being A Senior Engineer
Want to Know Difference Between a CTO and a VP of Engineering?
Thoughts on the Technical Track
The Update, The Vent, and The Disaster
Bored People Quit
Strong Opinions, Weakly Held

Firefox for Android: Collecting and Using Telemetry

Firefox 31 for Android is the first release where we collect telemetry data on user interactions. We created a simple “event” and “session” system, built on top of the current telemetry system that has been shipping in Firefox for many releases. The existing telemetry system is focused more on the platform features and tracking how various components are behaving in the wild. The new system is really focused on how people are interacting with the application itself.

Collecting Data

The basic system consists of two types of telemetry probes:

  • Events: A telemetry probe triggered when the users takes an action. Examples include tapping a menu, loading a URL, sharing content or saving content for later. An Event is also tagged with a Method (how was the Event triggered) and an optional Extra tag (extra context for the Event).
  • Sessions: A telemetry probe triggered when the application starts a short-lived scope or situation. Examples include showing a Home panel, opening the awesomebar or starting a reading viewer. Each Event is stamped with zero or more Sessions that were active when the Event was triggered.

We add the probes into any part of the application that we want to study, which is most of the application.

Visualizing Data

The raw telemetry data is processed into summaries, one for Events and one for Sessions. In order to visualize the telemetry data, we created a simple dashboard (source code). It’s built using a great little library called PivotTable.js, which makes it easy to slice and dice the summary data. The dashboard has several predefined tables so you can start digging into various aspects of the data quickly. You can drag and drop the fields into the column or row headers to reorganize the table. You can also add filters to any of the fields, even those not used in the row/column headers. It’s a pretty slick library.


Acting on Data

Now that we are collecting and studying the data, the goal is to find patterns that are unexpected or might warrant a closer inspection. Here are a few of the discoveries:

Page Reload: Even in our Nightly channel, people seem to be reloading the page quite a bit. Way more than we expected. It’s one of the Top 2 actions. Our current thinking includes several possibilities:

  1. Page gets stuck during a load and a Reload gets it going again
  2. Networking error of some kind, with a “Try again” button on the page. If the button does not solve the problem, a Reload might be attempted.
  3. Weather or some other update-able page where a Reload show the current information.

We have started projects to explore the first two issues. The third issue might be fine as-is, or maybe we could add a feature to make updating pages easier? You can still see high uses of Reload (reload) on the dashboard.

Remove from Home Pages: The History, primarily, and Top Sites pages see high uses of Remove (home_remove) to delete browsing information from the Home pages. People do this a lot, again it’s one of the Top 2 actions. People will do this repeatably, over and over as well, clearing the entire list in a manual fashion. Firefox has a Clear History feature, but it must not be very discoverable. We also see people asking for easier ways of clearing history in our feedback too, but it wasn’t until we saw the telemetry data for us to understand how badly this was needed. This led us to add some features:

  1. Since the History page was the predominant source of the Removes, we added a Clear History button right on the page itself.
  2. We added a way to Clear History when quitting the application. This was a bit tricky since Android doesn’t really promote “Quitting” applications, but if a person wants to enable this feature, we add a Quit menu item to make the action explicit and in their control.
  3. With so many people wanting to clear their browsing history, we assumed they didn’t know that Private Browsing existed. No history is saved when using Private Browsing, so we’re adding some contextual hinting about the feature.

These features are included in Nightly and Aurora versions of Firefox. Telemetry is showing a marked decrease in Remove usage, which is great. We hope to see the trend continue into Beta next week.

External URLs: People open a lot of URLs from external applications, like Twitter, into Firefox. This wasn’t totally unexpected, it’s a common pattern on Android, but the degree to which it happened versus opening the browser directly was somewhat unexpected. Close to 50% of the URLs loaded into Firefox are from external applications. Less so in Nightly, Aurora and Beta, but even those channels are almost 30%. We have started looking into ideas for making the process of opening URLs into Firefox a better experience.

Saving Images: An unexpected discovery was how often people save images from web content (web_save_image). We haven’t spent much time considering this one. We think we are doing the “right thing” with the images as far as Android conventions are concerned, but there might be new features waiting to be implemented here as well.

Take a look at the data. What patterns do you see?

Here is the obligatory UI heatmap, also available from the dashboard:

Patterns of Effective Teams

I have been lucky to build software products for a few different companies, each with a distinct culture. It’s help me form opinions about people, tools and processes that make teams effective at shipping software products.

Who makes up a software product team? Mileage may vary, but I like to include:

  • Developers
  • Testers
  • UX Designers
  • Project Managers
  • Product Managers
  • Support

Lots of companies organize people into functional groups: All the developers in a group, all the testers in a group, all the designers in a group… and so on. This doesn’t make it easy to ship software. It can create walls and make it harder to communicate. You also lose the “team” feeling, as well as the focus and drive that comes from that.

Product-centric teams seem to be more effective at shipping. These multidisciplinary teams embed members from the various groups on the team, all working together to create and ship a software product.

Over the years, I’ve seen productive teams using a few basic concepts. Some are process related, some can be aided by tools, but most deal with relationships between people:

  • Trust each other: Each member has a role, and members need to trust in each other’s ability to perform.
  • Talk to each other: Lots of open communication is important. The team is a safe place, so there are no stupid questions. Meet as a group often to discuss progress.
  • Support each other: You win and lose together. Help others, even if not asked directly.
  • Be passionate: The team needs to be passionate about succeeding and hungry to ship a great product. There will be rough spots on the way. There always are, but the team needs that passion to be able to power through.
  • Move as a single, focused group: Speed is important. Decisions, implementation, feedback – all need to happen ASAP. Distractions kill speed.
  • Plan work as a group and document the plan: If everyone is part of the planning, everyone is committed to the plan. Keeps the team focused.
  • Create a roadmap: You need a Big Picture too. What’s the vision and strategy? It helps set the tone for everything else.
  • Break work into small tasks and track the tasks: Small tasks are manageable and trackable. Small tasks are easy to scope and keeps the team focused. Watch out for scope creep.
  • Create milestones and track progress: Deadlines are a good thing, even if just internal. Forward progress is essential for shipping and milestones are great for tracking progress.
  • Adjust as needed: Don’t be afraid to adjust anything: schedule, milestones, tasks. You are collecting data every day. Use it to make informed decisions ASAP. Triage your work often.

I like to keep things lightweight. This includes tools and processes. Focus more on your product and the work at hand. Processes and tools can be distractions. The best ones are those that stay out of your way.

Update: Taras reminded me indirectly about the importance of passion, so I added it to the list.

Add-ons: Binary Components and js-ctypes

Lots of discussion going on recently about the affect of 6 week development cycles on binary XPCOM components in add-ons. I don’t want to re-hash those discussions, but Daniel Glazman brought up an interesting point in a comment on Wladimir’s post. Wladimir was suggesting binary component developers start moving to js-ctypes. Daniel pointed out that there are two classes of binary XPCOM components:

  • XPCOM wrappers around 3rd-party binary libraries: We use this model for exposing external binary functionality into JavaScript so add-ons and applications can access the libraries. Using js-ctypes should provide a simple, non-breaking way to expose the libraries. You create a simple JavaScript wrapper in a JavaScript XPCOM component. We need more examples of using js-ctypes to do this, but it works.
  • Pure binary XPCOM components built only using the Mozilla platform: Sometimes the functionality you want to expose is actually locked away in the Mozilla platform itself. Maybe there is no public nsIXxx interface or the existing interface has a [noscript] attribute on a property of method. This model shouldn’t be required anymore, in my opinion. Mozilla is pushing JavaScript based components and we should be exposing as much as possible to chrome JavaScript. I would encourage add-on developers to file bugs and lobby to expose binary-only parts of the Mozilla platform to chrome JavaScript.

JavaScript ctypes has come a long, long way since it was started back in 2007. Let’s start leveraging it more. The ctypes model has been used quite effectively in other languages.

Zippity – Using the Crowd to Collect Performance Data

If you’re serious about improving your application’s performance, you need to collect data. At Mozilla, we use Talos. The Mobile team has been using Talos to track performance on Maemo (using N810 and N900) devices for a few years and recently started tracking Android (using Tegras) as well. Here are a few of the metrics we track:

  • Startup (Ts)
  • Pageload (Tp)
  • HTML Rendering (Tdhtml)
  • Sunspider (Tss)

We build the application every time code is checked into the source repository. After each build, we run these performance tests and watch for changes. Regressions happen and need to be fixed as soon as possible.

One limitation of the current Talos system is the hardware variety. For Firefox Mobile, we run tests on two (2) types of hardware: N900 (Maemo) and NVidia Tegra (Android). One problem is that this doesn’t map to the real world very well. Don’t get me wrong – the current system minimizes measurement noise and we can see regressions fairly easily. However, without testing on a variety of devices, we just don’t get a good picture of how Firefox Mobile runs on your device.

Hello Zippity!

One idea that has talked about is to get the community involved in testing performance. If we could leverage many different people, using many different phones, we might get a better picture of how Firefox Mobile performs in the real world.

I created a simple add-on (Test Harness) that can install in any Firefox 4 for Mobile release and can run a series of tests. Then it posts the result to a public data server (Zippity) and anyone can view the results. When a test result is posted to Zippity, we capture a few bits of information:

  • Test Type: Pageload, startup – whatever the test might be tracking.
  • Test Manifest: What was the actual sequence performed? What series of pages did you load? This is important if we want to try to minimize noise in the results. If you want to only compare test runs that used the same input data, it can help reduce noise and make trends easier to see.
  • User Key: In case you want to only see results submitted by a particular user. The user key is completely optional. It can be used to help filter extraneous results.
  • Device Metadata: Information like device type, OS, OS version. Again, helpful for filtering.

I have been running Firefox Mobile nightly builds on several Android devices using a static set of webpages, hosted from a local webserver. Very controlled, to minimize noise and try to see small changes over time. You can see the results here.

Anyone can run the same set of live, real world pages using the add-on. Because of differences in devices, networking latency and changes to the actual website content – the results won’t be the same. That’s OK! The real world results are just as meaningful and provide just as much valuable insight.

Currently, the Test Harness add-on only sends pageload (Tp) data to Zippity, but plans are underway to extend the test types. If you want to play along, just install the Test Harness add-on. It is already configured to run the pre-defined pageset and submit the results to Zippity. Just push the button:

Zippity is still evolving and doesn’t have a lot of features. The source code should be in my Mercurial user repo soon. I’ll post more about Zippity and the Test Harness add-on as things improve.

Comparing WebRunner and Adobe AIR

Okay, maybe my tongue-in-cheek, roasting of Ryan Stewart didn’t go over so well. But, at one point in the post/rant I mention working on a project called WebRunner that, IMO, is a competitor to Adobe AIR. I, unsurprisingly, feel that WebRunner does a better job complementing the Web and is a better fit for web applications in general. Both systems allow developers to run rich internet applications (defined as you wish) on the desktop. That’s in itself is both novel and interesting to many developers. However, I think WebRunner and AIR are suited to slightly different applications.

I see Adobe AIR as a desktop runtime for Flash applications. This is a nice boon to Flash developers who have long waited to escape the browser as a runtime. Although AIR supports running DHTML/AJAX/Ajax style web applications through its embedded WebKit renderer, this will likely be overshadowed by the Flash-based applications. Of course, this is just me talking, but most “applications” I have seen running in AIR are Flash-based.

WebRunner, on the other hand, is completely built around a browser – the same browser found in Firefox. Any existing web application should operate in WebRunner without any changes. Create an instant desktop version of your web application. Of course, its not really a desktop version – its just running in a Mozilla browser without any traditional browser chrome. WebRunner doesn’t change how you build web applications. It doesn’t impose a new stack of technologies, it works with the Web.

WebRunner does have the ability to add “features” to web applications such as:

  • Support for simple desktop integrations like displaying alert popups, registering as content handlers for local files, and access to the file system.
  • Greasemonkey-like application scripting to tweak the UI, access the simple desktop integrations or add offline features, not already built into the web application.

Its important to point out that these “features” are completely optional and can be added in a very mashup-friendly manner that most web developers and power users are already using.

There is definitely more to come from WebRunner.

Firefox 3 – Offline App Demo

Firefox 3’s offline capabilities have been getting some attention lately: Chris Double’s post on porting Zimbra to use offline shows-off the potential. Robert O’Callahan and John Resig give details on the different pieces of the capabilities, namely: Offline Cache, Offline Events and DOMStorage.

A Simple Demo – Task Helper

I put together a simple offline application sample to help illustrate how the features can work. Call it – explanation through code. The application is a simple task list system. The current functionality includes:

  • Add tasks – Enter text in field and press ‘Add’
  • Complete tasks – Mark completed tasks and press ‘Complete’
  • Remove tasks – Mark tasks and press ‘Remove’
  • Data is stored as JSON and XHR is used to interact with server (PHP)


Nothing fancy. However, the application is “offline-aware”, meaning:

  • Application resources (HTML & JS) are tagged as offline cacheable
  • Before interacting with server, online status is checked. If online, use XHR. If offline, use DOMStorage
  • Online/Offline status is monitored using events. When the application switches form offline to online, data is resync’ed with the server

There is really nothing fancy about the offline stuff either, but putting it all together does make for a neat application. Using the latest Firefox 3 alpha, you should be able to:

  • Use the application while online.
  • Go offline using the “Work Offline” menu. The event log should show that the app went offline
  • Continue to use the application while offline
  • Go online using the “Work Offline” menu. The event log should show that the app is back online and has updated the server with any offline changes

If you were using a version of Firefox 3 with the offline cache patch (bug 367447) you would be able to a little more:

  • Use the application while online.
  • Go offline using the “Work Offline” menu. The event log should show that the app went offline
  • Continue to use the application while offline
  • Restart browser
  • Switch to offline (or have no network connection)
  • Enter URL to app (or use a bookmark)
  • Start using app with data from last session
  • Switching to online will update the server with offline data

Currently, the offline events only fire when offline is toggled via the “Work Offline” menu. Getting the events to fire by watching the network connection is the goal and would make this much more unobtrusive for the user.

Here is the source to the task helper application: taskhelper-src.zip

XUL Validator – Beta

I put together a simple XUL validator based on the ideas in my last post. There is a primitive front end where you can put XUL into a textarea and validate it. The validator is completely client-side JavaScript with a simple JSON rules file. You can view the source and the rules from the validator “Tests” page.

Currently, the validator checks the structure and semantics of the XUL using the rules given in the last post, with caveats I’ll discuss in a moment. The validator only tests for a subset of the best practices checks. The current tests are listed on the validator “Tests” page. I’ll be adding the rest of the best practice checks soon.

Now, the caveats for the structure checks. There are very few “hard-fast” rules when it comes to XUL. Elements can appear nearly anywhere. Elements can contain any one-off attributes, even made-up ones, which can be useful for CSS selectors. Attribute events, such as “onclick”, can appear on parent elements because events can bubble. The XUL validator, tries to apply some logical warnings and checks, primarily based on the XUL Reference and Tutorial, as well as what I have seen in the Firefox/Thunderbird source code itself.

The current logic is definitely open for discussion and revision! Feedback appreciated.

Update: I made some fixes to the code/rules based on feedback. I also added a small set of options so user can turn off types of checks.

UI Frameworks Are Pigs

Why are UI frameworks pigs? Because they don’t love you back.

You know the story: You meet this new UI framework. It seems so fresh and exciting. You start spending more time together, just doing small stuff. Things are so easy, not forced or boring like with your previous frameworks. So what if it acts a little immature. Before you know it, you’re writing specialized controls from scratch, embedding large amounts of business logic and enjoying every minute of it. You’re head over heels. Next thing you know, you’re crying yourself to sleep and listening to Barry Manilow.

Come on buddy! Snap out of it. UI frameworks don’t love you back. They don’t care about you. You need to watch out for yourself. Protect yourself. Isolate your code.

If your not ready to drop your current UI framework and switch to something else right now because it could take person-years to port, you have problems. If you have team members that love your current UI framework too much to want to change, you have problems.

They don’t love you back.

The experiences above are not about me, but I have this friend…