Update: Nov 16th 2015 - Added an extra row in the table for React under production conditions. The good news: it's 3x slower than vanilla, yes, but in actual terms I'd say it's fast for TodoMVC! The Polymer TodoMVC sample was also updated to version 1.2.2 today, and that, too, is faster.

If you prefer watching to reading, here's the video of the talk (you can also get the slides, too, if you like):

If you prefer reading to watching, well, keep reading...

The benefits of frameworks #

Earlier in the year I wrote about React's performance characteristics as the tree size it has to manage gets larger (TL;DR the bigger the tree, the more computation work it has to do). The responses I got to that post really varied, from appreciative and constructive to, well, the other end of the spectrum. The responses gave me some additional insight, though, which I think applies to all frameworks (to a greater or lesser extent), and which I'll now try and summarize:

• Frameworks are fun to use. I don't think I know any developer who actively seeks out a more challenging and less fun experience! And yes, absolutely, ergonomics are important.
• Frameworks make it quicker to build an MVP. This one is important, particularly for smaller, fledgling organizations or companies. Having a fast bootstrap may be the difference between shipping and not shipping.
• Frameworks work around lumps / bugs in the platform. I think this is less of an issue these days (though not irrelevant) as browsers have rallied around standards very well in the past few years. Certainly this is less painful compared to my early career, where we were dealing with IEs 5 and 6, and early versions of Firefox, Safari and, of course, Chrome.
• Knowing [insert framework] means I get paid. If you're looking to get hired, saying that you know the latest and greatest framework(s) may be the difference maker.

The key message I heard over and over, sometimes explicitly, and often implicitly, is that ergonomics are the most important factor for many developers.

The key message I heard over and over, sometimes explicitly, and often implicitly, is that ergonomics are the most important factor for many developers. If I can paraphrase, the sentiment was "if I have an easier time I'll build something better for my users". At face value that sounds appealing, but I think it misses a lot of important points.

Developer Ergonomics vs User Needs #

Users have needs, too, though. Here are a few of them that come to mind:

• Sites and apps should load quickly.
• They should have smooth interactions.
• They shouldn't slow down my phone.
• They shouldn't crash.
• They should have features I want.

So immediately we're into a potential trade-off situation: on the one hand we have our convenience, our ergonomics, and on the other hand we have our users' needs. And we have to trade, whether we like it or not, since frameworks aren't free.

A quick side step #

At this point I'm going to park libraries, and leave them out of this discussion. The reason is that, in my view at least, libraries can be swapped out and replaced if they prove problematic. Don't like the way a library does date formatting? No problem, just switch it for something else. Frameworks, on the other hand, tend to be a lot more difficult to swap, often requiring a rebuild of the app in question. And, by their nature, they're a lot bigger and more involved.

The cost of code #

All code has costs, but I think there are particular costs that are unique to frameworks.

• Learning it. You have to take time out to learn any new framework: what it means to write "idiomatic" code for that framework; its patterns, practices, and ways of doing things. This is no different to the web platform itself. There's no free ride, you're just swapping one set of issues for another.
• Re-learning it. One day you're just bimbling along, developing your app, and in the console you see a warning telling you that there's been a deprecation, and that you need to update your code. That often requires figuring out what the latest version of the framework needs you to do differently, and that may also mean updating projects that shipped with older versions of the framework.
• Debugging it. Frameworks, like any software, have bugs. That's to be expected, and it wouldn't be any different if you went vanilla and wrote your own. Debugging a framework, however, often ranges from difficult to impossible. If you find the issue, you may have to keep your own patched version until a "real fix" lands in the framework's production build.

Aside from the developer costs, there are costs to the user, too:

• Time. How long does it take to bootstrap the thing we sent down the wire?
• Bandwidth. How much bandwidth does it use up?
• CPU usage (battery). How CPU intensive is it? CPU utilization is going to correlate to battery usage: burn the CPU and the battery will get drained.
• Frame rate. How well does the code update the screen? (We know that users like smooth interactions.)
• Memory usage. How much memory does it use? Will it cause other apps to be evicted from memory? Perhaps even crash a tab?

Bringing data to the table #

So with all that in mind, I wanted to start off a conversation about measuring some of those costs, and my hope is that together we can build up more of a picture about the trade-offs we choose to make.

I decided to take a look at the bootstrapping time for frameworks on mobile. The theory being that, given a like-for-like app, one should be able to see how long it takes each framework to boot. I opted to use TodoMVC for the test, because in my view it's an MVP web app, and from a user's point of view, each variant is functionally identical.

Bootstrapping / Time-to-interactive #

Of the costs listed above, I decided to take a look at the Time, Bandwidth, and CPU usage for booting up some frameworks on a Nexus 5 and iPhone 5S.

I decided to measure the time it takes to evaluate and execute each framework's initial payload of JavaScript, and the processing and setup of the initial data set into a model. I decided to ignore the cost of styles, layout, paint and so on since they shouldn't vary all that much between the different TodoMVC implementations. I was also not testing transfer time.

Methodology #

I used WebPagetest to load the pages on a Nexus 5, and I requested a timeline file for each run, which I then passed to Big Rig for processing. The iPhone results I did myself, and had to tally up manually, because unfortunately Safari's JavaScript profiling doesn't seem to support any timeline / trace file export.

This, incidentally, is why I built Big Rig. I wanted to make it easier for all of us to start processing this kind of data quickly.

Repeating the test #

If you want to do your own test like this, at least for a device running Chrome, you should do the following:

• Install the Big Rig CLI: npm install -g bigrig.
• Go to WebPagetest.
• Set the location to Dulles.
• Set the browser to Nexus 5 - Chrome (or Beta/Dev).
• Under the Chrome settings, check "Capture Dev Tools Timeline".
• Run the test.
• Download the Timeline file on the left hand side of the results.
• Run bigrig --file /path/to/timeline.json --pretty-print.

If you prefer, you can watch me getting a Timeline file from WebPagetest:

The results #

Here's how the various frameworks stacked up in my tests:

Framework	Size	Bootstrap time N51,3	Bootstrap time iPhone 5S2,3
Polymer v1.1.4	41KB5	409ms	233ms
Polymer v1.2.2	47KB5	155ms	232ms
AngularJS v1.4.3	324KB	518ms	307ms
React v0.13.3 [JSX not transformed]	311KB	1,201ms	1,463ms
React v0.13.3 [JSX transformed via Babel]4	162KB	509ms	282ms
React v0.13.3 [JSX transformed; prod build]4, 6	160KB	174ms	118ms
Backbone v1.2.2 [inc. jQuery & Underscore]	139KB	248ms	168ms
Ember v1.10.0-beta.3	580KB	1,992ms	1,440ms
Vanilla	16KB	50ms	33ms

1. Tests done on a Nexus 5 running Chrome 47.
2. Tests done on an iPhone 5S running Safari 9.
3. All bootstrapping time includes handling initial todo list data.
4. JS Transformer stripped; JSX files transformed via Babel.
5. Excludes Web Components Polyfill (12KB).
6. React switched for minified production build.

For me the results are pretty clear: there appears to be a pretty hefty tax to using Frameworks on mobile, especially compared to writing vanilla JavaScript. The fastest is Polymer 1.2.2, which is awesome, but it's still 3x slower than Vanilla. React is pretty similar to Polymer, but I still have concerns over its scaling properties.

To make things super clear, here are a couple of notes:

• TodoMVC doesn't JSX Transform React, so I did. The reason there are three entries for React is that the TodoMVC example doesn't transform JSX, and instead includes the JSX Transformer library. To make things fairer (because the runtime JSX transformation is expensive!), I made a more 'productionized' version of the example, and tested that, too. The downside is that it's not the minified version of React, which is different again. So I also switched out the build of React for the production version, and that's the third build in the table.
• These times do not include transfer time. The only thing I was measuring was the time in JavaScript to bootstrap the framework and get the initial view built. In fact, some of the frameworks in TodoMVC aren't minified. For a more discussion on framework transfer sizes, check out the Filament Group's post from last year.

Possible Objections #

I figure there's likely to be some objections to the test, which are definitely worth talking through:

I’d argue your use-case is probably more expensive than TodoMVC.

• "TodoMVC isn’t idiomatic." I checked this one out, and from what I understand every implementation is idiomatic, and framework experts file PRs against them when they aren't.
• "TodoMVC isn’t my use-case." That's probably true, but I'd argue your use-case is probably more expensive than TodoMVC. Paul Irish recently did a performance audit on Reddit's mobile site, and he discovered, amongst other things, that 22 seconds were going into just booting up React's components on mobile!
• "A Nexus 5 / iPhone 5S isn’t what our users use." That's also probably true. I have the privilege of using good hardware, and I would imagine that many people don't have access to top end phones, so I'd expect these numbers to be even worse in those cases.
• "It’ll be better in the next version of <insert framework here>." This may well be true, but it doesn't do much for anything you've shipped with the current version.

The $64k question #

So, inevitably, the question comes: "Should you use a framework?"

I can't answer that question, because I think it's entirely your call. There are a million and one reasons why you may feel you need to use one. But, for what it's worth, here are my thoughts:

• Frameworks contribute ideas and concepts. Frameworks are a crucial part of understanding what approaches work, and what don't, and that ultimately drives improvements at the platform level. In that regard I think they serve as important testing grounds for the platform changes of tomorrow and let us figure things out before they become embedded in the web permanently.
• Frameworks are an inversion of control. The reason I parked libraries earlier on is that you can swap them out. Frameworks, on the other hand, invert control. They control the lifecycle of the app, and give you entry points where your code runs. You're still responsible for the final code, but you're not in control.
• Frameworks are expensive on mobile. Compared to vanilla, at least. For me, prohibitively so, but everyone has their own limits of what they're willing to tolerate.

Investing in knowledge of the web platform itself is the best long-term bet.

As I look at frameworks, I see the ergonomic benefits (and those are important, I agree!), but I can't help but feel that, for many developers, investing in knowledge of the web platform itself is the best long-term bet. Frameworks come and go, it just seems to be the ebb and flow of the web, and, as I said above, they do contribute ideas and patterns. But if you ever find that the one you use no longer works for you, or has a bug that remains unfixed, being able to understand the platform that underpins it will help enormously.

The bigger conversation #

When I wrote the post earlier in the year about React, I said this:

It seems to me that developer ergonomics should be less important than our users' needs.

I still believe that. As much as I love the idea of having an easier life, I don't want to ship something to people that doesn't run well, and I don't want them to pay the cost. Today I'm concerned by the costs of booting up frameworks on mobile.

I'm concerned by the costs of booting up frameworks on mobile.

This is only a first step. There are other metrics besides bootstrapping I've barely had chance to look into: memory usage, long-term CPU usage, and impact on frame rate to name three. Overall, I think we have more to do to properly assess the impact of code that we ship to our users and the costs we pass on to them.

If we can achieve fast-booting, low-memory, smooth-executing frameworks with good ergonomics we'll be onto a winner. Until then, for mobile at least, I'll be sticking to the vanilla web platform.