Bad Benchmark, Right Result

A benchmark is a benchmark, right? How wrong can it be? Turns out I got hoodwinked by one, but even when I unpicked it I discovered some interesting bugs. Come with me on a voyage of API discovery!

  • Last updated: 1 Dec 2013
  • Est. Read Time: 7 min
  • Tagged: #perfmatters, #scrutiny

The other day WebKitCSSMatrix was pointed out to me. If you've never come across it it's a really useful way of manipulating matrices for 3D transforms. It's available in Chrome, Safari and Opera. Firefox doesn't seem to have the corresponding Moz- or unprefixed version, and Chris Love informs me that Internet Explorer has MSCSSMatrix.

In short you create one of these matrices and then you can manipulate it like so:

// make an identity matrix
var matrix = new WebKitCSSMatrix();

// access any of the 16 components
// in the 4x4 matrix. Let's change
// the Y translation to 20.
matrix.m42 = 20;

// or use some built-in functions
matrix = matrix.translate(0, 20)

// although they make a new matrix
// which is bad for GC...

Now you have this matrix all you need to do is apply it to an element's transform:

div.style.WebkitTransform = matrix;

If you want to change the matrix's components after apply it you can do that, but you need to set the element's transform again once you do. Sadly, you can't just grab the current WebkitTransform and manipulate it (shame, I know) because you'll just
get a string to work with.

There was also a comment from a Chrome engineer on a Chrome bug that I found interesting:

[They should be] us[ing]: style.webkitTransform = new WebKitCSSMatrix("matrix(blah, blah)") or using the attributes to set only the pieces of the matrix they need w/o going through the CSS Parser.

This was suggesting the use of CSSMatrix in contrast to the (virtually-always-used) alternative of creating strings all over the place like so:

div.style.WebkitTransform = "translate3d(" + x + "px, " + y + "px, " + z + "px)".

I bounced over to Twitter all excited, because from a developer point of view this is clearly way freakin' awesomer than the string gymnastics.

The benchmark #

However, when I visited the benchmark in question I saw problems. Roughly speaking it appeared 70% slower to use WebKitCSSMatrix than to create strings. On the other side I had the comment in the bug, which definitely indicated that using a matrix should be faster than passing in a string. The theory being that internally we can detect the use of the matrix and can bypass the CSS string parser.

Scrutiny #

It shouldn't have taken me an hour to figure it out why the jsPerf benchmark was squiffy, but it did. The clue lay in the benchmark's preparation code. See if you can figure it out faster than I did.

In case you couldn't see it, here's what I (eventually) realised: the element doesn't exist in the render tree, so any style changes will be ignored. This test is therefore akin to checking whether it's faster to set a property of an object with a string or another object!

When the element has been added to the DOM and exists in the render tree, we can actually test how CSSMatrix compares to using strings because we can measure the time spent in recalculating the styles. In theory using the matrix should be way faster (no CSS string parsing, yo.)

In the end I created my own quick tests (one using strings, one using CSSMatrix) that animated 1,000 visible DOM elements, all in the hope of clearing matters up.

But all was not as it seemed... (Ooh, feel that tension.)

The right result #

What is already a budget buster for desktop would be an fps killer on mobile.

I'm on a decent Macbook Air and I was seeing ~20ms of recalc style for 1,000 elements in both the string and CSSMatrix versions. On mobile that would be in the region of 6-8x, so basically what is already a budget buster for desktop would be an fps killer on mobile.

The chart below shows the cost of Recalculate Style and time spent in JavaScript vs the number of DOM elements when you use strings:

But now see what it looks like for CSSMatrix:

There are two things to notice here:

  1. The JavaScript cost of using CSSMatrix is really high in Chrome. I did a little digging here and it's definitely where you go and assign the matrix to the element's transform.
  2. Probably related, but when I used the Chrome DevTools flame chart I noticed that a toString() method was being called inside my requestAnimationFrame callback. That wasn't from me, so it appears internally Chrome calls toString on the matrix and passes it over to the CSS parser, which is exactly what we were hoping to avoid.

Therefore because we're essentially bouncing down to a string here the Recalculate Style cost is virtually identical to setting a string directly. But for some reason (must confess here, I don't fully understand), the JavaScript cost of CSSMatrix is higher than using a string.

Conclusion #

For me this was an interesting journey. Having to rationalize seemingly contradictory claims was fun, and actually getting to the bottom of it was satisfying, even if the conclusion was the same: use strings not CSSMatrix. I've filed a bug with all my findings (where Eric Seidel, one of our engineers has confirmed the toString() theory... can I call that my String Theory? No? Too confusing. Gotcha.) and my hope is that our engineers will be able to allow CSSMatrix to be faster than it is today.

But here's a super critical point: if you benchmark visual APIs, make certain they operate on visible DOM elements that reside in the render tree.

Share this on: Twitter