Bad Benchmark, Right Result
A benchmark is a benchmark, right? How wrong can it be? Turns out I got hoodwinked by one, but even when I unpicked it I discovered some interesting bugs. Come with me on a voyage of API discovery!
The other day WebKitCSSMatrix was pointed out to me. If you’ve never come across it it’s a really useful way of manipulating matrices for 3D transforms. It’s available in Chrome, Safari and Opera. Firefox doesn’t seem to have the corresponding
Moz- or unprefixed version, and Chris Love informs me that Internet Explorer has
In short you create one of these matrices and then you can manipulate it like so:
// make an identity matrix var matrix = new WebKitCSSMatrix(); // access any of the 16 components // in the 4x4 matrix. Let's change // the Y translation to 20. matrix.m42 = 20; // or use some built-in functions matrix = matrix.translate(0, 20) // although they make a new matrix // which is bad for GC...
Now you have this matrix all you need to do is apply it to an element’s transform:
div.style.WebkitTransform = matrix;
If you want to change the matrix’s components after apply it you can do that, but you need to set the element’s transform again once you do. Sadly, you can’t just grab the current
WebkitTransform and manipulate it (shame, I know) because you’ll just
get a string to work with.
There was also a comment from a Chrome engineer on a Chrome bug that I found interesting:
[They should be] us[ing]: style.webkitTransform = new WebKitCSSMatrix(“matrix(blah, blah)”) or using the attributes to set only the pieces of the matrix they need w/o going through the CSS Parser.
This was suggesting the use of
CSSMatrix in contrast to the (virtually-always-used) alternative of creating strings all over the place like so:
div.style.WebkitTransform = "translate3d(" + x + "px, " + y + "px, " + z + "px)".
I bounced over to Twitter all excited, because from a developer point of view this is clearly way freakin’ awesomer than the string gymnastics.
Kris Gustaveson responded to my initial tweet with a query:
Sure enough, when I visited the benchmark in question I saw exactly the same thing as Kris. Roughly speaking it appeared 70% slower to use
WebKitCSSMatrix than to create strings. On the other side I had the comment in the bug, which definitely indicated that using a matrix should be faster than passing in a string. The theory being that internally we can detect the use of the matrix and can bypass the CSS string parser.
It shouldn’t have taken me an hour to figure it out why the jsPerf benchmark was squiffy, but it did. The clue lay in the benchmark’s preparation code. See if you can figure it out faster than I did.
In case you couldn’t see it, here’s what I (eventually) realised: the element doesn’t exist in the render tree, so any style changes will be ignored. This test is therefore akin to checking whether it’s faster to set a property of an object with a string or another object!
When the element has been added to the DOM and exists in the render tree, we can actually test how
CSSMatrix compares to using strings because we can measure the time spent in recalculating the styles. In theory using the matrix should be way faster (no CSS string parsing, yo.)
But all was not as it seemed… (Ooh, feel that tension.)
The right result
What is already a budget buster for desktop would be an fps killer on mobile.
I’m on a decent Macbook Air and I was seeing ~20ms of recalc style for 1,000 elements in both the string and
CSSMatrix versions. On mobile that would be in the region of 6-8x, so basically what is already a budget buster for desktop would be an fps killer on mobile.
But now see what it looks like for
There are two things to notice here:
CSSMatrix is really high in Chrome. I did a little digging here and it’s definitely where you go and assign the matrix to the element’s transform.
2.Probably related, but when I used the Chrome DevTools flame chart I noticed that a
toString() method was being called inside my
requestAnimationFrame callback. That wasn’t from me, so it appears internally Chrome calls
toString on the matrix and passes it over to the CSS parser, which is exactly what we were hoping to avoid.
CSSMatrix is higher than using a string.
For me this was an interesting journey. Having to rationalize seemingly contrary claims was fun, and actually getting to the bottom of it was satisfying, even if the conclusion was the same: use strings not
CSSMatrix. I’ve filed a bug with all my findings (where Eric Seidel, one of our engineers has confirmed the
toString() theory… can I call that my String Theory? No? Too confusing. Gotcha.) and my hope is that our engineers will be able to allow
CSSMatrix to be faster than it is today.
But here’s a super critical point: if you benchmark visual APIs, make certain they operate on visible DOM elements that reside in the render tree.