Real World RWD Performance – Take 2

About a year ago, I ran a test comparing the performance of websites using Responsive Web Design (RWD) across different screen resolutions. The test consisted of loading the same RWD sites in Chrome using 4 different screen resolutions, triggering different “views” of the site, and comparing bandwidth and load times in each.

The test results didn’t bode well. They showed that practically all RWD websites downloaded the full payload of the desktop website regardless of what was displayed. These websites would download hundreds of KB of unnecessary data to a small screen, only to hide or shrink them on the client, resulting in abysmal performance on mobile devices.

Over the past year, RWD performance has been discussed at length, with great posts from Jason GrigsbyTim Kadlec and many others. New tools such as PictureFill and Adobe Shadow make optimization and testing easier, and some of the conversation even made it into the HTML5 draft.

So, with all this progress, I figured it’s time for a retest – are we faster yet?

Quick Test Details

Similar to last year, I used http://mediaqueri.es/ as a repository of responsive websites. using WebPageTest, I loaded each of the 471 websites currently inventoried (an increase from 347 sites last year) in a Chrome browser, resized to one of 4 different screen resolutions each time. I then collected the results, and looked at the number of bytes needed to download each page in each resolution.

You can see more details on the test methodology in last year’s presentation, and can rerun the test on a given site with the instructions on this slide.

Problem Solved? Not really.

The results showed that, despite the better tools and conversation, most RWD websites still downloaded the full content of the page on every screen resolution. As you can see from the chart below, RWD pages loaded on the smallest screen were only 9% smaller than those on the largest screen (on average), despite huge differences in the amount of content shown and the physical size of images.

2013-page-size-per-resolution

Looking across websites, less than 7% of websites were at least 2x smaller when loaded on the smallest screen compared to the biggest screen. Another 22% weighed between 50-90% of the large-screen page size when loaded on the smallest one, and the majority (72%) were roughly the same size on the smallest and biggest screens.

2013-page-size-small-vs-big

Not good, but better

While these results are definitely not good, they’re still better than what we saw last year. As you can see in the chart below, the percentage of websites that are much smaller or slightly smaller in 2013 was double what it was in 2012. In addition, the average page in 2012 weighed only 6% less on a small screen than on a big screen, compared to 9% today – again a notable improvement.

This means the enhanced evangelism and improved tooling for RWD performance did have an effect, which is a significant achievement. That said, we probably need this growth to be exponential, not linear..

trend-page-size-small-vs-big

Unfortunately, while the cross-resolution page size stats improved, the overall performance did not. The HTTP Archive clearly shows how pages are constantly getting bigger, and this trend holds true for RWD sites as well.

The chart below (last one, I promise!) shows the average page size by window size. As you can see, the average page size has increased significantly across all the resolutions. Note that the sites included in each year’s averages are not exactly the same, but this trend holds even if we only look at the URLs shared across the two tests.

trend-page-size-per-resolution

Summary

The bottom line is – our work is not done. We’re seeing some fruit of our performance preaching labor, but the vast majority of RWD sites still don’t devote enough attention (or time) to performance. As a result, if you happen to browse a responsive website on your smartphone, chances are it’ll be considerably slower than if you were browsing a dedicated mobile site.

We need to keep pushing and promoting RWD performance, keep building tools and making it easier to be fast, and find a way to improve RWD performance faster than the pace we make pages slower in general…

So for all my fellow performance enthusiasts out there, keep going, we’re not there yet!

Side Note: If you’re interested in the raw test data, you can download the detailed excel sheet with test results, pivot tables and links to the individual tests here.

 

Posted on March 7th, 2013
  • Tjorriemorrie

    So the performance is improved with:
    1) smaller pictures
    2) less content

    Is that right? I was thinking that I can’t really strip out any of my css/js files.

  • http://christian-fei.com/ Christian Fei

    Today I set my goal to decrease the size of my website, and my first thought was to trim the image fat. I totally agree when you say that the battle is half won, so thanks, I’ll follow your guidelines :)

  • Pat O’Callaghan

    Interesting results. Have you considered using different user agent strings in your test rather than just resizing the browser? One thing is that some sites may contain a RESS component which alters markup or resizes images on the server-side using UA sniffing rather than the client side.

    • Guypo

      In my test last year I spot-checked quite a few sites with a real iPhone and iPad, using Mobitest, and they gave me the same results as the resized window.

      I didn’t do this sanity check this year, but I doubt it’ll show different results on any substantial number of websites… You’re more than welcome to download the excel and try out a few of the websites yourself if you disagree.

  • http://twitter.com/paddy2k Paddy O’Reilly

    At dotMobi we’re developed a too to help developers test their page size across multiple devices. As your post shows, we’ve found that many RWD sites offer no page size optimisations for smaller screens.

  • anynigma

    Awesome analysis. What do you think are the biggest barriers to increased benefits of RWD? You mention better tools. Do you have an example of a very well implemented RWD site? Just one nit-picky suggestion, you used blue for the 2012 bars on the “Page Size on Smallest vs. Biggest Screen” chart, but then changed to red on the next chart, and reversed the order of the data (2013 data is now to the left of 2012). Just threw me for a loop, and I wanted to let you know.

    • Guypo

      Sorry about the colors – I didn’t even notice it till you pointed it out!

      As for an example of a good site, it’s a bit alarming but I don’t have an actual role model to point out.

      http://ethanmarcotte.com/ does a good job staying lean and using responsive images, but it’s pretty light in content/function, so it’s not a good example of how to keep a complex RWD site fast.

      http://vogue.co.uk/ does a good job decreasing payload size as screen size shrinks, but even on the small screen its over 2MB in size, so it’s not a good example of a performant site.

      http://www.kavaruzova.cz/ does a good job adjusting weight to screen size, and is fairly lean, but it does so using a blocking JavaScript file, which makes performance across all screens less than stellar.

      This is actually a very good question and a very concerning answer – I’ll post it on twitter and see if others can point out better examples

  • http://tjm.org Tim McCormick

    It seems that total page file size isn’t necessarily a good indication of load time or user experience. Many sites, particularly any monetized “content” site like news, are huge assemblages of items from dozens of sources which may vary widely and unpredictably in latency. Ad serving networks are notorious for adding long delays to page load. What type of component and how it’s incorporated may make all or part of the page stall loading until it’s done.

    Even more broadly, I’d suggest that ubiquitous and continual mobile bandwidth should be considered an ideal, not normal scenario. Connectivity fluctuates and interrupts all the time, and even if realtime network information is not available by API to Web design elements, the general design approaches might take this reality into account much more than they seem to now. I think user experience can potentially be greatly improved by maximizing pre-loading and offline storage, e.g. via HTML5.

    My personal experience is that a large portion of all Web material is now unusable on mobile, and I go to it only as a last resort after apps & offline stored content. It seems Web design has substantially not made a transition to the mobile web yet, and “responsive web design” has mostly focused on layout issues and not network conditions, while encouraging publishers to stop doing dedicated mobile site versions that *did* work well.

    —-
    Tim McCormick
    @tmccormick tjm.org Palo Alto, CA