Responsive Images Meetup – A Subjective Summary

Earlier this week I was at a W3C Responsive Images Community Group (RICG) meetup in Paris. It was a great opportunity to connect with smart folks, some of which I’ve never met before, and I had some great conversations, both as part of the agenda and over drinks.

For those who weren’t able to enjoy the Parisian views, this blog post holds my (not at all official) summary of what took place, plus my opinions of it. Flo also wrote a blog post with his summary, which I highly recommend. I suspect there will be more detailed and accurate reports coming from the W3C RICG.

A bit of background

If you haven’t been following the conversation, let’s start with a quick background on where things stand. This is a short (and again, subjective) summary, you can find the full details on the RICG website,.

The need for Responsive Images can be broken down into three use-cases:

  1. DPR Switching: Serving higher res images only to devices with higher Device Pixel Ratio (aka Retina devices)
  2. Viewport Switching: Serving smaller images to smaller or lower-resolution screens
  3. Art Direction: Serving a different (often cropped) version of an image to a smaller screen, highlighting the important parts of a picture instead of just shrinking it.

Today, responsive images can only be achieved using JavaScript, CSS or server-side detection, which are harder to implement and have performance implications compared to regular img tags. The current efforts by the RICG aim to enhance HTML (or HTTP) to support responsive images without requiring such workarounds. However, there’s more than way to skin this cat…

Here are the primary not-mutually-exclusive alternatives on the table:

  • Picture: an HTML element similar to HTML5’s <video> tag, which uses media-queries to determine which image to load and is uber flexible.
  • srcset: An <img> tag attribute specifying different URLs to load for a given DPR and possibly resolution.
  • Client Hints: HTTP header(s) the browser would send, indicating the client’s DPR (and more), allowing the server to serve the correct image.

There are a few other proposals (like Yoav Weiss’s “Magical Image Format”) which have their own merits, but are not as far along.

Current Status

The first part of the meetup included presentations about real world usage of responsive images today, given by Mat Marquis (Filament Group), Mark McDonnell (BBC News) and myself (Akamai). While interesting, they reviewed the past and present and not the future, so I won’t discuss them further here. The next part of the meetup was dedicated to current status of each of the standardization proposals.

Right now, no browser supports any of these proposals, but there has been some progress in their implementation. There are JavaScript polyfills for picture and srcset, and code patches for Chrome & Firefox were written or are underway. Client Hints is a bit behind, but also has a Chrome Extension to simulate it and a Chrome “intent to implement”.

However, srcset has been getting more love than that. Both WebKit & Firefox implementations of srcset for DPR only (no viewport) are “almost ready”. The Apple rep there couldn’t confirm (or deny), but I got a strong sense that this feature will show up in Safari soon enough.

We also briefly discussed the Resource Priorities proposal, discussed in the W3C Web Perf working group, which suggests new “postpone” and “lazyload” attributes on various elements. Resource Priorities includes a way to avoid downloading hidden images, making it very complementary to srcset (which offers no such method). Note that Resource Priorities is only in draft mode, and has no implementations I’m aware of.

The Jaded “Implementer”

Amongst those who showed up for the meetup were members of the Chrome, Firefox, IE and iOS/WebKit teams, and I found it especially interesting to hear their perspective. Ted O’Connor & David Baron, from Apple and Firefox respectively, were particularly blunt in voicing a somewhat jaded perspective on anything that was proposed… Which was sometimes depressing but still good to have. Below are three concrete cases where such feedback was given.

The picture element was practically sneered at by the implementers. It was seen as a “kitchen sink” proposal, which tries to address every possible use-case for this specific need. This may sound appealing, but some needs are actually broader than responsive images, and should be addressed in a platform wide approach (like Resource Priorities). Other use-cases may be doable in other ways, and others still are simply not important enough to modify HTML for.

When discussing the use of media queries for determining which resource to load, whether as part of the picture element or not, there was… some unease from the browser folks. Media queries are more complex to evaluate, and thus would require more CPU; they can’t always be evaluated by the preprocessor (think about viewport width in an iframe); and they may be enhanced in the future in ways that would be even harder to process. Unlike picture, though, I didn’t feel like using media queries was entirely dismissed. If there’s enough push and value, I think browsers may agree to take this path.

Lastly, on Client-hints, there was push back against adding headers. One explicit statement was that “while we (browser implementers) aren’t saying we’ll never add new headers, we’re only one step away from that”. They pointed out that while some header-based content negotiation techniques worked in the past (e.g. Accept-Encoding), many failed (e.g. Accept-Language, Accept-Charset), leaving a sense that this is a wrong approach. Personally, I don’t think past failures are a reason not to try again, and feel like an opt-in mechanism for Client Hints would mitigate much of the concern.

Srcset – the most likely path

As open as the discussion was, at the end of the day these decisions are at the mercy of the browser implementers. And the implementers like srcset, and have actually been implementing it (as implementers often do). And so, it seems pretty clear that srcset will happen.

DPR support in srcset is likely to start showing up very soon. However, the extended srcset spec, which includes viewport sizes, is not a foregone conclusion. The main problem lies with the hard-to-parse micro-syntax used in srcset, used to describe permutations of viewport dimensions and DPRs. For example, consider this img tag:

<img src=”a.jpg” srcset=”b.jpg 400w 2x, b.jpg 800w 1x, c.jpg 600h 2x”>

This small example demonstrates multiple problems:

  1. The format isn’t easily readable. You need to know, for instance, that the specified dimensions indicate the “max” dimensions (e.g. a screen 400px wide or less, with a DPR of 2x or less), and not a “min” dimension.
  2. There is no way to modify the definition to be a “min” dimension, which conflicts with how “mobile first” media queries are written.
  3. The format requires repetition of URLs. In this case, b.jpg was created for screens that are 800px wide. This includes both a 400w/2x screen and an 800w/1x screen. Long URLs and multiple permutations can really make the attribute long.
  4. The mix of height and width create ambiguity. For instance, if a 2x device was 400×600, should it load b.jpg or c.jpg? The algorithm dictates what to do, but it’s not at all intuitive.
  5. The resolution is always specified in pixels (implied), not supporting large font sizes or zoomed in views.

This conversation was, in my opinion, the most productive one in the meetup, and came up with a few conclusions that most people agreed with:

  • For #1, we’ll just deal with it through education. It’s not brain surgery.
  • For #2, again, we’ll just deal with it. You can still describe the same conditions, albeit by using funky numbers like 3.99.
  • For #3, we discussed supporting OR conditions, but eventually dropped it. We don’t want to over complicate the format, and gzip compresses repetitions very well anyway. We also discussed using media queries, but – as mentioned above – that didn’t go over well.
  • For #4, everybody agreed we should drop “h”, but keep “w”. This makes the ambiguity go away, and nobody could figure out a strong need for “h” anyway.
  • For #5, we could support other resolution units, but there wasn’t much discussion about it.

One important clarification is that srcset is a recommendation, not a rule. This means a browser can choose to download a 1x image despite being on a 2x display. It may do so to preserve battery life, reduce bandwidth, accelerate the page, and may do so automatically or based on user preferences. This is different than media queries, which the browser must obey.

This flexibility means all images in a srcset should hold the same image, differing in quality only, since you can’t be certain which one the browser would load. This, in turn, means srcset cannot be used for Art Direction – which gets me to the next topic…

Not heading in the Art Direction

Some of the conversation was around priorities. Pretty much everybody agreed that the top priority was addressing high DPR (Retina) displays, and most agreed that handling smaller viewports was a close second. However, there was a strong lack of conviction about the Art Direction use-case.

The most common use-case for Art Direction is cropping – only showing a portion of an image on a smaller screen. Cropping can be achieved by serving a different image, but it can also be achieved with CSS overlays, showing only a portion of the downloaded image. This approach means you’ve downloaded wasted bytes, but if the image is already adjusted to the client’s resolution, it’s unclear just how much you’d save with server-side cropping.

For the use-case of showing different images in different viewports, you would need to stick to JavaScript or CSS. The sense in the room was that this is not a strong enough use-case to justify modifying HTML for it.

At some point I suggested adding an optional marker (in a separate attribute or in the micro-syntax) that forces the browser to respect srcset – let’s call it the strict marker for now. The strict marker would allow us to serve different images to different devices, including art direction needs. The image we serve would need to cover all the viewports in this device (e.g. landscape & portrait), but that’s still smaller than the full image in many cases. We debated this a bit, but didn’t reach any final conclusion.

Personal Perspective

While the conversation was very interesting, we didn’t really change the direction much… Browsers will stay on course implementing DPR for srcset (likely to show up soon), and my expectation is that they’ll add support for viewport width shortly after.

Personally, I arrived at the meetup favoring picture, but left it favoring srcset… assuming we add viewport and the strict marker, and that we implement Resource Priorities. With that combination, I believe we tackle the main performance needs: serve low res images to low res devices (srcset); avoid excessive client-side cropping (strict); and avoid downloading hidden images (Resource Priorities). I think it’ll be nice if we also added support for non-pixel viewport definitions.

I still think there’s room for Client Hints, especially if you want your images optimized through an automated tool, like Akamai’s FEO or Pagespeed. In addition, Client Hints is useful for more than just responsive images, and we need to judge it based on its complete value, not a specific hint.

All in all, I found the meetup very educational, and – while they were somewhat antagonistic – I saw a fair bit of merit in the implementer remarks. Hopefully you find this summary useful as well.

Posted on September 13th, 2013
  • tiebasq

    Browser vendors are such crybabies. Handling of fiew source elements is not hard. They’re implementing amazing JS engines, video codecs and GPU acceleration and …find getElementsByTagName(‘source’) hard. What a joke.

    • Tab Atkins Jr.

      You seriously think *that’s* the part they find hard? No, it’s handling all the corner cases around dealing with mutations/additions/deletions of elements at all points of the document’s timeline, particularly while you’re in the middle of parsing the element and its children. These corner cases aren’t *useful*, but they’re still necessary to define and properly implement unless you want random behavior from browsers.

      In contrast, attributes (whether one or several) are much easier, because they’re parsed all at once.

      I mean, for chrissakes, browser vendors aren’t *lying* when they say it’s hard. It’s not even really *hard*, per se – it’s just very fiddly and complex for little to no benefit over alternative syntaxes.

    • Guypo

      The concern they voiced wasn’t implementation complexity, but rather processing time and future usage. Once you add something to a browser, you pretty much can never take it back, because some websites would be using it. This means any change needs to be very carefully considered, since you’ll need to live with it forever.

      Another factor is that some problems are not unique to responsive images, like lazy loading resources, and so it may be better to provide an HTML-wide solution for it, as opposed to a specific solution that only addresses part of the overall problem.

      That said, I do think they’re being over-cautious of some of these proposals.

  • labomat

    Can you elaborate what you mean by “You can still describe the same conditions, albeit by using funky numbers like 3.99″ regarding the lack of min-width syntax? How do numbers like 3.99 help with the mobile first approach?

    • Guypo

      You can describe any breakpoint spit with either min-width or max-width. For instance, if you want style A for screens under 400px wide and style B for larger screens, you can express that with either min-width: 400px or max-width:400px.

      The only gap is the actual width where you draw the line. In this example, devices that are exactly 400px wide would get style A when you use max-width, and style B when you use min-width.

      However, you can use max-width: 399px to achieve something almost equivalent to min-width: 400px. It’s less elegant, but it works.

      I hope that helps clarify it.

      • labomat

        That means if I want to have a.jpg on small screens an b.jpg on screens bigger than 400px I have to write something like:
        img src=”a.jpg” imgsrc=”a.jpg 400w 1x, b.jpg 9999w 1x ?

        • Guypo

          No – the src attribute is used if no srcset condition is met, or you could use a srcset condition that uses only DPR

          • labomat

            Hm. In my example, at screens < 400px the 9999w is met (because it's like max-size). So if I don't supply a 400w for a.jpg b.jpg would be used on any screen size, wouldn't it?

          • Guypo

            Sorry if I wasn’t clear – your example would indeed achieve what you stated (except on ridiculously big screens), it’s just not the most concise way of doing so.

            For starters, you can achieve it by omitting the width parameter. Here’s the text from the spec: ‘omitted width descriptors and height descriptors are considered to have the value “Infinity”‘.
            So you could use:

            Second, while the spec text is very confusing, I’m pretty sure the browser would use the “src” attribute if none of the srcset conditions matched, meaning you’d be able to achieve your goal with this text too:

            Small comment: your example uses imgsrc instead of srcset.

          • labomat

            imgsrc Ups :-)

            Ah – I didn’t read the infinity part. Your last part would work as expected only for browsers understanding the scrset syntax. All other browsers would use b.jpg what is not as intended mobile first.

  • Tom

    Seems like it would be useful to consider the img src attribute as an implied base for srcset. that would allow would allow a400.jpg to be served from the same path as a.jpg without repeating it.