WebM vs. H.264: A First Look

Quality Trials
When WebM was first announced, I compared a WebM file against an H.264 file produced by Sorenson Squish. I concluded that "H.264 still offers better quality, but the difference wouldn't be noticeable in most applications." Now, I've spent a bunch of time producing both formats, and I've reached the same conclusion.

Note that Sorenson Squeeze uses the MainConcept codec, which has been the highest quality commercially available codec in my comparison tests. To supplement these tests, I also produced comparison files with the x264 codec, using the QuickTime-based x264Encoder version 1.2.13 (dated 6/27/2010) set to the highest quality, slowest encode preset (Figure 3).

In my tests, x264 produced slightly higher quality than the H.264 files produced by Squeeze with the Main Concept encoder, which was slightly better than WebM (Figure 4). In my view, however, slight differences in quality are irrelevant if the typical viewer wouldn't notice the difference absent side-by-side comparisons at normal data rates.

To explain, I produce my 720p test file at 800Kbps, which means that I'm allocating .029 bits per pixel in the 29.97 frame per second file. In comparison, YouTube produces its 720p H.264 video at about 2Mbps, which means an allocation of .072 bits per pixel, 2.5 times higher than mine. Why are my test files compressed so highly? Because if the data rate is high enough, all technologies look good, and it's impossible to differentiate.

What's the most relevant test? In my recent review of video files produced by a range of broadcast and corporate sites, the lowest bits per pixel allocation that I found was 0.43, with most well above .07. Apple produces its iPad advertisements at .168 bits per pixel, about five times higher than my test file, while ESPN produces at .173 and CNN at .106. The notoriously penurious Tiger Woods publishes at .136, though perhaps he'll tighten this up after losing $750 million in his divorce.
Long story short, if YouTube produces its WebM-based 720p files at 2Mbps at 720p, only the most discriminating viewer will be able to distinguish it from H.264, and then, only with side-by-side comparisons, which, of course, viewers never have. It's not whether one technology is better than another; it's whether it's sufficiently better to make a difference for the typical viewer.

I should point out that some highly respected sources don't share this opinion. For example, the Graphics and Media Lab of Moscow State University produces a codec comparison every year; this year it included VP8. In terms of H.264 encoding quality, the report concludes that "[t]he x264 encoder demonstrates better quality on average, and MainConcept shows slightly lower quality," which was my primary motivation for including x264 in this evaluation. Regarding VP8, the report concludes, "When comparing VP8 and x264 VP8 also shows 5-25 lower encoding speed with 20-30% lower quality at average." I just didn't see that.

Then there's x264 developer Jason Garrett-Glaser's extensive analysis of VP8 and comparison to x264, which concludes that x264 is 28% better than VP8, though his comparisons seem to focus on 1080p delivery, so it's unclear how much you can generalize these results to streaming. In any event, Garrett-Glaser's analysis is wonderful reading for anyone who wants to understand the inner workings of the VP8 codec and WebM spec, as well as the patent issues that WebM may be facing.

Both these comparisons rely primarily on automated quality measurements such as Peak Signal to Noise ratio (PSNR) and Structured Similarly (SSIM), which compare the encoded frame to the original and produce a comparative numerical score. I produce the files with the different technologies, making sure that they're within 5% of the target data rate without dropped frames. I then grab frames for comparison purposes and watch the files side by side to assess the presence of motion artifacts. You can view my HD comparisons at www.doceo.com/HD_Comps.html and my SD comparisons at www.doceo.com/SD_Comps.html, and comparative frame grabs are at www.streaminglearningcenter.com. Draw your own conclusions.

Overall, I'm sure that Garrett-Glaser can coax more quality out of x264 than I can. But find comfort in the fact that the Moscow study concluded that MainConcept "shows slightly lower quality" than x264, which is consistent with my results. Certainly, if you're using a MainConcept-based tool such as Squeeze, the quality difference between VP8 and H.264 will be meaningless at most relevant data rates.

Playback Requirements
Which takes us neatly back to playback requirements, which is where we started. Let's introduce the computers and browsers.

Briefly, I tested on four computers. The MacBook Pro has a 3.06GHz Core 2 Duo CPU, 8GB RAM, and was running OS 10.6.2 with an NVIDIA GeForce 9600M graphic chip. The iMac has a 2.0GHz Core 2 Duo CPU, 2.5GB of RAM, and an ATI Radeon X1600 graphics chip while running OS 10.6.

The HP 8710w was running a 64-bit version of Windows 7 on a 2.2GHz Intel Core 2 Duo CPU with 2GB of RAM and an NVIDIA Quadro FX 1600M graphics controller. The Acer Aspire One is a netbook running Windows XP Home Edition with a 1.60GHz Intel Atom CPU, 1GB of RAM, and an integrated Intel 945 Express Chipset for graphics. All computers except the Aspire netbook were measured playing back the 720p file, and Table 3 shows the browsers used in the respective tests.

During testing, I followed this procedure:
• I turned off as many background processes as possible.
• I updated my graphics card drivers.
• First, I loaded the page and waited until the video completely downloaded (watching the little bar thingie on the bottom of the player).
• Next, I played the file, monitoring and recording the CPU usage on Windows Task Manager and percentage idle on the Activity Monitor. Obviously, to compute CPU utilization for the Macs, I subtracted the percentage idle from 100%.

Table 4 shows the results grouped by format and playback environment. Pulling some conclusions out of these disparate numbers is challenging, but here goes. Most obvious is the fractured nature of the HTML5 market, which still lacks a single codec that can play across all platforms. With Microsoft only supporting VP8 playback on IE 9 if the codec is otherwise installed, and Apple squarely in the H.264 camp, Google's open sourcing of VP8 has done nothing to reduce this logjam. As you have probably heard, Adobe has announced that a future version of the Flash Player will support WebM at some point; that may end up being the only easy way to display WebM video across all relevant browsers. However, one suspects that you could buy a Tastee Freez in Hades before WebM will play on the walled garden of Apple iDevices.

If you're a Mac owner who consumes lots of video, you can see that results vary by browser and if your Flash implementation is hardware accelerated. If it is, Safari is your best option; if not, go with Chrome. As a random thought, people watching Flash-video on a nonaccelerated Mac via Opera or Safari probably consider Flash a CPU hog-I wonder if their opinions would change if they did switch to Chrome. Ditto for HTML5-based H.264 playback on the Mac, where Safari is the efficiency king and Chrome the laggard.

Regarding WebM, on the platforms without Flash GPU acceleration (the iMac and Aspire), the most efficient WebM implementation required less CPU than the most efficient Flash implementation, though the WebM implementations varied more widely. Absent hardware acceleration, CPU utilization on playback appears to be a wash.

As I mentioned at the beginning of this article, when playing a WebM 720p file on the HP 8710w via Media Player, CPU requirements dropped to 18%, about a third of the requirement of the most efficient browser and within spitting distance of GPU-accelerated Flash playback. On the Aspire for the 640x480 file, CPU load dropped to 24%; 30% with hardware acceleration disabled, which is much lower than any other tested technology. If any of the browser vendors can enable this level of efficiency for web-based playback, WebM would have a significant competitive advantage over H.264, whether Flash- or HTML5-based. If not, given that many newer computers and mobile devices offer some measure of Flash acceleration, WebM may trail in quality, encoding speed, and playback efficiency.

Analysis
As it stands today, WebM's value proposition is basically that it's free and better than Ogg Theora; though it's way too tough to decode as compared to the rapidly expanding base of GPU-accelerated H.264. If the browser vendors and Google can reduce CPU playback requirements to levels shown in Media Player, however, the story changes considerably, at least for pay-per-view and subscription-based video distributors, who currently pay a fee to deliver H.264 video. Ironically, given the lack of universal HTML5 browser support, the most efficient way to distribute WebM to your customers may be via the Flash Player, assuming that the Flash Player's CPU playback requirements are competitive. You'll probably still have to write checks to MPEG-LA for delivering to Apple's iDevices, however.

As you probably know, H.264 is royalty-free for free internet distribution, at least through 2015, so there's no financial incentive to switch. If your organization wants to migrate toward HTML5, WebM doesn't provide that single codec solution, is still lower quality than H.264 (however small the difference), and takes longer to encode. Overall, for those not charging for their video, H.264 is still a better solution, and given the rapidly increasing size of the GPU-accelerated installed base, it will likely remain so, unless and until Google creates distribution channels that you can't access with H.264.