Cracking the H.264 Codec

Beyond this, when choosing your H.264 encoding parameters, there are three inter-related things you should consider. First, you want to deliver the highest-quality video. Second, since advanced encoding options produce a file that’s harder to decompress, you want to ensure that your file plays smoothly on your target computers. Finally, for those producing massive amounts of files, you want a set of configuration options that will produce your files in a reasonable amount of time.

The first question I wanted to answer was how much of a quality difference exists between a file encoded using the Baseline Profile and a file produced using the Main Profile with all quality-related options enabled. Then, the question became, "How much slower did the file play back on mid- to low-end computers?"

To test this, I encoded two files to two different sets of encoding parameters. The first was my standard DV codec test file, which I produced at 640x480 resolution, 29.97 frames per second, at 468Kbps video/32Kbps audio. The second was an HDV source file that I first produced at 720p at a data rate of 4,000Kbps video/128Kbps audio, then produced again at 1,500Kbps video/128Kbps audio. In both cases, I produced the file twice, once using the Baseline Profile and next using the Main Profile with all quality-related options enabled.

Let’s discuss the low-resolution file first. Note that I produced the two low-res comparison files in four encoders: Adobe Premiere Pro 3.1.1, Episode Pro (4.4.1), Sorenson Squeeze 5, and Rhozet Carbon Coder 3.0, which offered, by far, the most configuration options. For this reason and in the interest of time and space, I’ll make the most comments about the Carbon Coder results.

In terms of quality, the tricked-out Main Profile produced better results, though these were primarily visible during high-motion sequences. In low-motion, talking-head scenes, the two videos were nearly identical. Note that this finding was relatively consistent among the various encoding tools.

So, we know that the advanced encoding parameters deliver better quality, but what does it cost us in playback speed? To measure this, I played the files back on four different computers, ranging from an old Dell Latitude notebook to a Dell Precision 390 workstation. This is a pretty grim and conservative range, ignoring all the quad- and eight-core systems sold over the last 12–18 months.

During playback of the same 1-minute section of the file, I recorded the high and low figures for processor utilization as presented by Windows Task Manager and Apple’s Activity Monitor. For good measure, in addition to testing the H.264 files, I tested Windows Media 9 Main Profile and FLV (VP6) files encoded to the same general specifications.

On low-end systems, the difference between the two H.264 files was real but not profound and moved into the irrelevant range when I got to the Precision 390, which was the most powerful computer on the planet for one shining moment back in 2006 but is now decidedly pedestrian. Perhaps even more interesting was the fact that at these encoding parameters, both versions of the H.264 file were almost always easier to decode than either the FLV or WMV files.

Overall, the quality difference between the two H.264 files was less than I expected, but so was the difference in required decompression horsepower—which, in any event, was less than that required for VP6 and Windows Media. At or below 640x480 resolution, I’d recommend using the Main Profile and optimizing any quality-related parameters available in your encoder, which I discuss below. The only exception would be if you were targeting very old systems, but even then, you’d probably be better off producing at a lower resolution with the Main Profile.

High-Resolution Results
High-resolution results differed from SD in several critical ways. First, at 4Mbps, I saw no quality difference between the two files, and at 1.5Mbps, I saw only a slight quality improvement using a tricked-out Main Profile as opposed to Baseline. It’s possible that encoding at lower rates may have increased the differential, but given the high playback demands of video encoded using the Main Profile, I’m not sure I would recommend that approach even if quality were noticeably better.Where SD H.264 playback was relatively trivial for the three most relevant computers—the xw4100, MacBook Pro, and Precision 390—HD playback was clearly a strain, and using the Main Profile significantly boosted the CPU horsepower required to play the file. That said, on the xw4100, playing back the file created with the Baseline Profile required more CPU cycles. Overall, however, unless I was targeting only high-end computers for playback, I’d be more concerned with playback smoothness than quality at HD resolutions and would use the Baseline Profile.

Interestingly, except on the Mac, the HD Windows Media file (here, I used the VC-1 codec rather than WMV-9) was clearly the easiest to decode, and on the MacPro, it was relatively close to the H.264 file produced with the Baseline Profile. If playback performance on low-power systems is a priority and your HD codec decision isn’t yet set in stone, then you might consider WMV/VC-1. That said, at least when encoding with the VC-1 presets from Carbon Coder, VC-1 doesn’t deliver the same quality as the H.264 files encoded using the tricked-out Main Profile configuration.

To summarize, when producing H.264 at lower resolutions—say, 640x480 and below—it appears that optimizing your file for quality is the best decision. In contrast, with HD video, since playing back H.264 video is so processor-intensive on even moderately powerful computers, it makes the most sense to optimize for playback frame rate.

What about encoding time? This varied by encoder. With Carbon Coder, the differential was minimal (on an eight-core HP xw8400) while on Episode Pro (on a Core 2 Duo-based MacBook Pro), the file produced using the tricked-out Main Profile took about twice as long. If you’re a high-volume shop and plan to produce using high-quality parameters, run a few quick tests on your encoding station to determine if you can afford the time to do so.

Optimizing H.264 Encoding Controls
With this as background, all that’s left is to figure out how to optimize for quality and display rate. By way of background, let’s start with an overview of the H.264 encoding ecosystem. Specifically, understand that most developers of encoding software license their H.264 codecs from third parties such as Dicas and MainConcept.

These H.264 vendors implement the encoding techniques they feel are most valuable, then supply code modules to the encoder vendors with all configuration options exposed. From there, the encoding software vendor essentially decides which encoding parameters to allow end users to configure. If a configuration option isn’t exposed, the encoding vendor either chooses a universal default value or allows multiple controls to be manipulated by a single, more general control.

For example, let's examine the advanced encoding parameters available in Rhozet’s Carbon Coder, which licensed its H.264 codec from MainConcept. (Adobe also licensed its codec from MainConcept.) Let’s take a moment and review some of the most frequently exposed controls, starting with Entropy coding mode because we’ve already covered Profile and Level.

Briefly, with entropy coding, CAVLC is your only option with the Baseline Profile. CABAC is available with the Main Profile and beyond and produces better quality but creates a stream that’s harder to decode.

The "Number of B-Pict