How to Use Objective Quality Measurement Tools
Interestingly, the “Talking Head” video showed only a 17 percent reduction in data rate as compared to the highest data rate files. This differential may increase for your hardest to encode sports or other high-motion videos, but at 17 percent, I wouldn’t recommend a different adaptive group for talking head videos than the other live action videos in this group.
Before leaving, let’s observe that in this example the VQM results aren’t that helpful because the numbers don’t correspond with any subjective quality level. While the lower numbers for the screencam and tutorial videos indicate higher quality than the others, there’s no correlation with any subjective evaluation.
Configuration of 1800 File
Table 2 presents another reason not to encode screencam and tutorial videos the way you encode live action videos; simply stated, they look worse when subsampled to lower resolutions. But let’s back up.
If you look at the adaptive group recommended in Apple Technical Note TN2224, you’ll note a significant gap between the 640x360@1,200Kbps file, and the next largest 960x540@3,500Kbps file. If you’re concerned about mobile viewers getting the best experience, and you should be, you might want to fill in that gap with a file around 2500Kbps. The obvious question is, what’s the best resolution for that file, and that’s what we test in Table 2.
Table 2. Choosing the best resolution for our 2,500Kbps file
To complete the table, I encoded all 1080p source files at the various resolutions to 2,500Kbps, and computed their SQM scores for the iPhone 6 Plus and iPad Air 2, which are shown in the table. As you probably suspect, the worst scores have a red background, while the highest scores have a green background. The Delta column shows the difference between the highest and lowest score, and I’ve highlighted the results for the tutorial and screencam video.
Intuitively, we know that these various options trade detail for more data per pixel, which translates to higher-quality pixels. That is, the 480p video has the lowest detail, but the highest quality per pixel, while the 1080p video has the most detail, but the lowest quality per pixel. In all cases, the 480p video delivered the worst quality on the two tested devices. However, the results were substantially worse on both devices for the screen-cam and tutorial videos, which contain lots of detail that blurs or otherwise loses quality when subsampled to lower resolutions.
Given this data, I would avoid the 480p and 540p files for the 2,500Kbps file in all cases, and might consider scratching the 540p resolution suggested by Apple for the 3,500Kbps file. It seems that jumping directly to 720p for both might be the best option.
For “Screencam” and “Tutorial” videos, which look great at very low data rates, it might make more sense to eschew an adaptive group altogether, and simply deliver a single 1080p or 720p file at 300 to 400Kbps. Given the SQM results, I would look long and hard at delivering these files at less than 720p.
CBR vs. VBR
The next issue involves constant bitrate encoding (CBR) versus variable bitrate (VBR) encoding. Within the context of an adaptive group, many experts recommend using CBR to avoid spurious stream changes that relate to data rate fluctuations rather than actual changing bandwidth conditions. The big question is, how much quality are you losing with CBR? Quite a bit, it turns out, in two critical ways.
To test the quality differential, I encoded the test files at 720p@2,500Kbps using single- and dual-pass CBR encoding, and 125 percent, 150 percent, and 200 percent constrained VBR encoding. The results are shown in Table 3 using the VQM metric, again, where lower scores are better.
Table 3. CBR vs. VBR encoding
Cells with red backgrounds indicate the worst quality (highest values with the VQM metric), while cells with a green background indicate the highest quality. The total difference between the highest and lowest quality is shown in the column to the far right.
The table reveals two interesting points. First, CBR always produces the lowest quality, though you can mitigate that in most cases by using two-pass rather than one-pass encoding. Second, 125 percent constrained VBR encoding delivers almost the same quality in all tests as 200 percent constrained DVR, while producing a much more ABR-friendly file. If you do opt for VBR, 125 percent constrained might be the best option.
Beyond the numbers, CBR files often exhibit several short sections of very low-quality frames that were produced when the encoder has to encode high-motion regions at very low data rates to meet the requirements of the restrictive encoding scheme. An example of this is shown back in Figure 1. If you study the figure, you’ll note that the play head is placed at a section where the quality curves of the CBR and VBR files differ substantially. If you looked at the frames in the video files at this point, you’d see a substantial quality difference between them, even though the average difference might be modest. So not only is the overall quality lower with CBR, many files will exhibit one or two areas where the quality differential is substantial and very noticeable, albeit for only a few frames.
Does that mean you should switch to VBR encoding for your ABR schemes? It’s certainly worth trying, especially since many producers have long used up to 200 percent constrained VBR in their adaptive groups without problems.
How High Is High Enough?
The last issue addresses how high is high enough? Specifically, when you’re producing an adaptive group of files, what’s the optimal maximum target data rate at the top of the group? Obviously, this number is key because it’s the highest bandwidth file and costs the most to deliver.
Table 4 shows the results of tests performed for a recent consulting client, where their adaptive group included three 1080p files, encoded at 5,000Kbps, 6,500Kbps, and 7,500Kbps. Looking at the SQM results, all three rate in the excellent range, with about 0.25 percent difference among them. The VQM scores showed a greater differential, but comparing the files in the Results Visualization screen showed very minor differences throughout. In both cases, the 7,500Kbps file costs 50 percent more to deliver than the 5,000Kbps file, but provides no real quality improvement. Perhaps in a premium service the difference might be worth it, but for most other services, 5,000Kbps should be sufficient.
Choosing the top data rates for your 1080p files
Final Thoughts
The basic message is that for most important compression-related decisions, objective quality metrics provide useful data at a fraction of the cost of subjective testing. There are some caveats, of course. First, take the results shown above as examples, not fact. Your results will vary by codec and encoding tool, as well as test clips. Second, never take the numbers completely at face value; before making any key decision, always play and watch the clips in real time.
Third, choose your test clips carefully. In my view they should represent a good range of both simple and challenging clips. Test clips that are too challenging could push your data rates to levels required only by 1 percent or 2 percent of your video. Certainly you want to know when you’re pushing the quality envelope, but try to test for your entire library, not just the hardest sequences.
Along these lines, it’s good to articulate the quality of video that you expect to deliver, particularly at the high end. If your goal is to avoid noticeable quality problems in all streams, that’s achievable, but only at very high data rates. Perhaps a better goal is the one that most producers seem to have adapted, that even satellite and cable TV streams get ugly on occasion, and it’s okay if your video does as well.
Whatever quality level you seek, there’s no reason fly blind any longer when it comes to how your tweak your encoding parameter and configure the files in your adaptive group. Objective quality benchmarks provide exceptionally useful data for any producer wanting to substitute fact for untested opinions.
This article appears in the 2016 Streaming Media Industry Sourcebook.
Related Articles
Counting viewers has always been a difficulty for streaming publishers, and today's multiplatform world makes the issue more demanding than ever. The answer is to radically rethink video analytics.
12 Oct 2016
As a rule, higher data rates equal better quality. But there comes a tipping point where the extra quality might not be worth the extra cost.
07 Apr 2016
For compressionists who want to see the image quality differences a tool measures, SSIMWave can feel incomplete. An upcoming update may change that.
13 Oct 2015
How can video compressionists assess the quality of different files? Only by combining objective mathematical comparisons with their own professional judgments.
13 Jul 2015
Never heard of it? Learn why we call this video encoding analysis tool invaluable, and a must-have for anyone serious about encoding or compression.
30 Dec 2014