MediaMelon QBR Review: Optimizing Video Streams One-by-One
I was uncomfortable using MediaMelon-provided data so I used the Charles Web Debugging Proxy to identify the actual segments being retrieved using QBR in both modes, and without QBR, and plugged the data into a Google Sheet. The Charles Proxy data showed that at 15Mbps bandwidth, the ABR player retrieved mostly 8.1Mbps segments, while both modes of QBR retrieved primarily 4.2Mbps segments. Table 1 summarizes the results, which confirm MediaMelon’s numbers. In this case, the two QBR modes actually reduced the number of bitrate changes during playback of the 2-minute file, though the differences weren’t significant.
Table 1. My results for the talking head clip calculated from data retrieved by Charles Proxy.
Since the results were nearly identical for Quality and Bitrate modes, I only tested Quality mode at 4Mbps. Here, the ABR player without QBR retrieved 41.76 MB compared to 29.83 for the QBR player, a savings of 29%. Looking at the fragments in Charles Proxy, QBR retrieved primarily the 1800 kbps 720p stream, while the regular ABR player retrieved the 2.6Mbps 720p stream.
Did MediaMelon make the right decision? At both data rates, unquestionably yes. To quantify this, I computed the VMAF score for all deployed streams. At 15Mbps, MediaMelon retrieved the 4.2Mbps stream, which had a VMAF score of 96.07, rather than the 8.1Mbps stream, with a VMAF score of 97.09. In the 4Mbps tests, QBR retrieved the 1.8Mbps stream (VMAF=90.88) rather than the 2.6 Mbps stream (VMAF=91.66). Since it takes 6 VMAF points to create a single just noticeable difference, in both cases, viewers wouldn’t have noticed.
Worst Case - Bandwidth Savings
Next I wanted to check performance with a more challenging clip, so I tested a five-minute segment from the movie Zoolander. Table 2 tells the tale. With a data rate throttle of 15 Mbps, QBR saved between 18 - 19%, depending upon mode. With a throttle of 5 Mbps, QBR essentially broke even.
Table 2. Results from the movie Zoolander.
Perhaps more concerning was the number of bitrate changes in the four QBR results. Traditional ABR dogma states that you achieve the best quality of experience finding the optimal bitrate and staying there, as stream switching is often obvious to viewers. Of course, in a clip with varying amounts of motion, staying at the same bitrate would mean slight improvements and degradations in quality as scenes get more or less complex.
MediaMelon’s assertion is that these frequent stream changes produce more consistent quality over the duration of the clip. It’s hard to argue with that, but if you’re beta testing the technology, observe the playback streams and draw your own conclusions.
Checking the Uplift
Next I wanted to check MediaMelon’s claims that QBR delivers “complex scenes at higher bitrates than the average available data rate, improving perceived quality.” In essence, I wanted to see if QBR would look ahead, and send smaller segments of easy-to-encode scenes in order to later send larger segments of harder-to-encode scenes. The QBR player tracks this during playback and calls it “uplift.”
MediaMelon sent a screenshot from one of their test clips, which showed an uplift of 24.84% (Figure 4). Using the same clip, with the bandwidth throttle set at 4 Mbps, I was able to duplicate the results.
Figure 4. MediaMelon’s test clip showed an uplift of 24.84%.
As a check against the results achieved with the MediaMelon clip, I tested with a clip that mixed 30 seconds of talking head footage with 30 seconds of ballet, thinking that QBR would send smaller segments during the talking head sequence, and larger during the ballet. Table 3 shows our results at the two test bandwidths.
At 15Mbps, QBR showed 38.8% savings, but no uplift. Basically, QBR retrieved all talking head segments at 4.3 Mbps, and all ballet segments at 5.7Mbps. I guess at that bandwidth, QBR didn’t have to plan ahead to download larger sequences from the ballet, so it reported no uplift. In contrast, without QBR, the player retrieved all segments at 8.1Mbps.
Table 3. Results from the mixed talking head/ballet test clip.
At 4Mbps, QBR retrieved all talking head segments at 1.8Mbps and all ballet sequences at 2.6Mbps, and again showed no uplift. Without QBR, the player retrieved all segments at 1.8Mbps, making QBR 20% less efficient in this instance.
During three days of testing, I tested a range of clips and throttle conditions, and though I saw some uplift in my test clips, I never came close to the 24.84% shown in Figure 4. A quick glance to the right of the Figure shows why-MediaMelon’s encoding ladder used 720p for all layers. In contrast, my encoding ladder, shown on the right in Figure 3, used the more traditional approach.
Why does that matter? As mentioned at the top, as tested, QBR can only switch streams that use the same resolution. In the MediaMelon encoding ladder, this meant that the player could switch between all streams in the ladder, from 500Kbps to 3.8Mbps. In my encoding ladder, ladder, the player could only switch between a maximum of three layers. At the tested 4Mbps, all segments retrieved by the QBR player were either 720p @ 1.8Mbps, or 720p @ 2.6Mbps, which really seemed to limit how much planning ahead QBR could do.
If you’re reading this in late 2017 or thereafter, MediaMelon may have already extended their technology to cover all layers in the encoding ladder, including those with different resolutions. If so, the technology benefits should extend to a much broader range of encoding ladders and playback bandwidths. Until then, to achieve the maximum benefits that MediaMelon claims, you’ll have to change your encoding ladder to one similar to that shown in Figure 4.
Real World Results
To get a feel for how QBR delivers in the field, MediaMelon put me in email contact with two professionals who have experience with QBR. First was Alan Young, who had worked with QBR when he was CTO of Encompass Digital Media. Young declined to specify bandwidth savings, but did state, “Put simply, QBR reduces the bandwidth (and therefore cost) whilst at the same time improving quality and reducing buffering.”
Next up was Todd Collart, president, global delivery for Deluxe Entertainment Services Group, who had just completed a successful proof of concept with a major telco that combined Deluxe’s OTT capabilities with QBR. Collart reported that QBR saved between 20-40% of bandwidth costs, depending upon the amount of motion in the video.
I specifically asked, “MediaMelon claims to be able to shape traffic by sending low bitrate segments for easy to encode scenes to “save up” bandwidth for harder to encode scenes. Are you seeing that?” “Yes, this is inherent in the technology,” Collart said. “MediaMelon offers a useful demonstration application that showcases this capability. Of course, Deluxe implemented its own test environment and was able to separately verify this worked as advertised.”
Where’s that leave us? QBR is an interesting technology that could become much more effective once it can address layers with different resolutions. At its best, it should be able to reduce bandwidth costs without impacting perceived quality, and perhaps even shape traffic to improve overall quality. Interestingly, true per-title encoding should be able to match the bandwidth savings, but not the traffic shaping.
All that said, more than any technology that I can remember, your results with QBR will vary according to formulation of your encoding ladder and the retrieval bitrate of your viewers, as well as the efficiency and performance of your video player.
Related Articles
The benefits of per-title optimization aren't just for the major players, anymore. Streaming Media reviews the first solution for smaller content owners and finds the results promising.
31 Aug 2016