How to Measure Video Encoding QoE
You don’t have to be a Hollywood filmmaker to know that video is a hit-or-miss proposition, but at least when you deliver your video in a theater, you know it will play smoothly. Unfortunately, when you distribute your video over the web, you have no such assurance. So, unless you measure the quality of experience (QoE) of your viewers, you have no way of knowing if your underperforming videos are failing because of content or because of some technical delivery problem. If your video is serving some mission-critical purpose, that’s unacceptable.
It used to be both technically cumbersome and expensive to monitor the QoE of your viewers. As you’ll read in this tutorial, via a service called Mux Data from a company named Mux, you can start monitoring QoE by adding a just few lines to your live or video-on-demand (VOD) embed code and will pay only $0.001/view to begin. To be sure, you may want to start with the Pro plan (at $499/month for 1 million video views a month) to access features such as longer data retention, access to industry benchmarking, and alerting, which still probably costs less than the price of even a minute’s worth of production for some promotional videos.
In this tutorial, I’ll first show how to integrate Mux monitoring into your video playback and then review the type of data you can access from the system.
Mux Integration
When you add Mux Data to your videos, you do so within what’s called an Environment, which is defined as the “highest grouping of data you want to combine and compare within.” In essence, Environments allow you to cordon off test and other staging integrations from actual production integrations for analytical purposes. So, while creating your iOS or Android integration, you can exclude data from these integrations from actual live productions.
You will add each video as a separate integration. To start, click Add Integration within an Environment, and you’re taken to the screen shown on Figure 1 to choose a software development kit (SDK). As you can see, Mux currently has integrations with the HTML5
Choose a player software development kit.
Video Element; Video.js (which also supports Brightcove players version 5.x); JW Player 7; Bitmovin Player 5.x, 6.x, and 7.x; Ooyala Player V4; THEOplayer 2.x (in beta); iOS 8+ and tvOS 9+ (Objective-C); and ExoPlayer v2 (in beta). If Mux doesn’t currently support your player, it will either create the integration for free or provide documentation and support for your own developers, which the company states should take a day or so.
Integration complexity varies by SDK. For my tests, I added Mux Data to a simple HTML5 player in about 10 minutes (see Figure 2), aided by a useful YouTube video produced by Mux co-founder Steve Heffernan (go2sm.com/muxintegration). There are two aspects to the link. The first script (to mux.js) ties the video to the Mux system. The second allows you to add metadata to the integration, either for informational purposes or to provide details about the video or environment that can’t be detected by the Mux system and may help diagnose problems.
Adding Mux to a simple HTML5 player
For example, you can use the experiment name field to separate different configuration experiments when implementing a new player or debugging a problem. You can also add video titles and IDs to quickly identify video-specific problems.
Data Tracking and Reporting
Mux tracks a number of common metrics like time to first frame, exits before video start, player and page load time, playback failures, rebuffer frequency and percentage, and seek time, as well as metrics like upscale and downscale percentage. As you’ll see in more detail in a moment, Mux can sort this data by dimensions like browser and browser version, country, and OS, as well as via much of the metadata added to the video, like player name and version, video title, etc.
Data is collected continuously and is updated to the analytics about 2 minutes after playback ends. You access the metrics in the Mux system via the screen shown in Figure 3, although you can also access the data via an application programming interface (API) to view or to integrate it into your internal systems (go2sm.com/muxapi). As you would expect, you can access data for any time frame as controlled by the calendar function at the upper right corner of the screen. I’m showing data from a small Mux customer for Sept. 1 through Jan. 12.
Accessing metrics in the Mux system
As you can see on the left in Figure 3, there are four main categories of data: metrics, errors, views, and alerts, which I’ll touch on further below. Metrics are dominated by the viewer experience score, which averaged 69 over the reported period and varied, as you can see in the graph. For perspective, the faint gray dotted flat line shows the average score for all Mux customers over the previous 90-day period. The graph shows that, while this customer’s viewer experience score flirted with the overall average for about the first month of operation, the experience improved thereafter and was consistently higher save for the big drop that occurred on Friday, Dec. 2. We’ll come back to that in a moment.
Related Articles
To create high-quality, broadcast-like streaming for the home, the solutions are already here. Discover what content-aware encoding, CMAF for low latency, and edge scaling can offer.
29 Jul 2019
Encoding and transcoding are at the heart of every OTT and online video workflow. The first article in this three-part series gives an overview of the technologies and a look at three major players in the space: Harmonic, AWS Elemental, and Telestream.
08 Nov 2018
Demand for on-prem encoding is waning, and vendors are responding with innovative hybrid approaches that offer the best of on-prem and the cloud.
23 Jul 2018
Mux CEO Jon Dahl talks about the company's QoE flagship QoE product and Mux Video, a new API to video hosting and streaming.
17 Apr 2018
When it comes to encoding settings, don't just "set it and forget it." Mux Video dynamically encodes videos when needed to provide the best results.
14 Nov 2017
Mux makes it simple for media companies to learn exactly what problems their viewers are experiencing, and then find solutions.
31 Aug 2016
Companies and Suppliers Mentioned