Measure it, Improve it: For Video Publishers, QoE and QoS are Critical
According to Marc Maycas, NPAW’s VP of customer engineering, the company’s competitive strengths lie in the granularity of data, especially archived data, and the flexibility of the multidimensional filtering. He also pointed to the breadth and depth of the SmartModules as particularly valuable to NPAW customers, noting that while most other tools focus on the operations manager, modules like the SmartUser allow the system to provide valuable insights to the content and marketing managers as well.
Natxo Imaz Erdoiza, CTO at Mediapro/ beIN SPORTS, relies on the SmartAlerts module to identify problems and allow his team to take corrective action. As Imaz explained, “We have alerts configured, which we use to balance CDNs when we detect any issue related with CDNs, ISPs, or carriers. It is quite common that ISPs fail at just one certain moment or interval. These alerts allow us to identify problems and offer a fast solution to just one ISP in a couple of seconds.”
Providers that want to automate CDN switching can use NPAW’s SmartSwitch module. According to Maycas, this module allows customers to define business rules to identify the best CDN for each customer request. These rules can be based on simple strategies for load balancing, a prioritized schema for managing costs and guaranteed commitment levels, or a fully granular decision tree that chooses the best CDN for each viewer based upon QoE metrics for a specific piece of content, considering the ISP and device of the target viewer.
Maycas also touted the company’s new SmartUser module, which deploys a churn prediction algorithm to predict churn before it happens so you can take actions to prevent it. The module provides visibility into end users’ behavior and lifecycle, with extensive segmentation, including geographic, devices, demographic, and others. You can then track the effectiveness of various customer retention strategies— like changes to customer care or content recommendation engines, coupons or discounts, or UI changes—with full visibility into users’ churn data and churn prediction trend lines in the SmartUser’s dashboards.
Pricing for the core service starts at $750/month, which includes features like real-time, multidimensional filtering for up to 500,000 views per month.
Nice People at Work’s YOUBORA Infinity and SmartModules
MUX
Mux is the new kid on the block of QoE solutions, founded by the team that developed the Zencoder cloud encoding platform and the Video.js player. While Mux doesn’t offer the broad functionality of NPAW, or the scale or informational scope of Conviva, it’s the most affordable of the three and is currently used by sites like PBS, Funny or Die, Livestream, and Wistia. According to at least one user, Mux provides more actionable data than other alternatives, which allowed it to more quickly identify and resolve player-related issues.
Specifically, while working at IGN, then-senior video software engineer Zac Shenker (now with CBS Interactive) was charged with implementing a new player, moving the company from version 4 to version 6. Shenker first used Mux to benchmark the performance of different players they were considering, and once they selected a player, to fine-tune and debug any problems.
According to Shenker, Mux’s error-reporting data enabled IGN to identify long-standing issues and solve some common error cases. In addition, Mux’s alerting feature quickly identified problems in production deployment, API issues, or bugs. Beyond fixing problems, Mux data allowed Shenker’s design team to focus on improving critical QoE benchmarks, delivering the improvement shown in Figure 4.
Improvements to the IGN player, enabled with help from Mux
As Shenker related, “We were paying a lot of attention to some of the metrics like startup time, exits before video starts, and looking particularly at rebuffering to understand when a user was having an interruption during the playback experience. Through the whole process, Mux has been an incredibly useful tool for us, in measuring a baseline and then understanding how things were improving as we were rolling out this new player across the site.”
In terms of feature set, Mux is expanding, both directly and via partnerships with other services. According to co-founder and head of product, Steve Heffernan, Mux is integrating with Cedexis to allow Mux-based QoE data to be incorporated into the CDN switching algorithm deployed by shared customers. Although the Mux beacon hasn’t been included in as many platforms as Conviva or NPAW, the company currently supports multiple OPV services; players like Video.js, THEOplayer, and JW Player; and offers SDKs for iOS, Android, and Chromecast. For players and platforms that aren’t currently supported, Heffernan estimated an integration time of “a couple of hours.”
For pricing, Mux offers a free plan that includes up to 100,000 video views per month, with additional views costing $0.0001/view, though this doesn’t provide alerting, A/B testing, or industry benchmarks. The $299 Pro plan includes all of these for up to 1 million video views per month ($0.0001/view beyond that), with a $2,499 plan adding support, service-level agreements, and other features for up to 10 million views ($0.00005/view beyond that).
Quality of Service (QoS)
Where QoE measures actual performance on the player using actual video files, most QoS technologies use synthetic players and processes to benchmark network performance, thereby identifying the best resources for delivering a customer’s video content. By synthetic, I mean the processes like the Touchstream pollers that retrieve the manifest file and fragments, but not using the services’ player or the actual end-user connection. One exception is Cedexis.
CEDEXIS
Cedexis is a general-purpose, web load-balancing solution that’s particularly valuable for companies deploying all forms of content, including video, via multiple CDNs. Unlike other QoS solutions that only use synthetic players located in data centers or other points of presence, Cedexis measures performance at the viewer’s actual computer, tracking over 14 billion measurements per day, which the company called Real User Measurements, or RUM. This data is then used to identify the best CDN to deliver video data to that user. Cedexis users include some of the largest companies in the world, like Microsoft, Bloomberg, LinkedIn, L’Oreal, and France Télévisions.
All data starts in the Cedexis data collection architecture called Radar. More than 800 organizations have joined the Cedexis Radar community by deploying the Radar tag on their websites and apps, totaling more than 50,000 origination points. On all these websites and apps, 2 seconds after the page load completes, the Radar tag downloads instructions from Cedexis relating to synthetic test objects to be retrieved from multiple CDNs and similar sites, which provide up to three measurements.
For example, a 43-byte test object measures latency and response time, while a 102,400-byte test object measures throughput. These test objects are retrieved from every major CDN and cloud provider and some relevant private platforms. This crowd-sourced data provides a real-time map of internet performance, and data regarding the delivery performance from multiple CDNs to the specific viewer later requesting video, or those proximate to them as shown in Figure 5.
This Cedexis crowdsourced internet performance map shows latency, throughput, and availability events.
Cedexis OpenMix is the load-balancing solution that utilizes this data. Within OpenMix, Cedexis customers identify the CDNs available for delivery and the business rules for choosing among them. As Simon Jones, marketing leader at Cedexis, explained, the factors that can be incorporated into the decision are very broad and can be augmented by third-party data via Cedexis Fusion. For example, the latency, throughput, and availability data provided by Radar would obviously be considered. Beyond that, if two CDNs were both delivering acceptable performance, you could choose the cheaper option.
Note that Radar is a free service; you only start to pay once you use OpenMix. In March, Cedexis added video QoE tracking to Radar. According to Jones, Cedexis will track start time, failure rate, rebuffer ratio, and bitrate via a simple beacon that should easily integrate into any HTML5 player, but won’t be available for mobile or OTT devices for the foreseeable future.
Beyond Radar and OpenMix, Cedexis also offers Cedexis Impact, an analytics and visualization service that helps customers tie performance improvements to key performance indicators. For example, via Impact, if a customer adjusted his use of cloud and CDN content delivery services to reduce buffering or wait times, he could see how that correlated with increased engagement or revenue.
OpenMix is licensed on a SaaS basis, with most contracts running for 2 years, and price is based on the volume of decisions. According to Jones, companies can get started for just a few hundred dollars a month.
There are several other QoS vendors worthy of note, including IneoQuest, Tektronix, and Cisco. However, most use synthetic players located in AWS data centers or the like, so they don’t provide data for the last mile, as QoE vendors do with beacons and Cedexis does with RUM measurements and the recent additions of QoE data points.
Different producers obviously use different data points to make critical deployment decisions, and there is no right or wrong approach. Still, at the end of the day, all that matters is the actual user experience. To modify Peter Drucker’s quote, if you’re not measuring that directly in some fashion, you won’t be able to improve it.
[This article appears in the July/August 2017 issue of Streaming Media Magazine as "Quality Control."]
Related Articles
Interra Systems and streaming media industry expert Dan Rayburn conducted a survey in 2022 to better understand the steps media companies are taking to quality control (QC) and monitor their content. The survey collected data from media industry decision makers. This white paper digs into the 155 participants' responses and explores key findings, including why companies are implementing QC and monitoring, the extent to which they're doing so, and the key challenges they're facing.
30 Mar 2023
Streaming quality has made gains, but isn't yet a match for pay TV, finds a survey. Viewers have little patience for stalls and buffers.
11 Oct 2017
Netflix's compact mobile download files look surprisingly great. Here's how video creators make their own low-bitrate files look just as impressive.
29 Jun 2017
Using biometric tests, Akamai learned exactly how disengaged and unhappy viewers become after experiencing even minor streaming issues.
21 Jun 2017
Churn rate is the most important indicator of a streaming service's health, and over-the-top video providers can't take risks with the user experience.
17 May 2017
From spatial to temporal to perceptual, optimizing video depends on how it's measured. This how-to guide explores recent advances taking place in video quality.
13 Apr 2017
The sheer number of video quality measurement tools makes it difficult to choose the right metric. Here's a quick overview of some of the options and what they offer.
07 Sep 2016
Companies and Suppliers Mentioned