QoE Working Group to Deliver Standards Document by End of Year
Measuring quality of experience (QoE) is easy when showing how one provider improves over time, but harder when comparing multiple providers. That's because each platform has its own definitions and standards. Even a common problem like buffering is impossible to compare when measured in different ways. To solve for that problem and help drive the entire industry forward, a working group will issue a document by the end of this year spelling out definitions.
That group is officially known as R04 WG20 Streaming Media Quality of Experience, but more commonly as the Quality of Experience Working Group, and it's overseen by the Consumer Technology Association (CTA) with participation by the DASH-IF. Steve Heffernan, co-founder and head of product at Mux, is one of the group's members and he spoke to StreamingMedia.com about its goals.
"It's very difficult today just to do an apples-to-apples comparison between two major streaming platforms. Even if they were reporting the metrics themselves, you would have no way to be sure you're actually talking about the same thing," Heffernan says.
To solve the problem of making video quality comparisons, the working group will create strict definitions and reporting methods. The idea is for video platforms to report their own metrics to third parties, helping companies and consumers make an educated choice about the product. If a platform has an outage during a major live event, as YouTube did during the World Cup in July, that generates a lot of bad press but may not represent that platform's general level of quality. Having easy comparisons would show whether an outage is a fluke or typical of a larger problem.
The working group's document will make clear all the details about how quality is measured, defining terms like stalling or buffering, and ensuring that all platforms measure issues in the same way, such as when a user closes his or her laptop and the platform doesn’t get all the needed information. In those cases, small details can have a major impact on overall quality measures, so it's important to have a uniform procedure.
The report will spell out three layers of metrics reporting: First, the platforms themselves will integrate these metrics and report their own numbers publicly to users. Second, vendors like Conviva and Mux will gather this information and create industry-wide reports. And third, creating self-reporting tools in browsers and video applications to report exactly what the viewer sees. It's all possible, Heffernan says, and only requires integration. Most platforms already report the details necessary to track these metrics.
Overall video streaming quality has been improving for years, but having access to detailed metrics will help content owners keep viewers satisfied.
"While, for the most part, it might seem like most platforms are performing all right, there are points where customers start to drop off and start to find other places to watch the content," Heffernan says. "That can be as early as two seconds of startup. If they're not committed to a platform and they're just trying to watch some random video, then if they're waiting for two seconds often they'll start dropping off even after that instead of competing to watch a video and your ads. It can have a pretty big impact on the bottom line."
Besides helping content owners, standard metrics will be an asset to viewers, as well. Looking ahead, Heffernan imagines tools that show when video is being throttled, so people can see if their phone carriers are giving them the unlimited high-definition video they were promised or choking them back to 480p. Viewers could run their own quality checks on their network and device.
The working group is taking comments and suggestions now, so the document released at the end of this year should be final. Platforms will be able to take action immediately. While the working group doesn't have the ability to enforce its decisions, many of the biggest companies in streaming are part of the group and will adopt its recommendations.
"The companies that are involved in the standards process are some of the bigger companies that you'd also want to be adhering to it," Heffernan says. "Google and Apple and Comcast and Verizon and ATT and all these companies in some form or another are participating in these standards, and that helps throw weight behind them because you can say, 'Here's the recommendation backed by these companies,' and give it some clout."
Related Articles
To create high-quality, broadcast-like streaming for the home, the solutions are already here. Discover what content-aware encoding, CMAF for low latency, and edge scaling can offer.
29 Jul 2019
Comcast Technical Solutions Architect Ribal Najjar defines video QoE both in terms of subjective experience and qualitative measurement in this clip from Streaming Media East 2018.
08 Oct 2018
If you're not using a video quality measurement tool, you're behind the curve. Here's a look at the most popular tools and how they work.
01 Dec 2017
How can publishers compare video quality at different resolutions? There's the theoretically correct answer and then there's how it's generally done.
18 Oct 2017