Video: What Are the Most Important QoE Metrics for Player Optimization?
What are the key metrics to follow to assess and optimize player performance? Rebuffering? Error rate? And how important are industry-wide standards for these sort of assessments? Panelists from Vimeo, CBS Interactive, Hudl, and the New York Times discuss at Streaming Media East 2017.
Learn more about player optimization at Streaming Media West.
Read the transcript of this clip:
Zac Shenker: Error rate is a very critical metric to track, particularly when you're going from being blind on your video performance, to starting to have metrics coming in, understanding where users are starting to encounter critical failures in video playback, because often those are occasions where a user may not necessarily reload the page. They may abandon at that point, so certainly trying to avoid those sort of problems, and from that, also trying to understand when you have any regressions in your video players or your video stack that maybe start to impact certain platforms, or browsers, or things like that.
Matt Fisher: Rebuffering rates are obviously in a very important ... Very important metric to ensure that the quality is where you expect it to be. The amount of plays is obviously a nice metric to ensure that the content is being, or the platform, is being utilized to expectation, but then you know, maybe a half step deeper, things like stream switching and understanding yes, the video play that played quickly, how well was the experience? That's kind of what we're talking about today specifically, is yes, everything works, but how well did it work, and how much can we understand without every user calling us and saying "Yes. This was a great experience."
Brian Kaiser: For us, rebuffer is number one. Start time is number two. The only reason they're in that order for us is because we have a subscription service business, so they're not just going to bounce off, right? Once we get them in there, we want to keep the experience a lot higher. We're also maybe a little weird that stream switching is really important to us. We have coaches that are analyzing minute details in the film, so if they expect high quality and it drops, even just temporarily and comes back up, that's actually a bad thing. We'd rather take a buffer hit on that and not have that disoriented experience.
If their qualities actually need to drop because their bandwidth is down, then that's totally fine, but like we don't want those momentary blips. Short stream switches are really bad for that, so we try to track that also.
Fabio Ribeiro: Yeah. I've been looking at the fact that if the people switch a lot between levels, they stop watching the content to watch the quality. Startup time is something that's really hard for us, because we do have pre-roll, and this hurts a lot, the startup time.
Zac Shenker: We're all talking about a lot of the same metrics here, but in general, it's still a big problem in the industry of actually defining how exactly you measure each of these metrics. Is the startup time that we're measuring the same as what anyone else is doing, or the same when understanding what a rebuffer is, or what any of these sort of metrics actually mean and exactly how they should measure ... You know, I think it's good that there's groups like the CTA working group that are getting together now to try and actually define standards and write papers around suggested ways to actually measure these, that hopefully eventually the industry at large and multiple vendors can actually implement in the same way, so that we all have consistency when we're talking about what these things actually mean.
Related Articles
SSIMWAVE Chief Science Officer Zhou Wang discusses the shortcomings of traditional QoE metrics in this clip from his Video Engineering Summit presentation at Streaming Media West 2019.
23 Aug 2019
At Content Delivery Summit 2017, Jon Alexander explain why rebuffering metrics are so critical to gauging end-user QoE and pinpointing delivery issues, and how analysis of those metrics can help content publishers optimize delivery with great specificity.
22 Feb 2018
Conviva's Ed Haslam discusses the combination of QoE and other metrics that contribute to a data-driven experience strategy for looking beyond content to improve user engagement with online and OTT video.
04 Jan 2018