Buyers' Guide to Streaming Playback Testing Tools 2019
I've been working in the digital and streaming industry for about 20 years, the last 15 as CEO of RealEyes Media. In that time, our company has worked on three Olympic Games, two Super Bowls, and thousands of live events in between, in addition to developing streaming media applications.
Get instant access to our 2019 Sourcebook. Register for free to download the entire issue right now!
I love the work most of the time, but I hate it when things aren’t going well. My goal is always to find ways to minimize issues in our live streams. To this end, I’ve assembled a video engineering toolbox with solutions for diagnosing, fixing, analyzing, and interpreting problems with streaming video. Some of these tools were created by other companies, and others were developed by our team at RealEyes and are available to the industry to use.
The solutions in my toolbox break down into three categories:
- Network-type “over the wire” tools that analyze what’s being transferred to and fro in a live stream
- Tools that go “under the covers” and attempt to identify what’s actually in the video that’s causing problems with the stream
- Tools that address what I call “the last 2 feet,” players that provide diagnostics on issues that are specific to delivery.
To learn more about “over the wire” and “under the covers” solutions, check out my Video Engineering Summit presentation from Streaming Media West 2018, Tools & Techniques for Diagnosing & Resolving Problems in Large-Scale Events.
These “last 2 feet” issues can prove particularly tricky, and that’s what we’ll focus on in this roundup. I don’t use all of these solutions every day, but I’ve used all of them at one point or another, and each one offers different features and benefits that I’ll discuss in this article. For each tool, I’ll talk about what to look for, why you would want it, and what you need to be able to use it. Some of these tools are pretty basic, but still useful in their own way; others get pretty deep.
VLC Player
We’ll start with a player that should be familiar to just about anyone reading this article. Although limited in many ways, VLC Player (Figure 1) is a great player for getting started with testing, just to see how things work.
Figure 1: VLC Player
It doesn’t give you a whole ton of insights, but it’s a good baseline to make sure your video is playable in a general sense. The tricky thing with VLC Player is that I’ve seen it play a lot of things that other players can’t handle. It’s a little bit too good in that regard.
Is it a great litmus to test to say, “OK, well, if it plays in VLC it should play anywhere”? No. More likely, it’s the opposite. If it doesn’t play in VLC player, you can be fairly sure it’s not going to play almost anywhere, unless it’s due to authentication, token issues, that type of stuff. But that’s why you use your network tool up front.
Beamr View
Unlike VLC, Beamr View (Figure 2) is designed for comparative quality analysis. It’s very useful for comparing quality between different bitrates, or when you have two different videos and one has a problem, but they’re supposed to be the same. It’s also good when you’re just encoding video and you want to be able to visually validate stuff. Beamr View is not a conclusive tool but instead provides an excellent way to make subjective visual decisions in a comparative A/B fashion, and it does that very well.
Figure 2: Beamr View
One of the great things about Beamr View is it’s free. Beamr doesn’t make it sound like it’s free when you go to the site, but it actually is. You just hit them up and say, “Hey, I want to use this, is that cool?” and they’ll say, “Sure, go for it. As long as you’re not doing something you really shouldn’t be.” Beamr also has amazing encoding tools that may also be free, depending on your level of usage.
Beamr View’s comparative analysis is mostly A/B, and it does frame-by-frame. You can get indications of data up on top, as shown in Figure 2. In the clip shown in Figure 2, even though these are separate resolutions, I’ve told Beamr View to scale to match so I can see. It also has a couple different ways that you can look at your videos, such as moving the vertical line that splits the screen between the two different versions to see the quality benefit of one versus the other.
Beamr View also has a Butterfly View that flips one clip horizontally so you see the clips in a mirror image of each other. There’s also a Dual View and Overlay. You can toggle between all of these views. Split View (shown in Figure 2) is my favorite because it lets you do the back-and-forth.
Beamr View is also good for testing playback, plus handling emerging codecs such as HEVC. So when you’re wondering, “Am I really going to get a benefit from HEVC versus AV1 or H.264?” this can be a really good place to start evaluating that. (Let’s be clear: You should get a benefit.)
Telestream Switch
One of my personal favorite “last 2 feet” tools—and the one I use probably the most—is Telestream Switch (Figure 3). It’s a paid player, and not a cheap one ($499.99 for the Pro version that includes the diagnostic tools).
Figure 3: Telestream Switch
What justifies the cost is Switch’s ability to fix things, especially subtitles, closed-caption data, and so forth. Switch is one of the few tools that actually allows you to pull out your 608/708 data that’s in-band and fix the problems it diagnoses.
Figure 3 shows the player in Telestream Switch Pro. All the little markers you see are actually showing your I-frames, P-frames, and B-frames. The yellow lines represent the I-frames, the purple lines the P-frames, and the blue lines are the B-frames.
Why is this important? If we have content that doesn’t start on a keyframe, bad things happen. We always need that I-frame in the beginning, or a number of issues can arise as a result of that. One of the first things I always have to check is, what’s our keyframe interval? And is that keyframe cadenced properly? If we set a keyframe to appear every 2 seconds, we need to make sure that that happens and continues consistently throughout the video for each segment. Not doing so can cause issues in certain platforms, especially when you start switching between content, such as ads. We have seen a lot more issues on OTT set-top devices struggling with improper or missing keyframes on segments, which can lead to application crashes mid-stream during a live event.
You can see additional keyframes in Figure 3. Typically, these occur on scene changes. It’s fine to have additional keyframes when we need them, but we need to always make sure to maintain our cadence with the set-interval keyframes. One weird thing that happens sometimes in Switch (for reasons I have yet to figure out) is that the player shows a blank keyframe in the beginning. But if I run the video through other tools, such as FFprobe, it won’t indicate that there’s an extra keyframe there.
Extra keyframe aside, I find Switch to be a super-valuable tool. It has helped me identify a lot of problems in the past, especially when we’re trying to do intermixed content, or testing for potential encoding issues. If a stream gets cut and packaged improperly, Switch is a great visual tool to allow me to actually see that.
Also, if we’re seeing bad things happening when we play back the video—bad blocking, for example—being able to play it frame-by-frame in Switch is really handy. But it does have its downsides, specifically around its cost.
RealEyes MOE:Viewer
Figure 4 shows a tool called MOE:Viewer that we put together at RealEyes. MOE stands for Media Orchestration Engine. This is what we use for live or VOD and monitoring our clients’ streams. One thing we’ve found with live events over the years is that we really need to pay attention to what’s happening in the manifest. Is it updating properly? Did we have a stall? How is what’s in the manifest relating to the content? Being able to test this stuff very quickly while carrying out all of the other tasks involved in a live stream just became too much to handle reliably.
Figure 4: RealEyes MOE:Viewer
We can keep curling, and we can keep loading in the browser, but we wanted to have a tool that provided these features and correlated to visuals. So we created MOE:Viewer.
In MOE:Viewer, which we used to call Manifest Viewer (Figure 5), you can save your sessions via a drop-down in the Manifest Viewer dialog. If put in a stream that I want to be able to continually test, I can just save it as a session and come back to it anytime I want. The one shown in Figure 5 is a VOD file. If you’re going to use XMLHttpRequest (XHR) credentials, you can set a global token in there, if you have token access.
Figure 5: A saved session in Manifest Viewer
Related Articles
Stream live with low latency. Collaborate on edits. Distribute to multiple platforms. Analyze data to make better choices. Here are the innovation leaders every streamer needs to know about.
08 Nov 2019
RealEyes CEO David Hassoun discusses more playback-testing tools, including RealEyes MOE:Viewer, in this clip from his presentation at Video Engineering Summit at Streaming Media West 2018.
11 Feb 2019
RealEyes CEO David Hassoun discusses Beamr View, Telestream Switch and other tools you can use to test playback for your streaming content in this clip from his presentation at Video Engineering Summit at Streaming Media West 2018.
08 Feb 2019
Advances in monitoring and analytics anchor superior OTT user experiences. Here's a road map for pre-event testing that will ensure high-quality performance across the board.
04 Feb 2019
RealEyes CEO David Hassoun discusses Charles, Wireshark, Apple TV Proxy, and other network diagnostic tools in this clip from his presentation at Video Engineering Summit at Streaming Media West 2018.
16 Jan 2019
Streamroot's Nikolay Rodionov walks viewers through the key elements of an effective A/B testing workflow for testing video players during development.
19 Dec 2016
Even SMBs need to think about load testing these days to ensure they can serve audience demand. Here are three questions to consider.
03 Oct 2016