Video: Network Diagnostic Tools for Your Video Engineering Toolbox
Learn more about video engineering at Streaming Media's next event.
Watch the complete video of this presentation from Streaming Media West, VES101: Tools & Techniques for Diagnosing & Resolving Problems in Large-Scale Events, in the Streaming Media Conference Video Portal.
Read the complete transcript of this clip:
David Hassoun: When we first start digging into some of the stuff that's over the wire--how many people have used Charles? A good chunk of you. If you haven't, go take a peek at it. It is, I would say, a pretty industry-standard tool. You also have your network diagnostic tools and your browsers and so forth, but if you're working on devices as well, this testing is also meant to cover Fire TV, Apple TV, mobile device, and so forth. Charles is a really good, relatively easy-to-use tool for monitoring your network traffic and being able to see what's happening, as well as taking it further to allow you to test and force different scenarios to see what happens.
Charles Proxy
We use Charles a lot when we need to test our stream failover capabilities. If we have multiple run dish and sets within a manifest and we want to make sure that everything's failing over from A to B to C or whatever, often times we'll work with Charles and set up rules within there that we can turn on blacklisting. Or we'll test bandwidth fluctuations at an extreme. Charles is a great tool for that. It's relatively cheap as well, pretty robust, and works pretty dang good.
There can be some issues one the Linux side, but normally it works very, very well. It can also be used, like I mentioned, you can proxy stuff over when you're using things like Fire TV or your mobile device. Some of them are easier than others, to be able to proxy those into your machine, but it's doable and we're going to talk about some options there.
Charles also will open up a HAR file, just in case you guys didn't know that. A HAR file, if you have your network diagnostic tools in a browser and you're saving that out, that's normally going to be in a HAR file. It a relatively open interchange format, and that an be handy. We'll also see a little bit more about that later on.
Wireshark
When you need to go deeper on the network, Wireshark. I'm sorry to anyone who's had to use Wireshark a lot. It can be a bit overwhelming. They key thing about working with Wireshark is you see everything going on with the network. Not just your normal HTTP, HTTPS, TCP-based traffic. The key thing, if you ever have to work with Wireshark, is get a grasp on the filters. The bare minimum: HTTP and TCP; put that in there. That will be helpful. That will at least narrow it down from the fire hydrant to a fire hose. There's still room for growth in there and it really comes down to using those filters appropriately to dig stuff out.
Ultimately though, with either of these type of tools, or your network tools in general, the idea is that when you're working on a major live event, specifically, or even if it's on-demand content, the base level thing is you're looking for things other than 200s. Right? So 400s, 500s. Those are your HP status codes that indicate that something's failing, that it didn't get delivered.
Generally most of these tools make these things in red, in some way so that you can see them a little bit quicker. Identify those and then you can kind of work your way back from there as to, "Okay, why is this happening, and who do I need to call and yell at?"
Fire TV
A couple of little hidden gems for Fire TV. If you don't know about this, there are some hidden depth tools on Fire TV, a little key combination or command line, you can open those up. And it actually allows you then to be able to set proxy so it makes it very easy to send that traffic through to your personal machine so you can network monitor it, as well as, it shows you things like your CP usage, your memory usage, and so forth. It lets you overlay this stuff on top of your whole Fire TV device. Really, really handy and it's super-valuable. It also shows you your network throughput and so forth. It's one of these things that I get surprised that it's not well-know--but once you know it, you're never going to forget it.
Apple TV
For Apple TV, that can be a little bit harder sometimes to proxy. Working with Apple TV can be a bit harder to proxy at times, but still doable.
Network Log
One of our challenges though is that whenever we're working with any customers, the first thing when we start seeing problems is like, "Well did you get a network log? Can we see it? Can you send that over to us?" Those of you who worked with Charles, you can know, especially working streaming video, these can be logs that get very large, very fast at one point and sharing them can be difficult. Honestly ,if you want to be able to do this proxy and often times, especially on some of the Apple devices to go to yours, you end up having to actually use a Mac to be able to do that. It can become very expensive, overall, to start having a bunch of these proxy devices.
RealEyes MOE:Cap
We have been working on these little guys; we have a bunch of these raspberry pies and we set up this distributed tool called MOE:Cap. We can just plug these in with a network cable and these show up as a wireless access point. We say, "Okay, great. From my internet on my device or whatever I want, just use this as my access point." It allows us to monitor and get logs very, very quickly and easily. We can then save those logs to a HAR file and then share those.
But the cool thing as well with this is that we have five of these in our office right now and we set up our organization. Even from here, we can actually look at the live network log going on at a streaming device in my office, and be able to save and pull that out here. So I can give these to our customers and say, "Hey guys, listen, we want to do some testing with you guys. You're in a separate network, even if it's behind VPN and so forth, if you can set up one of these guys and use this as your access point for the devices you want to test on, we can do this collaboratively with you and see that network traffic coming through. It's really, really handy."
Ultimately, you can just view stuff randomly, but you can start saving your captures. When you save them, you can either convert them to HAR if they haven't been or download a HAR if they have. Once you start creating a session, you give it a name and it'll date it. Then you can see a GUI to go in here and actually be able to start working with it. Once it's done, you can come back and say, "Okay, stop capturing," and then you can convert it.
Like I said, I had one of my people kick on one of the devices beforehand so here you can see these are all the devices that we currently have in this group. There's four that are online, two that aren't. I know I'm PI-19, but here I can just go ahead and lets take a look and lets see if this thing is still actually running and capturing traffic. So the nice thing with this, I'm able to actually monitor network log for devices in a separate location.
Relatively real-time, it's pulling in the data right now. Since this thing has been running for a while, it can take a little bit. Scroll down, we can see new data coming in and this is it's streaming out content as we go right now. I can click onto these, I can see the response. I can get the same type of data. I can download this as a HAR file for an individual response right now, or I can create and download the whole session that's been recording now for and hour plus from our dashboard. You can see here this new task is currently capturing and it keeps going until I want to stop it.
So this is something that we've had a lot of fun building. It's really cool because it's all distributed but it gives a good route for working with these type of network captures. We're just opening up a private BETA on some of this stuff too, so if you guys want to start playing with some of these devices, can let me know. Like I said, they'll be a form in the end and you can play around with those too.
Related Articles
The CEO of RealEyes shares his personal video engineering toolbox, which contains solutions for diagnosing, fixing, analyzing, and interpreting problems with streaming video.
03 Jun 2019
RealEyes CEO David Hassoun discusses more playback-testing tools, including RealEyes MOE:Viewer, in this clip from his presentation at Video Engineering Summit at Streaming Media West 2018.
11 Feb 2019
RealEyes CEO David Hassoun discusses Beamr View, Telestream Switch and other tools you can use to test playback for your streaming content in this clip from his presentation at Video Engineering Summit at Streaming Media West 2018.
08 Feb 2019
RealEyes CEO David Hassoun discusses Elecard Stream Analyzer, FFprobe, and other tools for deep content analysis in this clip from his presentation at Video Engineering Summit at Streaming Media West 2018.
23 Jan 2019