Cloud Production for Live Sports

Learn more about cloud production and live sports streaming at Streaming Media East 2022.

Read the complete transcript of this clip:

Eric Schumacher-Rasmussen: I'd like to welcome Jef Kethley, live producer extraordinaire, who is coming to us from his fancy studio in a trailer in Florida. Hey, Jef.

Jef Kethley: Hey there. Great to be here, actually. Glad we could make some time to get together.

Eric Schumacher-Rasmussen: Yeah, I'm glad we could, too. You were on a panel yesterday about live production in the cloud and Corey Behnke moderated it. Cory's name will be familiar to some of the people on this who are watching now. I'm sure. And he made a really good point that we've talked a about how things have changed since the pandemic began and how the trend toward remote and distributed production has really, really taken off. And of course it has, but live producers like you and Corey have been working and moving towards remote and distributed production workflows for quite a long time. So what led you to where you are today?

Jef Kethley: Well, it happened about nine years ago, actually. We've been doing this tennis contract for professional tennis with my company LiveSports for almost 10 years now. And what happened was, we booked our local freelance talent like most production companies were doing at the time. Well, they became sick and could not make it to the event. Then it was like, where do we find somebody to take their spot and take over their responsibilities? It's chasing the scoring and stuff. It's not just throwing anybody down and letting them follow because you have to kind of know the game a little bit.

Eric Schumacher-Rasmussen: Just to make clear, a large portion of what you do is tennis, right? You're covering live tennis.

Jef Kethley: Yeah, before March 13th, 2020, the magic, day we were doing over 45 tournaments a year--hours and hours and hours of professional tennis. We had to keep up with the scoring and everything at the time. And the graphics process, very key on that, because our sources were going out to gaming houses across the the world. So we looked around, tried to find somebody local, and we just didn't have the staffing in there to take their place. So we did one day, then I immediately called up one of my guys that was back in the office that was getting ready for another show and said, "Hey, can you jump on TeamViewer and just watch this for an hour? I'll give you video and audio over Skype so you can see and hear what's going on and do the updates for the scoring so we can kind of catch our breath and stuff."

And he said, "An hour? Yeah, sure. No problem." So he jumped on and it was like three hours later, and he said, "Do you still need me?" I said, "Is it working?" He said it was, and I said, "Okay, well, I'll plan on this for the next few days."

Jef Kethley: And that was the start of it all. And it clicked: We could actually do distributed workflows from anywhere. And so we started moving deep into it. Doing more and more graphics was easy. We started moving into doing just monitoring of the sources as an encoding engineer from our master controls or, or from people's homes even.

But the next big step was about five years ago when we invested in Mark Roberts Motion Control cameras, with camera heads that actually are robotic. They're all IP-based. So we could do recordings and all our shooting and everything from inside the truck--which is a 1000-2000 feet away--on fiber, but it's IP-based. So immediately I thought, "Can't we just stretch that IP a little further?" And that was the next step. And we were controlling cameras remotely, and that was something that very few people had done until the pandemic hit. We were slowly doing proof of concept in developing and making it our workflow.

The next step really came around when we had to. We stepped into the pandemic, and we moved from things that would have taken us five years to do, and we did them in what felt like five days. And so a lot of our applications that we were already doing things remotely, or just in remote operations--that was what propelled us really fast to move that away in the cloud.

Eric Schumacher-Rasmussen: In one of our previous discussions, you mentioned that there's, there's all sorts of side benefits to that for you covering so much tennis, you're getting your crew off of 120-degree courts.

Jef Kethley: Absolutely. That was one of the first things. It was like, "Hey guys, you, you can work in the A/C in the truck." The technical director and the rest of the graphics guys, and everybody else is there. But the big thing was they got off the court, and they were like, "I'm not nearly as tired as I was." They could do a 10 or 12 hour day and not feel like they're just completely burned out from being out in the sun at 120 degrees.

And then the next part that came after that--and this is where the really the big investment in distributed workflow and remote workflows came in--was allowing them to work from home or work from the office. That allowed them to have a better quality of life. And that was more important me than saving a flight here and there, or hotels or food or whatever. That was way more important because I've had staff that have stayed around after we made the switch where I was burning through camera operators and things like that after months. It was rough. It's a tour, so it's every week, every day, for weeks at a time moving crews around. It's a tough life. I think of bands moving all over the place--that's the kind of tour we were doing. So even though we had tour buses, and we were doing everything we could do to make it as easy on the crew, we found that by giving them the opportunity to work from home or work from the office and be home with their family and sleep in their own bed, it became a bigger thing. That was where it's just clicked. It's not about saving the money. It's about making it better for the crew and the employees. And I think everybody's kind of seen that now, that work-from-home is not a fad. It's a thing that's gonna stay around for a while.

Eric Schumacher-Rasmussen: For sure, for sure. And all of this led you to what we're going to focus on today, which is a new or relatively new platform-as-a-service that you're bringing to market called MasterControl.Cloud. I'm gonna let you talk about that and show it all.

Jef Kethley: We're not gonna do a lot of showing off today because it really is a very customized system. But let's look at it this way: Whenever we build systems out, we build workflows out. We try to do that in everything. As a longtime system integrator, over 30 years, we would always qualify with the client to find out exactly what their needs are. What are their pain points? What can we do to help them? And it's the same thing we do when it comes into our cloud infrastructures. Some people just need a lot of replication. Say, for instance, our Tony Robbins clients. Our Eric Worre with Worre Studios. Those guys just needed a lot of distribution. So that was one place where we just excelled in the cloud during the pandemic because nobody else was doing it. We were setting up Zoom farms instead of 10 Microsoft laptops 10 Apples or whatever, or 100 Apples, whatever it was, you couldn't find the gear, you couldn't get it together. We replicated that workflow in the cloud. We actually wrote as many as 180 Zoom machines running in the cloud that were replicating out all this content that we were setting up from their studios. And so that's one direction we went.

As a business owner and with LiveSports, we were always looking at how much could we bring to the shoot, to the sit,e and minimize the amount of big trucks. And so we went from a crew and our tour buses--we had a crew of 6, 7, 8 people that were traveling around--down to a crew of three, or two sometimes. And now we're on an average of two that could do the same job. We might have 2 people on site, but we may also have 5 or 6 people that are offsite that are working with the crew producing the event. So we went from these big vehicles to smaller vehicles. And then the next step after that was saying, how small can we go? We could actually make this really efficient by bringing all theswitching infrastructure and all the graphics and all the replay and everything like that up into the cloud, because the cloud is where we can be even more flexible.

So if we need to have three different events running, no problem. We just copy and paste and make three different master control systems. Then the next step would be, maybe we need to have 20 different events running. And that's where we started working with the other clients, as consultant, building out AWS infrastructure, Azure infrastructure, putting our NDI processing engine in, which is from Sienna. That's one of our big pieces of glue that kind of hold everything together. And then, as we were building those out, they were coming back to us, saying, "Okay, we've got that one set up, but we don't have the crew to help do more. And we need some help. So we just started offering this platform that we're talking about now, which is the same tools that we've been using.

So it might be a Viz Vectar video switcher in the cloud. It might be a NewTek 3Play for instant replay, or it may be a couple of vMix machines, or maybe 10 vMix machines. The same tools that we're using, we're just duplicating and using in the cloud. But my team is managing it and helping them run the show. And it could either be a hand-off-the-keys, here's the login information, go have fun, and call me if there's a problem. Or other times, they'll say, "Well, how many vMix operators do you know?" It's like, "About 50, how many do you need?" And the thing with the Viz Vectar Plus, is it's a TriCaster in the cloud. So there's a lot of TriCaster users out there. It's very simple for them. Once they get in their mindset, that they're just controlling a virtual machine they don't have to be sitting in front of ... Once they get to that mindset, then they see it's the same workflow. It's no different. It's just NDI sources in, NDI sources out, audio in and out, a bunch of different ways. It's just little slight tweaks in there, but that's where it really kind of moved into our platform of being able to give people that, that backing of, "Well, you need more? Sure, no problem. Let's turn up some more."

Eric Schumacher-Rasmussen: We've been talking about the cloud for years. It always surprises me when I go to someplace like NAB or IBC, and I'm reminded that there's still some resistance to the cloud. What do you think are the biggest concerns people have with moving workflows to the cloud, and how do you overcome those concerns when you're talking to them?

Jef Kethley: The biggest obstacle, without a doubt, is just lack of understanding. They there are people that dive in really quick because they see the small numbers that Amazon puts out there like, "Oh, you could rent this virtual machine for $2 an hour." But you don't just turn on a machine for an hour broadcast. You need to actually have it up and do prep and do some more work after the event for post editing and things like that. So it's more than that, but it's also bandwidth cost, it's infrastructure cost. But most people, when they start thinking about cloud, they think it's harder than it really is. It is hard. I'm never going to belitle that fact. It's not a simple thing, but it is one of those things that, once you find people that are like us that are really fluent in it, it becomes easier and easier. That's our goal with the MasterControl.Cloud. That is gonna be the way to get people into cloud production to make it easy. And that's what we're trying to do, using the same tools that they're used to using in an on-prem type of environment. They could get that in the cloud and we could manage it for them, or we could hand them the keys and they could go to town, whichever way.

When we developed MasterControl, it really is about more than just a switcher. It's not just vMix in the cloud. It's Viz Vectar tied in with the Sienna NDI processing engine. It's tied in with replay, if we need replay. It's tied graphics. We've got Kairos working in the cloud, we've got Ross XPression working in the cloud. We've got Vizrt in the cloud with the Trio. And then the last little piece of smaller software that the last two or the live BlueFx, which some people are familiar with and CharacterWorks. So we've got five different graphics packages, depending on what you need. That is what we're trying to do. We're trying to tailor it to what they're familiar with, and then also give them the exact tools they would have in a production truck.

Eric Schumacher-Rasmussen: Sure. I would guess that you've seen demand for what you're offering just skyrocket over the last couple years.

Jef Kethley: It definitely has. At first, we were using this all internally, and we've been doing the same platform for four or five years now. It really was one of those things where we started off not really thinking about it as a product, as something that other people would be into, but it really was the pandemic that turned that around. People are asking for this and they need it. I looked it as a way of improving our own efficiency, and as we became more and more efficient and better at using it, it helped us. And now I see as an advantage that we can help other people too.

Eric Schumacher-Rasmussen: I'd like to welcome Jef Kethley, live producer extraordinaire, who is coming to us from his fancy studio in a trailer in Florida. Hey, Jef.

Jef Kethley: Hey there. Great to be here, actually. Glad we could make some time to get together.

Eric Schumacher-Rasmussen:

Yeah, I'm glad we could, too. You were on a panel yesterday about live production in the cloud and Corey Behnke moderated it. Cory's name will be familiar to some of the people on this who are watching now. I'm sure. And he made a really good point that we've talked a about how things have changed since the pandemic began and how the trend toward remote and distributed production has really, really taken off. And of course it has, but live producers like you and Corey have been working and moving towards remote and distributed production workflows for quite a long time. So what led you to where you are today?

Jef Kethley: Well, it happened about nine years ago, actually. We've been doing this tennis contract for professional tennis with my company LiveSports for almost 10 years now. And what happened was, we booked our local freelance talent like most production companies were doing at the time. Well, they became sick and could not make it to the event. Then it was like, where do we find somebody to take their spot and take over their responsibilities? It's chasing the scoring and stuff. It's not just throwing anybody down and letting them follow because you have to kind of know the game a little bit.

Eric Schumacher-Rasmussen: Just to make clear, a large portion of what you do is tennis, right? You're covering live tennis.

Jef Kethley: Yeah, before March 13th, 2020, the magic, day we were doing over 45 tournaments a year--hours and hours and hours of professional tennis. We had to keep up with the scoring and everything at the time. And the graphics process, very key on that, because our sources were going out to gaming houses across the the world. So we looked around, tried to find somebody local, and we just didn't have the staffing in there to take their place. So we did one day, then I immediately called up one of my guys that was back in the office that was getting ready for another show and said, "Hey, can you jump on TeamViewer and just watch this for an hour? I'll give you video and audio over Skype so you can see and hear what's going on and do the updates for the scoring so we can kind of catch our breath and stuff."

And he said, "An hour? Yeah, sure. No problem." So he jumped on and it was like three hours later, and he said, "Do you still need me?" I said, "Is it working?" He said it was, and I said, "Okay, well, I'll plan on this for the next few days."

Jef Kethley:

And that was the start of it all. And it clicked: We could actually do distributed workflows from anywhere. And so we started moving deep into it. Doing more and more graphics was easy. We started moving into doing just monitoring of the sources as an encoding engineer from our master controls or, or from people's homes even.

But the next big step was about five years ago when we invested in Mark Roberts Motion Control cameras, with camera heads that actually are robotic. They're all IP-based. So we could do recordings and all our shooting and everything from inside the truck--which is a 1000-2000 feet away--on fiber, but it's IP-based. So immediately I thought, "Can't we just stretch that IP a little further?" And that was the next step. And we were controlling cameras remotely, and that was something that very few people had done until the pandemic hit. We were slowly doing proof of concept in developing and making it our workflow.

The next step really came around when we had to. We stepped into the pandemic, and we moved from things that would have taken us five years to do, and we did them in what felt like five days. And so a lot of our applications that we were already doing things remotely, or just in remote operations--that was what propelled us really fast to move that away in the cloud.

Eric Schumacher-Rasmussen: In one of our previous discussions, you mentioned that there's, there's all sorts of side benefits to that for you covering so much tennis, you're getting your crew off of 120-degree courts.

Jef Kethley: Absolutely. That was one of the first things. It was like, "Hey guys, you, you can work in the A/C in the truck." The technical director and the rest of the graphics guys, and everybody else is there. But the big thing was they got off the court, and they were like, "I'm not nearly as tired as I was." They could do a 10 or 12 hour day and not feel like they're just completely burned out from being out in the sun at 120 degrees.

And then the next part that came after that--and this is where the really the big investment in distributed workflow and remote workflows came in--was allowing them to work from home or work from the office. That allowed them to have a better quality of life. And that was more important me than saving a flight here and there, or hotels or food or whatever. That was way more important because I've had staff that have stayed around after we made the switch where I was burning through camera operators and things like that after months. It was rough. It's a tour, so it's every week, every day, for weeks at a time moving crews around. It's a tough life. I think of bands moving all over the place--that's the kind of tour we were doing. So even though we had tour buses, and we were doing everything we could do to make it as easy on the crew, we found that by giving them the opportunity to work from home or work from the office and be home with their family and sleep in their own bed, it became a bigger thing. That was where it's just clicked. It's not about saving the money. It's about making it better for the crew and the employees. And I think everybody's kind of seen that now, that work-from-home is not a fad. It's a thing that's gonna stay around for a while.

Eric Schumacher-Rasmussen: For sure, for sure. And all of this led you to what we're going to focus on today, which is a new or relatively new platform-as-a-service that you're bringing to market called MasterControl.Cloud. I'm gonna let you talk about that and show it all.

Jef Kethley: We're not gonna do a lot of showing off today because it really is a very customized system. But let's look at it this way: Whenever we build systems out, we build workflows out. We try to do that in everything. As a longtime system integrator, over 30 years, we would always qualify with the client to find out exactly what their needs are. What are their pain points? What can we do to help them? And it's the same thing we do when it comes into our cloud infrastructures. Some people just need a lot of replication. Say, for instance, our Tony Robbins clients. Our Eric Worre with Worre Studios. Those guys just needed a lot of distribution. So that was one place where we just excelled in the cloud during the pandemic because nobody else was doing it. We were setting up Zoom farms instead of 10 Microsoft laptops 10 Apples or whatever, or 100 Apples, whatever it was, you couldn't find the gear, you couldn't get it together. We replicated that workflow in the cloud. We actually wrote as many as 180 Zoom machines running in the cloud that were replicating out all this content that we were setting up from their studios. And so that's one direction we went.

Jef Kethley: As a business owner and with LiveSports, we were always looking at how much could we bring to the shoot, to the sit,e and minimize the amount of big trucks. And so we went from a crew and our tour buses--we had a crew of 6, 7, 8 people that were traveling around--down to a crew of three, or two sometimes. And now we're on an average of two that could do the same job. We might have 2 people on site, but we may also have 5 or 6 people that are offsite that are working with the crew producing the event. So we went from these big vehicles to smaller vehicles. And then the next step after that was saying, how small can we go? We could actually make this really efficient by bringing all theswitching infrastructure and all the graphics and all the replay and everything like that up into the cloud, because the cloud is where we can be even more flexible.

So if we need to have three different events running, no problem. We just copy and paste and make three different master control systems. Then the next step would be, maybe we need to have 20 different events running. And that's where we started working with the other clients, as consultant, building out AWS infrastructure, Azure infrastructure, putting our NDI processing engine in, which is from Sienna. That's one of our big pieces of glue that kind of hold everything together. And then, as we were building those out, they were coming back to us, saying, "Okay, we've got that one set up, but we don't have the crew to help do more. And we need some help. So we just started offering this platform that we're talking about now, which is the same tools that we've been using.

So it might be a Viz Vectar video switcher in the cloud. It might be a NewTek 3Play for instant replay, or it may be a couple of vMix machines, or maybe 10 vMix machines. The same tools that we're using, we're just duplicating and using in the cloud. But my team is managing it and helping them run the show. And it could either be a hand-off-the-keys, here's the login information, go have fun, and call me if there's a problem. Or other times, they'll say, "Well, how many vMix operators do you know?" It's like, "About 50, how many do you need?" And the thing with the Viz Vectar Plus, is it's a TriCaster in the cloud. So there's a lot of TriCaster users out there. It's very simple for them. Once they get in their mindset, that they're just controlling a virtual machine they don't have to be sitting in front of ... Once they get to that mindset, then they see it's the same workflow. It's no different. It's just NDI sources in, NDI sources out, audio in and out, a bunch of different ways. It's just little slight tweaks in there, but that's where it really kind of moved into our platform of being able to give people that, that backing of, "Well, you need more? Sure, no problem. Let's turn up some more."

Eric Schumacher-Rasmussen: We've been talking about the cloud for years. It always surprises me when I go to someplace like NAB or IBC, and I'm reminded that there's still some resistance to the cloud. What do you think are the biggest concerns people have with moving workflows to the cloud, and how do you overcome those concerns when you're talking to them?

Jef Kethley: The biggest obstacle, without a doubt, is just lack of understanding. They there are people that dive in really quick because they see the small numbers that Amazon puts out there like, "Oh, you could rent this virtual machine for $2 an hour." But you don't just turn on a machine for an hour broadcast. You need to actually have it up and do prep and do some more work after the event for post editing and things like that. So it's more than that, but it's also bandwidth cost, it's infrastructure cost. But most people, when they start thinking about cloud, they think it's harder than it really is. It is hard. I'm never going to belitle that fact. It's not a simple thing, but it is one of those things that, once you find people that are like us that are really fluent in it, it becomes easier and easier. That's our goal with the MasterControl.Cloud. That is gonna be the way to get people into cloud production to make it easy. And that's what we're trying to do, using the same tools that they're used to using in an on-prem type of environment. They could get that in the cloud and we could manage it for them, or we could hand them the keys and they could go to town, whichever way.

When we developed MasterControl, it really is about more than just a switcher. It's not just vMix in the cloud. It's Viz Vectar tied in with the Sienna NDI processing engine. It's tied in with replay, if we need replay. It's tied graphics. We've got Kairos working in the cloud, we've got Ross XPression working in the cloud. We've got Vizrt in the cloud with the Trio. And then the last little piece of smaller software that the last two or the live BlueFx, which some people are familiar with and CharacterWorks. So we've got five different graphics packages, depending on what you need. That is what we're trying to do. We're trying to tailor it to what they're familiar with, and then also give them the exact tools they would have in a production truck.

Eric Schumacher-Rasmussen: Sure. I would guess that you've seen demand for what you're offering just skyrocket over the last couple years.

Jef Kethley: It definitely has. At first, we were using this all internally, and we've been doing the same platform for four or five years now. It really was one of those things where we started off not really thinking about it as a product, as something that other people would be into, but it really was the pandemic that turned that around. People are asking for this and they need it. I looked it as a way of improving our own efficiency, and as we became more and more efficient and better at using it, it helped us. And now I see as an advantage that we can help other people too.