Streaming the North Texas Irish Festival: A Cloud Production Case Study

Tim: We have some other issues as well with SRT and getting it set up. It's just different. I think you had a fun little challenge with every time you sent an SRT signal to a port and stopped it, and you could never use that port again on that machine without rebooting it. It's just weird stuff, right?

Anthony: The whole thing was a learning experience in many different ways. SRT in and of itself is a learning experience because there's the caller, and there's listener, but then there's a separate set of terms like the caller and the receiver, and all four of those terms seem to be completely interchangeable depending upon who can actually set their port forwarding. So, whoever can set their port forwarding, then I can call to that port, even if I'm the listener, I can call, which makes no sense in terms of being logical. But that's the way it works. And also, as you said, an end-user, we all had backups in place, because it was, "Well, we were going to go local, but then we don't know about power, so let's go with the backup, which would have been AWS." But then it was, "Let's just go with that as our primary and use our local as a backup." But having backups in place regardless of what your plan is, is always important. I had my entire show loaded up on my laptop, and at any time I could click Play on that, and push things.

Tim: Yeah, definitely don't choose a festival of 30,000 viewers as your first time to run AWS. That's my advice to folks. There's just a lot of funky little things you have to learn and figure out, and a whole suite of things that can go wrong that you would have never thought of running it on your own desktop.
Anthony: It also brings up a very key point in terms of AWS, in that, making a change takes time.
Michelle: It does, it does. It takes at least 24 hours usually, but we were able to do it in two, which is unheard of. It depends on where your server is built, because some of them have them available and some of them don't, so they have to go make sure that that particular location has the servers available to do it.

Anthony: Right, because technically, there's so many millions of people using AWS that it isn't like you walk into a room and all of the server power is readily available only to you. They have to allocate your resources among everybody else's resources. And if you're on server A and your allocation just fits into A, but if now you went bigger and you you don't fit into A anymore, they have to literally move your allocation around between machines. And as far as I know, it's actually a manual process--

Michelle: I don't know about that. That's a great question. I do know that the G series--that stands for graphics, so those are the ones that have the GPU. They're in high demand, probably because of all of us producers going, "Wow, this is cool. Let's try it."

Tim: I know we're talking about the little glitches that occurred. But in all, we pulled it off, right? We had this fantastic experience. We had 39 hours streamed in a day across six channels. We had a little bit of a hitch with our feed from Ireland. So we haven't really talked about that. But one of the unique things I convinced the Irish Festival to do was to actually send a live feed from a stage in Ireland that hosted a whole bunch of local acts (Figure 4, below).

Figure 4. The North Texas Irish Festival's virtual stage streamed direct from Ireland

The bands that were there and even the headliners, all streamed from that location. And we actually had a Boxcast server purely coincidentally hosted in AWS that crashed and took down just one of our six streams, but it also took down 10 or 20 of their other customers. And so they had to scramble, and it was almost an hour before we got service back up again on that sixth instance. That's a lifetime in a live stream.

But the good news is, folks had five other channels to go watch and we got the channel back up and everything was good. Boxcast was in constant communications with us about what was going on, which was fantastic. That's why I selected them--their support is legendary. When you have an issue you can get people instantly. You're not waiting two hours or 24 hours to get a response.

Anthony: In a live event, that's critical. You've got to have answers. One of the key things that I think you did to, as they say, "de-risk" this event is have a lot of content that was prerecorded, but looked live, that was included in this event. And even my show--the Ireland show--because they were using a live stage and they would have to change the stage between acts, they prerecorded certain things so that when one act finished, I'd go to a small recording so that they could move stuff around, add a third seat, and things like that. And then we'd go back to the live show. So that prerecorded content really does help in terms of, "Listen, I need to shift something. Let's push a video." And we could do something in the background.

Tim: We were originally planning on doing it all live--having all these stages, and live feeds from every one of them. And I just got cold feet. I just said, "There's so much risk in this," and especially because of the ice storms and everything else. And of course you run into these issues with COVID and making sure you have to keep things socially distanced and masked up. And so you just can't have a big presence, in these venues--at that time, anyway.

In Texas, pretty much all of the events were prerecorded in the few weeks prior. So they're all dedicated, recorded for the festival. They weren't reused material. It was all original material, but we recorded it. I think we ended up having almost 150 different video segments that we were able to use and bring in. And then what we did is ... I'm not a fan of live streams of purely prerecorded material because I just feel they don't have the energy. So for all of those local artists that that we're recording, we actually brought them in on vMix Call for about five or seven minutes to have a little interview, talk about the music, talk about themselves. And that was went really well. We got huge praise for that because a lot of people had never met the bands. People have been going to this festival for 40 years. Some of these bands have been playing for 40 years. There's a band who played 40 years ago, that was in our group. But people had never met them or talked to them, and they had a chance to actually hear a live interview with them introducing the prerecorded material. That was a really good balance of energy with, reduced risk. We also could make the effort to get their music. We recorded the music in advance on a stage. And what we did with the bands is we recorded 55 minutes of music and we let them pick 45. So if there were tunes that they messed up, they could scratch them.

That gave them a little bit more comfort, because none of these bands had played in a year. So they were all really nervous. And so they all wanted us to let them get four or five takes and we said, "No, we're going. We're trying to give a live feel to the event. This is what we're filming. You can scratch a couple of them, so don't screw up too bad." But that was huge, in terms of de-risking it and giving us the ability to juggle when we needed to juggle.

Michelle: I would say that making sure you have more time to test--and that was really hard in this instance because of the storms and the lack of power and all that went down--but I think that we just didn't have enough time to go through the whole show. We should've gone through it a few times, but we didn't have any time. So, that would be my other advice--just test, test, test, and try to break it and test more. That's the way to go.

Anthony: So, testing and rehearsals. I've done shows where we didn't do a full-on rehearsal. They switched the tech check into setup day. And so we did that, and the next day was production and the first half of the day went great. And then, in the second half, where we had four callers on one machine, four callers on another machine, NDI going back and forth, and NDI going to an out-of-machine record and all this stuff. And then our internet on that one machine, the NIC got overwhelmed and would drop out for a couple of seconds. And they asked, "Well, what's going on?" And I said, "We didn't actually like have a moment where we did all of this and connected all these people at the same time to uncover that for whatever reason, the NIC in this one machine wasn't going to do a gigabit reliably. So, you gotta actually do it to know that it can do it.

Michelle: You do, and you have to also hit the Record button and hit the Stream button at the same time if you're using them both off the computer. Get those five callers in at the same time you're going to roll tape, or roll record. You should completely do exactly what you're going to do the day of the show, if you can.

Anthony: And then, like we were saying, if that seems to work, add a sixth caller, add a second record, a second stream, and find out: Where is the breaking point? Do you actually have overhead? Are right at the limit? And then look at the computer Task Manager, see where you are and everything. If you're doing all of this and your CPU and GPU are down at 30%, you're probably good.

Michelle: Exactly. And it definitely depends. You had an SRT and a WebRTC, and your computer instance was fine with that, right? .

Anthony: Well, like I said, we didn't even use the SRT. So I just had playbacks, record, and the one call in that was solid all the way through the day. It was the most magical thing. It was beautiful. It looked good. It sounded good. You couldn't tell the difference between the vMix Call and the recordings they had sent me. You just couldn't tell.

Michelle: That's great.

Anthony: I don't want to keep you too long. I just want to cover the audience, the distribution. How did it come across? How big was the audience? Where did it go?

Tim: It was fantastic. So, a normal Irish festival has about 60,000 people attend over three days. And we had 30,000 watch it live from 18 countries, which was really special because normally it's a very local audience that comes. We picked up a whole host of people who had gone to it over the years and moved away and were on the east coast. We had a lot in Europe as well (Figure 5, below). And they were able to watch it for the first time in many years. We got a whole host of emails from folks thanking us, and it was people that had gotten on in years and couldn't move around the festival like they used to, but were able to enjoy the festival experience for the first time in a decade.

Figure 5. Geographic distribution of the North Texas Irish Festival 2021 audience

It was really, really heartwarming to see just the huge response, and the positive response that we got. it was funny as well because, I've worked with the Irish festival for years, and at the end of the festival, there's always 30 or 40 people complaining that food was too expensive, you didn't have my favorite beer, parking was too far, whatever. You just get a lot of folks that that grumble a bit. We didn't get one complaint. That's astonishing.

We used it as a fundraiser as well to cover the cost of the event, and it far exceeded our goals. It was really well received by by the public. The board were just ecstatic. They put their trust in me. I don't think they exactly knew what they were getting until the day of. But when they saw it, and were able to interact and jump the stages and all of that stuff, they were just ... I got a really touching note from one of the board members basically saying, "The video quality actually made it better than the real thing in some respects. If I was sitting in a crowd in Dallas, there's no way I could have seen exactly how this person's fingers move on that accordion, or gotten that close." To him, it was a superior experience. And that's exactly what I was going for--that you could actually have a better experience than going there in person. So that was really appreciated.

Anthony: You gave me a note that says, "The festival is the largest in the state and normally gets around 60,000-plus attendees every year. And that's over several days." Now, the festival you created online was one afternoon long, and the Ireland part went into the evening, but the festival part was an afternoon long. And we had about 30,000 people in one-sixth time. And it continues to be viewed as well. So like you said, the reach is much different. And how do you think this will change how the festival is done in the future? When we can, do you think it's going to go back to completely in person, go hybrid, or continue to be online?

Tim: They really want to look at hybrid. As we're looking at next year, they've asked me to come up with some ideas for them. We're making the assumption it's in person and people will be able to move freely. But the ability to get access to maybe even the performances after, they're talking about having that live feed from Ireland every year, going forward, just because it was such a great connection to to have. And so we're talking about, "Well, how do we do that? Do we know what we're talking about?" Actually having a stage that has a broadcast from Ireland and chairs in front of it and a good sound system. Those are some of the things that we're trying to look at to help make that connection between Ireland and Dallas and also provide some some online capabilities for folks to interact and come in.

But the risk is, how do you do it without discouraging in-person attendance? Because you're carrying all of the costs, it would be catastrophic if 30,000 people went online and 30,000 people came in person. How can you find a way to encourage the attendance, but not sabotage the financials? That's what we're trying to figure out. I suspect, like I say, at least some of it will be streamed on the internet at the time. There'll probably be replays available, or an ability to pay for access to all of the concerts after the fact, things like that. We've got a little bit of time to figure it out this time.

Anthony: Yes. That's the best thing. But I think the online audience for an event that's only online includes a lot of people who would much rather have been there, and all those people, if given the opportunity to be there, will still go. There will be a small shift, I think--and this is just my opinion--at a certain percentage who say, "I like going, but I don't like the parking or the beer. I'm just going to watch it online and drink my own beer at home."

Tim: "I hate that shade of green of the beer. I'd never use that shade of green."

Anthony: But then you also gain people who would not have ever gone or can't have gone. Like you said, the people who reached out to you after the fact that were able to attend from other states or other countries. You gain all of that, which you don't normally have if it's just an in-person event. So I think hybrid probably gives more than it takes away to an in-person event.

Tim: I'm hoping. An idea I'm kicking around is instead of it being a three-day festival, maybe it's one day online and two in person, and that would give something special on one of the days, give everyone a bit of a flavor, but it's also a teaser to get people to come out. So it could actually work as form of promotion to maybe encourage more attendance. Those are all things we're stewing on, and we're just waiting to see how this whole COVID thing ends up as well. I'm sure it may have a thing or two to say about this next year at this time.

Anthony: All right, Michelle, I want to thank you very much for sharing your time, sharing your experiences. I think us going through this and sharing this will benefit a lot of people who are trying or interested into AWS who are trying it out. "Oh yeah, it's just a virtual computer." "No, it's a bit of a learning curve."

Michelle: Exactly, exactly. My pleasure to be here. Thanks for inviting me. I really appreciate it. It was a great experience and it was great working with you and Tim and the whole group. Lots learned, that's what we do in this business. We often the hard way, and that's just the way it is.

Anthony: Alright, Tim, thank you very much for your time. I appreciate you letting us, and the viewers know some of the details about how you put this big event together: six live streams, international feeds as well. I think you really put together a great show. We had great producers all around and we had great capability that handled all the challenges that came in front of us. And me personally, I look forward to doing it again next year.

Tim: I'd love to.

Anthony: Alright, thank you very much. And thank you everyone for watching.