NAB 2018: Adobe Talks Premiere Pro and AI

Shawn Lam: It's Shawn Lam here for Streaming Media Producer at NAB 2018. I'm here with Steve Forde from Adobe. We're here today to talk about Premiere Pro and the artificial intelligence Sensei. What new features do you have?

Steve Forde: My two favorites, especially with Adobe Sensei, which is where we're using artificial intelligence and machine learning to really enhance the creative process. One of the key things is called shot matching. This is where we take the color workflow and make sure that, as you apply a color grade across, say, a series of clips, we'll actually detect the face. We center it around basically face detection and how the human interacts in the scene, and then we orient the color grade around that.

So, what we're doing is we're doing matching to the skin tones, that kind of stuff, and it really brings out the color profile. As an editor, you can basically rely on the fact that the color grade is oriented around your talent and the focus of the shot.

Shawn Lam: That's really where it should be, right? The talent is the focus. That's where you need to get those skin tones just right. How accurate is it as a starting point?

Steve Forde: It's actually fairly accurate. What we do when we mean machine learning or artificial intelligence is that we actually teach Premiere Pro to watch TV. We teach the application to watch and go through thousands and thousands, especially if you know of our other products like Lightroom.

Every user of Lightroom, which is a lot, is uploading every photo that they take into our Creative Cloud. We then take our algorithm and machine learn, based on their photos, to understand facial structures, skin tones, and all those types of things. Then we take what that algorithm learns, and we put it into a product like Premiere Pro, so that as the editor works with the shot, they can know that we're actually fairly accurate to a really wide array of different user types because of what the algorithm has learned by scanning thousands and millions of photographs in Creative Cloud.

Shawn Lam: So, since it has applications in color matching, where else do we see Sensei working in Premiere Pro?

Steve Forde: In audio. Another key thing is how we do understand when you have, say, a voice as we're talking now, but you want to auto duck the background noise, or you want to be able to use a music bit. The right thing is, for us, again, Premiere Pro ... in this case, actually Audition. Audition learned to listen to what it was working with. Basically, as people were working with their content, and they allowed us, of course. We don't do this just out of the gate.

But basically, Audition as an algorithm would understand and learn when people were trying to apply these things, and as a result, we can then take that algorithm and apply that in the product. As other people bring in their content, we're taking everything we've learned from before, and we're being able to do an auto keyframe, basically at a volume level and then really separate the tracks.

As an example, you can then tweak your track's keyframing if it's not quite accurate. But the results are pretty astounding. It's come out more accurate than I thought it was going to be at the beginning.

Shawn Lam: That's exciting too because I'm sure there's applications or there's use cases where, without having to review everything, you're making sure you're not peaking, right?

Steve Forde: That's right.

Shawn Lam: The additive audio when you're adding multiple layers of audio on there, that can be very punishing.

Steve Forde: And that's what we're trying to do with machine learning. We're not trying to replace the creative process with algorithms. What we're trying to do is enhance the creative process so that the tedious tasks are getting out of the way. Those are the types of tasks that machines are really good at, so why don't we let the machine do all that stuff, and then you can focus on the creative. It just really raises the bar for the kind of content that somebody can create.

Shawn Lam: Now, on the Adobe Media Encoder side, H.264 has been a codec that's tried and true. Everyone's using it on a daily basis. Moving forward with next-gen codecs, HEVC, AVI codecs, they're really slow to process right now. They don't have the hardware acceleration. What's happening in the future with Adobe? Where do you see those codecs going?

Steve Forde: One of the things that we really focused on for this release was orienting the H.264, start with that.

We wanted to be able to say, by decode, just in your playback, but also in the encode process, we've added the hardware acceleration to that. In other words, in many cases, we're up to five times faster and pretty much we're on par with any other application, if not faster, with H.264 and coding.

Decode, the same thing, especially now that Apple released with the iPhone, HEVC is native in iOS 11, so we really wanted to up the game in terms of performance. As users are bringing not just professional cameras, but iPhones and all that kind of stuff into the mix. Hardware encode and decode is a key priority for that. I think we've made a lot of progress on encode. You'll see more decode coming, so that we get that really smooth 4K, 8K playback, using the hardware.

But we also did one key differentiator, so I think one thing that separates our products from a lot of the other competition in the market is that we oriented on-call to be first and then performance. We wanted to make sure that you can dial it back because when you do a hardware decode or encode, you are dependent on the hardware profile. Sometimes there's slight variances in one hardware decode to another hardware decode. What we wanted to be able to do is allow the user to dial it back and say, "Okay, it's going to be slower, but the pixel perfection will be there as well."

Now it's really a choice. Do I want this fast, and I can live with the output? And the output is really good. I'm not saying it's bad. But if you need that fine, fine pixel quality, especially if you're using it in post, H.264 is a highly compressed format, so we wanted to be able to allow you to get the highest quality output and go from there.

I think we're now best in class, and we're not going to stop there. I think, as HEVC and other codecs come online, Premiere Pro's claim to fame is that it can process any format and do so natively in real time. That is our story, and we will always continue to do that.

Shawn Lam: All right. Thank you very much, Steve. This has been a look at Adobe Premiere Pro at NAB 2018. I'm Shawn Lam for Streaming Media Producer.