Video: How to Build an A/B Testing Workflow
Streamroot's Nikolay Rodionov lays out the basics of A/B testing and how to implement it in HTML5 player development: how to present different versions of your player to different users, with custom condition variations that you can control and document their effectiveness.
Learn more about streaming player dcevelopment at Streaming Media East.
Read the complete transcript of this video:
Nikolay Rodionov: Let's take a look at how to actually create the A/B testing split. The concept here is very simple. Basically you need to be able to give different versions of your player to different users with custom conditions that you can parameter and control. There are several methods to do it. I am going to talk to you about two of them today. The first is the dynamic config injection, and the second is basically provide different version or builds for your player via reverse proxy.
The first method is to create a service that injects a different configuration into your player and overrides the default config. Basically what happens is that instead of just bundling your configuration with the JavaScript player file, you make your player a request via config from A/B testing broker service that is completely separate from your player build. Basically, that way you can customize the configuration file you send to each of your users, and you can make it depend on the stream they're watching on the ISP or city they're in or any other parameters you want.
This method has a lot of advantages. The biggest one is that it’s very easy to setup and use because basically you just work with a simple config file. It's also easy to deploy and roll back frequently because it's a completely separate service from your player build. Finally, it's also very scalable because the only thing that this A/B testing service does is basically send a 20-line config JSON at the beginning of each session of your users.
On the other side, being able to only change the config for the users, of course, gives this method a quite limited scope, so you can fine-tune your config parameters and even test new features via flags, but it's not a very good workflow because you don't want to have code that has a lot of conditionals and depends on how the flags are. You don't want to like to go too deeply into that for new features.
Related Articles
The CEO of RealEyes shares his personal video engineering toolbox, which contains solutions for diagnosing, fixing, analyzing, and interpreting problems with streaming video.
03 Jun 2019
Brightcove's Matt Smith explains how to meet the challenges of live event streaming from technical to QA to process issues to securing bandwidth in this presentation from Live Streaming Summit.
03 Aug 2017
Streaming Learning Center's Jan Ozer compares four approaches to testing video compression quality in this clip from Streaming Media East.
20 Jul 2017
Streamroot's Erica Beavers gives content owners migrating from Flash to HTML5 a whirlwind tour of HTML5 player components and UIs, likening an HTML5 player's inner workings to the component of a loaded hamburger.
16 Feb 2017
Streamroot's Erica Beavers explains what A/B testing is and how it can benefit organizations building custom video streaming players.
15 Dec 2016
Nice People at Work's Diane Strutner, Studio71's Mike Flynn, and WillowTree's Jeremy Stern debate the merits of in-house vs. third-party analytics tools for video brands.
01 Dec 2016