Work in Parallel to Streamline the VQ Measurement Workflow
Every once in while you write a column knowing that a certain percentage of readers will think you’re a simpleton for using a certain workflow for so long, while another percentage will secretly thank you for showing them a better way. This is such a column.
To explain, most video quality measurement tasks involve multiple operations. First, you encode the file, then if it’s a new format (like AV1) or a certain tool (like vmafossexec, the open-source Netflix tool for computing Video Multimethod Assessment Fusion, or VMAF) that only accepts YUV files, you convert to YUV. Then you run the objective quality measurement tool and compute the score. Then you delete the files and start on the next batch.
There are several positive outcomes of performing these tasks serially. First, the batch files are simple to write, which minimizes the risk of error. Second, often you have checks to run on the encoded files before you process further, like checking the file size. However, there are also several key negatives. First, you have to be there to trigger the subsequent events, so if encoding stops at 3 a.m., you probably won’t start measuring quality until the next morning, losing valuable machine hours. In addition, because YUV files are incredibly large, even if you have very large SSD disks you often have to limit the number of files you process to avoid running out of disk space, losing more hours.
I started with this serial workflow back in 2014 when I began working with video quality metrics. I have thousands of batch files on my various test systems, efficiently labeled encode _.bat for encoding tasks, scale_.bat for scaling, and MSU_.bat for measuring VMAF, Structural SIMilarity (SSIM), or peak signal-to-noise ratio (PSNR) with the Moscow State University Video Quality Measurement Tool (VQMT). Since deleting large files is easily managed through Windows Explorer, I simply delete manually, avoiding yet a fourth class of batch files.
Just recently, while working a consulting project, a colleague sent some of his batch files for my review and verification. Rather than working serially, he worked in relative parallel, first encoding a file, then converting it, then measuring quality, then deleting the YUV file. When I saw his approach, I didn’t slap my forehead and proclaim “D’Oh!” but I felt that I should have, since it cures several of the key problems of my serial approach and had several key advantages.
First, since his single batch accomplished all four tasks, I didn’t need to be there to launch the separate batch files, keeping the process going day and night. Second, it eliminated the disk space issues, so I could process an almost-unlimited number of files. Finally, when his process was done, it was done; all that was left was to record the data. With my serial approach, I had to take three separate steps before I could harvest my data.
Of course, every silver lining has a dark cloud, and this parallel approach is no exception. First, the parallel batch file is much more challenging to create and debug, increasing the risk of failure and taking much longer to create. Second, you really can’t build file-checking into the batch file, so if the file’s data rate is too low or two high or if the file is otherwise defective, you’ll process it anyway, costing you machine time. Finally, many times you want to visualize the YUV or Y4M files in a tool like VQMT and you’ll have already deleted them. That said, conversion to YUV is generally a fast operation, so you can quickly recreate them for viewing. (And if you need some scale_.bat files, I’ve got thousands I can send.)
For the most part, however, I’ve seen the light and am changing my workflow from serial to (closer to) parallel. This will undoubtedly streamline future projects, but it leaves me uncomfortably wondering how many other tasks I’m performing equally inefficiently simply because no one has ever shown me a better way.
[This article appears in the April/May 2019 issue of Streaming Media Magazine as "Streamlining Your VQ Measurement Workflow"]
Related Articles
Tektronix Applications Engineer Andrew Scott discusses objective and subjective methods and tools for picture quality measurement in this clip from his presentation at Streaming Media West 2018.
19 Apr 2019
Video quality measurement is more crucial than ever. In this buyers' guide, we'll identify the most commonly used metrics and discuss tools that deploy them.
29 Mar 2019
This video analysis tool—VideoQuest for short—offers deeper metrics and is more configurable than others on the market, especially when exploring the differences between two encoded files.
20 Dec 2018