Moscow State Updates VQMT With New Metrics
Moscow State University’s Video Quality Measurement Tool (VQMT) has always been one of the least expensive and most functional quality measurement tools available. The latest updates (10.0 and 10.1) add new metrics and provide significant user interface, performance, and feature enhancements that make it a must-have upgrade for all serious streaming video producers. Like previous versions, the product costs $999 with upgrade pricing available for some owners of previous versions. Here's a brief overview of the new features, which we explore in a video at the end of this article.
By way of background, VQMT is a Windows application that also runs via the command line in both Windows and Linux environments. VQMT is a very flexible tool that includes over 15 metrics, both reference and non-reference, many with different variations. The UI version can compare one or two compressed files to a single source, which is great for comparing different encoding techniques or encoding configurations.
In the UI, VQMT graphs the results (see Figure 1) so you can gauge performance over file duration and identify problem areas. For example, Figure 1 compares the same file, an excerpt from Tears of Steel, encoded using constant bitrate encoding (green) and 200% constrained VBR encoding (orange) measured using the new VMAF metric. Though the average scores are quite close, the big green drop off around 30% into the file indicates a potential problem area as does the sharp downward spike on the extreme right.
Figure 1. The results plot comparing CBR and 200% constrained VBR. Note the multiple tabs on the upper left, discussed below.
You can move the slider to any point in the graph, and then click Show Frame on the bottom right to view the source and two encoded frames. It’s a great way to identify any problem areas and to see if the scores actually reflect artifacts that would be noticeable by your viewers.
Key New Features
In terms of new metrics, Netflix’s VMAF is very high on the list. I’ve been working with VMAF since it appeared in the 10.0 beta, and at first the analysis was single-threaded and exceptionally slow. Since then, MSU has done a great job accelerating performance on multiple-core computers like my HP Z840; while VMAF is still slow because it involves multiple metrics, it’s now quite usable. New in 10.1 is the NIQE metric, reportedly one of the best non-reference metrics available.
From a UI perspective, version 10.0 represents a complete rewrite, which initially is jarring but ultimately is vastly superior. There are multiple usability enhancements, including the ability to drag and drop files into the main screen, a nice convenience. More important is the ability to run multiple tests simultaneously, as evidenced by the multiple tabs on the upper left of the figure.
That is, one you start the metric calculation, you can load new compressed and source files into the program, and start another analysis. In previous versions, the program was locked until the initial computation completed. Though you could load multiple instances of the program (and you still can), this was cumbersome and prone to crashing. I’ve run ten simultaneous analyses in version 10.1 without so much as a hiccup, a great way to leverage the 40 cores on my Z840.
The other great usability enhancement in the command line is the ability to analyze files of disparate resolutions without first converting them to Y4M format. For example, to compare a 720p file against the 1080p source in previous versions, you had to scale the 720p file to a 1080p Y4M file. It’s simple and fast, but it’s an extra step, and if you’re analyzing 100 files overnight or over a weekend, and/or working with 4K or larger files, you may not have the disk space available to do so. In version 10, you can now compute any metric without first converting to Y4M.
As an example, I recently completed a 2700p60p VR project that involved 3-minute test files, each of which was 150GB in size in Y4M format. If I had to pre-convert to test, I could test about 5 files without running out of disk space, a severe limitation. With this new capability, disk space is no longer an issue.
Though these features may sound modest, they vastly increase the testing throughput possible with VQMT. If objective quality metrics are an important part of your configuration and testing practice (and they should be), these make the latest upgrade absolutely essential.
Related Articles
The head of the Moscow State University Graphics and Media Lab—the people behind VQMT and Subjectify.us—offers his insights into objective and subjective metrics, as well as VMAF hacking and AV1.
13 Dec 2021
The good news: As always, Moscow State's codec studies are some of the most comprehensive available. The bad news: Unless you're TikTok or Tencent, you won't have access to some of the best performers.
11 Jun 2021
If you're serious about experimenting with different codecs and/or encoding parameters, MSU's Video Quality Measurement Tool is an essential tool, and version 13 brings some welcome improvements.
01 Jun 2021
In rigorous tests of video codecs, only one produced vastly different results during objective and subjective examinations. What could cause that discrepancy?
15 Nov 2019
This video analysis tool—VideoQuest for short—offers deeper metrics and is more configurable than others on the market, especially when exploring the differences between two encoded files.
20 Dec 2018
Moscow State University makes subjective comparisons less time-consuming with an online service that enlists other people to watch and rank videos.
26 Oct 2018
Latest codec quality comparison study finds AV1 tops on quality, but far behind on speed, and that VP9 beats out HEVC. Moscow State also launched Subjectify.us, a new service for subjectively comparing video quality.
30 Jan 2018
In addition to comparing HEVC codecs, report also compares HEVC to VP9 and x264, with potentially controversial results
09 Nov 2015
Never heard of it? Learn why we call this video encoding analysis tool invaluable, and a must-have for anyone serious about encoding or compression.
30 Dec 2014