Review: Gomez Active Streaming XF

Without QoS, then, these node-based or embedded reporting systems are very important, and it will be several years before QoS tools fully make it into streaming. In fact, Windows Media Services for Windows Server 2003 uses the less-reliable Type of Service (ToS) and doesn’t take advantage of routers and WAN transports that have QoS baked in, although Windows Media Services has been updated to use QoS policies in Windows Server 2008 to manage outgoing network traffic.

The Company Line
This review is based on two factors: First, iStreamPlanet's David Morel and I had a walk-through with the company’s chief technology officer, Imad Mouline. Then, David put up a series of test files as a baseline for our testing.

One of the slides that Mouline presented during our walk-through was the most telling. It shows a series of bars, in a Gantt chart-type of horizontal bar graph. The far left of the lower half of the screen shows important information that’s needed to get the stream launched—various times are available for DNS time, connect time, and the time to receive initial data, all in milliseconds. This information shows how long it takes for the CDN to deliver the first data to an end user (or, actually, to a node, as this is the measurement technique for Active Streaming XF). Another set of charts, which can be drilled down to if desired, shows the difference between standard CDN delivery and content acceleration delivery.

However, the real power of the buffering chart I mentioned is the amount of time in which the streaming file played over the total delivery of the content and the amount of time in which the program was idle and buffering content. This rebuffer time is critical to determine if content is properly encoded (to avoid spikes in bitrate) and properly delivered.

The power of this particular type of test is even more impressive when one realizes that the output can be viewed in real time. CDNs have historically had to crunch through numbers from SQUID files after the fact, which takes a significant amount of time and provides, in the opinion of some, very little in the way of business intelligence. Windows Media live counters can be used in a real-time scenario, based on details provided by Windows servers running a plug-in as part of Windows Media Services.

Another type of information that can be extracted by the Gomez system is streaming errors. There is a period of time before the video actually begins to play. While abandonment by the user is an important factor in determining whether to use content acceleration, the error codes note that the content could not be reached at all or that the content never reached the play stage because of buffering or other potential problems. Active Streaming lets the tester drill down into these particular errors to see additional information as well as to view node location and to determine whether the problem is localized.

Taking It for a Spin
After our walk-through, we set up a series of tests. Active Streaming XF supports Windows Media, progressive Flash, and Flash streaming, so we chose a variety of content to test. In its testing description, Gomez recommends that users of its system test "high-performance or high-value" streams, meaning that key or representative streams should be the ones chosen to best cover general issues or specific problems faced by a particular high-value stream.

At first, I expected I would have to ask Gomez to change out tests if we wanted to test a variety of content, but I was pleasantly surprised to find that the company provides a flexible pricing model and self-service administration. This means that we could decide the frequency of testing (by the hour, day, week, or month), the length of the test, and the number or combination of streams we tested. We could, for instance, choose to monitor 10 discrete streams every hour, and we could change up the actual streams or frequency of testing to allow for additional stream tests within the basic pricing.

In setting up the tests, we noted that the interface is not unlike other Gomez products, so users of previous Gomez testing tools will be familiar with the slightly cumbersome but functional user interface. Creating a test is the same point-and-click scenario, with a few new selection criteria added to it, based on the nature, type, and location of the stream. All in all, the process of creating a test was simple.

Once the tests were configured and running, we let them run for several days. We set up our first round of tests to expire automatically—a handy cost-savings feature if testing is to occur over a set period of time. We viewed the data a few days after the tests expired, which brought about a surprising result: Tests that had expired were only available for editing, so it was necessary to re-enable the tests to view the legacy data. Once the tests were re-enabled, we were able to view the key data around our test streams, including availability, DNS time, connect time, initial buffer time, total rebuffer time, buffer count, and total play time.