Review: Gomez Active Streaming XF

All of these are presented in numerical and graphical displays, and the ability to review multiple pieces of content in one report is an added benefit, as it allows CDNs and their clients to track performance among different delivery vehicles. For instance, traditional, single-origin delivery versus CDN delivery is a common test that is done during a presales decision cycle to show the benefit of a CDN, as is the testing of two or more CDNs against one another.

We found the statics around availability, buffer count, and total rebuffer time to be the most compelling since this gave us a sense of what the user is facing, although a combination of Active Streaming and Actual Experience would probably provide a fuller picture. In a CDN-versus-origin or CDN-versus-CDN comparison, it’s relatively easy to compare QoS based on buffer counts and total rebuffering time.

Much like static-object testing, the frequency of testing can have a tremendous impact on the quality of the results. Here, we ran into a bit of a conundrum: All CDNs have differing algorithms in their edge machines, so once-an-hour frequency may result in missing content placed in and dumped from a cache on one CDN while content may be held longer in the cache of another CDN. The question is whether CDNs with algorithms that dump the cache more quickly will modify their algorithms to allow for longer caching or whether they would just set up traffic inspection to recognize Gomez testing and adjust their caching behavior for the testing period only. However, since the cost of increasing testing frequency is solely at the expense of the subscriber to the Gomez Active Streaming service, it may make the cost of testing multiple times every hour to try out the various CDN caching algorithms somewhat unreasonable.

One other area we found is a lack of monitoring performance under heavy load. Load testing, though, is something Gomez may address in future releases, as all of its monitoring and reporting services are software as a service and can be easily updated, much like the number of nodes on a variety of CDNs. This last point is critical: The number of nodes, their locations, and the particular CDNs they sit on is somewhat limited in the current system, which rolled out in late September 2008. We recommend getting a demo of the system when you read this, to determine if the number of nodes has increased in or near the location of your biggest client base.

A Step in the Right Direction
In summary, the Gomez Active Streaming XF solution solves a niche need that was previously available from only dedicated streaming monitoring solutions and leverages the fact that Gomez has long been considered a leader in the website-performance-monitoring game. This first foray into large-object and streaming-delivery testing for CDNs seems to be a good initial launch, as it appears Gomez has seen the value it can add in the CDN space by providing testing of both live and video-on-demand streaming content. Its initial implementation seems to capture the primary requirements that will benefit CDNs during the presales cycle (sort of a Good Housekeeping seal of approval) and can potentially provide the content owner with some value as a monitoring tool in an ongoing use case. While Gomez believes that its solution may serve as a monitoring mechanism to enforce service level agreements, we think it’s too early to determine this, given the limited number of nodes.