Taking Control of Big Data: The Challenge of M&E Data Management
When deciding on the best topological approach, organizations must consider many key variables, such as organizational philosophies, quality-of-service requirements, desired service level agreements, target financial structures, security, and long-term sustainability. CSM solutions provide flexibility and environment abstraction so that organizations can focus less on where and how assets are physically stored and more on efficiency, creativity, and maximizing value and usage of assets.
Archive eXchange Format Guarantees Universal Access
Another big advantage of advanced CSM solutions is that they rely on the open Archive eXchange Format (AXF). On the cusp of SMPTE standardization, AXF is the physical embodiment of the object store concept and includes key preservation characteristics that ensure long-term protection. It was designed for the transport, storage, and long-term preservation of complex file-based collections regardless of their origin or use. On-premises or cloud-based CSM solutions that leverage AXF guarantee long-term asset accessibility. Any application that conforms to the emerging AXF standard can access, interpret, and recover AXF objects stored by other systems for years to come, mitigating risk and ensuring long-term security and accessibility of valuable file-based assets.
AXF is like an advanced ZIP package that encapsulates any type and number of files/folders in a well-structured, self-describing data object. The design ensures that AXF objects are universal and can be stored on any type of technology, including datatape, spinning disk, flash media, and even in the cloud. AXF also adds self-describing characteristics to the media itself, which means that it can be accessed and read on any system, not just the one that created it.
AXF is already being used on dozens of diverse storage and network transport technologies. Also, because it was designed as a streaming-centric technology, it is being used extensively to transport complex file collections across WAN networks as well as to and from the cloud.
Why HSM Is Not Enough
Hierarchical storage management (HSM) systems generally manage the migration of any type of file from more expensive, less scalable spinning disk storage to lower-cost datatape archive storage. HSM systems were built to address storage infrastructure management in traditional IT environments but lack key features and functionality required in these demanding media-centric ones. They are file-based by their very design and therefore cannot address the complex object-store challenges in big data environments. A few limitations:
- HSM systems are ruled by simple migration policies that determine when the system should move a file from datatape to spinning disk, such as when it has not been accessed for a certain period of time or if it has a specific file extension. Often these policies assume that files become less relevant as they age -- something that certainly does not apply in complex, media-centric environments.
- HSM policies also don’t consider the fact that specific files are often components of assets and not the assets themselves. As file migration policies are satisfied, the missing context of the object can cause components of a single asset to be scattered between storage layers and among several datatapes in the datatape library tier. And the problems snowball from there. Because this simple, policy-based file movement between storage tiers fails to account for so many variables, it is not nearly sophisticated enough to handle big data. Add to this the proprietary storage approaches and lack of support for AXF, and long-term accessibility is at risk.
- In terms of storage abstraction, several HSM solutions address some aspects of the concept, but few also natively bridge the gap between commodity storage and the application-specific devices that create and consume content.
- Content-aware features of CSM solutions are simply not available in HSM solutions, and therefore they must rely on other applications outside of the storage management layer to facilitate interoperability, sharing, and collaboration between workflow silos.
- HSM solutions are limited to the environments in which they are deployed, which means they cannot leverage the elastic nature of the cloud to bring agility to the organization. While some HSM providers offer cloud services, those services are nothing more than simple implementations that age files from more expensive cloud-based disk storage to datatape.
These limitations not only make it impossible for HSM to overcome big data challenges, but in the long run they can be catastrophic to the operation and its valuable media assets.
CSM Has the Clear Advantage
Advanced CSM solutions embody the object store philosophy while providing true storage abstraction, unifying complex infrastructures and providing a rich set of features tuned for high performance and demanding big data environments. With a pedigree in M&E big data handling, these advanced CSM solutions apply anywhere intelligent, object-based storage abstraction and management are needed.
The CSM approach allows a level of asset storage, handling, and preservation that no traditional HSM solution ever could -- empowering organizations to be ever more agile in the face of a rapidly evolving landscape.
Related Articles
Streaming services are collecting huge volumes of data on viewer preferences and histories, then using that data to drive new content production decisions.
05 Aug 2019
Some of the biggest video publishers around are sitting on several years' worth of viewer data that they're only now beginning to sift through.
12 May 2015