The Time to Prepare for Recovery Is Before Video Files Go Missing

Earlier this year, one of my clients messaged me with an urgent issue: The person had accidentally deleted a server-side recording of a live sporting event from the previous evening. By the time he had sent the message, several hours had already passed, and other live events had continued to stream and record to the same server. (This is an important detail that I’ll come back to.) He wanted to know if I could help recover the missing recording. I had built a post-event transcoding pipeline for this client, and he had access to the shared network file system (NFS) from which the transcoder server would pull source recordings. He was hoping I could “undo” the mistake.

While I wasn’t able to recover that specific recording from the NFS (or from any other temporary volume used in the transfer process for postprocessing), I did rediscover the various approaches to file recovery, specifically for video files. I was amazed at how much material was recovered by open source tools, fully intact and playable. As a result of that experience, I’m going to provide two sets of recommendations for others who may need to solve this same dilemma.

What to Do When You Accidentally Delete a Remote Video File

If you delete a remote video file that you still need, I recommend taking these five steps:

1. Immediately stop all write activity to the volume where the file was stored. This means you need to stop any process on the server that could be writing more data to the same volume, potentially overwriting the bits of your lost file. If you’re trying to recover a file on a media server that is still streaming live video, you may need to wait until your other event(s) have finished in order to shut down the media server.
2. Download and/or purchase a file recovery product and run its recovery scan as soon as possible. If you are attempting recovery on a Linux system, you can try Foremost, an open source tool. (You can read a tutorial on Foremost.) Here’s a command that looks for MP4 and MOV files on /dev/disk1s1 and saves recovered files on a separate volume, /dev/disk4s2: foremost -t mp4,mov /dev/disk1s1 -o /dev/disk4s2
3. Copy the recovered data to your local computer so you have a duplicate of it and you can run further tools on the recovered video file, if necessary.
4. Review the playability and content of each file. Recovered files don’t usually have the same filename as the original file that was deleted, so you won’t have an easy way to identify a specific file among several other recovered video files. If you can’t open or play a recovered file, put that file into its own folder/directory on your computer for the next step. If you were able to find your lost file and it worked as expected during playback, thank the video gods and celebrate that you get to be the hero in this story. However, if you weren’t able to find your file among playable ones but found nonplayable files, there’s still hope.
5. If you have nonplayable recovered files, you can use video repair software to attempt full or partial playback. The software I’ve used most is Untrunc, an open source tool. This tool (and several other licensed products easily found with a Google search) usually wants a reference file to compare to the damaged file. In the case of my client’s lost recording, we located a similar recording made by the media server with the same publish settings (dimensions, frame rate, bitrate) as the lost recording. To date, I’ve only managed partial recording recovery with Untrunc, but it can often recover audio portions that are more intact than the video portions.

What You Can Do to Avoid Future File Loss

If you haven’t lost any files yet, don’t count on luck to continue your streak. Take these steps to avoid file loss in the first place:

1. Build redundancy into as many components of your video pipeline as possible, from onsite recording to remote server recording. If you rely on outsourced vendors, make sure they’re providing redundancy.
2. If you make long live recordings, configure your recorder to segment the files based on time. While many systems allow you to record a live stream into one file, any hiccups could corrupt the entire stream. If you use segments, then any corruption can be isolated to one particular segment.
3. Create copies of files during encoding processes, backing up the source file to a local cache folder for the encoder to use while keeping the original source location intact. If possible, create copies of files on different disks or partitions from the original location.
4. Build restrictions into your pipeline to prohibit manual interventions by your organization’s staff. In my particular example, a project manager was able to access the remote file system directly and delete the source file that was being used during a transcode process.
5. Have recovery tools pre-installed on any systems that read/write your video files.
6. Back up video files to a cloud storage service such as Amazon AWS S3.

I hope you don’t find yourself in need of video file recovery in the future, but if you do, keep calm and proceed as quickly as possible to retrieve the bits that can be found. The longer you wait, the less likely it is that the entire video will be recoverable and playable.

[This article appears in the June 2018 issue of Streaming Media magazine as "Prepare Yourself for Recovery."]

Project Managers Are Essential for Any Video Development Project

It's a thankless job, but PMs keep projects moving forward in a thousand ways. When projects don't have a PM, things can get seriously out of whack.

16 May 2019

Video: The ROI of Redundancy in Large-Scale Live Streams

Verizon's Joe Einstein and Right Brain Media's Deke Cooper discuss redundancy best practices they've used for webcasts that are too big to fail, such as the Grammys, the Academy Awards, and the Masters.

19 Oct 2016

Best Practices for Adding Redundancy to Live Encoding and Delivery

Learn the best practices for reacting and responding in real-time to issues in an encoding and delivery workflow to ensure a seamless experience for viewers.

19 Jun 2015