lld: lossless decimation / deduplication of all-I-frame codecs on Linux (ProRes, etc.) with ffmpeg
lld is a bash script that does lossless de-duplication of ProRes or other all-I-frame video files (whatever ffmpeg can handle).
Download the script here. Current version: 1.0
You will probably want to chmod the file to be executable: "chmod ug+x lld". If you have any questions/comments/improvements/bugs, please let me know. Big thanks to poisondeathray and shekh on videohelp.com (the most helpful site on the entire internet) for the assistance.
Background:
I had some silent Super 8 film scanned to digital recently, and a mix-up at the transfer place meant that the footage was scanned to 29.97 fps with duplicate frames: a pattern of 1-2-2 so that the 18 fps would appear at normal speed when played at 29.97 fps (and apparently the 1-2-2 pattern isn't totally consistent because of course it's not 18:30 but rather 18:(30/1.001) ). This is not telecine, since there are no combined frames, so I just needed to de-deuplicate the footage.
There are many options for de-duplicating, one of which is ffmpeg's mpdecimate filter. But ffmpeg can only apply filters while transcoding. My footage was in ProRes 422, which is an all-intra-frame codec, so in theory we should be able to do this losslessly.
I'm told that VirtualDub FilterMod can do lossless de-duplication of ProRes (and presumably other codecs), but it only works on regular frame patterns: "drop every nth frame", which obviously won't work in this situation.
I've also found threads implying that scripts or EDLs can be used with Final Cut (?) on a Mac to accomplish this, but that's not an option for me.
I found one solution that seems to work great. It is a bash script that uses ffmpeg to losslessly de-duplicate the 29.97 fps footage to a new 29.97 fps ProRes file where each individual frame of the new file is a single frame of the original 18 fps footage. This script assumes no audio associated with the footage -- it would need to be modified to handle footage with audio. As far as I know, it will work on any codec that ffmpeg can losslessly extract frames from (i.e. all-intra-frame codecs). This includes ProRes in .mov, but I haven't tested others.
I'm told that such manipulation of ProRes might generate incompatible or "illegal" ProRes files, so I'm not sure if this will work all the time, but my test files seemed to work great for me. Unfortunately, changing frame rates losslessly is apparently non-trivial for ProRes. This may not be a big deal, though: you can import the resulting 1:1 framed 29.97 fps file into your NLE and re-time it as desired, and you should be able to render out a 1:1 at 18 fps (or whatever frame rate you desire). I say this as a non-expert, note, but it seems logical to me.
You can see the discussion where I worked this script out at this videohelp.com thread link.
Usage:
Requirements:
- ffmpeg, which typically comes with the also-required ffprobe
- the calculator program "bc" -- I believe this is standard on all distros, but I'm not sure.
To use the script, cd to the directory that holds the file in question. Invoke it like "/path/to/lld file.mov 18", where 18 is the fps you believe the footage to actually be. (This fps specification doesn't affect the processing; it's only used to generate an error estimation of the expected-vs-actual number of single frames found. You can omit it. My footage had some kind of lead-in that wasn't in 1-2-2 so there is a -0.1% error for me.) lld will create a temporary directory where it does its work. This directory will be automatically removed, but if you add the term "debug" at the end, you can preserve it, which may be useful if you are tweaking things in the script. Note that you must specify the expected fps if you use the "debug" option... it's only a bash script, after all. :-)
If the starting file was file.ext, The resulting file will be named file.ext.dedup.ext.
This script is relatively smart, in that it checks for existing files/directories, and exits if unexpected things happen, but it's really kind of a hack, so use it with caution.
In terms of tweaking, you may want to
The whole process happens roughly on the order of 0.33 X real-time for a given video file on my i7-4770K. A 1-minute video file takes a bit over 3 minutes to process for me. A successful run may look like this:
$ ./lld file.mov 18
lld version 1.0
29.970029970029970 fps detected. Detecting duplicates...
Detected 853 good frames out of 1421 total frames = detected actual frame rate of 17.982017982017982 fps
-0.100% error from expected 18 fps.
Generating segment video files...
Generating segment logfile for concatenation and concatenating...
Deleting temporary directory. (Use e.g. "lld file.mov 18 debug" to prevent this.)
Removing temporary directory: file.mov.tempdir
Done. Resulting filename: file.mov.dedup.mov
How it works:
First lld will run the mpdecimate filter in a debug mode that doesn't actually generate an output file. It parses the output of this run to understand where the duplicate frames are, and where the "keep" frames are temporally located. These time offsets are then adjusted back in time slightly to avoid rounding errors when targeting the frames in the following step, where ffmpeg losslessly generates a video file with a single frame in it for every kept frame from the original (losslessly extracted, assuming ffmpeg is capable given the codec in use). Then ffmpeg uses the concat demuxer to (losslessly) join the individual one-frame video files.
Related links that may be useful: