HD For Indies

HD4NDs Research: Update on uncompressed 35mm telecine project (10 bit RGB 4:4:4 log)

Hey all -

here's an update on what's been up with that 35mm to uncompressed 10 bit RGB 4:4:4 log project I've been working on:

Monday:

Got together with Sam (director) and David (editor) to talk workflow.

I'd done some similar testing, but it was nice to do some testing with actual footage.

What we had was the 10 bit RGB 4:4:4 log footage, captured in the BlackMagic 10 bit RGB codec. Hooray team. Problem was, this is 190 MB/sec footage at 23.98, and is too awkward to edit with - plus it is about 1.7TB of data total.

For offline editing, this is unwieldy to say the least.

So I set up a batch operation to create offline footage - that worked. He wanted 23.98 DV anamorphic, and after some minor bulk tweaking on the footage's settings in FCP, it worked fine and was ready for editing.

Then I created a bunch of subclips from that DV offline footage, then dropped those onto a timeline and set random in and out points. Then I set up some cross dissolves and cuts, etc., to simulate an actual edit.

Then I rendered out a reference QuickTime to have a freestanding copy of the edit.

Then I went through my matchback process to make sure I could convert that DV offline edit back into an uncompressed HD edit with all edits, dissolves, trims, etc. intact - and it worked as planned, like a charm. Better yet, it took about one minute to do.

Since I'd also created a set of DVCPRO HD offline files, that means whenever he wants to see his edit in HD, he can conform to DVCPRO HD in about a minute and play it back to see it (he could edit in DVCPRO HD, but he feels more comfortable with DV, as it is a better fit for the hardware he has).

So that was Monday, and Monday was good. Sam and David left, confident that they'd be able to match back to their HD content, even if their HD original content got destroyed and had to be re-telecined (that was a longer conversation, but I talked with them about how we could precisely recreate the HD footage if needed, using the hole punch as reference).

Thursday/Friday:

Sam called me to tell me he'd successfully backed up his uncompressed data to a third location (there's my set we captured onto my RAID, his second set that we copied off onto FireWire 400 drives, and that took a LOOOOOOOOOONG time, and the third set to another set of drives). So I was now clear to delete my set if I wanted to in order to free up space on my systems. I had dumped a backup of an important data set (all the footage of the capture from the Texas HD Shootout last month)

But....I didn't want to just flush that data set, I still wanted to mess around with it some for workflow R&D. I didn't want to just dump it, but I sure didn't want it to be taking up space a backup of the Texas HD Shootout footage could be. So I decided to cook off a full raster backup to some other format.

I originally set up my Dual 2.0 GHz G5 to cook a set of JPEG2000 codec'd QuickTimes. Doing some quick testing, it was interesting to realize that JPEG2000 is a.) slooooooow to compress, and b.) doesn't save all that much file space at higher quality settings. In fact, in the 90%+ range, the files were only about 1/3 the size of the originals. Hmm. An improvement, but not as much as I was looking for. I did several tests with a short segment (15 frames or so), and decided on 25% quality for the offline I wanted (it would be good enough for my testing needs). I started it around 11pm Thursday night on the Dual 2.0 GHz G5 (PCI-X), and it estimated 18 or 19 hours to go.But as of 1pm today, it estimated 22 MORE hours to go.

So, getting bored, I copied off one reel (12 min 9 secs) to the Quad G5 and started doodling with using plain PhotoJPEG. While JPEG2000 is a more efficient codec and creates fewer artifacts, it is sloooooooow to compress (as I was seeing) and it does NOT play back in real time, not by a long shot.

But PhotoJPEG plays back in real time, and compresses MUCH faster.

First test was to the PhotoJPEG setting, it was 25%, and looks like crappy offline - full raster, but LOTS of compression artifacts. Icky. You could cut with it, but you can do better. It converted my 136.2 GB source file down to 1.6GB - a huge file size savings, but waaaaay too much compression going on. It was a little slower than 2:1 to convert - about 27 minutes for the slightly over 12 min source.

I then changed the setting to 75% Quality, and it took a little longer - about 33 minutes to compress, and made a 7.7GB file.

A quick note on how to tell how quickly your footage is converting: assuming your drive system is faster than the stated read numbers (looking at Disk Activity in Activity Monitor), the higher the read, the faster it must be converting - otherwise why would it be needing to read in so much data so fast? For instance, converting to JPEG2000, the read speed is about 8 MB/sec - because it takes so long to compress each frame, it doesn't have to read in the next frame very often. Just knowing that the data rate is 190 MB/sec, dividing 190/8=23.75, so it's taking about 24:1 to convert to JPEG2000 at 25% quality setting. With 2 1/2 hours of footage, that implies it'll take about 2 1/2 DAYS (24 hours per one day) to convert. Yuck. High quality but time consuming.

PhotoJPEG, on the other hand, is tons faster - the read speed was about 95 MB/sec for 25%, and about 70 MB/sec for 75%. That's about 2:1 for 25%, and 2.7:1 for 75%. This tells us that higher quality settings take longer.

So, breaking it down:

Source File:
Read speed: realtime playback data rate is 190 MB/sec
File Size: 136.22 GB
Bit Depth: 10 bit RGB 4:4:4

JPEG2000 @ 25% quality: 8 bit, assumably 4:4:4?
Read speed during conversion: about 8 MB/sec
File Size after conversion: 17.71 GB
Compression ratio (file size): 7.7:1
Compression ratio (time to compress): approx 24:1

PhotoJPEG @ 25%: 8 bit 4:2:2
Read speed during conversion: about 95 MB/sec
File Size after conversion: 1.6 GB
Compression ratio (file size): 85:1
Compression ratio (time to compress): approx 2:1

PhotoJPEG @ 75%: 8 bit 4:2:2
Read speed during conversion: about 70 MB/sec
File Size after conversion: 7.67 GB
Compression ratio (file size): 17.7:1
Compression ratio (time to compress): approx 2.7:1

Also, in terms of compressing efficiency on the CPU:

-JPEG2000 - appears to be single processor threaded, running 97-99% of one CPU utilization on the Dual 2.0 GHz G5 looking at Activity Monitor.

-PhotoJPEG 75% running on Quad G5 - appears to be running 45-50% of one CPU, so appears to be single processor threaded as well and either isn't very efficient or something is slow in the pipeline - can't be the drive system, it can do far higher data rates than this is using.

So unless/until Apple redoes their codecs (perhaps for the Intel Macs, and I STILL don't have my v5.1 for my MacBook yet), it would NOT do you any good to throw a big multiprocessor machine at this task for these codecs at this time.

It's also interesting to note that except for a couple of mathematically lossless codecs that only compress 30-50%, there are no 10 bit codecs to save file space - Cineform for Mac, where are you?

(Update 5:30pm Friday- now the dual 2.0 GHz G5 working on JPEG2000 conversions estimates 25 hours to go...Update Saturday 9:30am - now it estimates 36 hours to go....JP2K to slow for archiving! About a day to compress an hour....harrumph!)

-mike