HD For Indies

...so earlier I wrote the previous piece (see below or here) talking about how the Red Drive is now a small two drive RAID with 320 GB capacity. Almost certainly two 160GB drives. If they'd said 360GB, that would open the door to three 120GB drives, with possible fault tolerance (third drive for RAID 3 parity storage). But they didn't, it looks like a two drive solution. So that got me thinking about about what kind of storage times you'd get. And that got me thinking about other recording formats and what kind of datarates, storage requirements, and storage time on Red RAID you'd get.

UPDATE - so since yesterday when I posted this, the Red One Formats Chart has been updated - scroll to bottom for latest info

So I put on my propeller beanie, and got to thinkin':

Redcode RAW compressed 4K@24p is 27.5 MB/sec, thus 1.15 MB/frame. This is what we've been told by Red on the record, but all specs subject to change, yadda yadda. (I said uncompressed earlier, my bad.)

4K frames are 4096x2304, or 9,437,184 pixels, and only one channel (RAW, remember?)

What if you want to shoot 1080p or 1080i?


1920x1080 is 2,073,600 pixels, but is RGB - so three channels. Multiply by 3: 6,220,800 pixels.

So, if the compression ratio juju is all the same, short of any more firm data from Red, it looks like 1080pRGB could/should be about 2/3 the amount of data. Assuming everything is constant (which it probably isn't, but lets wing it for now), that would imply 1080p RGB frames would 2/3 the size of 4K RAW frames - so about 0.755 MB/frame. 1080p24 would thus be a bit over 18 MB/sec, 1080p30 would be 22.65 MB/sec, and 1080p60 (from which 1080i60 would be derived) would be 45.3 MB/sec. That's a lot of datarate! Oddly, note that the format required to record broadcast standard 1080i60 RGB is higher than that of 4K@30p RAW.

What about 720p?

Following the same kind of math, 720p (1280x720 RGB) frames are 29.3% the size of 4K RAW, so about 0.337 MB/frame. Thus:
720p24 Redcode RGB = 8.1 MB/sec (cake!)
720p25 Redcode RGB = 8.4 MB/sec
720p30 = 10.1 MB/sec
720p50 = 16.8 MB/sec
720p60 = 20.2 MB/sec
and the maximum 720p120 recordable onboard is 40.4 MB/sec

So given all that, the biggest challenge is 1080p60 RGB - and all the broadcast folks will be wanting that (but don't forget you can spit that out the HD-SDI live as well).

So back to the Red Drive - at 320 GB, I'm presuming it is two drives, thus 160 GB drives. Looking at Tom's Hardware, there are two 160 GB 2.5" SATA drives on the market:

the Hitachi Travelstar 5K160:

Min sustained read: 23 MB/sec
Max sustained read: 46 MB/sec
Min sustained write: 18.4 MB/sec
Max sustained write: 45 MB/sec

and the Fujitsu MHV2160BT:

Min sustained read: 15.6 MB/sec
Max sustained read: 41.3 MB/sec
Min sustained write: 20.3 MB/sec
Max sustained write: 41.3 MB/sec

The challenge for a storage device like this is that it needs to be able to read and write the worst case scenario ideally across the entire capacity of the disk - the fastest outer tracks to the slowest inner tracks. Whatever you can write you better be able to read in real time as well to play it back. So you have to find the WORST performance of minimum read and write and go with that. Since we're talking a 2 drive RAID (at least, I'm presuming two drives), we can almost double the performance of the drives. In my experience there is some overhead penalty vs a single drive - on my 4+ drive RAIDs under Mac OS X, which probably isn't the case here, but let's assume the worst for the moment - a 10% hit in RAID config.

SO:

Of these two drives, the Hitachi looks preferable - it's lowest performance is an 18.4 MB/sec write. Hmm. In a two drive RAID, with idealized performance, that is still only 36.8 MB/sec - well short of the 45.3 MB/sec required (and that with no overhead or safety margin!). So I'd guess off the bat that if they are using the Hitachi drives in a RAID, they'd have to partition it to NOT use that too slow portion of the drive - I've described coping strategies for this kind of thing here - you partition at the point beyond which performance is below target data rate + overhead.

So you'd have to figure out where performance is over 48.3 MB/sec plus a safety margin - let's say 20% to pick a small but plausible sounding figure - thus about 58 MB/sec. I don't have a chart handy specific to the drive falloff of the Hitachi, but based on typical performance of other 2.5" hard drives, I'd guess that point would be very roughly around 70% capacity. So in the end, you'd sacrifice about 30% of the capacity of the array (about 90 GB) in order to ensure ALL formats could be recorded to ALL portions of the array at ALL times.

Otherwise, it would get far too confusing for the typical shooter - imagine you've been shooting for 1080i60 in the morning, switch to 1080p24 for a while, and the Red is telling you there is still 20 minutes of 1080p24 space available. You switch over to 1080i60 mode, and it oops, it is too full, the array is too slow now, no 1080i60 recording availble. Who'd want to deal with that?

So that knocks about 30% off the possible recording times. The stated data rates haven't changed, just how much of it will reliably fit on the array with realtime performance.

SO - based on all my potentially erroneous assumptions above, here's what recording times might be with a 2x160 Hitachi based 320 GB RAID 0, IF it can only use 70% of it (about 207 GB):

720p24- 7.1 hrs
720p30- 6.9 hrs
720p50- 3.5 hrs
720p60- 2.9 hrs
720p120 (highest frame rate possible recordable onboard) - 1.4 hrs - but played back at 24p, that's over 7 hours!
1080p24- 3.2 hrs
1080p25- 3 hrs
1080p30- 2.5 hrs
1080p50/for 1080i50- 1.5 hrs
1080p60/for 1080i60- 1.3 hrs (these two are the biggest surprise, so a common format records almost the least amount of time)
2Kx1080 RAW @ 24p - 6.4 MB/sec - about 9 hrs - wow, what a deal!
2Kx1080 RAW @ 25p - 6.7 MB/sec - about 8 1/2 hrs
2Kx1080 RAW @ 30p - 8.1MB/sec - bit over 7 hrs
2Kx1080 RAW @ 50p - 13.4 MB/sec - about 4 1/3 hrs
2Kx1080 RAW @ 60p - 16.1 MB/sec - about 3 1/2 hrs
2Kx1080 RGB has been removed from the format chart - no longer an option - see bottom of article for details
4K@24p or 4K@25p - a bit over 2 hours
4K@30p- about an hour & 40 minutes

An interesting note for folks interested in lightweight packages that can get a lot of recording time per day - windowed 2K compressed RAW is actually the most space efficient format the camera can shoot! 2Kx1080 windowed RAW at 24fps is only 6.4 MB/sec based on these calculations, that's about 23 GB/hr. Note that is BETTER than 720p24, which you'd think would be the most efficient shooting format. NOPE. Since 720p is RGB, thus 3 channels, and 2K RAW is only 1 channel, 2K RAW is actually lower datarate. With a lightweight PL mount 16mm lens and a single Red Drive, assuming you have enough batteries you could shoot for NINE HOURS nonstop 2K windowed RAW 24p! Whereas is you shot 1080p, you'd have to be recording in RGB, and would only get roughly a third of the recording time. 2K windowed RAW is a DEAL when it comes to shooting efficiency - the only tradeoff is that the 1080p RGB COULD (doesn't have to) be derived from full sensor so you'd get 2x oversampling in both dimensions - 2K RAW you MUST shoot windowed, no oversampling. But....it is always nice to have choices!

Conversely, 1080p60 RGB for 1080i60 delivery is surprisingly "expensive" in terms of recording time - IF these calculations are correct, you'd only get about an hour and 20 minutes of record time on a Red Drive. If you could convince your producers to go for 24p on the project rather than 60i, you'd get nearly SEVEN TIMES the recording time on the Red Drive. Sounds crazy, but true.

I also suspect that lower res formats may need to be run a little "rich," on data rate, whereas higher res can get away with being run a little leaner based on the nature of wavelet compression, so that may well influence all this too.

Keep in mind, there's a boatload of presumptions and assumptions in all this, so this is just the sketchiest of guides. Those decimal places are misleading - these numbers aren't that accurate. - the idea being, a 320GB RAID won't yield a full 320 GB of storage capacity - reason one being formatted capacity (knocks you back to 296 GB), reason two the slowest part of the array may need to be partitioned off and not used in order to guarantee performance. If they are using some other drive, or can get clever with counter-striping (one drive starts writing at inner tracks, other drive starts writing at outer tracks), the situation may be much better. But my understanding is that Medea, now owned by Avid, has a patent on such implementations, and they may not be in a mood to license that technology to other vendors. (Anybody know for sure? Clue me in if I'm wrong.)

If they DID counter stripe, and it worked the way I think it might, they'd be in luck - if they could use the full capacity of the array, those times would all go up 40-45%.

The other issue with this announcement - a RAID 0 (still what I'm presuming until I learn otherwise) is an inherently riskier data situation than a single drive. Not only are you doubling your risk of failure (one drive fails all data lost in RAID 0), BUT in the event of a drive failure, RAID 0 DATA IS NON-RECOVERABLE, UNLIKE A SINGLE DRIVE. Or at least, that is what I've been told from data recovery firms when I've asked. With a single drive that fails, you grit your teeth and pay several hundred to a few thousand dollars and get your data recovered if you have to. With RAID 0, I've been told it isn't possible.

Bigger is awfully convenient in the field, but also is a bigger basket with all your eggs in it. A tape goes completely bad, you lose one hour of footage worst case scenario. A Red Drive goes bad, if my above conservative calculations are correct, you could lose up to 10 hours of footage (full capacity usage of 720p24 footage) - ouch! So protecting against a head crash, directory damaging power loss, or other risks will be crucial on the Red Drive. I've been running RAID 0 arrays with 8 to 10 drives in my studio for a couple of years now - I presently have two 8 drive arrays and one 10 drive array that I own in the studio, and loaner/review units of 4 and 6 drives in the studio. I have yet to have a data losing crash or failure with any of my arrays, BUT:
-I am careful
-all are ALWAYS on a UPS
-the important data is backed up at all times
-MOST importantly - they all just sit perfectly still and are off when not in use.

With 2.5" drives in laptops, which is a similar situation as compared to Red Drive - it gets moved, knocked around, much higher risk of getting dropped - so the life expectancy is MUCH shorter. I've had a laptop drive fail on me in less than two years (my 12" PowerBook), I've also had THREE friends have their laptop drives die on them in less than three years, and they then came crying to me. Point being, life for a hard drive used in a portable environment is TOUGH. Putting that in a two drive RAID 0 doubles the risk, so again, shock isolation will be key, and it'd be good if there were some way to know how much use a given unit had been through (rental market especially?) to know when it was time to retire it.

Of course, if any of that makes you nervous - just get the Red Flash recording module - only 32 GB to start, but solid state - MUCH more secure recording technology - just higher cost per GB - we've been told 32GB will start around $1000. Drat, now I'll feel compelled to make up a chart on that one...maybe I'll update later in the day...UPDATE - now with Red Flash estimated times: yeah, I couldn't resist - I added a third image to the Excel sheet screen grabs, is on the same page as the Red Drive rates (see link next paragraph). The surprising news - for narrative shooters, 4K @ 24p you get about 18 minutes. But for broadcast folks wanting to record 1080i, you get as little as 11 minutes if recording in RGB. Ouch! For broadcast types, that's pretty damn slim. Of course, you SHOULD be shooting 2K RAW windowed 60p (for later crop & conversion to 60i) and you'll get half an hour of recording time - much more reasonable. And 2K windowed RAW 24p? A whopping 77 minutes - take THAT, P2! : ) And solid state has a consistent transfer rate (AFAIK), so no partitioning worries there. And of course, 64 GB would double all these record times.

To figure all this out, I made an Excel sheet and took some screen grabs, peruse it if you wish - based on all my guesses in here, has MB/sec data rates, GB/hr storage requirements, and recording times for a Red Drive with and without partitioning based on assumptions. For the zillionth time, this is my guesses, all extrapolated from public info, is not official from Red, etc.

On a related note, there's a thread over on DVXuser about footage handling on location & archiving footage with the Red camera. I was in a pessimistic mood and got all frothy in posts 41 and 45. I talked about the practicalities of freelance shooters working with digital data and the issues I'd experienced in the past when having to hand over a lot of data to clients on relatively pricey media.

-mike

PS- for the curious, here's some links used in researching this article:

One new spec... - DVXuser.com -- The online community for filmmaking (Red Drive RAID announcement and discussion)

footage handling on location & archiving footage - DVXuser.com -- The online community for filmmaking (my stuff starts on page 5)

Tom's Hardware - this is 2.5" hard drive minimum sustained write speed comparison chart, with Hitachi and Fujitsu 160 GB highlighted. You can use the pop-ups at top to see other specs (min/max sustained read/write was what I was most interested in) and to highlight other models. Super duper handy page! Also available for 3.5" disks as well, see test link above chart.


200 GB, 2.5", SATA: Fujitsu's MHV2200BT | Tom's Hardware - drive performance falloff chart. Shows how drive speed falls. I used the shape of this arc as a guideline for estimating Hitachi falloff point, scaling to pertinent values of Hitachi.


2.5" Hard Drives by Toshiba and Western Digital Rev Up | Tom's Hardware - another similar chart, used to double check for the reasons stated directly above

RED / RED ONE Format Options - this chart updated a few weeks ago, and I used it to verify max frame rate/frame size/format stuff


UPDATE TUESDAY MORNING -

So the Red One Formats chart has been updated - it used to be this, now it is this

Here's what has changed:

2K RGB has been DROPPED entirely - you can either record 4K RAW, or 1080p RGB for full aperture, or 2K windowed RAW with 16mm/B4 lenses, or 720p RGB.

1080i RGB recording has been dropped entirely - as Stuart mentioned in an interview I did the other week, 1080i formats are derived from 1080p recordings - so shoot 1080p50 for 1080i50 and 1080p60 for 1080i60.

OK, so what about 1080i formats out the HD-SDI? Losing that would have a huge negative impact on the viability in the broadcast market for live events. RECORDING 1080i is clearly out, but what about passing it through? I'd be very surprised if they dropped it, but confirmation that it still exists would be nice. UPDATE: CONFIRM FROM RED - 1080i OUT THE HD-SDI IS STILL THERE.

Perhaps never a clearer example of "all specs subject to change at any time" - I think I may have hastened the chart update with this article.

Since 2K RGB was sendable over the HD-SDI, I was presuming 2Kx1080 since that is a SMPTE standard that I know can be sent over HD-SDI. But 2K RAW couldn't - perhaps it is 2Kx1152 or somesuch? Why the discrepancy? Because 2Kx1080 isn't 16:9, it is just what would fit down HD-SDI. 2Kx1152 is a true 16:9 aspect ratio. If it were 2Kx1152 not 2Kx1080, then my datarates for 2K raw should be about 7% higher to account for the taller frame.

I've edited this article a bit since originally posted, and I will update the graphs shortly on the linked page...

Also, my math is demonstrably wrong somewhere - both Jim and Jarred have told me that 1080pRGB is about twice the datarate of 2K windowed RAW, not nearly 3 times as I've calculated. I'm not sure where my error lies, but know that my guesstimates are wrong - probably on the 1080p data rate side. I'll double check my math...

-mike