HD For Indies

For a few years I've been writing about using SATA based RAIDs for uncompressed HD projects. Slooooooowly but surely, some RAID 5 (fault tolerant, don't lose all your data if a drive craters) solutions have been coming to market.

At first, they were PCI only (during the PCI-X G5 years). So they were too slow for uncompressed HD (this was before DVCPRO HD native editing was popularized).

Then there were PCI-X cards, but they were still too slow - 80-110 MB/sec write speed for RAID 5 - far too slow for the serious work I wanted to do (10 bit 1080i60 is about 160 MB/sec, need about 200 MB/sec throughput for reliable performance with overhead).

Then I got a Highpoint card, which was a bit slow - one of the 1800 series something or other cards (I'm writing this in Newark airport on way to Spain, can't exactly look on the Shelf Of Old Tech to check).

Slowly but surely, more choices. I recently visited a client's shop that had been running one of the newer Highpoint cards (the 2314 was it?) in RAID 5 mode with PLENTY of throughput for uncompressed HD.

I just read this review on CalDigit's HDPro, the writer runs it through its paces with an 8 bit 1080i60 onlining project.

Then I got an 8 drive, 4TB review unit from Dulce that I am overdue to finish my write up and return the unit, but it has been performing splendidly as well.

I think we're there in terms of affordable, fault tolerant storage. They don't have the maintenance features we'd like, you should probably still wipe the RAIDs between projects (still utterly unrealistic in most working situations with overlapping jobs), but the price is getting closer to doable for a broad swath of budget constrained indies.

Give it another year or so for fault tolerance to be a "yeah, of course, throw that in" option on storage.

I'll have more to say when I publish my full Dulce unit review when I get back from Sitges (Spain).

-mike

UPDATE Shane Ross wrote in to ask how I missed his review, in 4 places no less, of the same CalDigit review.

It is MUCH more thorough, go read that one for a better explanation and better testing, including throughput testing on both a mac Pro (over 300 MB/sec!) and a laptop (over 150 MB/sec. Laptop!).

Here's his:

LITTLE FROG IN HIGH DEF: REVIEW-CALDIGIT HD PRO

Labels: hardware, storage

from Apple online store description:

The Mac Pro RAID card offers improved performance and data protection to your Mac Pro system — up to 304MB/s of sequential read performance in RAID 0. Ideal for video and creative professionals with demanding storage needs as well as for tower server applications, this hardware RAID option supports RAID levels 0, 1, 5, 0+1, and Enhanced JBOD. It has 256MB of cache and an integrated 72-hour battery for protecting the RAID cache. The card occupies the top PCI Express slot (slot 4) and connects to the four internal drive bays.

To enable your Mac Pro for hardware RAID, select the Mac Pro RAID card option and two or more hard drives in bays 1 through 4. Each RAID level has minimum requirements for the number of hard drives:

RAID Level Drive Requirements Benefit
Enhanced JBOD One to four drives A non-RAID configuration with the ability to migrate to a RAID set at any time
RAID 0 (striping) Two to four hard drives Maximum performance and capacity for the most demanding I/O requirements
RAID 1 (mirroring) Two hard drives Maximum protection for critical data
RAID 5 Three or four hard drives Data protection, up to 199MB/s of sequential read performance, and efficient capacity utilization
RAID 0+1 Four hard drives A mirror of striped drive pairs providing performance and data protection
The Mac Pro RAID card supports the creation of multiple RAID sets in a system and multiple volumes per RAID set. For optimal disk utilization in a RAID set, all hard drives should be the same size. Your Mac Pro system ships with each hard drive individually configured in the Enhanced JBOD level with Mac OS X installed on the drive in bay 1. Using Apple's RAID Utility software, you can migrate the drives into a RAID set without reinstalling Mac OS X or reformatting the drives, or you can customize your RAID volumes to meet your exact requirements.

Please note: The Mac Pro RAID card occupies one of the available PCI Express expansion slots.


Key thing of note - 199 MB/sec read speeds under RAID 5. Magic number for uncompressed 1080i60 10 bit 4:2:2 video: 200 MB/sec is the usual recommended number. For 1080p24 10b444 RGB: about 230-240 MB/sec. Whither write speed, Apple? Write speed is almost always slower in RAID 5 than read speed, so if read is about 200, and that's the minimum for uncompressed HD, where's the write speed? It is probably lower, and that's a bummer.

Also, that 199 MB/sec - will it hold that through the capacity of the array, or slow down as the drives get full...like most other storage? Remains to be seen.

More later, I'm testing an Octo Mac with a Highpoint 2322 RAID 5 right now, and it works pretty darn well....

There's new versions of iWork, now with a spreadsheet, and iLife, now organizing stuff by Events and yielding TONS more storage space for .Mac accounts (with my 2GB bumped to 20GB, and those who paid for 4GB bumped to 30GB). That's now enough space to back up all my photos...I THINK (double check).

iPhone output is mentioned in iMovie, and you can now FINALLY store all our video in one place in iMovie, organized by Events. Gotta read more, but I'll pick it up at a store ASAP to doodle with. I hope it has improved multi-machine sync capabilities as well...but I doubt it.

-mike

Labels: Apple, hardware, Mac Pro, storage, workflow

One Terabyte Hitachi SATA drive compared to others

Bare Feats takes a good look at some of the new modern hard drives, which I haven't done in a while, and DAMN, they are fast!

For instance, the Western Digital 750GB SATA drive can read/write up in the mid-90 MB/sec - a year or so ago, the faster drives were good for maybe 65 MB/sec - this represents a roughly 50% increase. You'll still have fall-off in performance as they fill up (and data is written closer to the slower data transfering hub as opposed to the faster edge of the platters), but that certainly made me perk up.

In some recent testing of a Mac Pro, for instance, which had a 3x5000GB drive RAID 0, I was playing back uncompressed HD from a 3 drive RAID...without dropped frames! Very impressive. Of course, this was on a nearly empty array, so all data was at the fastest part of the array, and I didn't get a chance to do a zone test (test performance fall-off across the capacity of the array), so I don't know how sustainable that would have been, but newer, modern drives are definitely worth checking out for the speed improvements they yield.

Go read this article to see how the Hitachi 1TB, the Western Digital 750 GB, Seagate 750, and Maxtor 500 all stack up.

By the way - Seagate has been the most often cited as most reliable brand to me in terms of not failing as often as other brands. If building a RAID 0, that is worth noticing. In this lineup, however, that'll cost you over 20 MB/sec per drive.

-mike

Labels: hardware, storage

Writeup from Creative Cow on SAS RAID from Dulce Systems:

Does 250ish MB/sec writes, 360ish MB/sec reads in fault protected 8 drive array setup - nice!

The Before Scenario:

"As so many of my clients use SATA drives on their editing systems (both AVID and FCP), I started to test out some of the Raid protected SAS solutions. I was initially disappointed. On one hand, they were inexpensive, and they were fast. On the other, they were difficult to set up, they never warned you that a drive had failed, they had too many rules for what you could do and not do, and when it came time to rebuild your broken RAID, it was a manual process. For those of you familiar with older working RAID solutions from Apple, Medea, StorCase and HUGE Systems, this was just unacceptable."

The After Scenario:

You click on "Quick Function/Quick Create" and select "RAID Level 3."Another menu that says "64 bit LBA", and a menu that says "Volume Initialization Mode/NO Init", click on SUBMIT, and in FIVE SECONDS, the RAID 3 drive array is there. I then deleted the RAID by clicking on "RAID Set Function/Delete RAID Set", and created a RAID 5, using the same few keystrokes.

IN FIVE SECONDS!!

I couldn't believe it. Well, I really DIDN'T believe it, so I loaded up some media, and yanked out one of the drives. The drive array started to beep, indicating a failure. I opened up the Dulce RAID Console, which indicated that there was a failure. I popped the drive back into the Dulce (with power on - no powering down) - and IT STARTED TO REBUILD all by itself.

This sounds VERY promising, and could be a good solution for a lot of folks. Unfortunately, they do NOT sell bare systems - you have to buy the fully configured card & enclosure with drives.

Mike's Analysis: A 2TB system (8x250GB presumably) starts at $4400, an 8TB is $8000.

Formatted as RAID 3 or 5, this should allow for a usable, formatted capacity of 1624GB for the 2TB model yielding $2.71/GB, up to 6475GB for the 8TB model, yielding $1.23/GB.

Compare this to Apple's fibre channel equivalent cost of a 7TB model for $12,400, pus $500 for the card for a total of about $12,900 for a RAID 50 capacity of 5550GB at a cost of $2.32/GB. The 10.5TB model formats to 8325GB at a cost of 14,300 for a price/GB of $1.71.

There are other differences between the models, but this is a place to start analyzing from for single station utilization.

Sounds very promising, I want to get a review unit.

-mike

Labels: hardware, RAID, storage

Media Distributors to Demo Holographic Storage Solutions June 21 | Studio Daily

"Media Distributors, America's preeminent distributor of professional products and services for entertainment and enterprise, will present a First Ever Technology Showcase of Holographic Storage at the company's Studio City headquarters starting at 4 PM on June 21st. The announcement was made today by Richard Myerson, President, and Tom Evans, Senior VP, Marketing, Media Distributors.


-partners are In-Phase, DSM Terastore & PoINT Software
-live working demo
-only North American demo
-shipments start next month

Read above link for full details.

Labels: hardware, storage

MacNN | MicroNet debuts 5TB eSATA RAID

Micronet has a new OS X compatible eSATA RAID capable of up to 5TB of storage in a RAID 5 config. It has a dedicated controller on board, and is $2350 for a 2.5TB, $3K for a 3.75TB, or $4500 for a 5TB model. PCIe and PCI-X host cards available.

The catch is how fast is this to read/write in a RAID 3/5 config?

-mike

Labels: hardware, RAID, storage

LITTLE FROG IN HIGH DEF: CalDigit S2VR Duo Review...and Firewire VR

Long detailed review from Shane Ross. If I recall correctly (and I may not), he has a similar relationship with CalDigit as I do with Red - worked the booth at NAB.

Lots of details, so if you're considering their stuff, a good read

Labels: hardware, storage

Studio Daily | Archion Announces Synergy HD4

Need fast storage for your Avid Unity setup? This might be what you need - 8TB for $20K, Unity certified.

-mike

Labels: Avid, hardware, storage

Studio Daily | Tips for Implementing Storage - short description here, but a very useful article.

-mike

Labels: storage