HD For Indies

I was scanning through my news stuff online yesterday, and came across an updated version of Au Naturel, a set of plugins for FCP, AE, and combustion. The set of plugins does some useful things, including (to my interest) some more realistic blur and exposure controls as compared to what FCP or After Effects do on their own. Internally, the plugin works in a 32 bit/channel float space (extended detail in shadow and highlights, this is the best way to get accurate results). I emailed to ask the author (Darrin) whether the plugin would return 10 bit results if used on a 10 bit timeline in Final Cut Pro, and 16 bit results if used with a 16 bit in After Effects. As expected, the answer was as follows.

Darrin responded:

Yes, it will return 16-bit per channel data to AE (After Effects) if you're working in 16-bit per channel mode. And sadly, in FCP, it only has the option to return data in 8-bit RGB. It can't produce 10-bit RGB or 10-bit YUV.


I then asked for further detail about FCP, he responded:

I'm not sure there is a way to do all these things (generate 10 bit 4:2:2 or 4:4:4 results in FCP) natively in FCP at this time. Basically, they'd have to be done using FCP's FXScript. The problem with FXScript is that you can only use it for full-frame actions. You can't get at the individual pixels. For something like the blurs, that's probably not too much of a problem (except for edge handling), but for the compositing and transformations, it would pretty much be impossible to do them, as far as I can tell. Plus, you wouldn't get the extended range allowing you to go above the maximum or below the minimum values. Realistically, I don't see it being possible unless Apple makes some big changes and starts giving developers a lot more information. (Or, if they just upsampled the 10-bit data to 16-bit data and passed that to the plugin, it could work right now.)

And that in a nutshell explains some of the limits of what you can do within Final Cut Pro at this time. It's own built-in effects will only handle RGB at a maximum of 8 bits/channel (256 levels of gray or color) as opposed to 10 bits/channel (1024 levels of gray of color). Apple gives developers two ways to make plug-ins (pieces of code that integrate into the program and extend it's capabilities) for Final Cut Pro: FCP supports the After Effects plugin model, so plugins that work in After Effects will (usually) work in Final Cut Pro, or programmers can write plugins specifically for Final Cut Pro using FXScript.

The After Effects plugin stuff only supports 8 bits per channel, period. So no 10 bit anything, and everything 8 bit gets processed from YUV to RGB and back to YUV, which is bad, because there isn't a way to convert YUV-RGB-YUV and have the same colors you started with, even if you didn't "do anything" to the color information while it was in RGB.

The FXScript stuff does let you manipulate 10 bit data (in YUV or RGB as well? Not sure his "10bit data to 16-bit" fully answered that), but it has it's limitations. Apple would need to extend the API (Application Programming Interface) to give developers the hooks into the code they needed to drag out the control they want.

So all this means that Final Cut Pro is hindered at present from being a one stop solution for precise color correction work. That is why I wrote up that post on how to get Final Cut Pro HD timelines into After Effects using the Automatic Duck products.

That's it for today. Or actually, wow, that's it for the year. See you all next year, have fun tonight!

I'd like to give a heartfelt thanks to everyone who's read the blog this year, and especially to everyone who's written in and contributed. In the past four months, I've had over a quarter of a million pageviews, so somebody out there is reading, and it's appreciated.

-mike

If you liked my NAB 2004 coverage, you'll like my MacWorld 2005 coverage.

I'm going to San Francisco, where I lived for a bit during the dotcom heydey craziness.

I'll be covering the show with an eye towards digital video, especially high definition video.

I expect to see exciting new options in low cost high speed disk storage, the anticipated debut of X-San, a new version of iMovie that is expected to handle HDV and 16:9 DV footage, possible faster G5's, and whatever else Steve will pull out of his hat.

If you're going to be attending drop me an email at mike at hdforindies dot com and we'll see if we can hook up and discuss what we've discovered.

-mike

Think Secret is reporting the following:

iMovie 5 is expected to be renamed iMovie HD with its new support for HDV, panoramic 16:9 aspect ratios, and MPEG-4.

in their article on MacWorld Expo San Francisco 2005 (coming up in January 10-14)

This, combined with the report earlier this year at IBC in September when Apple surprisingly announced features of the next version of Final Cut Pro (HDV support, DVCPRO HD 1080i50 support, IMX suport), makes me think that it MIGHT be MAYBE possible that the next version of Final Cut Pro might be splashed at MacWorld. It would be awkward/embarassing to have features in the consumer iMovie HD that weren't present in the professional Final Cut Pro HD. But NAB in April is really the show for that kind of thing. Have to wait and see. Maybe they will announce but not ship until later, so NAB could be release time for FCP 5, good press attention for that. Sometimes Apple previews and releases months later, sometimes they drop the bomb and ship immediately. Have to wait and see.

After posting this, a reader wrote in to point out that the X-San delay might have something to do with this - that maybe they wanted to launch X-San & FCP 5 both at the same time. But MWSF is SO not a video show, NAB is....

I will be attending MWSF and hopefully blogging from the show floor, wireless access permitting.

-mike

OK, it's That Time of Year, so before MWSF comes along and changes the game (which will be fun to see what happens & changes), I wanted to just check in and spell out what I recommend these days for HD editing (on the Mac at least, keep watching for PC recommendations). Without further ado:

Apple Mac G5 dual 2.0 or 2.5 GHz: faster is better, buy what you can afford - $2500-$3000 base price - don't get the fat video cards unless you are sure you need/want them, such as for Motion, since it would limit you to only two PCI-X cards. Do NOT buy one of the dual 1.8 models, it Will Not Do. While the older single processor 1.8 models had PCI-X slots (a crucial requirement), two 2.0 GHz processors are listed as required for the HD cards discussed below. I had someone trying to argue with me the other day that since their 1st generation 1.8 GHz G5 had PCI-X slots, it should work, "since the processor isn't really involved in capturing footage." I didn't know where to start other than to sigh and point at the manufacturer's required specs. Just get the right box, people. A G4 has NO affordable way to monitor HD footage other than connecting to a $25,000 (and up) deck connected to an HD monitoring solution. And that's not very affordable. I don't care how much you luv your G4, it's not the right/cost effective box to finish/color correct your HD project on for multiple reasons. The G5 was a BIG step up in a variety of ways, and you can't upgrade a G4 to get there.

Final Cut Pro HD, or better yet Production Suite: It costs so little more, you're really better off buying the Production Bundle, because not only do you get DVD Studio Pro (you're making DVDs, right?) for less than list, you're getting Motion for free effectively. Final Cut Pro HD is $1000 by itself, spend the extra $300 on the Suite.

BlackMagic Design DeckLInk HD Pro (single or dual link) - $1500-$2000 - (are you REALLY going to be working with 4:4:4 media? If not, the single link is fine.) The single link is probably fine for most folks. My favorite HD card. The AJA Kona2 is the other contender, but it costs more and doesn't suport 4:4:4 with it's drivers yet (last I heard.) The trade-offs are this: Kona2 has realtime 3:2 pulldown on SD output, DeckLink HD Pro doesn't (but I think they're working on it). Kona 2 costs more - $2500 last I heard. Kona2 has a $300 breakout box, DeckLink HD Pro doesn't have one available. (BlackMagic offers the much more fully featured Multibridge, which is hardly fair to call a breakout box, but it's $2000). Kona2 has 4:1 compression QRez codec for offline work, BlackMagic doesn't but supports variable compression PhotoJPEG, which Kona2 doesn't. Both Kona 2 and DeckLInk HD Pro have hardware support for 12 bits/channel, but neither offer codecs, and Final Cut Pro doesn't support it anyway. The two boards are very, very similar. I generally give the nod to BlackMagic, but check your workflow needs carefully and see which best suits your needs. Check to make sure your required features are supported in software.

RAM: Buy at LEAST another GB of RAM, preferably 2 or more. $190 & up. I buy from Crucial.com and have been reasonably happy with that barring some shipping oddities, but find your own favorite vendor.

Storage

SATA Host Card

Sonnet Tempo-X 4+4 SATA card: $200. These are a WORKING 8 port SATA card, reliable so far. No longer recommending Highpoint RocketRAID 1820a until they get their act together. This card is fast and reliable so far in my tests. The only downside is no hotswap - gotta reboot to change drives. If you're going to use the internal ports as well, buy the $20 SATA 8 Channel PCI Cable Bracket Assembly findable on this page.

OR

the upcoming Firmtek 4 port card. Details to follow or see my earlier report. You CAN use two of these at a time in slots 2 & 3 and not hinder your HD performance by putting the DeckLink HD Pro card in slot 4. Careful chosing drives, SSC enabled drives won't work. But Firmtek supports hotswap, Sonnet doesn't.

Drive enclosures:

There are options. Both Granite Digital and MacGurus offer fixed and hotswap enclosures for up to 8 disks at a time. Granite Digital even has an 8 bay hotswap 3.2 TB model for $5900 (ouch! A bit pricey). I'm also very partial to the new Firmtek 1EN2 hotswap enclosure, which gives you a two bay hotswap enclosure with low cost drive trays. Very easy setup, and the drives are connected as directly to the host card as would be a fixed (non-hotswap) setup. Since the SATA 1.0 spec was designed to be a direct (one cable only) connection between host SATA port and drive, the fewer cables involved the better I think. The 1EN2 enclosure can be used with any card, multiple enclosures can be RAIDed together. It's just drives, there's no magic. So you could have 4 enclosures connected to a Sonnet card, or 2 enclosures connected to each Firmtek 4 port card. The upcoming Firmtek card supports hotswap, so that's a plus - you can unmount a RAID from the desktop in the Finder, pop out the drives, put in new drives, and they'll show up. Very slick. The one downside on the Firmtek card is that they are not compatible with drives with SSC enabled. More on that later.

In general, I'd recommend either a 4 or 8 drive fixed array, or a 4 or 8 drive hotswap array, depending on your needs. Too many variables to give a quick answer. Need 10 bit 4:4:4? Use more than 4 drives. Need a lot of space? 8 drives, not necessarily all in the same RAID. Need expandability or multiple concurrent projects? Hotswap. You could get clever and have a fixed 4 drive for scratch work (editing footage and rendering), and archive on a 1EN2. Be careful with the 1SE2 card, however, as it is only PCI not PCI-X and can slow down system performance if not carefully configured correctly.

Hard Drives

I'm partial to buying from ZipZoomFly.com, they have great prices, free second day shipping, and have been good to me so far through about 20 hard drives ordered. Leading contenders for HD:

Maxtor Maxline III 300GB: This is what I have and use and like at the moment. The price went up since I got mine for $220, so they aren't as good a deal as they once were. Fastest drives out there, good warranty, long life expectancy. About 65 MB/sec performance at the faster outer tracks, about 38 at the slower inner tracks. $308/ea and backordered at zipzoomfly.

Maxtor DiamondMax 10 300GB: Performance identical to the above but rated for 8 hrs/day 5 days/week rather than 24/7 for the Maxline III's. $204/ea. Best deal for the speed. Rob-Art reported that Maxtor said these drives were "mechanically identical" to the Maxline III drives. So does that mean they are the same and it's marketing hype that differentiates them? Or more likely that the Maxline III's have better bearings & higher quality parts? If I were being professional I'd say I don't have conclusive data on that. If I were writing a blog, my comment would be "Dunno."

IBM/Hitachi 7K250 250 GB and 7K400 400 GB drives: Identical performance from what I've read - a bit over 60 MB/sec at the outer tracks, about 33 MB/sec when full (inner tracks). Good bang/buck on the 250 GB model at about $135, and the 400 GB model is, well, 400 GB on one drive, which ups your maximum RAID capacity. The 400 GB model is presently $338.

Seagate 7200.8 models: This is the dark horse contender - I hear that these are very fast but haven't seen any test results yet. The 400 GB model is $429, the only drive here over $1/GB. The 300 GB model is $239, which if it's fast would be my new favorite based on the specs & price.

And NEVER buy a refurb drive. EVER. ("See this scar here? Refurb." : ) )

Data Backup for RAID 0: I've switched my recommendation from plain FireWire drives. Either a FireWire dock with cheap removable drive trays, or for a faster solution, external SATA hotswap like the Seritek 1EN2 enclosure. Pros & cons: Everyone has FireWire, not everyone has SATA. SATA is faster. For serious backups, data isn't safe until it's on two separate pieces of media and the two pieces of media are stored in different places.

Computer monitor:

Bigger, brighter, sharper, higher resolution is better. LCD is more expensive but much sharper. I like the La Cie CRT line with their sharp, bright screens and ambient light hoods, but Sony, Viewsonic, and others make nice monitors too. For LCDs, either the Apple 23" ($1900-2000) or the HP based on the same panel at lower cost and more features (around $1600). Digital (DVI) is far sharper than analog (VGA) connections.

If you can afford it, get a second monitor (smaller is fine) for bins etc. Extremely useful, and it doesn't have to be super nice.

Again, budget, space, etc. define what you can do. There's lots of stuff out there that can be found for less, but I'm a quality nut - if I'm going to be staring at this thing for many hours a day, I want it to be sharp and bright. I can't buy a new set of eyeballs if I wear these out (my ears are already shot).

HD monitoring solution:

I recommed a combo approach: get an HDLink from BlackMagic Design, and get the Apple or HP 23" LCD (I like the Apple better). THEN get yourself a proper standard definition studio reference monitor with component inputs. I'm still researching for a favorite. This way you can evaluate detail on the LCD and color on the studio CRT, using the split HD/SD monitoring capabilities recently added to the BlackMagic Decklink HD Pro drivers. You can connect to a cheap consumer HD monitor with the HD analog component outputs on the BlackMagic card just to see what you're doing, and get/rent/borrow the HDLink & LCD and studio CRT at the end of the project for color correction.

Speakers: harrumph. Get something decent. Near field studio reference monitors. I have some Alesis Monitor Ones I'm very happy with, but they aren't made any more. Go research this somewhere else. But recognize that the 3 piece (two sattelites and a subwoofer) typical PC setup is not, Not, NOT going to be accurate and you'll be VERRRRRRRRRY surprised if you try to do your mix on those, the same way you'll be very suprised if you try to color correct on the consumer HD set you edited on because you couldn't afford the HDLink/CRT setup.

Other goodies: I like the Contour Design ShuttleProv2 and ShuttleXPress jog/shuttle wheels. I like the specialty editing keyboards that have the Final Cut Pro commands printed on them in color grouped sets. I'm too lazy to look up a URL, but they're out there (use the Google Bar at the top of this page for "editing keyboard.") I like the Matias Tactile Pro keyboard for typing, it's the best EVER.

OK, did I miss anything? Let me know.

Budget setup:

Now here's the fun part:

DON'T BUY ANYTHING UNTIL YOU SEE WHAT'S OUT AT MACWORLD. That's when lots of new Mac products get released, so Hold Off Until. And even then, if you don't have to have it yet, wait and see what happens at NAB in April.

I'm just amazed at the amount of mail from people that have super detailed implementation questions for a project they aren't shooting until next summer. LOTS WILL HAPPEN between now and then, and I'm SURE my advice will change between now and post-MWSF, and especially between now and post-NAB. Keep your eyes open, keep reading this and other sites that are on top of all this stuff, and don't buy until a reasonable amount of time before your production (enough time to get it in, set it up, troubleshoot it, practice & get comfortable on it).

Here's another online tutorial from Automatic Duck. This one is to get Final Cut Pro projects into Avid without having to redigitize. I'm not an avid Avid user, so I just skimmed this one. One thing I did note was that Avid can't create individual files larger than 2GB using this process. Kinda limiting if you're wanting to do heavy work.

But it's a way to port your project. My last post was about how to do your own high quality finish using Final Cut Pro HD and AFter Effects. It would be a really, REALLY long render if you tried to do a feature this way.

If you had a bigger budget and/or were in more of a hurry and/or wanted the convenience and realtime feedback of an online suite, consider the possibility of doing a DV offline of your project in Final Cut, using the above linked process to pipe it over to an Avid project, then taking that Avid project to an online suite with your HD tapes. I _think_ that should work, but I haven't tried it nor read the tutorial in depth. But Avid project files can "hand up" to the bigger Avid machines, like a DS Nitris that could be used for realtime color correction on HD media. Those systems will do 10 bit realtime color correction, right? Right? (Not absolutely sure on that - somebody confirm? Email me - mike at hdforindies dot com).

This is different from the usual process of export and EDL and take that with your tapes to the online session. If you've done tricky things in your FCP timeline, they are more likely to be supported via Automatic Duck's conversion than by a simple EDL (edit decision list) export. EDL's are pretty non-robusst in terms of feature support, usually.

Patient control freak geek that I am, I'd be inclined to do the self serve model described in the last post.

-mike

So you've got a bunch of high res footage, such as HD transferred to disk from a service bureau, or film scans, or whatever. You don't have/don't want the expensive equipment to view/edit/conform it in it's native format, but you want to be able to have tweaky control of your footage to do things like Color Finesse for highest possible quality color correction. What do you do?

One option is to use Automatic Duck's Pro Import AE 3.0.

It's demonstrated in this online tutorial.

Mike's Comments: I'd run it differently - instead of the accurate but painfully lengthy process of hand rendering out each clip using After Effects to create your low res proxy footage, I would use Final Cut Pro's Media Manager capabilities, and take advantage of it's Recompress capabilities to generate the low res proxies by compressing a whole bin of footage in one command. You could downsample to DV for ultimate portability (but low res), or DVCPRO HD for HD resolution (sorta, 960 instead of 1280 pixels wide, or 1280 instead of 1920 pixels wide) and still have realtime color correction and transitions on your system. If you don't have a 1 GHz+ G4 or better, use DV. If you don't have a good way to monitor HD, use DV and plug into your monitor/TV/whatever you have.

Do your offline edit, use Color Finesse for your color correction (it's higher quality than the Apple realtime stuff), relink to the HD media, THEN use Automatic Duck to carry the project over to After Effects for your high quality final render.

Make it a 16 bit/channel composition and render out to your favorite 10 or 16 bit/channel codec or file format for finals.

Doing a filmout? After Effects does a better job of scaling than Final Cut Pro, can render RGB in greater than 8 bits/channel (FCP can't), and if you want better quality scaling consider using Algolith's Content Adaptive Scaling plugin to get to the film compatible 2048x1536/1556 final size typically used for filmout.

I'd stare long and hard at anyone who said this wasn't a reasonably high quality way to get good quality final results at a reasonable cost. It wouldn't be fast, though.

-mike

OK, for those keeping score - HD-DVDs were announced that could have dual layers - one for standard definition DVDs, one for high definition DVDs. This was touted as an advantage of the HD-DVDs over the competing Blu-Ray format.

Now, the Blu-Ray group is announcing that they, too can do hybrid disks with both SD & HD content on them. But Blu-Ray can do hyrids better - up to 33GB on a single side of a disk on one layer for HD content, up to 9GB for SD content, all on the same side of the disk if I'm interpreting this right. The hybrid HD-DVD SD/HD solution doesn't provide for as much SD content space since it only allows 1 layer (4.7GB) of space for standard definition content.

Read the article for more details if you wish.

Mike's Comments: The one-upsmanship between HD-DVDs and Blu-Ray continues. X-Box 2 is supposedly going to use HD-DVD, Sony's Playstation 3 is supposedly going to use Blu-Ray (since Sony is a major developer of the Blu-Ray standard). Both systems can paly back the same the video codecs, both can do hybrids. Blu-Ray holds more, does hybrids better, but is likely to cost more to produce from what I've gleaned. Sony is at risk of getting into another Betamax/VHS war...and is on the same side it was before - the better but more costly solution.

-mike

I'm going offline for a few days for family/holiday stuff, just wanted to say thanks to everyone for reading and writing in this year.

Season's greetings all - wreck the malls with cows on harleys.

-mike

Quick update - after that last report when I said that I had trouble getting a Sonnet Tempo X 4+4 card based RAID 0 to mount, I haven't had any trouble with it since if I power up the drives before the G5 (which is proper procedure anyway).

I've captured a bunch of video and played it back, including a one hour and 8 minute long single capture, which didn't drop any frames. In various testing I've played back at least 10 hours of footage and had maybe 3 or 4 dropped frames under various harsh testing regimes, so that isn't bad at all.

-mike

Recently Martin Euredjian (inventor of the eCinema device for monitoring HD signals on computer LCD panels) wrote an interesting post on one of the CML mailing lists positing that we were a couple of years away from a good solution for LCD monitoring for HD. I followed up with a series of questions for him and he was kind enough to email back detailed responses. I'm presenting in Q&A form below:

Question: How good do you consider the Apple 23" LCD (or Sony or HP) for in field usage to judge exposure and focus?

Answer: We now have lots of EDP100/Apple based monitoring systems deployed in critical viewing applications. We have, for example, a customer who is editing and color-correcting a weekly episodic show that's going out to six million viewers every week. There's absolute reliance in our color calibration for this post facility. We also have very prominent shows, such as "Star Trek: Enterprise" using our monitoring system on set. All evidence shows that we are doing a good job of providing a solution that can be relied upon. The performance of the panel by itself cannot be considered. The critical component is what makes it work as a video monitor. Simple inexpensive converters cannot even begin to address the special drive requirements that lead to more accurate color and motion rendering.

Mike's Comments: I'm very curious to see/know what can be done with a 60 Hz progressive display to enhance motion rendering in terms of signal processing before it hits the display. Beyond 3:2 pulldown, what can be done that won't introduce artfiacting of it's own that would look different on a true 24p display solution?

Question: Do you think it's worth having these pixel-for-pixel solutions while shooting, or are they overkill?

Answer: If you are shooting HD, you should actually SEE HD. Yes, seeing every pixel is important. It is also important that color and accuracy be addressed by the solution in question.

It is also important to understand that lookup tables (LUTs) alone cannot and do not address color calibration on LCD monitors. There are fundamental reasons for this being the case, some of them being trade secrets that cannot be discussed here. Using LUTs to affect color significantly can seriously affect the fundamental gamma function of the display, thereby distorting the resulting image. We use proprietary, sophisticated (and patent-pending) techniques to bring color and contrast into check without crippling the output image.

Mike's Comments: First off, of course Martin's going to recommend using devices like this on set since he sells them. : ) (I agree, I'm just poking fun.) As for the whole gamma/LUT thing, I'll have to take his word on it, I'm not technical enough to comment with authority. The big debate seems to be between folks like Martin, saying it's hard and complicated, and manufacturers of lower cost competing products, saying their device works, it's not that hard. I don't know the answer to this, I haven't seen them side by side, and on top of that I'm not a video engineer anyway. But everyone that I have spoken to (at last year's NAB & via email) that has worked with Martin ALWAYS speaks highly of his knowledge and expertise.

Question: Would you mind if I posted your comments about "within 2 years" on my HD post related blog?

Answer: A question I get asked relatively frequently goes something like "When are we going to have a true replacement for a CRT-based monitor?". Despite my best efforts --and I do try hard-- I can't say "NOW". There are still technological gaps in TFT LCD's that prevent us from achieving the last few percent of performance we seek. I'm talking high-end reference-grade here. Everything else is available COTS today.

Anyhow, we recently conducted a very exhaustive side-by-side evaluation of our latest LCD technology against the tried-and-true BVM-D24 with a group of Thomson camera engineers (both US and Netherlands). We spent all day running just about every test we could come-up with. Imagine, that? Eight hours evaluating, testing and quantifying image quality and signal processing. The ultimate LCD vs. CRT test. Certainly the toughest crowd I've had to perform for, by far. These tests served, among other things, to further identify the strengths and weaknesses in BOTH technologies. To be sure, perfection does not exist. Where one excels, the other lacks. Kind of like real life, you know?

Cutting to the chase, here it is:

Based on all of my work to date, as well as data from many instances of exhaustive testing (such as the session described above) I think that we will have better-than-CRT performance TFT LCD replacements within two years. The bad news? They will not be as thin and slim as the trend in consumer-land seems to indicate. This due to the hardware required to optimize the solution. Also, they will cost $25,000 to $35,000, without a doubt. Why? Because this "ultimate" device simply cannot be made with current off-the-shelf LCD technology and low-cost electronics. Specialization will elevate the cost basis and this will translate into pricing not too different from that of current high-end CRT's.

Current high-grade TFT's are excellent, of course. When coupled to our EDP100 Display processor, they produce images that can be regarded as Reference grade for most applications. If you need that last 10% in performance though, you'll have to wait a little while longer.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Martin Euredjian
eCinema Systems, Inc.
www.ecinemasys.com

Mike's Comments: Thanks Martin for taking the time to respond.

Been thinking about how one might post and do digital intermediate process on 10 bit 4:4:4 RGB footage from F950 or Viper Filmstream on more of a Do It Yourself basis. Final Cut Pro can capture and play back this kind of footage, but it can't process it (color correction, filters, transitions) at anything better than 8 bits per channel. I'd rather work in 10 bit 4:2:2 (which FCP can handle just fine) than 8 bit 4:4:4.

I did some thinking about what could be done with the tools I know about. What about using Automatic Duck's tools to port the edit to After Effects, where you can process the footage in 16 bit/channel color space (suitably more than the 10 bit source footage), and use Color Finesse for color correction (it's slow, but phenomenal quality). My thought was that AE could be used to process the footage, kick out a file with color correction and transitions finished in 10 bit quality. Render it all out (slow) but then import the render back into FCP for final layback to tape.

If you had logarithmic (as opposed to linear, like video is usually done) footage from a Viper in Filmstream mode, in theory you could do basic editing in FCP and use HDLink to properly preview the log footage in a "viewable" way on a 1920x1200 LCD. Then you could use Automatic Duck to port the edit to combustion to do your picky color correction. As an added bonus, combustion is file format compatible with it's higher end siblings in the Discreet product line - flint/flame/inferno/fire/smoke/lustre. I don't know offhand exactly which products are file format compatible with combustion, but I know at least some of them are.

However, there are problems with this approach - one of the biggest is that the 3-way color corrector in FCP, which is realtime when working with your DVCPRO HD or 8 bit 4:2:2 proxy footage (which is still HD sized and 23.976 fps, by the way), does NOT translate when using Automatic Duck. This isn't Duck's fault - it's just that there is no equivalent tool in AE or combustion to port those settings into. Color Finesse, while slow, at least flows seamlessly from FCP to AE (in theory, haven't done it myself yet).

I've got 90% of the workflow figured out in my head right now, a few tough spots I'm trying to resolve on how to actually sit down and do it. (Paying consulting clients would of course get the full benefit of everything I haven't written on the blog).

One other option for heavier work would be to port to Apple's Shake and do your serious color grading there. But Shake is what, $3000? Ouch. But it used to be $5000, so that's better.

Stu Willis and I have been having an email conversation, he pointed out some options:

They're all free. I have not used any (yet):

http://www.highend2d.com/shake/tools/

Lists a number of shake tools, including ones to convert EDLs to shake, but the FCP one is:

http://homepage.mac.com/bjornfh/trans/howTo.html

(Its also listed on the 2d site).


I've also heard some gossip from around the web that Motion was a practice run for what Apple is planning on doing with Shake - that we'll see a lot of GPU (graphics card) driven realtime effects stuff at a grander scale than what Motion can do. Hopefully we'll see better integration with Final Cut.

Come on Apple - you make'em both, say they're both for serious filmmaking - so make'em work together in the tight, beautiful Apple way, alright?

So that's (most of) what I've got so far.

I need to reality check these on some actual productions. If you have a project in or around Austin, TX that you want to do a high quality HD finish on, drop me a line (email below).

If you have further info, thoughts, insights, links, strategies, by all means, please email them in and let's get this all figured out together.

-mike@hdforindies.com

Quickie - I recently had to handle a huge stack of sequential 16 bit TIFF files, which After Effects couldn't read in - I had to do a batch conversion via Photoshop to sequential PSD files. Hassle.

This fixes that problem, and would be useful in a variety of situations. It is also faster than Adobe's TIFF writer. Features:
(from VersionTracker.com description)

Support for 16-bit TIFFs
Zip compression support
Single-channel (grayscale) saving option
Floating point TIFF support (for High Dynamic Range images)
30% faster than Adobe's plug-in
Better integration with AE's Output Module dialog

Funny how these things time out - I was just thinking the other day that I hadn't heard anything about X-SAN, which Apple said they would ship in the fall. Well, tomorrow is the official last day of fall, and it ain't happened yet.

So right on cue, Apple officially announced that it won't be shipping X-SAN on time.

Duh.

I'd been thinking that if they'd waited this long, they might as well wait until MWSF next month, and I'm betting that's what they'll do - Steve will get up on stage and be all proud of himself and announce that it is available is my guess.

-mike

If you want to keep on top of what's up on this site, use the RSS feed and an RSS viewing application. With it, you can see whether any new articles have been posted to this site, and see the headline and first paragraph or so. Quick and easy.

My favorite RSS reader/browser is NetNewsWire, which lets you skim though all the headlines from all the websites you subscribe to (the list you make). You get to read the headline and the first paragraph or more of the article. Anything you want to read more of, just press return and it opens in the background in your browser.

-mike

Wired has this article about England's plan to put twice as many digital projectors in England as there are in all of the U.S.

An ambitious plan to promote smaller, more independent films that can't take advantage of the economies of scale that come from a full scale feature film with it's hundreds of film prints, media campaign, etc.

It's a very interesting plan, would love to see more of these efforts in the U.S. Mark Cuban, with HDNet and 2929 Films, is trying this in the U.S. but is hungry for content.

-mike

the headline says it. This article talks about the variety of ways digital dailies are being done and handled now, and how they might be in the future. 2K dailies over the internet, the future of 4K, lots of good stuff in here.

If you're thinking of making a high res digital movie (720p and up), this is an excellent read.

-mike

In a hurry here - this article talks about using the latest generation of camcorders - HDCAM, XDCAM, Panasonic SDX900 - as used on a variety of TV shows etc. Good practical advice on shooting & production, much of which applicable to indie filmmaker crowd.

-mike

This guy set the two cameras up side by side and did some test shots, DV vs HDV.

Interesting to see the difference in clarity.

-mike

Another snippet on the low cost Panasonic HD camera that I mentioned the other day. A Guy I Know who had the inside skinny on last year's Panasonic move (DVCPRO HD support via FireWire into FCP) wrote this in a public forum the other day:

Panasonic announced at the last IBC conference that their countermeasure to HDV would be a comparably priced offering using DVCPROHD. I suspect we'll see that product unveiled at NAB


So it sounds like he's pretty confident that it'll happen. That little factoid I missed at IBC - so they ARE planning on having a response to HDV, and it'll be DVCPRO HD based. Wow, now things get really interesting. I don't know how inexpensive they are planning on making the new camera, but if it records DVCPRO HD, and it's Panasonic, it's probably going to be 720p based. How inexpensive could that be? Not sure. I would GUESS that it would be more expensive than the current HDV offerings. Looking at the camera they show, it's prosumer not professional (no interchangable lenses, no serious glass on the unit), so circa $5K is in the realm of the possible. 24p would be a definite possibility -

DVCPRO HD is a MUCH better format to work with than HDV. Uncompressed audio, real timecode, native editability, native FCP support NOW, real decks that can capture in a "normal" type post production workflow, etc.

This would kick us into a REAL HD DV environment - cheap cameras, high res.

Oh, and here's the thread that discusses this HDV vs. DVCPRO HD stuff. Noah puts the smackdown on a lot of HDV misinformation. The one thing that HDV does have going for it is 1440x1080 native data format, whereas DVCPRO HD (if it's 1080i) is 1280x1080, or if 720 res only 960x720. BUT the HDV cameras have 960x1080 sensors, etc. etc. etc. We'll just have to wait and see whether it's 1080i or 720p.

-mike

Got some reader mail about somebody wanting to transfer an HDCAM project to D-VHS on the cheap. I don't know off the top of my head of anyone that has a dubbing service for this.

If you don't have a Kona2/DeckLink HD/similar card and disk array, there are alternatives.

I was thinking that the person could send their completed HDCAM project to a place like bonolabs.com where they can capture your footage to disk. (They can also transfer film to disk, but then you're doing a "blind" telecine session, that makes me nervous about the look of what you'll get back).

Once they recieved the footage (on a FireWire drive), it would be pretty straightforward to use LumiereHD to encode the footage back to an MPEG-2 TS stream to go to D-VHS deck.

At least, I'm 95% sure that could be done without sitting down and doing it myself.

-mike

I've been thinking more about production stuff, here's some notes from email I've sent lately to readers about projects. Yeah, it's rambling. This is a BLOG!

: )

There's 4 kinds of media:

Truly unique footage - data that comes from a place that can't be recreated no matter what - such as if you captured video directly to disk from camera.

Expensive to reacquire unique footage - data that would be difficult to reacquire, but could be done so but it would take time and money. Such as having to rent a deck and then recapture all the footage from your batch list in FCP. Things in this category take hands on machine, actual people hours to get, and often there are hard costs of equipment involved too. This is the kind of stuff that keeps you at the studio over the weekend. (You could argue that you could always reshoot anything, but that's REALLY expensive, and you Get Shot in the post world if you screw up that bad).

Regeneratable footage - time consuming to regenerate. Notice I don't say recreate, just regenerate. It takes machine time, not people time, to recreate this for the most part. So FCP renders for color correction/effects/transitions would fall into this category. Rendering output from a 3D or compositing program falls into this category. It would suck if you lost it, it would take time, but it could cook over the weekend while you went water skiing. You get the difference.

User/operator decisions - this is the REALLY unique work - your editing decisions, your compositing/3D files, Photoshop files (that you've edited & done stuff to), etc. These are the smallest files, but the most important. I back these up very regularly to "little" media like CD. If there's a lot of Photoshop work involved, maybe a DVD-R. These are the files I'm most concerned about.

Here's what to do about them -

Truly Unique Footage - you HAVE to have very up to date backups of this stuff. If it is shot footage, it's usually all coming in early in the project. The bad news is there's tons of it, the good news is it usually comes in early so you can get it all backed up to disks or whatever, and then you're done with backing it up. For your specific project, it'll come trickling in over time I'm betting.

Expensive to Reacquire- this is usually footage from tape. Tape can be a backup, but it's very time and manhour expensive if you lose your data. Having solid timecode, NOT using "Capture Now" without timecode, and good discipline in logging makes sure this data is recapturable. If you own/have ready access to the deck, you can potentially treat that as your backup, knowing it'll take a bunch of time to recover.

Regeneratable Footage - IF you can survive the time hit if you lose this footage, having it on a RAID 0 or individual drives is bad if you lose it but recoverable. If you can afford backup, use it.

User/operator Decisions - But for GOD'S SAKE, have your source files (AE, combustion, FCP, etc.) backed up DAILY, worst case WEEKLY. Think about it - what's the most you could survive without, or having to recreate? And could you get it back just the way you liked or, or the director liked it, or the DP liked it? Yeah. Back that stuff up daily.

So it boils down to risk management - what could you survive without, and what are the odds of losing it? How bad would it be to lose it?

I've been thinking about the different kinds of storage, too.

Lots of readers are assuming they need all their footage online, all the time, on disks fast enough to capture and play back the footage. NOT true.

For instance:

Apple's X-RAID, when configured to be fault tolerant (otherwise why buy it?) in a RAID 50, can write about 200 MB/sec, but can read something close to 300 MB/sec.

10 bit/channel (30 bit color), full res 444 RGB at 60 fields/sec (or 30 frames/sec) is about 240 MB/sec, the heaviest HD signal generatable today that is still HD video based. To give some safety headroom, disk systems need to be capable of about 290 MB/sec. This is too much to capture on a single X-RAID.

BUT...you could capture to a low cost SATA RAID 0 (sub $3000), then copy the data over.

AND...it IS fast enough to play it back in realtime.

This faster reads than writes is The Deal when working with fault tolerant storage that does RAID 3, 30, 5, or 50.

Say you have a ton of footage you need to capture and store securely. Say you have 2TB of footage you need to capture from that expensive tape deck you don't want to have to rent again until it's time to lay back to tape, and you need to do VFX work on it.
(I could get more subtle and say analyze what does and doesn't need to be at online quality vs offline quality - you COULD capture everything to HD res offline (DVCPRO HD for instance), then capture at online quality for just the shots that need FX work)
Or, however it's going to be generated (say you're doing stopmotion, timelapse, whatever) you're going to need to stash 2TB. Capture it with whatever speed you need, but consider storing it on something cheaper. FireWire disks, hotswap SATA drives in a dock/drivetray setup, SyncRAID XL (maybe, still testing) fault tolerant array on a cheapie G4, etc.

Think about your options and how you need to work.

Some options:

Highpoint RocketRAID 1820a using the RAID 5 capability: nice in theory - fault tolerant storage, online speed. Real world: flaky, flaky, flaky! I could barely get mine to function, and when it functioned it corrupted data. Supposedly others are using this feature on Mac, but I can't get it to work, and I kinda sorta know what I'm doing. RAID 10 fares a little better, in that it works. I have a client that used RAID 10 for a week on their uncompressed HD project. But with any level RAID, getting the RAID to show up and stay up and not get corrupted is a problem.

FireWire drives - what I used to recommend the most. Voluminous, immediately accessible (plug it in, drive shows up, copy off/open data right away). Cons - enclosures can be $50-$150 per disk, increases cost/GB. VERY convenient, everyone has a FW drive. FireWire 400 tops out around 35 MB/sec, FireWire 800 tops out around 50 MB/sec.

FireWire drive sled - it's a FireWire drive, but "open" so that you can re-use the enclosure and bridge board (the chipset that translates FireWire signal to IDE signals). The drive sits on a cheap $10-$20 sled to protect the guts on the bottom. I think Granite Digital sells a FW drive sled. Very convenient for the reasons above so long as you keep the enclosure part around.

External SATA drives - about the same functionality as the FireWire drives, faster (native SATA is faster than FW), but less convenient since others have to have an external SATA card.

External SATA on removable SATA drive tray - my favorite fast/cheap solution of the moment, but does requires a PCI or PCI-X card with external SATA ports. These are coming out more and more, check MWSF for more. $170 dual bay enclosures, $22 for drive sleds.

Cheap nearline storage server - take a used G4, put up to 3 SyncRAID XL cards in it (not sure multiple cards cooperate, haven't tried it yet myself). Put external SATA drives, be they fixed or hot swap, and you can have up to 4 1/2 terabytes of fault tolerant storage for under $9000. For render intensive projects, this can work pretty well - I did some render tests, and using footage linked across a network rendered just as fast as content on a local drive. Editing? Haven't tried that, don't even know if DV would work in this context over a GigE network.

After going on and on about how tape was obsolete, I'd basically given up on tape backups as too slow, too small, and too expensive. Until somebody mentioned LTO-2 tape the other day on the CML (Cinematographer's Mailing List). Uncompressed, 200 GB per $50 tapes, 40 MB/sec, drives (even autoloaders!) under $5K street price. This sounds exciting, until I realized you'd have to back up about 11 TB before you save any money over a hotswap SATA dock setup like the Seritek 1SEN2 combo (or others like it). And that's about 17 hours of 1920x1080, 24p, 10 bit, 4:4:4 RGB footage (Viper Filmstream or Sony F950 footage), or 25 hours of 10 bit 4:2:2 YUV Footage (Viper in HD mode or Sony F900 recorded to HDCAM SR), or 33 hours of 8 bit 4:2:2 (HDCAM footage).

OK, enough rambling.

-mike

UPDATED - SEE BOTTOM
So I've mentioned in the past that there were three primary obstacles to inexpensive editing of HD material (skipping over the whole massively expensive cameras and decks part).

1.) HD-SDI capture cards: this has pretty much been solved by the presence of cards from AJA and Black Magic. AJA's Kona2 lists for $2500, BlackMagic's product line runs from $600 to $2500. This is nothing new, I've been writing about these cards for 6 months or more.

2.) Storage - fault tolerant RAID arrays capable of more than 200 MB/sec across their entire capacity (good enough for all 10 bit 4:2:2 HD work) are out there but expensive. Prices start around $7000 to $8000 and go up from there. Cost per gigabyte of storage starts around $7/GB and gets down to about $4/GB in the larger arrays costing $12,000 to $15,000. My suggestion for those on a budget is to try native Serial ATA (SATA) arrays on cards like the Sonnet Tempo X 4+4 connected to an external SATA enclosure like the Burly Box from MacGurus or somesuch. But then to have a backup of all of that data on either FireWire or removable (dockable) SATA drives. I've talked about this before. Latest twist - set up a 6 or 7 drive fixed array, and leave that last port or two for dockable storage, such as the low cost Seritek EN2 setup I wrote about a few days ago (scroll down or search). The EN2 comes with two drive trays, extras are $22/ea. 200 GB drives can be had for as little as $118 at zipzoomfly.com, so that's $140 for 185 GB of real world backup - that's about 75 cents/GB. Cheep. Or, and I just now thought of this, have a bay (or two?) in your Burly Box set up with a hotswap tray (available from MacGurus) and use their drive trays. Yeah. That might be tidier, but I like the more direct connection of the Seritek's docking solution, and their drive trays are less expensive.

3.) Monitoring HD Video Signals - the low cost champ solution has been to get a $700 BlackMagic HDLink and connect it to a 1920x1200 23" LCD, such as from Apple, Sony, or HP (HP has the lowest cost unit, around $1600) to monitor 1920x1080 pixel HD video signals. This provides an extremely crisp, sharp view of your HD video signal, but doesn't have the contrast, dynamic range, or subtle tonal reproductions of a CRT. This is nothing new, I've been talking about it for months as well.

Latest twist - The latest version 4.7 drivers for the BlackMagic DeckLink HD Pro Dual Link ($2000) allow for simultaneous output of HD and SD monitoring signals. If you're working in 50 or 59.94 fields/sec HD video, the DeckLink HD Pro Dual Link can be configured to output 1920x1080 to the HD-SDI (useful for HDLink viewing on LCD for sharpness and detail), and ALSO simultaneously send out an analog SD signal over the component analog wires for viewing on a standard definition monitor.

That way, in theory, assuming the colors aren't changed during downconversion, and you're using a calibrated studio grade monitor, you could judge your color on the SD image, and judge your pixel level detail stuff on the HDLink connected LCD monitor.

Also, for those working with DVCPRO HD footage and looking for a way to monitor it, you can use the built in down conversion capabilities of ANY of the BlackMagic cards, even the $295 DeckLink card. So for $300, you can monitor your DVCPRO HD material on an SD monitor. This is a HUGE improvement, since before, the only way to monitor DVCPRO HD footage was either via FireWire back through the Panasonic AJ-1200A DVCPRO HD deck to a studio monitor, or with a $600 plus DeckLink HD card. You still have to be on a G5 for this to work according to the tech notes, but $300 is a very modest addition to an existng G5 based editing station that is set up to edit DV with a monitor with component inputs.

I haven't tried it on my own system yet due to a nightmare of cables and gack in the studio. But this certainly does sound cool and useful.

As always, for the frugal, you can do your editing onscreen and rent monitors at the end of the production cycle to do your color correction and use the realtime capabilities of Final Cut Pro HD.

I haven't tried this setup yet, and I don't know if the simultaneous HD/SD monitoring is only via the twin SDI ports on the dual link HD Pro cards, or whether the analog component is kicking out SD or HD. Hopefully the analog is kicking out SD, that way it could hook up to a less expensive monitor, and/or wouldn't require an SDI to analog converter.

Other stuff - you can now output NTSC/PAL composite output with the "Y" connector on the analog breakout from the card for SD monitoring.

The one thing that DOESN'T work the way I'd want it to for sure is that the realtime down conversion is for NTSC & PAL framerates ONLY, there is NO support for 24p in this simultaneous HD/SD monitoring configuration. Maybe they'll get really clever and figure out some 3:2 pulldown magic with this, but we'll have to wait and see.

-mike

UPDATE 12/17/04 - A couple of anonymous Commenters on the blog wrote in to say the following:

Anon 1:

The DeckLink HD Pro Dual Link card can down convert HD to output standard definition video using either hardware or software.

Hardware down convert gives you Ch A HD (serial digital), Ch B SD (serial digital) and SD on the analog component/composite output.

Software down conversion gives you SD on all outputs.

Also your choice of letterbox or anamorphic.


Mike Says: Very cool! This is maximum flexibility/convenience. To have SDI or analog for HD is very useful, and definitely allows for maximum flexibility in workflow

Anon 2 wrote:

Mike - great site. A note on the downconversion topic. With the Blackmagic cards, yes, they don't support 24p downconverted to 59.94i downconversion, but the Kona2 card will happily do this. It's a great piece of kit.


Mike Says: Good to know. Other than Q-Rez (1/4 data rate alternative offline codec specifically for use with Kona2), this is the only feature point of differentiation between the Kona2 and DeckLink HD Pro Dual Link that I'm aware of offhand.

AJA folks or fans, if you know of any other points of differentiation, please let me know. I've been hyping the BlackMagic stuff because I was able to get my hands on one to evaluate, then bought two of them with my own Hard Earned. Kona2 didn't start shipping until mid-October from what I heard. Last I heard a few weeks ago, Kona2 still couldn't do dual link correctly (disabled in software). AJA has held off shipping hardware mostly until features were implemented, BlackMagic tends to ship hardware early then fleshes out (or fills in, depending on your perspective) the feature set via software updates.

-mike

Update: a little more context:

This feels too cool to be true. Mysteriously, all I got on this was a link from Christopher Barry in Australia. What you see here is as much as I know:

Cinematography.com page.

Here's the photos they link to:

Photo 1

Photo 2

Photo 3

Total Conjecture Follows

What it LOOKS like is a lower end camera that records to DVCPRO HD data format onto Panasonic's P2 memory cards. I'd guess it might be 720p since that's what the Varicam does (and it's lower data rate, more suitable to P2). IF this is real stuff, this could be a sub-$20K camera...in theory.

Are these real? Are they merely marketing mockups? Is this "roadmap demonstration" or actual hardware coming to market? I recall seeing some faked up dummy stuff at NAB similar to this.

OK, that's as out on a limb as I'm willing to go.

Might be prototypes of future stuff, might be wooden mockups.

If nothing else, it's Panasonic testing the winds for industry feedback.

IF this were where they were going, here's what I WOULD think: unless P2 is cheaper to produce than tape formats, I'd rather have tape formats and ingest via FireWire. P2 requires a specific docking station to read those cards. All Macs have FireWire and PCs have it or can add for $50. If they go DVCPRO HD on a lower end camera, I'd love it, and I'm sure indies would love a low cost high res solution. Sony's HDV looks good, but is so compressed it causes me concern (especially the compressed audio, remember that!), so this could be a good intermediate camera that's easier to work with.

If it exists.

: )

-mike

UPDATE LATE FRIDAY:

Reader Jerome Olivier wrote in:

Mike,

Regarding the Panasonic cameras, just thought you might be interested in a translation of what the panel reads in front of the camera in the 3rd picture:

Small Type Memory Card
Camera Recorder
"P2Palm" HD-MPEG type

Looks like there's two models on display in those pictures, one that saves to MPEG and the other that saves to DVCProHD. Consumer / prosumer versions?

MacWorld is carrying a story about NEC's prototype HD-DVD drive, suitable for computers, that will read and write HD-DVDs and read and write DVDs and CDs. The story didn't mention whether DVD-R or DVD+R was supported (or both). To be clear the drive will support rewritable HD-DVDs, too, so it could potentially be a good data backup solution.

This is just one more step on the road to high def DVDs. If you're shooting HD or HDV, you'll be wanting to use this technology in a year or so. In the meantime, there are literally dozens of players that will play back some kind of Windows Media 9 based encoded video. I recently read an article somewhere about how to set these up on a PC, but I don't recall where. If anybody knows, email me at mike@hdforindies.com

-mike

BlackMagic Design today announced that their DeckLink Multibridge is shipping to customers.

It does of ton of things, but the short of it is that it will convert between HD & SD from either analog or digital inputs to analog or digital outputs. Want to input analog HD and kick out SD-SDI? Can do. Want to uprez a VHS and capture it via HD-SDI uncompressed HD? Can do.

Partial from their website:

Dual Link SD/HD-SDI 4:2:2 and 4:4:4 input and output.

Analog HD/SD YUV RGB 14 bit 4:4:4 component video output, switchable to S-Video.

Analog HD/SD YUV RGB 10 bit 4:4:4 component video input, switchable to NTSC/PAL. Analog sync input.

Analog composite NTSC/PAL 14 bit 5x oversampled output.

HD format support:1080 lines at 24p, 25p, 30p,48i, 50i, 720 lines at 30p, 50p, 60p

SD format support: NTSC, PAL.

SDI compliant with SMPTE 292M, SMPTE 259M, SMPTE 296M and ITU-R BT.656.

Color precision support: 4:2:2 10 bit, 4:4:4 10 bit, 4:4:4 12 bit

Color space support: 4:2:2 YUV, 4:4:4 YUV, 4:4:4 RGB

4 Analog balanced audio inputs and 4 analog balanced audio outputs on XLR connectors.

8 Channels AES/EBU digital audio in and 8 Channels AES/EBU digital audio out.

2 Analog consumer audio outputs using RCA connectors.

Operates as SDI embedded and de-embedder when not used for video.

Firmware upgrade, calibration and settings via USB 2.0 for Mac OS Xô and Windows XPô

Real time video processor for down conversion and other tasks.

Reversible rack mount in 1 Rack Unit size. Mount for break out box or traditional converter operation.

Two models, Dual Rate HD/SD or lower cost Standard Definition only.


It does a lot of stuff, so check out their website.

-mike

Media 100 has shipped their Media 100 HD system. Highlights:

From their website:

Features

Multiple tracks, powerful effects
-Multiple video tracks (up to 99) with opacity controls in the timeline
-Real-time audio effects with audio dynamics and reverb
-Advanced keyer with RGB, YUV and HSL controls— and real-time SD performance
-Intuitive color correction with real-time SD performance
-Full motion alpha channel support with real-time SD performance

Broad support for HD and SD standards—with real-time format conversion
-10-bit uncompressed SDI I/O: 1080i, 720p, 525/60 (NTSC), 625/50 (PAL)
-10-bit uncompressed component video I/O 525/60 (NTSC), 625/50 (PAL)
-All-On-One Mastering: capture and conform to and from multiple HD and SD resolutions
-Real-time, broadcast-quality HD to HD, HD to SD, SD to HD format conversion
-Real-time aspect ratio conversion (when format converting on I/O)


-Simultaneous HD and SD output for program out and monitoring
-Professional 19" rack-mountable I/O break-out box with Genlock

Simply Compatible
-Media 100 HD-native software codec
-Media 100 i® and 844/X compatible
-QuickTime® support (including Animation, Motion JPEG B, FCP 10-bit, DV codecs)
-Mix any supported codecs in one timeline— without conversion or rendering

High Performance
-Faster-than-real-time rendering of many complex effects
-Real-time SD editing and multi-stream effects
-Accelerated HD editing and multi-stream effects
-Hardware and host-based acceleration support for maximum performance

Mike's Comments:Reading between the lines, here's what I glean from that:

1080i & 720p support - no 1080p support

realtime SD color correction - therefore no realtime HD color correction?

Mix codecs on same timeline - actually, that is a very nice feature I wish FCP had. Faster, easier, time saving, less drive space needed...a very cool feature.

It says realtime SD effects and accelerated HD effects. I read that as faster than the CPU could do it alone, but not realtime.

But with no realtime HD transitions or color correction, and ESPECIALLY no 1080p support, this system really is not useful, as I understand it, for indie filmmaker types.

Optibase bought the assets of Media 100 when they filed bankruptcy (I think that's right, maybe they were just on the edge of bancruptcy?). In any case, they were in financial trouble. Now that the product has shipped, I'm not massively impressed.

A Final Cut Pro HD system with BlackMagic DeckLink HD Pro or AJA Kona2 system can do everything this system can except for mix codecs on a timeline, and have additional crucial features (1080p support, realtime color correction and transitions) Media 100 HD lacks.

-mike

So I'm getting ready to send my review unit back to Firmtek, and I get curious before I let it go how it would do in one of my older Macs, a G4-867 single processor model. (the model before the Mirrored Drive Doors/Wind Tunnel model). I don't have a BlackMagic card in that box, so I used QuickBench 2.1.1. So the results aren't quite apples to apples.

But on this older Mac, using the same two drives that I tested with this card & enclosure the other day, I got very similar results.

Using the QuickBench utility, I can tell it to test transfer rates on various data transfer sizes. Using a sequence that tests from 20-100 MB transfer blocks, I was getting over 100 MB/sec. Not quite as fast as the results reported with BlackMagic's Disk Speed Test Utility, but pretty close.

I was surprised - I thought the slower performance of the PCI slots in this older G4 would be an impediment. But then I remember a discussion I had with another vendor about a different card, which boiled down to the fact that the maximum transfer rate for a 32 bit card (at least theirs) was around 115 MB/sec.

So my GUESS is that one MIGHT have a throughput limit of around 115 MB/sec, which is very easy to exceed with two SATA disks (the Maxtor Maxline III & DiamondMax 10 disks top out at around 66 MB/sec per disk, the Western Digital Raptor 10K tops out around 72 MB/sec). But, for those using a G4, you can't use a Kona2 or BlackMagic card anyway for HD in those systems, you're stuck with SD for SDI type I/O. So if you're editing any kind of standard definition video on a G4, the 1SEN2 combo will work for you, too.

And if you're doing FireWire based HD with DVCPRO HD, it'll work for that as well.

So DV, DVCPRO, DVCPRO 50, DVCPRO HD, and any other compressed or even uncompressed SD should work fine with a 1SEN2 combo.

-mike

Site Traffic Update:

I was checking my logs the other day, and realized it had been about 90 days since I had started seriously tracking traffic in a consistent way. In the 90 days since I started tracking traffic, I've recieved over 200,000 pageviews.

Wow! I had no idea there'd be this kind of interest.

Monday I had over 3000 pageviews, and those aren't final numbers. (It isn't midnight yet in time zone where servers are).

I'm a little frustrated at myself for falling behind on posting my own research resuls into HD workflow, performance of StorCase and XServe RAID and other products, and just posting a lot of industry news with commentary, so I want to get back to my own unique, nowhere else stuff on here.

But I'm very pleased to see that I have a growing audience of readers interested in this stuff.

Answering Reader Mail & Their Questions:

As always, feel free to write in, I respond to all reader mail, even if I don't know the answer to your questions I'll write back. There have been a few readers with such detailed specific questions about their own particular production or workflow needs that I've basically said I'll answer that but it'd be on a consulting basis. As a general rule, however, if you have an issue that applies to a goodly chunk of my reader demographic, I'll answer it or do some research to find the answer. Only when a question is so specific that I don't think it's of wide interest, it requires more than casual research/effort on my part, and/or it's for a specific project that's a for-money gig will I say that's a consulting question, not a freebie. But please don't let the "for consulting" part scare you off - write in, what's to lose! Worst case scenario I might not answer your unique-to-you question for free. But I never charge any money unless we discuss it upfront and both parties agree beforehand. I'm not going to surprise anyone with a bill just for answering a question.

On the other hand, if you do have a production coming up and need a consultant on your acquisition and/or post & effects and/or offline & online workflow strategy, I'm Your Man. Email me and we can discuss needs, rates, schedules, etc. It's surprising how much an all data-centric workflow can save. I've spent months of research into low cost, fast/efficient workflows designed specifically for budget/gear constrained indie productions. And those months of research are based on starting with over a decade of digital video production and post production experience.

So in general if you have questions or comments please do write in or use the Post A Comment link at the end of each article. If you don't want your questions public just email me directly at mike@hdforindies.com. If you want an answer and use the Post A Comment link at the bottom of the article, please do include your email address in the message you send me otherwise I'll have no way to respond (Post A Comment entries come through as anonymous from the blog, even if I get a name attached to it.).

So keep writing and asking questions, or floating suggestions, or sending in the links - always appreciated!

And to my steady contributors like Christopher Barry and Martijn Schroevers, please keep it up, very appreciated!

-mike

OK, this isn't really indie related, but interesting from an industry perspective. Thomson/Grass Valley, which also makes the Viper camera, and owns Technicolor and it's DI lab, and other post facilities, recently acquired The Moving Picture Company, which has done visual FX for movies (Alexander, all Harry Potter movies, etc.) and commercials (Coke and whatnot).

Sounds like they are trying to get more vertically integrated - own the production methodologies from acquisition, to editing technologies (Grass Valley), to post production and visual effects (MPC), through DI to film (Technicolor).

A bold move. Who else has the breadth to both make the camera, make some editing gear, actually do the post, and own the high end DI & film labs. The other name that comes to mind?

Sony.

Interesting.

-mike

HD Labs Report: Firmtek Seritek 1V4 Pre-Release Review

As I mentioned in my Firmtek Seritek 1SEN2 review the other day, Firmtek is planning on coming out with a 4 port Serial ATA card of their own, in both internal and external port models. (Hopefully both will be viewable at the MacWorld San Francisco conference next month)

I was a beta tester on the Seritek 1V4, which is the four internal SATA port product, and now that the cat is out of the bag, I can share my experiences with it.

The version of the card I tested had four internal Serial ATA ports. Since I'm not a big fan of internal ports, I routed the cables to the outside of the unit and connected them to my MacGurus.com Burly Boxes, with 8 Maxtor Maxline III 300 GB drives, my favorite bang/buck hard drives for uncompressed HD SATA work.

INSTALLATION:

It's a PCI-X card, which means that it won't slow down an adjacent card if mounted in slots 2 or 3 of a dual G5. This becomes important when you need to mount multiple cards in a computer.

Installation is simple, it takes about 10-15 minutes to shut down a PowerMac, put it on it's side, open it up, install the card, route cables, close it up again, and plug everything back in.

It's driverless, so that there is no software installation, and therefore no software driver incompatibilities. After installing the card and conecting drives, it just works.

One thing I really do like about their internal port cards is that they use the most solid, robust connectors for their internal connections as compared to everyone else's internal ports. The manufacturer claims these stouter ports will help protect against ESD (electrostatic discharge) damage. How likely is that to happen? I don't know, but since we're getting into the dry winter months, the thought of scuffing across the floor and static zapping my G5 dead as fried chicken is not an appealing thought. I just pulled off my fleece jacket and heard it crackle with static electricity - a possible death knell for computer circuitry.

As with all internal port cards, my recommendation is to route the cables out an empty PCI slot cover, anchor them with zip ties to the chassis to prevent yank damage/disconnects, and connect to an external SATA chassis. Firmtek's own Seritek 1EN2 is an interesting hotswap solution for 2 drives at a time. You can hook up 2 of these 2 drive enclosures to create up to a 4 drive RAID 0 stripe for the kind of performance I mention below. There are also external enclosure solutions from MacGurus and Granite Digital.

PERFORMANCE

Drives can be formatted as individual drives or striped into an array using either Apple's included Disk Utility or SoftRAID, which gives more options and controls. Performance is pretty much up to the drive - the card will pump data about as fast as the drive/s can disk it out. In my tests, I used four Maxtor Maxline III 300 GB SATA drives striped into a RAID 0 with SoftRAID 3.1, and got read speeds of 238 MB/sec, write speeds of 255 MB/sec using BlackMagic's Disk Speed Utility v4.6 on an otherwise empty array.

Doing my usual slice testing, which tests how drive performance falls off as the disks get full, here's what I saw:

(partitioned using SoftRAID 3.1)

Slice 1 (beginning of disk, fastest part): 238 MB/sec reads, 252 MB/sec writes
Slice 2 (20% full mark of array): 229 reads, 242 writes
Slice 3 (40% full mark of array): 212 reads, 225 writes
Slice 4 (60% full mark of array): 193/204
Slice 5 (80% full mark of array): 168/178
Last 25 GB (to check worst case performance): 139/147

As you can see, the performance is pretty impressive. Using two cards, performance should nearly double from what I've been told, but I haven't verified that myself.

What does that mean in terms of real world performance?

It should be able to handle multiple streams of SD uncompressed video just fine.

Obviously, multiple streams of compressed video should work OK as well.

Same goes for the compressed HD formats - DVCPRO HD, PhotoJPEG on BlackMagic, AJA's QRez, and eventually Avid's DNxHD should all do just fine, with support for multiple simultaneous streams (limited by and depending on seek times and throughputs of the drives involved, and the speed of the computer involved).

For uncompressed HD editors, it would work fine for 1080p or 1080i work just fine in 8 bit 4:2:2 video. If you are capturing at 10 bits/channel, you'd need to partition the array using SoftRAID to make sure that the performance would always be fast enough.

For 10 bit 4:2:2 1080p24 work, you'd be able to use 80-90% of the array's capacity.

For 1080i29.97 work, you'd be able to use at least 60% of the array with these particular drives (fast & large).

Maxtor also makes the DiamondMax10 300 GB drives, available for $206 presently at ZipZoomFly.com with free 2nd day shipping. These drives have nearly identical performance as compared to the Maxline III's but cost less (but aren't as enduring/robust, lesser warranty).

IN USE:

One thing in particular I REALLY liked about this card (in my limited testing) was that it never, EVER failed to mount drives upon reboot. I, and others, have had SIGNIFICANT problems with the Highpoint RocketRAID 1820a card failing to mount drives at startup or upon a reboot. I've had readers complain of 1820a arrays dropping offline in the middle of usage, with resulting complete data loss, a risk with ALL the RAID 0 SATA cards on the market.

Side note - I also recently had problems with beta hardware of the Sonnet Tempo-X 4+4 card I recently reviewed online, in that an array REPEATEDLY wouldn't mount upon multiple reboots (this happened after I posted my review). Sometimes I'd reboot and it would see 6 drives, sometimes it would see 7, but not all 8. I was trying to work on a project, so I just pulled the card and installed a RocketRAID 1820a for a while instead. Now, to be fair to Sonnet, that was with a beta, NON-RELEASE piece of hardware, and I might have just have had a loose cable somewhere. I just received my shipping unit Sonnet in the mail and I'll be installing and testing that soon.

One of the tests I want to put both pieces of shippng hardware through is a Reboot Test - simply reboot 20 or more times and see if it mounts the array every time.

REAL WORLD FINAL CUT PRO HD PLAYBACK TESTING:

OK, back on track here - I didn't get a chance to test as much as I would have liked to, but one of the things I did test was to see how well it would play back uncompressed HD in a real world setting.

So what I did was to set up various timelines in FCP HD 4.5 on my dual 2.5 GHz G5 and play back 10 minutes of HD footage. I had my settings such that if it dropped a frame it would abort playback and report the error. If it finished the sequence with no reported errors, that was a pass. If it didn't, that was a fail.

Working my way up from lowest to highest throughput media, here's what I got:

1080p24 8 bit 4:2:2 (24p HDCAM): PASSED all the way through to and including Last 25 GB of the array

1080i29.97 8 bit 4:2:2 (HDCAM 29.97 interlaced):: Not tested, but should pass based on the fac that it is less data than 1080p24 10 bit 4:2:2 (the next test below)

1080p24 10 bit 4:2:2 (HDCAM SR YUV, D-5): PASSED up to but NOT including the Last 25 GB partition (the last partition that passed was Slice 5)

1080i29.97 10 bit 4:2:2 (HDCAM SR YUV, D-5): PASSED through slice 3, slice 4 failed after a few minutes

1080p24 10 bit 4:4:4 (HDCAM SR RGB): Slices 1 & 2 passed, rest failed (so you could do SOME HDCAM SR work with this setup, just not a lot. The first sliced passed the first time, then failed after 6 1/2 minutes. Slice 2 passed. So if you were producing an HD short, you could offline then online in 10b444 RGB. Slice 1 & 2 are only about 35-40 minutes of footage. But any effects or color correction work gets downsampled to 8 bit RGB as I understand it...bummer.

Side note: after running these tests, I had a client tell me that they had trouble with their 1820a based system dropping frames after 10-15 minutes consistently. So testing testing methodogy isn't bulletproof - just because it'll play for 10 minutes doesn't mean it will play perfectly for an hour long show to record to tape. Doesn't mean it won't, either.

SUMMARY

PROS: low cost, high speed interface card (I like the future 4 external port better than the internal port version). Solid connectors. Driverless.

CONS: Internal ports a pain/risk/hassle to port to the outside of the chassis. Only 4 ports, not 8 (but you can use two in a G5). No RAID 10 support, so if the RAID 0 fails due to brownouts, drive failure, directory corruption, whatever, kiss your data goodbye (so back it up!). No RAID 3, 5, or 10 support via hardware or software.

If you are an indie wanting a low cost solution to do high speed uncompressed HD work, one of two of these (external) cards with an external enclosure looks like an excellent solution - fast, inexpensive, easy to deal with, provided you have a backup plan in place for your data. Although this has been the most reliable of the SATA cards I've tested to date (but I want more experience with the shipping unit), I still would like to see the industry provide a low cost fault tolerant SATA storage solution for Macs.

-mike

UPDATED AGAIN 12/22/04- SEE BOTTOM OF REPORT

Sonnet Technologies today announced their own 8 port Serial ATA card for Macs, the Tempo-X SATA 4+4. It requires no drivers whatsoever, and has four internal and four external ports.

Remember that article I wrote the other week about what was wrong with every Serial ATA card on the market? This isn't one of those cards.

I've been testing a pre-release version of this card recently and found it to be an improvement over the Highpoint RocketRAID 1820a card in the following ways:

1.) half the ports are external (too bad not all of'em)
2.) drives don't mysteriously go offline or fail to show at bootup
3.) performance is higher
4.) requires no drivers - truly plug 'n play

Robert over at Bare Feats also tested a pre-release version of the card, and he has a nice review of it here with pictures and graphs and a nice comparison chart of the available SATA cards available for Mac. It's good enough I'm not going to bother trying to recreate something that says the same stuff, just go look at it over there.

So what does this card do? It has 8 ports, 4 external (on the outside of the PCI slot cover) and 4 internal (that you can route calbes out of an empty PCI slot cover, even if you're using a "fat" video card like the NVidia 6800 series or ATI Radeon 9800 Pro 256 MB version).

It requires zero drivers - plug it in and go. Hook up to 8 drives to it using an external enclosure such as the Burly Box from MacGurus and away you go. While there are products that allow for internal SATA drives, I'm not a fan of them, they don't work for my intended workflows, so I ignore'em. If you want to go internal, check out the solutions from Wiebetech (G5 Jam) and, umm, somebody else I can't think of.

Performance is impressive - not only is the unit more stable (so far) than the 1820a card, it's also faster. Barefeats has posted some impressive numbers, but I got even better - I put the Sonnet card in slot 4 of my dual 2.0 GHz G5, and the HD card (a BlackMagic DeckLink HD Pro Dual Link, the 4:4:4 model) and used the BlackMagic Disk Speed Test utility to measure performance. This utility gives a pretty good approximation of the real demands of capturing video, which is my primary concern, so I use that for benchmarking.

The way I tests disks, I partition the array into 5 equal sections (444 GB for the 8 300 GB disks I have) and then cut the 5th one short to make a 6th partition of the very last disk. This lets me see how performance would degrade as the array got full (as is the case with virtually all RAID setups). Drives are fastest when they are empty, and get slower as they get full of data. I run the BMD Disk Speed Test utility a number of times and use the average of several runs.

Maximum performance comes from the first part of the array, when all the disks are reading/writing to the outermost tracks.

Using BlackMagic's Disk Speed Test utility, I measured read speeds of 463 MB/sec on my 8 drive array using Maxtor Maxline III 300 GB drives, and 506 MB/sec write speeds.

Damn!

As the array gets full, it slows down, but is still very fast. Here's the full test suite:

Partition 1: 462.8 reads, 506.2 writes
Partition 2: 440.4 reads, 469.2 writes
Partition 3: 411.3 reads, 440.2 writes
Partition 4: 357.7 reads, 383.5 writes
Partition 5: 328.0 reads, 351.9 writes
Partition 6 (Last 15 GB): 252.9 reads, 272.2 writes

As you can see, this disk array should be sufficent for ANY uncompressed HD needs, up to and including 10 bit, 4:4:4 RGB 1920x1080 at 60 fields per second (which is a 238.8 MB/sec video stream). Even with 20% safety overhead, that would call for 290 MB/sec. An 8 drive RAID 0 should provide that up to about 90% full capacity.

BUT...it's still RAID 0. And that's one of my few quibbles with this card. If any one of those 8 drives go out, all 2.4 TB of my data would be GONE.

What's the fix? Or at leats the low cost fix? RAID 10. RAID 10 is cheap and easy to implement, but has not been implemented on this card...yet.

RAID 10 is dumb but simple - you would have a 4 drive RAID 0, and then mirror it with...another 4 drive RAID 0. This cuts both throughput and capacity in half, but if it's done in software, is a MUCH cheaper solution than the RAID 3 and RAID 5 solutions on the market. If you have a failure, the other half of the mirror is there for recovery/restoration. If set up properly, it just gives you a warning and lets you keep working until you repair the damaged half and resynchronize.

RAID 3 and RAID 5 are more elegant...but tend to be more expensive per GB, and have lower maximum speeds (but stay faster longer than RAID 10).

As an aside, I've been testing Apple's mighty $13,000, 14x400 GB drive, 5.6TB X-Serve RAID, and while I'm not done tweaking on it, have yet to break the 200 MB/sec barrier for write speeds in a RAID 50 configuration. I can get 2 1/2 times that performance with a $2500 RAID 0...but in an unprotected, non-fault tolerant setup. But as with ANY RAID 0 from any vendor, that's kinda like saying my car goes 400 miles an hour...but has bad brakes, no seat belts and a giant metal spike on the steering wheel pointed at my chest. RAID 0 is vulnerable to catastrophic failure at any time a drive goes bad. But don't take those X-Serve RAID numbers as gospel - I need to have a serious geek to geek conversation with their tech guys before I sign off on that's all it will do.

Anyway, back to RAID 10 - it can be done in software (nobody has for Mac yet) or with a software/firmware combo, which is how Highpoint has implemented it with their RocketRAID 1820a card...although poorly (I could never get mine to work right, others apparently have though). I've been trying to pressure SoftRAID to offer RAID 10 in a future version, but apparently unsuccessfully. If you'd like to see SoftRAID implement software RAID 10, let them know by emailing them at sales@softraid.com.

Other things I'd like in this card - all external ports. Did I say that? I'll say it again anyway.

OK, bitching aside, this card is the most promising thing on the market, and at a very attactive price point. If you are an indie trying to do an uncompressed online with your HD project, this is the best shipping low cost solution I know of, IF YOU HAVE A BACKUP of all your data, such as with a FireWire drive system. If you are a facility, get an X-Serve RAID (or similar) instead.

(As a side note, the Highpoint RocketRAI