HD For Indies

This guy's blog has an entry about Final Cut Pro 4.0.2 (not the latest version) and it's support of After Effects Plug Ins. Final Cut Pro HD supports SOME After Effects plugins, but NOT all After Effects Plugins. Which ones work? This guy takes a look.

It gets pretty detailed and geeky if you have a specific question about a specific plugin.

-mike

Pixel Corps Online Beta is going online -they are going to offer at least 45 hours of training materials online this summer for only $75.

If you are looking for a good place and way to pick up your digital post production skills, this is a nice option. These folks have put out a lot of the dvGarage content and plugins over the years.

Worth checking out. From their website, kinds of stuff you'll learn:

Character Design
Photoshop Skills
Video/Film Compositing
3D Modeling, Animation, Rendering
Photogrammetry
HDRi Creation
Character Animation (including Motion Capture manipulation).
3D Matchmoving

You will also learn:
Production Pipeline Integration
Basic Unix skills
Effective Naming and File Structures

In an email conversation with Noah Kadner (friend of the site and demo artist for Panasonic at NAB this year), he mentioned that the Panasonic 1200A has just started shipping in the last week or so. The base retail price is $21,000, the FireWire interface is $4000, and the HD-SDI i/o board and SD output board (HD & SD on same board) is $6,000. So $30K with all the ins and outs you might want.

This is the deck that lets you capture the native Varicam footage into your Final Cut Pro HD system via standard FireWire 400. So even a 1 GHz PowerBook G4 can capture and edit HD. If all you are doing is straight cuts, it is EXACTLY as good as the $65,000 Varicam shot it.

I've had a couple of friends of the site looking for these - my guy in NYC said he couldn't find any anywhere...so this explains it. They only announced the decks at NAB this year.

-mike

UPDATE Sunday 5/30/04: This is from a very knowledgeable reader Rui del-Negro in reference to SMART drives:

"S.M.A.R.T. won't warn you about any errors before they happen. As long as you run an error check on your drives regularly (i.e. once every couple of weeks), you'll spot the problems anyway, usually before S.M.A.R.T. does. Windows has had support for S.M.A.R.T. for 8 years or so, and I've only seen it give a warning once (and it was for a drive that I already knew was dying).

I had another 5 drives fail (three of them gradually, two catastrophically) without S.M.A.R.T. noticing anything.

If you want data about drive speeds or reliability, go to http://storagereview.com/. "


UPDATE: Here is a nice article on it.

There is a technology available for ATA and SATA drives called SMART - short for Self Monitoring Analysis and Reporting Technology- available on many vendors' hard drives (such as IBM, Seagate, etc.) that allows the drive to pre-emptively tell when it's failing (from some but not all types of problems). I've heard of this technology for awhile but didn't know how to access it. Somewhere around OS 10.3.2 or 10.3.3, Apple included a line in their Disk Utility to tell SMART status - so now it will say that is either OK or failing. If failing, RIGHT NOW is the time to back up your data and never trust that drive again. I recently had this happen on the boot up drive on my old G4/867. I'd been using it as my primary email box for a couple of years, and it had 4 years worth of email, contacts, schedules, etc. on it, on top of the lengthy chunk of time it takes to get all my applications installed, configured, plug-ins installed, custom settings loaded, etc. etc. etc.

I was able to use Carbon Copy Cloner to back up the drive to a disk image on another drive in the machine, swap out the old drive for a new one, then disk image the old bootup disk's content onto the new drive (while booting from a FireWire drive).

So this saved all the data from being lost if/when the drive failed in the near future.

I am going to try to not buy any drives that don't have this feature in the future - and you shouldn't either.

I need to double check, but if Disk Utility lets you check SMART status on SATA drives in an array, it's a good way to monitor the array's health, and potentially avoid data loss.

Same thing (hopefully) applies to FireWire drives as well. I saw a link on a website to a tool to check SMART status of an ATA drive mounted in a particular FireWire drive vendor's enclosure. Since these manufacturers typically are using the same bridgeboards, it might work on other vendors' enclosures as well.

But definitely something to put on your specs list when shopping from now on.

-mike

So I've been continuing to think about online/nearline/offline storage for uncompressed and compressed HD. Latest thought: You can only capture so much footage in a day. Let's say you've just been shooting the day before, or a client comes in with tapes. If you can manage to log and capture 5 hours of footage, I'd be agog. If you can log and capture 2 hours of good takes, I'd consider that a very reasonable day's work.

Backing up to FireWire 800 at 35 MB/sec, it takes about 8 1/2 hours to back up a terabyte. If you have 2 hours of uncompressed HD work (which you wouldn't anyway, you'd be capturing offline unless this was your one and only chance to capture uncompressed), that would only be about 6 or 7 hours to back up 800 GB of 1920x1080 @24fps and 10 bits per channel (30 bit color). That's entirely doable overnight, even to a single FireWire drive such as the La Cie Bigger Disk which has a 1 TB capacity. And actually, it would probably only take 5 or 6 hours on FW800 with that drive.

Once you've done capturing, there's very little that you're creating that needs tremendous amounts of backup space. Think about it - the only stuff you need to worry about that is BIG is video that you can't easily and readily recapture (such as if you only rented the deck for a limited amount of time). All the other stuff - your bins, your Final Cut Pro project, any Photoshop, Illustrator, After Effects files (except for big renders) can easily be backed up onto a CD. If you had a lot of stuff, maybe a DVD. But still cake to do daily backups, so that if your system melted down, you wouldn't be screwed.

But for the big stuff - captured video that you either can't, or can't afford, or don't want to recapture, you're doing all your capturing up front at the beginning of your process in the first few hours/days of your editorial process (depending on how big your project is). And unless you're being a dork and digitizing every last little bit of footage you shot AND not logging it as you go, an hour or three of selects logged and captured would make for a very full day of work. Back it up overnight. Can't afford that La Cie drive I just mentioned? On a total shoestring budget? Back up with Retrospect or somesuch (or just some syncronizing software) to a FireWire 400 or 800 drive or drives. 300 GB drives are already out there, and 400 GB drives are RSN (Real Soon Now). My sources say the distributors have them on order but they haven't shown up yet. But days/weeks, not months.

So the workflow would be something like this: tapes come in, log and capture either uncompressed or compressed depending on your storage capacity and deck availability, and last thing before you go would be to get the backup rolling. If it all fits on one disk, just drag it over and leave...

And keep backing up at the end of the day after each capture session if you want to be sure to not lose that day's work in the vaguely likely, gonna happen once or twice a year, chance that a drive in the array choses to die that day. But you'll have backups, so you'll be OK.

Along those lines, having one extra drive sitting around unused that can be used to replace the bad drive isn't a bad idea at all. In big fancy nice arrays, this capability is built in. But I'm betting a lot of you folks reading this don't know that.

Then, after your capture/logging phase is done, or on top of that if you're editing footage from a feature as it comes in, you just use a simple CD backup for your Final Cut Pro project files, emails, budgets, etc. for the rest of the stuff you're doing. Or a DVD if you have a lot of Photoshop artwork, that kind of thing.

See? Easy.

and cheap.

-mike

HDCAM isn't the only format that downsamples before going to tape (1920 to 1440 horizontal resolution), but also DVCPRO HD downsamples it's 1280 res - it isn't 1280 horizontally, it gets downsampled to 960. Ick.

-mike

Third Update: Nah, I'm full of crap. Now I'm reading somewhere else that Sony F900 and F950 actually do image at 1920 pixels across. I'm going to shut up until I find out the truth.

Second Update - Monday 5/31/04Talking to a friend today, he said that the native resolution of the imaging chip in the Sony HD cameras was only 1440 pixels horizontally - so this makes sense why that is the detail they lay down to tape.

Update: another site said it's 1440x1080 - they just squish horizontally. More research to be done...I think the 810 may have referred to the number of pixels used for 2:35:1 aspect ratio...I need to read more. This isn't golden info. I could check, you know, the manufacturer's specs on this.

Forgot to mention this anywhere yet - in the 1080p24 res with HDCAM, the image is filtered down to 1440x810 (I'm sure on the 1440, not sure on the 810) before being written to tape. So it's not a "true" 1920 resolution.

That's one reason why the Thomson Viper and Dalsa Origin look so good - they actually record 1920x1080 pixels to hard drive...

While at the DFRG meeting talking about AJA's Kona2 board and it's ability to do a realtime transcode from HD-SDI to DVCPRO HD, DFRG member Jose Mata made a very clever leap - if you're shooting with Varicam, which only has HD-SDI and no FireWire (and won't anytime soon, like a year or two), rather than renting the Panasonic $25000 deck for $250/day or more, why not just go directly from the HD-SDI tap on the camera to the HD-SDI on the Kona2, and use it's realtime transcode to DVCPRO HD, and NOT have to rent the deck?

This sounded extremely smart, and I've been thinking about it since he said it.

Now, this is transcoding from decompressed DVCPRO HD in an uncompressed data stream (but from compressed data, don't forget) BACK to that same compressed format. This would be perfect, if and ONLY if the encoding algorithm used by Apple is EXACTLY the same as that done by the deck, AND that decompression/recompression results in an identical file. I dunno. Maybe it does, maybe it doesn't. Worthy of testing. While at first this seemed a brilliant shortcut, it isn't necessarily generating the same results as the FireWire capture methodology. If you're only doing this for offline editing purposes, you're golden - the timecode matches, and if you're going to go back and recapture "uncompressed" later (or straight DVCPRO HD) you're all set, because you can trust you'll replace, frame for frame accurately, with the best possible footage from the camera.

But if you want to use it for final quality stuff, maybe it's the same, maybe it's not...and if it's not, is it good enough for your purposes? I dunno.

Jose pointed out that Robert Rodriguez hooked up disk arrays to the HD-SDI tap/s from his camera while shooting some of his HD projects (I'd imagine for the greenscreen work in Spy Kids 2 and/or 3), and did NOT record to tape, thus bypassing the compression the deck does before the zeroes and ones go on the tape (so they'll fit).

This is a good idea. I'd feel perfectly safe using it to capture for non-final editing, but I'd want to test before I trusted this to be designated "good enough" for final quality output to film. I'd want to go back and recapture unless I was SURE that it was the same or so close that _I_ didn't care. And I'm kinda picky.

But still a great way to save $250 plus per day on an indie shoot....

Cheers, Jose, this one's for you...

-mike (glug)

UPDATE 6/1/04: Over the weekend, my old boss from frogdesign came by to fool around with FCP HD and was nice enough to bring his Cinema 23HD with him. Hooked it up to FCP HD, rebooted to finally get everything talking nice, and ran the demo DVCPRO HD footage I picked up at the Apple booth at NAB. The difference between my Apple LCD and his is that mine is 1024x768 (older 17") and his is a 23HD with 1920x1200 res. So we were able to preview every single pixel in the 1080i60 footage. It has some artifacting when previewing interlaced material since it is an inherently progressive display, albeit at 60Hz. Now, I had a BlackMagic Decklink card installed, with it's desktop preview running, so that may be the reason for the following - we saw white dots streaking across the display when viewing fullscreen on the 23HD. We saw these same dots on the non-fullscreen video preview window as well. Not elsewhere. Very wierd. I did NOT see these on my smaller display. Seemed like a hardware problem - other QuickTime movies didn't display this behavior, we only saw it in FCP HD with DVCPRO HD footage (granted, we didn't check any other footage). Single pixels at full white intensity, looked like some kind of scan update problem. I'll play with this some more. I've been too lazy to check out Apple's support boards to see if anyone else is mentioning this, and again it may be due to the presence of the DeckLink card, dunno. Just wanted to point that out. Anybody else seen/heard/read about this?

Somebody tonight at DFRG was asking about why buy the $1300 HD-SDI to DVI-D HDLink device (or the $8000 eCinema similar device), when there is already the Desktop Preview fullscreen function in FCP HD.

A couple of reasons:

1.) If FCP HD isn't set to Video Quality High, you're not seeing all the detail onscreen that's in the image. I'm not positively sure you're even seeing it then. I'll have to test to verify. On my laptop, it's also scaling the image down to fit my screen since my screen is only 1024x768, not big enough to show the 1280x720 resolution of my test footage. Nor 1920x1080 test footage. Only Apple Cinema Display 23HDs are that big.

2.) A question of color matching, CLUTs (Color Look Up Tables), and true output from HD-SDI where you KNOW you're seeing every pixel. How different are the two? I'll need to do some side by side matching. In a perfect setup, I've have 3 23HD monitors. That's $6K of test gear that I'm not exactly rushing out to buy, even though there is presently a $500 rebate if you buy one at the same time as a G5.

Testing to follow....as always.

-mike

codec testing:

I have a clip from a project I did last year, it was captured from HDCAM to BlackMagic's 4:2:2 lossless codec at the same resolution as the source footage - 1920x1080@24fps. Basically, short of HDCAM SR, this is the best quality you can shoot and (mildly) compress your HD, short of going to some sort of digital cinema camera.

I decided to try some tests:

In Cleaner, the industry standard high quality tool to convert resolutions, framerates, and codecs, I compressed and converted this 1080p24 lossless BlackMagic codec clip too 720p24 DVCPRO HD.

I'm late to the party - this is the first time I've actually sat down to mess with the codec.

It worked, and the footage looks quite good. The datarate seems amazingly low. While it's a lower framerate than DV (24 vs. 30), it's nearly three times the resolution, but it's still only about 150% of the datarate of DV. Wow.

Moving on:

There are 2 codecs listed in the standard QuickTime Settings dialog. One is Apple DVCPRO HD 720p, and the other is DVCPRO HD 1080i60. From the naming, it sounds like they intend you to ONLY use 1080i60 with the second codec.
So what if...

I decided to try to convert the file to 1080p24 using both of these codecs - the 720p and the 1080i.

The results were interesting - compared to the original, these DVCPRO HD files were SUBSTANTIALLY darker. I don't know what's up with that. And before you write in, I DEFINITELY didn't have any color/brightness adjustments going on in Cleaner. I triple checked.

What's that about? I don't know.

I need to read up on it, try to figure it out.

Closely scrutinizing the two files created at 1080 res:

The one from the 720p codec - looks pixelated and stretched, as if I were zoomed in to 200%....but I'm not, I'm looking at 100%. The datarate is only 5.4 MB/sec..which is the same data rate as the 1280x720p24 movie. So no joy there.

The one from the 1080i60 codec - it's 10.9 MB/sec, which is more appropriate for this larger file. It has some steppiness that could be field vs. frame processing. I'll need to squint at it some more and try some true test files made to show the problems.

I went back and compared my 720p file against one I generated out of Apple's own Compressor. It was NOT darker. So Cleaner is doing something funky to the files during transcoding.

Grabbing the same frame out of both 1080p24 conversions, taking them into Photoshop and staring closely at them revealed about what I would have guessed - the one from the 720p codec was smoother but less detailed, the one from the 1080i codec was more detailed but had more blocky, aliased, stair steppy artifacts, I'm guessing because it was expecting fields not frames.

I haven't tested these to see how well they'd play back in FCP HD (or even it it would balk at them), but I'd consider either of these viable to work with.

Apple provides realtime previews only for certain codecs - DV, DVCPRO HD, and uncompressed (well, sorta realtime - it's only render-free on your computer screen, NOT on your video preview monitor). If you want to have low data rate and realtime effects, DV or DVCPRO HD are your only choices. If you want to edit in 1080p24 without downsampling to 720p, you're kind stuck. The only codec left to use is BlackMagic's PhotoJPEG, since Apple's PhotoJPEG only wants to work at 720x480, no matter WHAT resolution you tell it to use in the Settings. And BlackMagic's PhotoJPEG doesn't allow RT Extreme functions. Oh, and that uncompressed realtime stuff? Only on 8-bit (per channel) footage, not 10-bit. So if working with BlackMagic, you can use PhotoJPEG at 1920x1080 for low data rate offline editing. Great! But without any realtime effects. Dammit. Go back and recapture your final selects when you've done your editing, and you can do realtime 8-bit color correction. Want to do 10-bit color correction, with 4 times the number of gradations of color in each channel? You can, but you get to render everything. No RT for you. (No soup for YOU!) So you're stuck - you can't get 1080p24 RT with the small compressed, or the highest quality uncompressed. Only the not-quite-so-good 8-bit files, which are still huge and require a RAID to watch play back.

So maybe these bastardized 1080p24 DVCPRO HD codecs can be made to work for offline? Maybe they'll work with RT Extreme? Maybe. I'm thinking it outta work, CPU horsepower permitting, but of course it may all just, pardon my french, shit the bed and totally not work. Tomorrow I can set up a test and see. Some of you may have already tried this. Feel free to gloat and tell me your results before I test this.

See how evil and treacherous HD can be? Yeah...it still needs a few years of faster CPUs, more mature software, bigger/cheaper/faster storage, better LCDs (faster pixel refresh, more saturation, more contrast) and we'll be there.

Hey all - right after the DFRG meeting tonight I went by Guerro's on South Congress to eat. While waiting for my Santa Fe enchilada (fried egg, verde sauce, chicken al carbon, whole black beans..yummm), I messed around with FCP HD with a 720p24 project on my laptop while running on batteries. I was using the demo file Apple gave out on DVD-ROM at NAB.

It played back, which was amazing enough, even though I'd seen it do it at NAB on a lesser machine.

With this 12" PowerBook G4, 1.33 GHz, 1.25 GB RAM, the 720p24 project Apple was handing out at NAB on DVD-ROM played without dropping any frames...AFTER I went into Energy Saver and changed the Options to run the processor at Highest, NOT automatic. This is the second machine I've seen Automatic underperform on - my dual G5 renders After Effects noticeably faster with Highest, NOT Automatic. Lesson learned.

Anyway, he's what I saw. Forgive me if you've all played with this by now, I was busy on other stuff.

The default Video Quality Medium setting in System Settings/Playback Control displays individual frames just fine when you aren't playing back video. But as soon as you play back video, you can instantly see the image get softer and fuzzier. Clearly (or fuzzily), you can see that Medium quality is sacrificing some image clarity in exchange for more CPU cycles to dedicate to RT Extreme performance. Sometimes desirable, sometimes not.

Changing the Video Quality to High kept the image quality the same for playing/not playing, so I presume that is the full detail of the file. Can't tell on my 12" screen, it's only 1024x768, and the footage is 1280x720. Along those lines, a 15" PowerBook WILL display all pixels of a 720p signal with it's 1280x854 (or whatever vertical res, it's enough) screen., I don't think I properly indicated that in my portable FCP editing writeup the other day.

I'll play with it on the G5 with the 24" monitor to see how 720p as well as 1080i files fare.

-mike

Hey all -

I'm going to redesign (or really, um, just design) this website starting in the near future.

Any website features, suggestions, organizational hierarchies, whatever, send it in.

I want a wiki, I want sections to organize this content, I want a page of useful links.

What do you folks want?

Let me know at mike@hdforindies.com

Last week, after seeing the release of the Mac MPEG Encoder from Main Concept, I hypothesized that Lumiere might want to look at using it as an encoder in a future version.

Interestingly enough, I received an email from Frederic Haubrich (the developer) telling me that in fact they had started a relationship with Main Concept starting at NAB, and will be releasing Lumiere version 1.1 utilizing Main Concept's MPEG-2 encoder in about a week (June 1st).

In my emails with Frederic, I asked him about using DVCPRO HD as a working codec with HDV footage. He said it works well and referred me to this link on their forums about working with HDV, and the pros and cons of DVCPRO HD vs. uncompressed as working/editing codecs.

Tonight at 7pm I'll be speaking at the Austin Digital Filmmaking Resource Group's monthly meeting.

TUESDAY, May 25, 2004
Yarborough Public Library
2200 Hancock Drive (just west of Burnet Road, between 45th and North Loop)
7:00 to 8:30 p.m.

Free and open to the public.

I'll be speaking about what HD is, and what it isn't for digital filmmaking. I'll be talking about some of the technologies that have come out recently that assist in making movies with HD at high quality and low cost, such as HDV, DVCPRO HD, Final Cut Pro HD, low cost storage options for compressed and uncompressed, etc.

MicroNet is now offering a multi-drive FireWire 800 enclosure in capacitites from 400 to 800 GB. These are just larger than usual hard drive cases that hold multiple drives and show up on the desktop as a single large spanned volume. They aren't RAID speed, the two drives don't read and write at the same time for a given file, they are just joined so that when one gets full the files just start writing to the next drive and the user never has to worry about it.

("It's called a change-over. The movie keeps on going, and nobody in the audience has any clue." -more Fight Club quotes. I dunno why. Voices in my head? Am I waking up in the middle of the night as a coolio indie filmmaker? Will actors start showing up on my porch with $300 personal career funeral money? Am I in this movie?. Digression.)

In any case, these types of drives are good solutions for large FireWire storage needs. As this thread points out, FireWire isn't always so plug and play, every once in a while it's Plug & Oh Shit! When dealing with crucial data, a good bit of paranoia isn't such a bad idea. Fewer devices on the chain is a Good Idea. Less stuff, fewer cases, less cabling, less power, better better better.

I'd recommend solutions like this for 24p DVCPRO HD for portable or additional studio usage (after I had added a second internal SATA drive to my G5).

-mike (tyler?)

MacNN has this story about a new NAS from Iomega, former manufacturer of Zip and Jaz drives. Not the best name in the field, but an interesting option for low datarate (DV? DVCPRO HD at 720p24? Dunno, not tested, unknown specs) over network connections. The better unit has a high speed SCSI interface.

Synopsis: This is a long rambling article, so I'll be kind and summarize: my 12" PowerBook is less than ideal for field editing. A 15 or 17 inch PowerBook with FireWire 800 drive would be better, but only for DVCPRO HD or other FireWire based media, including standard def SDI, component, composite, or s-video. For uncompressed HD, a G5 with PCI-X card is mandatory. Details on all these options follow below.

The full on, long bloggy version:

UPDATE 6/1/04: Here's a theoretical approach to resolve the issue of how to have more storage when you're using the single FireWire port on a 12" PowerBook G4 - use the USB 2.0 port. Even though it is crippled in use on the Macs (read barefeat's article on G5/eMac/PowerBook USB 2.0 performance), it will still do about 18 MB/sec. The drives are capable of far more than that, it's the interface that is messing it up. I plan on doing a test with FireWire input to the single FireWire port, then capturing the daa to a drive on a USB 2.0 interface. DV is only 3.5 MB/sec so that should hopefully work OK, the question would be how much could be reliably done with the DVCPRO HD deck (maximum datarate of 14 MB/sec for 1080i60 footage). Would 720p24 work? 720p30? 1080i60 (which is the "biggest" at 14 MB/sec)? If the drive does a real world 18 MB/sec, that gives SOME overhead. Remains to be seen how it would really work, especially on the little PowerBook, which shares a lot of motherboard guts with the 12" iBook.

END OF UPDATE

ORIGINAL POST BELOW

So I've had my new 12" PowerBook G4 (the new 1.33 GHz model) over the weekend, and so far I like it. If you're interested, here is MacWorld's review on it. The only thing that dawned on me was that while it is capable of editing HD footage with FCP HD, it's not the best machine for it.

The raw specs are there on the machine as I have it configured:

-1.33 GHz G4 (1.0 GHz required for editing HD with FCP HD)
-1.25 GB RAM (1.0 GB required to edit HD
-FireWire 400 port for hooking up to deck for DVCPRO HD capture
-60 GB internal hard drive
-DVI output, so can hook up a second monitor, including an Apple display for the Digital Cinema Desktop Preview feature (including the 23HD monitor, so I could display every pixel of a 1080p frame)

So this all sounds great, right?

Well....

I can run FCP HD. I can hook up one or more external FireWire drives to add hundreds of gigs of capacity to the system. BUT...if I want to capture from a deck, it wants to dominate the FireWire bus. It wants to be the only thing on that bus.

It has generally been reported in the past that you do not want to have a FireWire camera or deck on the same FireWire bus as a drive that is reading data that needs to go to the camera. The data actually goes from the drive, to the computer, then back out the cable to the camera, and this is too much data flowing around. Frames get dropped. I haven't heard this confirmed for DVCPRO HD, but I suspect it to be true.

So that means I'm capturing to the internal drive. And with the machine loaded up with all my usual production tools, NOT including the 12 gigs of LiveType and Soundtrack stuff, I have 36 GB of free disk space, which is about 1 3/4 hours of 720p24 DVCPRO HD footage. Actually, that's not too bad. If I'm shooting 30 frames per second, whether interlaced or progressive, I'd have about an hour and a half of footage space. Still not too bad. But if I need to capture more than that, I'd need to transfer it over to a FireWire drive. So I could envision a solution where I hook up the deck, capture to it, get the internal drive close to full, then disconnect the deck, hook up a FireWire drive, and copy the captured footage over to it. Dump the already copied footage off the internal drive, disconnect the FireWire drive, rehook up the deck, and resume capturing.

The clever among you realize I could have already had the FireWire drive hooked up but not in active use during capturing - this is true. So when editing and using footage off of that drive, you can edit with it, but now you're possibly in trouble if you're trying to preview by running the FireWire cable to the deck and the deck to your video monitor (not computer monitor). Again, I haven't heard this confirmed but I'll bet it's true.

So what's the fix?

I could have opted for an 80GB drive from Apple, which would have jumped me up to about 55 GB of free space on the drive before any project/work data was added, giving about another hour of 720p footage, or roughly 45 more minutes of 720p30. Apple charges $125 for the UPGRADE from a 60 to 80GB drive. Considering that a new 60 GB drive costs about $150 online, and an 80 GB about $175, Apple is charging quite the premium. My plan had been to sit tight with the 60 GB until I needed something bigger, then disk image it to a FireWire drive, purchase and swap out an 80 GB drive, image back to the internal using Carbon Copy Cloner, and buy an external bus powered case for my newly freed 60GB drive. Total cost: about $225. So rather than going Apple's route, instead I got a bus powered, shirt pocket sized 60GB external drive for about $100 more than Apple's "upgrade" that doesn't include keeping the second drive.

OK, that geeky diversion aside, what's the fix?

Get the 15 or 17" PowerBook. Besides a bigger screen, faster processor, faster graphics card, backlit keyboard, gigabit Ethernet, you get a second FireWire port, only this one is FireWire 800. And they start at about $400 more than the 12".

So then you get the best deal: You can capture DVCPRO HD over your FireWire 400 port, and capture directly to your FireWire 800 external drive. This way, you're not capturing to the same drive your application and OS is on, which has long been considered a Bad Idea, and invites dropped frames. Whether that is still true with these new faster machines I don't know, but the potential still exists.

So the ideal travelling edit kit would be:

15" or 17" (new 2004 model) PowerBook G4, 1.5 GHz processor. Pay extra to get the single 512 MB RAM stick in slot one (there are only 2 slots). Buy a third party 512 (or better yet, 1GB RAM stick (from crucial.com or other trusted vendor). If you have the space, don't mind the weight, and can live with the reduced battery life, get the 17" for it's bigger screen (1440x900). High res screens are incredibly useful for editing. And that size is big enough to show every pixel of a 720p signal.

Buy an external FireWire 800 drive. Speed isn't terribly crucial if working with DVCPRO HD (even 720p60 is only 14MB/sec). I have bought some La Cie d2 Hard Drive Extreme units lately and been satisfied so far. Or, possibly consider the larger but multi-disk, single mounted volume Big Disk for one terabyte that connects via a single FireWire 800 connection (they're $1200). That's about 50 hours of 720p24 DVCPRO HD footage.

Buy a second monitor - make it an LCD flat panel display. Multiple displays are INCREDIBLY useful in editing, especially to put bins on the smaller monitor. SERIOUSLY consider getting an Apple Display in 17, 20, or 23 inch size to use their Digital Cinema Desktop Preview feature. If you're doing a Varicam shoot and using DVCPRO HD, any of the Apple displays will do - even the $700 17" model does 1280x1024 res (but it's expected to be cancelled and replaced by some 16x9 aspect model soon, perhaps at WWDC in late June 2004). If doing a 1080 res project, only the Cinema 23HD display will show all the pixels at once (it's 1920x1200 res). So whether using it for preview or for bins (and there's a hotkey to toggle modes), it's damn useful.

If you're going to the trouble of toting a second monitor, might as well bring along a separate keyboard and mouse. The built in keyboard is so-so, but the trackpad blows for serious ongoing work (at least I think so). My favorite keyboard, if you aren't going to use the wireless Bluetooth model from Apple (or the wired version of the same), consider the Matias TactilePro. It's my current favorite, I just got mine a couple of weeks ago. SUPER clicky positive key action, much more akin to an IBM Selectric rather than the mushy action of Apple's standard keyboards. It's about $100, but I think worth it. It has a power button for the computer on it, which the modern Apple lacks. I'm guessing Steve Jobs wants you to leave it always on, and just use the switch on the computer itself, or the button on the Apple Display.

As far as mice go, I'm partial to the Logitech 4 button model here, it fits my hand nicely and I can configure depressing the scroll wheel to do This App Only Expose, and the 4th button to do All Apps Expose. Super handy for not having to reach over to the keyboard. Just enough buttons, not too many, fits my hand nicely. Of course, keyboards and especially mouse preference is a very personal thing.

But keep in mind, this is a road kit - even though you have tons of storage, 2 displays, deck interface, etc....it's still just a G4 laptop. An when it comes to doing editing, as soon as you do something that requires some serious CPU processing, the dual G5 just blows the doors off of these machines.

I plan on doing some side by side tests to compare in the near future do delineate the differences between my 3 Macs that aren't servers: a single processor G4/867 (old school, doesn't meet spec for HD usage in FCP HD), the king hoss dual 2.0 GHz G5, and this laptop I'm writing on, the 12" G4 PowerBook 1.33GHz.

Also, you're stuck to FireWire interface - you can do DV, DVCPRO, DVCPRO 50, DVCPRO HD, or perhaps even hook up to an AJA I/O device for uncompressed standard definition video. AJA I/O units can handle standard def SDI or analog component, composite, or S-Video. The do-it-all rackmount box is about $2000 I think (could be wrong, might be $2500). And since it is uncompressed standard def, you'd need it on it's own bus, and a FireWire800 array to capture to. Uncompressed 4:2:2 NTSC is about 21 MB/sec per stream. As long as I'm on the topic of FireWire devices, I should mention DV hardware codecs, little boxes that convert DV to analog and vice versa. The Sony DVMC-DA2 is my favorite (sorry, can't find a link). It is useful when you don't have a DV camera or deck to preview your editing work on a TV or NTSC monitor.

And hey - if you're dealing with a laptop here, who wants a rackmount box anyway?

So any kind of uncompressed HD is not an option with a laptop - you have to have a PCI-X card for that. While there are some PCI expansion chassis based on PC Card expansion slot technology, there aren't any that do PCI-X, and there won't be - PCI-X has throughput far in excess of what PC Card buses can handle.

And since DVCPRO HD doesn't have a 1080 progressive setup of any sort, and neither does HDV, there is no way (yet) to get a 1920x1080 movie at 24 progressive frames per second onto your laptop.

OK, so what about road worthy G5 solutions? There are companies that will roadcase your G5. A client of mine, High End Systems, has what they call a Catalyst Media Server that ships in a road case built by Calzone cases. It's based on a G5 with specific software, drive, and monitor capabilities. It essentially does 4 layers of realtime After Effects for lighting guys. If you're interested in road cases, email me and I'll send you the phone # of the Calzone rep in your area. The media servers High End builds are shipped all over with rock & roll bands by lighting guys, so they are QUITE robust and road/travel worthy.

If you're considering a "heavy" transportable install, a roadcased G5 is the way to go. Have a couple of LCD monitors, at least one of which is an Apple for the previewing functionality. Among the Apple stuff, I'd rather have one 23HD ($2000) than two 17 " ($700 each). I'd rather have 2 20" ($1300 each) than one 23HD. Unless I was doing 1080 res footage, then I'd need one 23HD regardless.

Then again, there's always this beast which I don't quite understand yet. It's a rackmountable G5 running OS X but in a custom case.

There are kits out there to do 4 drives in a 1RU standard 19" rack. I need to research a good SATA one. FireWire 800 is good enough for DVCPRO HD, but NOT for any kind of uncompressed HD. There are problems with the G5's FireWire 800 circuitry and support that prevent it from going as fast as it might, even with multiple buses used (by adding PCI cards with multiple FW800 slots). For full details, check out the articles on FireWire 800 over at barefeats.com. There are 3 or 4 articles dealing with the issue and documenting the crappy write speed on G5 FireWire800. You can play it back, but you can't capture it is what it boils down to.

In theory, the Apple 17" could be mounted within a standard 19" rack. It's less than 18 inches wide, but you'd have to take the back leg off. Marathon Computing makes a bunch of interesting rack and wall mount options for computers and monitors, but a retractable mount for an Apple 17" monitor isn't a product I saw at a quick glance.

As I've mentioned before, even though I haven't used it yet, I'm liking the idea of an Apple Cinema Display 23HD with Decklink's HDLink in conjunction with an HD-SDI capable card, such as those from AJA, BlackMagic (DeckLink), and Aurora. Too tired to dig out links for those right now, sorry. Search this web page and you'll probably find links. I haven't done side by side tests, but I like the DeckLink Pro and AJA Kona2 best so far. Yeah, I definitely need to do a serious specs comparison as well as real side by side testing with these units.

I'm still hoping to utilize SATA arrays for uncompressed HD work, although I haven't done sufficient testing as yet. There is purportedly a 4 external port PCI-X SATA card coming out this summer, but in the meantime, there is the Firmtek Seritek 1S2 card. It has 2 internal SATA connections that can be run outside the case to external SATA boxes like the one from PPA, Inc. (available at Fry's Electronics) or the one I mentioned a few days ago on the blog from Granite Digital.

If not, Apple's X-RAID is a viable option, but I never got a straight answer from anybody at NAB about what was the guaranteed baseline datarate I could count on from the thing if I wanted a RAID 3 or 5 (fast, but with protection from drive failure) setup. "What's the highest datarate I could count on in the worst case scenario with this thing, such as when it's nearly full?" was what I was trying to ask. I kept getting directed to the senior product managers whom I could never find. But I'm working on it still.

OK, that's enough for tonight. Anyone who wants to know more, feel free to email me at mike@hdforindies.com and ask questions, I'll answer them privately (by email) or publicly (on the site), whichever is OK by you.



-mike

There's an excellent article on HD monitoring and CRT vs. LCD for monitoring HD over at Digital Cinema.com, with emphasis on the new generation of devices that allow you to monitor HD on the Apple Cinema Display 23HD.

Mike's comments: This guy pretty much says it all. The only thing I could add would be to also mention Decklink's HDLink, a $1300 dollar similar device. I'd like to see a quality comparison of the two devices side by side. the eCinema device is $8000, the HDLink is $1300. Both purport to do the same general thing.

In any case, I HIGHLY suggest reading this to anyone that wants to do color critical work in HD...or even just do ANY kind of HD monitoring.

I'm growing increasingly convinced that Apple 23HD monitors are an excellent monitoring solution.

That article also references this one, which is a geeky technical discussion of zone plate signals. It also includes several high res images to compare CRTs to LCDs, and better yet, some images to let you see how good your own HD gear is.

Benjamin Palmer from the barbarian group in Boston wrote in response to my post on backup solutions.

He pointed out an excellent solution for long term backup systems - raw firewire docks - It is a small block with a FireWire interface on one side, with an IDE connector on the other. Wiebetech makes one of these FireWire docks for $120.

He said for their video projects, they'll buy two drives, and make a backup of the project on each drive. If you buy big enough drives to backup the entire project, so long as less than 300GB this works with a single drive per copy of all the project assets. They then have 2 backups, and store each one in separate facilities (office and home or somesuch). This is an excellent solution, and with 2 copies even covers you for facilities catastrophe.

For short form uncompressed HD work, or longer compressed work, this is a great archival solution - fast, capacious, cheap, online. Clearly the way to go.

Mike's Comments:

For long term solutions, this sounds extremely viable. However, for daily backups beyond the scale of a drive or two, this could be cumbersome.

For ongoing rotating backups of HD quantities of data, this is, I think, only quasi-viable. If you're only generating 250 to 700 GB every day or two, this could work. I'm a bit tenuous about recommending handling bare drives on a regular basis, as this is how they expect you to hook them up to the dock.

UPDATE Sunday morning:Now that I'm fully concious and reading the Wiebetech website, I can see that they offer a baseplate to protect the raw electronics on the bottom of the drive for only $12. This makes me feel a lot safer about using these things. So ignore the comments about handling bare drives, this makes it much more viable. Sorry Benjamin, didn't see those the first time. Also, check out Robert Morgan's writeup on it from 2 years ago on barefeats.com

For daily backups, whether in whole or incrementally, I wouldn't recommend this as a first line of defense against drive failure unless I didn't have any other options. I'd rather use FireWire drives, and even then it's a bulky, not elegant solution. Update: Nah, I partially rescind that. With baseplates, this isn't so bulky or awkward, now my only issue with it is the speed with which you can get stuff done, and the need to swap out drives.

Once again, the wisdom of RAID 3 or RAID 5 makes sense...I just loathe the expense.

When working with HD quantities of data that you don't want to, or can't afford to, store on redundant backups, and it is unfeasible or undesirable to recapture from tape, some kind of high speed backup is necessary.

These docking stations would work, but awkwardly - ideally the backups should be able to run unattended overnight without requiring human intervention (such as to load in a new drive). Even at 35 MB/sec, it still takes 8 1/2 hours to back up a terabyte of data. Thus the overnight factor. If you have to swap out drives 3 times, that just isn't feasible to do yourself (unless, of course, you have interns ; ) )

But, again, for long term project backups, FireWire docks sound great.

-mike

While researching that last post, I stumbled across another external SATA case, this one from Granite Digital. Appears to be of sturdier construction than the PPA, Inc. cases I've been using to date. Useful for building external SATA arrays, if/when a 4 external port PCI-X SATA card finally ships.

Check it out.

So I'm sitting here at 11:11pm shuffling tons of data around, nervous something is going to get lost.

5 years ago I bought an AIT+ drive for about $2000 that could back up 35GB natively per tape at a rate of about 3 MB/sec....then you get to wait for it to rewind and verify the data. VERY time consuming if you have a lot of data. Not very practical for uncompressed HD quantities of data.

Today, those drives go for about $850.

The media for those drives, however, hasn't dropped much in price. Poking around on the web today, I found individual 35 GB tapes went for around $40-$45 each. The best price I found for a 5 pack was $175, the next best price was $215ish.

So even if I ignore the cost of the drive itself (and I've backed up about 900 GB of data on tape), best case scenario the media costs $1/GB. And it is SLOOOOOOW. And you have to have storage to put the files onto once they come off of tape, because tape is an offline storage medium - you can't open or save directly to tape.

Looking at other, faster, more modern solutions, I see that SuperDLT320 drives, that store 160 GB natively (and might store as much as 320 GB compressed, but digital video doesn't compress much AT ALL). These drives write data at 960 MB/min (that might be the compressed datarate, therefore twice as fast as "real" transfer rate). So at best that's still about a gig a minute, so a terabyte would take 1000 minutes, or nearly 17 hours to back up, not including tape swapout time or verification time (so double it to verify data integrity). Ugh. Oh, and it costs $3500 for the drive and I don't know how much for the tapes.

Hmm.

OK, what if I want to back up to something online, fast, and easy to hook up?

I had previously thought about just backing up to ATA drives and shelving them. But G5s use SATA drives, and there are decent odds both slots would be full.

FireWire drives are viable, but the kits for the cases are a little pricey. But they would work fine.

I looked into a couple of FireWire hot swap docking stations, but they are $260-$300 for the base station, then $30-$50 for the dock. Not bad, but still not great. You get good transfer speeds (20-50 MB/sec, depending on FW400 or FW800), so a terabyte takes I'd guess about 9 hours to back up. Not bad.

Then I thought about the fact that I'm getting ready to put together an external RAID based on SATA technology. Hmm. Assuming there is an available SATA port (not always available), cheapie, good enough for backup but I wouldn't use everyday cases are $50. With a $200 250GB drive, that's, hey, whaddaya know, only $1/GB. But now it is literally about 10 times faster and online. But you have to have an available SATA port...not so convenient.

External FireWire cases run $80-$120 or so. So hmm, not quite so good a price bargain.

What to do?

I need to work out the price points at which it cost justifies to use a hot swap FireWire or SATA dock.

But it's 11:40 and I'm tired. Another day.

This is all slowly pulling me towards the wisdom of RAID 3 or 5, which I am loathe to do due to the costs involved. But if it is feasibly fast and costs about the same....maybe that's the way to go.

OK, this is just an incomplete starter kit, but is a place to get rolling from.

I downloaded the demo version of the Main Concept Mac MPEG-2 Encoder (see story below) and installed it.

If you go in and try to set the resolution to a 4 digit number, or set the datarate higher than 15000 bits/second, it won't let you. Harrumph.

The reason why is that you have to change a profile setting to allow this.

Here's what to do:

In the main window, there are two buttons that say Edit on the right side of the window in about the middle. Click on the lower of the two Edit buttons.

This calls up the MPEG options.

There are four tabs across the top - the first one is Basic, click on that.

The first pop-up is MPEG Profile.

Change that to MPEG-2

click on the button that says Advanced in the lower left of the screen.

This calls up the MPEG Output Settings screen.

There are two tabs at the top of the screen, Video Settings and Advanced Video Settings. Click on the Advanced Video Settings tab.

At the top left of the screen are two pop-up menus for Profile ID and Level ID.

Set the Profile ID to Main Profile, and the Level ID to High Profile. That's the magic step.

Now click OK and you can go back and set your MPEG-2 settings to 1280x720, 15 frame GOP, transport stream, etc. etc. etc.

When I get into my research further I'll post up some pre-made specs, but for now this should be enough to get anyone rolling who can look up the rest of the HDV spec.

I haven't had a chance to play with the Lumiere HDV product, but since it's encoder is based on open source stuff, perhaps the Main Concept offers better quality.

If there is a noticeable quality difference, I'd suggest Lumiere license the Main Concept encoder as their own, or at least offer a bundle with Main Concept for Mac, or offer a Lumiere Pro with the better encoder.

My $0.02, NOT having seen the quality of either software's output side by side.

-mike

Main Concept, a longtime player in the MPEG-2 encoder field, has released a Mac version of their MPEG-2 encoding software.

It's a standalone application with some of the features of Cleaner or Compressor, without their power or depth for pre-compression options, but for the MPEG-2 encoding process it offers a TON more compression settings, getting into the nitty gritty of GOP sizes, I,P, & B frames, and others things you've probably never heard of.

So why does this matter? Because HDV is an MPEG-2 transport stream with a GOP size of 15 and a frame size of 1280x720p or 1920x1080i @ 29.97 fps. If Main Concept can be used to compress to that codec (it'll do all the GOP and data rate stuff, can it do the large frame sizes, that is of substantial benefit for the Mac community.

UPDATE: Spoke to Mark at Main Concept, and he said that it WILL most emphatically do HDV compliant files. He didn't know off the top of his head if it had the presets in there or if you would need to make and save them on your own, but it would "do anything the MPEG-2 spec will allow, and even let you go beyond that if you want." ....as in set some non-viable, non-functional settings. But basically it isn't resolution limited, so that's great.

Now I'm REALLY looking forward to a side by side comparison between Lumiere and Heuris' HD Toolkit Pro, and see what's the best quality combo between those and possibly using the Main Concept encoder. We'll have to wait and see...

-mike

Slightly Off Topic: My experience with 2 of the new 12" PowerBooks, which, to keep this On Topic, CAN run Final Cut Pro HD.

If you're reading this for HD purposes, here's the executive summary: think carefully about what you buy where, and when to get an extended warranty. Repairs on modern computers are SHOCKINGLY expensive if out of warranty. Be picky, but nice about the quality of products you receive. Check them out thoroughly when you receive them before mucking around with them. Clearly, I need a blog entry on How To Buy A Mac.

Here's the Long, Ranty, Meandering Blog version:

UPDATE: Ahh, what a wired world we live in. So I get home from a long run and get online to check the FedEx progress of my replacement laptop. I get online, query the Internet, some computer probably in Atlanta, Georgia then comes back with the answer....it's sitting on my porch, not 8 feet from where I sit. I could have, of course, opened the blinds and looked...but I'm such a digital guy I check online first. Dork.

So on Saturday evening I ordered a 12" Apple Powerbook, one of the new ones with the 1.33 GHz processor, $1800. Plus an extra battery and the custom fitted Brenthaven backpack for it. Since I live in Texas and do video post production and film work, I qualify for tax exempt status for items used in direct production of said content.

I faxed in my form to Apple on Sunday, and have yet to receive confirmation that they will credit my credit card back for the sales tax amount, around $200.

I received my laptop today around noon, happily started unwrapping it, and noticed The Thing. Somewhere during assembly, somebody had bent part of the frame, a 1/5 inch strip of metal that runs just above the optical drive insertion slot on the side of the thing. The metal was bent over about a 2 inch distance, and just enough that it provided an edge to shave leg hair off if you flip it over wearing shorts (this was how I found the issue). So 15 minutes after FedEx left I'm on the phone to Apple saying This Won't Do. It took me an hour of convincing, but they were kind and gracious enough to send one out the same day once I had the first one on it's way back to them via FedEx. I should get my new one on Monday, we'll see how it does. There was another anomaly, however. I ordered this directly from the Apple website on Saturday, May 15th. When I called in to discuss the issue, they showed that it had been bought via some other channel on April 21st. We read the serial number to each other back and forth a few times to be sure we both had the same number. Strange. This could have affected my warranty status, so be sure to keep and print all confirming details of all your major purchases when done online.

Later in the day, I had errands to run. My friend Sondra wanted to get a new laptop (she's a 30-something going back to school) for school usage, so I said let's go out to Fry's and look at the choices of iBooks and PowerBooks. I also needed to return a hard drive - I had purchased a Western Digital WD2500JB hard drive (my favorite fast semi-cheap ATA drive) but I didn't quite have the right one - I had the 1 year warranty model, and I'd found the 3 year warranty model online at zipzoomfly.com for $10 less with free 2nd day shipping.

UPDATE: I should have bought an IBM Hitachi Deskstar 7K250 250GB drive, they are faster, and cost less at zipzoomfly.com ($183). Drat.

So we get out there and lo and behold, they have the 1.33GHz 12" Powerbooks, 5 in stock. And their extended warranty is better and costs less than Apple's (plus they give you a loaner if the repair will take more than 24 hours). AND they give you more free semi-free stuff (free after rebate, but rebates are all assigned to crappy sub-contractors these days - BEWARE REBATES! Another post in and of itself). She got a $170 UPS with Mac software, a $20 5-in-1 USB 2.0 card reader (for camera cards etc.), and a $50 printer. All of which came with full purchase price refunds, so all she paid for was tax on them.

So I'm seriously doubting my wisdom of buying a stock model computer directly from Apple. I've heard that there are some support circumstances where you'll get replacements rather than repairs if and only if you buy direct from Apple, but with several hundred dollars in price difference...I'd buy from Fry's from now on if I were buying a stock configuration item...assuming all the refunds come in as they should (be sure to do every last little thing they want), and Fry's has the item in stock...seems like a better deal. Apple's AppleCare extended warranty is $350 for a laptop...20% of the purchase price. Ouch. But repairs are hideously expensive on modern computers, especially laptops. When the "n" key on my Mom's blue iBook (the first generation colored toilet lid looking ones) died, it was $400 for a new keyboard...you have to buy the whole thing. I doubt if the machine was worth a whole lot more than that when we repaired it. BUT...for some items, the only way to get those configs is a custom order. For instance, another friend bought a 15" PowerBook. We custom ordered to get a single 512 MB RAM stick in it (to leave one free for upgrading later, since Apple charges way too much for RAM & hard drives)., the backlit keyboard and SuperDrive. I always buy from Apple the LEAST amount of RAM and hard drive I can and update those third party. I know what I'm doing and what I'm shopping for and am comfortable doing this, not everyone will be.

But then we got home and I start helping Sondra port her Vaio to the PowerBook and I notice the screen has a dead pixel. So she's taking it back right now to get an exchange.

If my Monday unit has a dead pixel, I'm going to just do a straightforward return and get one at Fry's and open it in the parking lot to make sure everything is OK.

So, the lessons learned are:

1.) Don't assume Apple Store is the best option. I like it because supposedly you can get better service...remains to be seen.

2.) Some good combo deals are out there, if the item is truly in stock (I've had many, many misleading conversations about "we're getting 23 in Tuesday...")

3.) Federal law states you have 30 days to return something. When you get a new machine, screen, hard drive, whatever, immediately put it through it's paces. When I get a new hard drive, I format it with the Options set to Write Zeroes. This forces it to write to every bit on the drive, and if there are any bad sectors they get "locked out" so they'll never be used again. If it fails this test, or formats to a drastically lower capacity than it should then I return it (keeping in mind all drives format to something like 7% less than stated capacity - computer industry math at work). If it's a new monitor, I check it for CRT aberations or if an LCD for dead pixels (there are software packages to help you identify these) and return them if they don't work. As a first line of attack, try to get them to cross ship you a replacement, if they say X # of dead pixels is within spec, escalate to their superior. If being nice doesn't work with whomever you escalate the case to, then return it with all the little doodads so they won't charge you for them being missing. Then buy another one.

4.) If you have trouble with a brand new product, STAY NICE, don't get huffy. I got myself all in a twist while I waited on hold thinking of things I could hold over their head if they didn't give me what I wanted, but in the end, I just stayed calm and polite and tried to sound reasonable. What really got things moving along was "I realize you can only do what you're allowed to do, but I just paid $2000 for a new laptop. You are offering to have it repaired...I paid for a new one, not a fixed one, and I think what I'm asking for is reasonable and fair." Now, if it had some tiny cosmetic flaw, I could see them saying "Look pal, that's how they all are." So be reasonable. But if it is functionally impaired, keep after them...but nicely.

5.) Apple designs products beautifully, but like most consumer electronics, they are sometimes assembled overseas by folks just a notch above loincloths on dirt floors. Quality control issues can happen. It is a strange world we live in - consumer electronics are (hopefully) painstakingly designed in the states by fairly well paid engineers, but aren't assembled by them. So weird stuff can happen. A friend of a friend works for a large computer company as a senior engineer and went to the overseas Asian production facility for final fit and finish check on a laptop. The guy who was hand assembling the unit for the overseas contractor was LITERALLY a guy in a loincloth sitting on a dirt floor, trying out diferently finished tiny screws that were visible on the exterior of the assembled product. When they needed a different screw or whatever, they woke somebody up across town at 3am to bring some over so the American engineer could sign off on them. Very unreal, very much Our Too Modern World. Also, keep in mind that just as it's not wise to buy a new car in the first year until all the QA stuff is worked out, I wouldn't rush to buy the first version of a new motherboard design from Apple (or any other computer manufacturer). Most computer companies are more concerned with getting the latest tech out the door rather than delivering perfectly functional units. Companies knowingly ship slightly flawed (sometimes not so slightly flawed) products all the time on the assumption that people want it sooner rather than perfect but later. When Apple (or anyone) starts a production run of a new computer model (not just a speed bump of the last version), the first few hundred or thousand off the line are almost always a little funky, a little out of spec, and have to be hand tweaked back into spec...which can make them twitchy and unreliable, and unless it's an obvious anomaly, devilishly tricky to hunt down the core difficulty. So while I braved into buying the first SuperDrive Macs, and the first dual G5s, I've been lucky and they've worked OK. Probably because even though I ordered on Day Zero (day Steve Jobs announced them), it took me MONTHS in both cases to get them. And in both cases, the first quirky batches off the line went to someone else. I thought of this factor when I heard the university building the super cluster with G5s was getting all the first units, including, presumably, mine. Also keep in mind that Apple tends to have, ahem, an abundance of optimism as to ship dates when they announce products. OK, now I'm just bitching, so I better stop.

But I digress, significantly. That's why this is a blog, and it's free. : )

Which computer manufacturer was it with Loin Cloth Man?

Just like in Fight Club:

"Which car company do you work for?"

"A major one."

-mike

OK, this is sliding into personal blog mode, but there is a valid HD & business thing going on here too.

I'm in the middle of reorganizing my studio, shuffling drives between machines etc. In doing so, I have a few hundred gigs of data that need to be stashed somewhere while I'm reformatting drives, moving data, etc. Think of it like this: I have 10 gallons of colored water and a variety of buckets that add up to 11 gallons of water holding capacity. I have to carefully pour stuff around to not mix different colors.

But while I'm doing this, I'm realizing that I have over 200 gigs of unique, not backed up data sitting on my kludged together 640 GB 4 drive SATA array (4 standard G5 issue Barracuda drives). THIS MAKES ME NERVOUS. While I've had this set up for awhile, I haven't done any serious work with it, and haven't had to, you know, TRUST it with important data. I'd be VERY concerned about having this kind of quantity of important unique non-replicable data (tons of notes for the book, all my HDForIndies.com email, test results, test footage I've created, etc.) sitting in a vulnerable location like it is. If you haven't read my earlier posts, the nature of a RAID Level 0 is this: if any one drive fails, the array fails, and your data is gone (unless you go through very expensive recovery services). And, since it is 4 drives instead of just one, the odds of failure for the whole thing are four times higher than for a single drive to fail.

IF THIS WERE CAPTURE FOOTAGE I wouldn't be so concerned - (especially if I had the capture deck handy) worst case I'd have to recapture it. But this is EXACTLY the kind of situation I wouldn't want to have in a production environment - unique data that I can't readily regenerate or recapture without substantial work. But since it is the contents of two startup drives and a laptop worth of research, it makes me uneasy.

This is why RAID Level 3 or 5 would make life better - if any one drive goes down, no sweat, all data can be perfectly and readily recovered. But RAID Level 0, being the cheapest and the fastest, doesn't have that benefit. However, while RAID Level 0 can be done on the cheap with just drives, system software and maybe some drive cases, RAID 3 or 5 requires a controller, and the cost per gigabyte goes up hugely - by a factor of 3 to 5 or more.

So in the future, I plan on only keeping replicable data on the arrays. I might stash renders, such as from Cleaner or After Effects on the array, but not the project files, or scans, or retouched Photoshop work, etc.

Get the idea? Your data is your time=your work=a chunk of your life. It is worth protecting a chunk of your life.

For more on this topic, read my discourse on "Backup strategies & risk analysis for HD quantities of data," it's in the 04/01/2004 - 04/30/2004 section of the Archives link at the top right of this web page.

Once you click on that link, do a search in your browser for the phrase

Backup strategies & risk analysis for HD quantities of data

and it'll take you right to it.

This is slightly off core topic but of use for heavy Mac users. I've been reading on macintouch.com about finicky FireWire performance, especially when multiple devices are on a FW bus, and especially when there is an iSight involved. I read somewhere on there that a powered FireWire hub was the answer, perhaps because the iSight was drawing overly heavily from the power provided by the FireWire bus. So I bought one tonight, a SIIG 6 port powered FireWire hub. It ain't pretty.

But I'm going to see how it all works together, now that I have a wierd panopoly of USB and FireWire peripherals. At present, I'm booting off a FireWire 400 drive (I'm running a 4 SATA array), with a 17" Apple LCD monitor running off a Dr. Bott's ADC to DVI converter, with multiple USB hardware keys (dongles), an IR remote (Keyspan), a camera card reader, a USB cradle for my voice recorder, an iSight, and an iPod mini. Looooooooots of peripherals in a funky setup. So I'll see how this goes and keep everyone posted. The good thing about the SIIG is that it's cheap, got if for $30 at Fry's. The only other one they had looked slightly better but cost $65.

More as I get it all going. If I don't post tomorrow, that's a bad sign

Hey all -

a kind reader pointed out that there were ads in my RSS feed the same day that I noticed it last week. They are identified by ADV: at the beginning of the description. Since I'm lame and just post this via Blogger, they only provide an Atom feed, not the handy standard RSS feed. My buddy set up a script to hit 2rss.com (I shant dignify them with a link) to use their free Atom-RSS converter.

As of last week they started inserting ads into the RSS feed. Not into my site, just the feed. Slightly clever, but clearly annoying.

My buddy that helps me with the web stuff has posted about this issue and when he returns from biz travel will be fixing it.

So the ads will continue this week, but we shall squash the annoying vermin this weekend when we get our own Atom=>RSS converter up and running on our own secure server.

In the meantime, thanks for your patience...and don't click on those ads. Let's not encourage them.

This summer the Digital Media Academy (DMA), a nationally recognized, Stanford University accredited... [MacNN | The Macintosh News Network]

Mike's Comments: I think these are the same people who were talking about having a class in Austin that I met at NAB last month...

The 1394 trade association has announced their plans to develop a wireless FireWire protocol.

Read all about it.

This could be fun/useful/interesting. Imagine shooting HD (either HDV or DVCPRO HD) and recording wirelessly to a drive or computer 30 to 100 feet way...and of course all kinds of useful inhome and in studio applications as well. Products will start showing up perhaps as early as beginning of 2005.

Small Tree Communications released a 10 Gigabit Ethernet adapter card for Mac OS X using the Intel 82597EX 10GbE controller. [MacInTouch]

Mike's Comments potentially useful in a SAN or mega-high speed networking environment - it's theoretically capable of the same speed as 25 FireWire connections. Obviously, you'd need some kind of a massive array to feed it this fast, but it's nice to know it's out there. Theoretically, you could capture raw Dalsa Origin output with this. But it's a $4800 card. Ouch. You can buy a server for that.

Apple has started offering the AppleCare Professional Video SupportLine, which provides customers wi... [MacNN | The Macintosh News Network]

Mike's Comments: Gets you 12/7 support, $800 for one designated contact person. Interesting.

Here's the direct link.

FireWire Depot released a FireWire to Two UTP5 Repeater, which enables 1394b signals to be transmitted and received across up to 100 meters of unshielded twisted-pair category 5 (UTP5) cable. [MacInTouch]

Mike's comments: interesting. Might be useful in some shooting situations where you want to shoot and record direct to drive at some physical distance. Maybe.

Read this article if you want the skinny on setting up and using this new FCP HD feature.

This lets you run your video (even HD) full screen on your display.

From Apple's site, this support nugget

In prior versions of Final Cut Pro, it was possible to preview video playback on NTSC or PAL video monitors connected to a video port on a graphics card.

In Final Cut Pro HD, video preview is handled by the Digital Cinema Desktop feature which is more robust, supports full screen playback, and works with no latency.

However, Digital Cinema Desktop only works in this mode with AGP graphics cards. Video monitors connected to a PCI graphics card do not appear as an option for video playback. For an analog monitoring solution other than the supported third-party video capture cards, connect the monitor to a FireWire DV converter box and select the converter box in the Audio/Video Settings window.

Mike's Comments: the new feature is great, but I'm always a little suspicious when a feature is deleted in lieu of a new one. I don't see, from my limited vantage point, why this feature needed to be removed in order to get the new one. But it wouldn't surprise me that Apple wants to sell more Cinema Displays, so this feature usurped the old one.

...but the new feature is really, really nice.

I'm looking forward to doing a side by side test to compare and differentiate this feature from the HD-SDI to DVI-D converters made by DeckLink and one2one. I suspect the hardware solutions will look better and be more accurate...but how much more?

From a press release:

AJA Video Systems Announces Support of New
DVCPRO HD codec in Final Cut Pro HD
New Kona 2 dual link card enables real-time uncompressed editing in HD

Grass Valley, CA (April 30, 2004): AJA Video Systems Inc. announced that their new KONA 2 SD, HD and dual link HD capture card fully supports Apple’s new DVCPRO HD codec in Final Cut Pro HD, which was announced at NAB2004.

Kona 2 enables Apple’s new DVCPRO HD format to be used with any HD-SDI source including existing HD decks such as Panasonic D5, Sony HDCAM and HDCAM SR. The DVCPRO HD codec allows online quality at significantly smaller file sizes and disk requirements. The DVCPRO codec supports both the 720p and 1080i HD formats.
Developed in close cooperation with Apple, Kona 2 can capture and output DVCPRO HD media to HD-SDI. Additionally, Kona 2’s hardware performs part of the DVCPRO HD processing – freeing up the CPUs for more effects power when outputting to HD-SDI.
With HD post-production on the desktop, editors have amazing power at significantly reduced cost, making HD accessible to more post production facilities. This is an exciting combination for professional HD production.

-----------------------------end press release---------------

Mike's Comments: So now you can capture from whatever HD SDI source (like HDCAM or D-5) directly transcoding as you capture to DVCPRO HD. Use it either as a very high quality offline codec that ALSO has support for the realtime effects in FCP HD, or just use it as your final if you are satisfied with the quality and/or don't have a large, expensive disk array.

A very good workflow option.

Previously, cards such as the AJA Kona and DeckLink products offered an annoyingly tricky set of trade-offs: if you wanted realtime effects, you had to work with enormous uncompressed files. If you wanted smaller file sizes, or had limited storage, or didn't have an extremely high speed disk array, you could use some other codecs...but didn't get realtime effects.

Even then it was limited - if I understand this correctly, as of Final Cut Pro 4.1.1, both AJA and DeckLink could do realtime color correction....but ONLY in 8-bit uncompressed video. Not in the convenient for offline PhotoJPEG, and annoyingly, not in 10-bit uncompressed...which was what you might have wnated to work with to get best quality results, and was why you bought that huge expensive array.

blah.

So this is a BIG deal for workflow efficiency.

It's cheap. Not only is it cheap, three weeks ago the recommended solution wasn't cheap. The cost of an hour's worth of footage dropped about 30 fold. How does it do this?

It works with the native camera codec of the Panasonic Varicam, so if doing straight cuts, is literally exactly as good as uncompressed if you were shooting with the Varicam anyway - since it is the camera's native codec.

Yeah yeah yeah, as soon as I do some color correction, titling, effects, transitions, etc. those portions of the footage are recompressed.

But the core competency is there - so the cost of an hour of storage of 8-bit, 1280x720 progressive footage at 24 frames per second just dropped from about $1000 to $1300 per hour to about $30-40/hr.

Where'd I get those numbers? Looking at the cost of disk arrays from Huge, Medea and Apple for arrays capable of 125 MB/sec+ (720p24 is around 90 MB/sec) as compared to a standard SATA drive capable of 14 MB/sec (child's play for a modern drive). I've recently seen Fry's selling the fast IBM/Hitachi 7K250 SATA drives for as little as $160 after rebate (sorry, deal's over, I missed it too).

And storage is a huge component of the cost of a system.

So a $3000 G5, with the addition of a few hundred dollars in RAM, can already do several hours of DVCPRO HD footage (at about 50GB/hr). Want to edit your movie? OK, (run the math here of shooting ratios, digitizing ratios, rendered footage for color correction, etc. etc. etc.).

Want to be able to capture from a deck with HD-SDI outputs? Somewhere between $1500 and $2500 for the card (assuming it'll realtime transcode to DVCPRO HD).

Want to preview pixel for pixel with HD SDI? Fine, a $2000 Apple 23HD Cinema Display and a DeckLink HDLink will plug into a card from Aurora, DeckLink, or AJA.

Throw in any old computer monitors, hook up your (potentially rented) decks and broadcast monitors and other paraphenalia (as needed), and you're ready to rock...for CHEEP.

And cheap is good.

Better than cheap, however, is cheap AND good....which this provides.

I'm sitting in the Albuquerque airport awaiting for my family to come in on their later flight so we can all drive to a relative's wedding. There's free wireless here - yeehaa!

So some interesting thoughts to pursue:

somebody I talked to at NAB said that the DVCPRO HD codec is not locked to a particular size or framerate. I need to verify this. So even if it can't (natively) go back out to 1080p24 with the DVCPRO HD codec, can I write to it effectively? Is it compression ratio locked, or compression data rate locked? DV-NTSC gets wierd results if you feed it HD sized files (all bad/screwy scaling/compression artifacting). Will DVCPRO HD suffer the same results? Can you use whatever framerate you want with it? I'd hope so since the Varicam (list price $66K) can shoot anywhere from 4 to 60fps.

Somebody may have already solved this, I just haven't had time to sit down and play with it yet. I have had a Cinema23HD display, a 640GB RAID, and an HD capture card sitting in my studio for about a month and still haven't had time to test/play with them. Does that tell you how busy I've been?

Somebody wrote in (thanks J!) saying they have a Super16 short they've shot and want to post it in HD for minimal bucks. What's the least expensive solution? It's a good rubber meets the road challenge. I thought about HDV, but it doesn't have a 24p solution, only 720p30. I'm looking into the technical difficulties of doing a frame for frame transfer to HDV, then worrying about retiming it in post, either in FCP or at worst in After Effects, or some other tool. A minor task of twiddling the frame rate metadata from 30 to 24 (or 23.976, or whatever) frames per second. If the source was 24p and you put on HDV 720p30 resolution, your footage is playing too fast. Damn, wish I had FCP HD on my laptop. This proves to me my next laptop MUST be capable of running FCP HD. Ill be buying in the next month or so. To handle DVCPRO HD, must have 1 GHz G4 and 1 GB of RAM at a minimum.

The AJA Kona 2 HD board will do on the fly transcoding from uncompressed to DVCPRO HD, in and out. That means you can capture from uncompressed source (D-5, HD-CAM, dual link HDCAM) to DVCPRO HD. Useful either as working codec or offline codec. I received a press release from them this week, I'll post it right after I post this.

Does the new Decklink do the same? I hope so.

--------------------------------------------------

Workflow comment - I bet there's no way Apple will release the DVCPRO HD codec for Windows, likely EVER. Apple is using their relationship with Panasonic with DVCPRO HD (native Panasonic Varicam codec) to make FCP HD a more attractive package. Apple really only sells FCP HD to sell more Macs. If DVCPRO HD were available on PC, it would reduce the Mac's advantage. Thus, no way.

Why this matters: if in an all-Mac environment for post & FX, can simply hand over source DVCPRO HD clips. If in your project someone is going to work on a PC to do effects, or anything with source footage, you'll have to render out to another codec, such as BlackMagic's lossless 4:2:2 codec, or Microcosm, or something that is lossless if you don't want to reduce image quality. MUCH larger files, takes more space and less convenient to work with. Bummer for the PC users out there.

--------------------------------

I'm now officially working on a book that will include all this info and more. As I'm starting to organize it, some interesting thoughts:

Your shooting format will affect your storage size and type. OK, that's obvious. But your storage format will influence not only what type of storage you should use, but also what kind of backup regime you should employ, if any.

Huh?

Let's say you shoot with Panasonic's Varicam, using DVCPRO HD codec. If you go with a SATA disk, it's pretty easy to back up, and worst case scenario if the drive craters you can recapture. But woops, that means having access to the camera or a deck. Is it readily available, or a rental? Using the "1% per day" rule of thumb, that deck is $250 a day to rent. A 250 GB drive is about 5 hours of footage...a day or recapturing.

Let's say you shoot HDV. Low cost, low data rate, all well and good. But if your storage fails you, you have to recapture by hand. Man, total drag, and you're at risk of losing continuity between your FCP project file and the media if you have to recapture. I need to read up on Lumiere and the Heuris HDV stuff to see how replicable it is.

Can you use DVCPRO HD as a working codec for HDV proxies effectively? Will you still get all that realtime effects goodness? Note to self: verify.

If you shoot HDCAM for longform, that's an expensive deck to rent (circa $1000/day following 1%/day guideline). It would be EXPENSIVE to have to go back and recapture a movie's worth of footage. Doable, but days of time and a lot of money. If you are cheap and have used SATA RAID 0 and a drive dies, all your data is gone. Are multiple FireWire drives a price viable backup strategy? Or is the risk of having to recapture worthwhile?

In the book, I'm going to discuss various storage solutions, and the backup strategies that make sense to match u