HD For Indies

Apple today announced the new iMac G5. It's basically a 17" LCD (same size as 17" PowerBook), with some guts underneath that bring the overall case aspect ratio to more like 4:3, and all the guts either underneath or behind the screen. It's less than 2 inches thick. Nice. There's also a 20" model. I bet you can find links for it without me taking the trouble. This isn't an HD workstation really. You could do DVC PRO HD stuff with it, but it's expansion capabilities are so limited this is about all the coverage I'm going to give it. 1.6GHz or 1.8GHz models in 17 or 20 inch sizes, respectively. Minimal RAM & HD: 256MB, 80GB drives. Clearly they were trying to keep starter costs low with these itty bitty RAM & HD specs.

According to MacinTouch, some folks who ordered the 30" Apple Cinema display have had their ship dates pushed back from 8/31/04 to 10/25/04. So if it ships the 25th, you'll get it around Halloween I'd imagine....

-mike

Algolith Software has announced the availability of some of their key software technology as After Effects 6.0 compatible plugins.

They have the following useful plugins for digital video (SD or HD) manipulation:

Anti-Aliaser to remove aliased edges from CG footage
Adaptive Deinterlacer to remove field tearing from interlaced footage and smooth the results
Content Adaptive Scalar this is a biggie - this is a much better algorithm to uprez footage, for instance from 720p to 1080p. I've seen the results of this filter, and it looks MUCH better than Final Cut Pro HD's methology, better than After Effects, better than Cleaner (my favorite scaling app)
Frame Rate Converter for conforming one frame rate to another
Time Warp for retiming shots
Noise Reduction ( Dynamic or MPEG to reduce noise in footage, either from film/CCD noise or any DCT type compression artifacts (such as from JPEGs or MPEG footage)

I hope to play with these in the near future, if they fulfill their promise, they will be great tools to improve the look/quality of digitally acquired imagery.

I am especially curious to see the results of 720p taken up to 1080p with this plugin.

mike

UPDATED WITH HDV INFO ON 9/8/04:

I've been looking into some stuff, so I decided to review what the various formats can do.

HDV

Not really a professional format, but I'm sure some indies will use it.

HDV resolution 1:

Playback frame size: 1280x720
Acquisition frame size: 1280x720 to tape
frame rate/s: 30 fps progressive

no interlaced 720 signal, no 24p

bit depth & color space: 8 bit, 4:2:0 (alternates which color channel gets recorded)
datarate: 19MBit/sec (about 2.375 MB/sec)
compression type: MPEG-2 transport stream, with GOP (group of pictures) 15 (think of it as keyframes every 15 frames)

HDV resolution 2

playback frame size: 1920x1080
acquisition frame size: 1440x1080 to tape (but the Sony upsamples 960x1080 to this)
frame rate: 60i (interlaced, no progressive options)
bit depth and color space: 8 bit, 4:2:0
datarate: 25Mbits (about 3.125 MB/sec, slightly less than DV)
compression type: MPEG-2 transport stream, with GOP (group of pictures) 15 (think of it as keyframes every 15 frames)

DVCPRO HD:

1280x720 10 bit 4:2:2 at 59.94 or 60 progressive fps. 24, 25, 30 fps also possible as standard stuff, or anything from 4 to 60fps if the hardware or software frame rate converter is used.

1920x1080 10 bit 4:2:2 at 59.94 interlaced

Reality:

10 bit 1280x720 4:2:2 progressive off the HD-SDI tap gets sampled down to 8 bit 960x720 at 4:2:2 colorspace then compressed to about 5 1/2 MB/sec
10 bit 1920x1080 interlaced only off the HD-SDI tap (gets sampled down to 8 bit 1280x1080 at 4:2:2 colorspace then compressed to about 14 MB/sec

HDCAM

1920x1080 10 bit 4:2:2, progressive or interlaced, 24p possible (psf)

1920x1080 10 bit 4:2:2 off the HD-SDI gets sampled down to 1440x1080 8 bit at 3:1:1 colorspace then compressed to something like 22.5 MB/sec (don't have hard #s)

HDCAM SR

1920x1080 10 bit 4:4:4, progressive or interlaced, 24p possible (psf)

Reality:

Hey, this one (finally) works well:

1920x1080 10 bit RGB 4:4:4 of the dual HD-SDI taps actually stays just that - 10 bit 1920x1080 RGB 4:4:4. It is then compressed with MPEG-4 SP (Studio Profile) with 4.27:1 compression, results in a 50 MB/sec datastream to tape. Uncompressed off the taps, that signal is about 180MB/sec for 24fps.

Panasonic D-5:

1920x1080 10 bit 4:2:2 progressive or interlaced to tape as I understand it. 1080p24 possible. The deck is $99,000. HDCAM SR seems like better quality, better deal.

Synopis:

I see two major categories/price points at this point in time (as I understand it so far, blah blah blah, disclaimer disclaimer). You can split hairs and make other choices, but in terms of final quality,

Budget level 1:

Go for the the Varicam. It can shoot variable frame rates, and while Sony originally touted the increased resolution of their system, they, ahem, don't focus on the fact that it subsamples the colorspace at 3:1:1. While the Sony starts with more pixels, it throws out lots of information before it compresses it. The Panasonic, as the numbers and user reports bear out, starts with fewer pixels but keeps more of the color data. Shooters like it's good color reproduction, and it's ability to do heavy color correction without having the image break up heavily. Panasonic's CineGamma capabilities have also been highly lauded.

But the post process is soooooo lovely to work with in Final Cut Pro HD when using the AJ-JD1200A deck. Source camera quality 8 bit 4:2:2 720p24 is only 5.7MB/sec with 2 channel audio, and FCP HD can do realtime color correction, FX, etc. with it.


Budget level 2:

If you can reach the next big threshold, go for a full size, full bandwidth solution. Shoot with Sony F950 to HDCAM SR in 4:4:4 RGB mode via dual link HD-SDI. For the money, this is a very nice solution. Full res RGB color with no color subsampling, and very very light compression. Described as "visually lossless". I'd like to see some heavy/extreme color correction tests to compare HDCAM SR to straight-to-disk solutions.

The data rate is quite a bit higher for this - if you're going to try to hang on to all that quality, you've got to go uncompressed 4:4:4 10 bit, and that's something like 180 MB/sec. Yowza. That's about 640GB/hr for online work. To quote Roy Scheider in Jaws:

"We're gonna need a bigger boat."

Beyond that:

The Thomson Viper Filmstream recording to HDCAM SR is a high quality but logistically complex solution. 10 bit log (not linear, as all else mentioned here is) 4:4:4 RGB. This is what was used on a lot of night footage in Collateral. They decided not to record straight to disk as impractical at this point in time. Unless you're shooting greenscreen, I wouldn't see a strong/practical reason to do so.

In post, you'd need even more space, since the source footage isn't really watchable/editable. I need to do some thinking on how this workflow would really shake out. The possibility of having an HDLink or One2One do a hardware based log==>lin conversion for monitoring makes the workflow easier, but it's still quite complicated. Not for the faint of heart.

Beyond that to the point of not yet: the Dalsa Origin shows a lot of promise, but it isn't a very realistic shooting possibility as yet. Shoots a 4K image, but it's a humongous datastream - takes a quad 2 gigabit fibre link just to carry the datastream.

But because it shoots log, it complicates shooting, posting, and color correcting SUBSTANTIALLY. From what I can gather this is the best image quality you can get with a camera that has a reasonably viable production methodology. The Dalsa (mentioned below) has to be tethered to a huge disk array at all times...not terribly convenient.

The Kinetta camera doesn't exist yet, and based on what I've seen, their hope for small drive modules is going to take a while to achieve. But I haven't read up on it since NAB.

UPDATE SUNDAY NIGHT: I forgot to mention - don't judge the quality on the data rate alone. Not all compression schemes are equal. For instance, the MPEG-4 Studio Profile used in HDCAM SR is supposedly better than the DCT based compression used in HDCAM (or so I read...somewhere...NOT gospel). So don't go comparing data rates in one format as if they were on the same "scale" as data rates in another format.

Links I found/used while working on this post:

http://www.creativecow.net/forum/read_post.php?postid=109108642875955&forumid=126

http://millimeter.com/mag/video_nab_12/

http://www.williamsdp.com/24p_difference.html (this one's pretty good)

http://www.digitaltelevision.com/publish/dtvbook/ch3.shtml (good one also)

http://www.hd24.com/compression_numbers.htm

http://videoexpert.home.att.net/artic3/262hdvr.htm

http://www.uemforums.com/2pop/ubbthreads/showflat.php?Cat=&Number=22149&Main=22036

http://catalog2.panasonic.com/webapp/wcs/stores/servlet/ModelDetail?storeId=11201&catalogId=13051&itemId=68620&catGroupId=14594&displayTab=O&surfModel=AJ-HD3700B&surfCategory=D-5%20HD%20Mastering%20Systems

http://www.vtpcorp.com/htm/panhdvtr.htm

http://www.thomsongrassvalley.com/products/cameras/viper/ (this is the Viper home page, lots of good stuff to learn, download the PDF FAQ, it's a good overview)

-mike

I've been going through all my started-and-never-finished posts from the blog, and here's one I definitely should have published. This is old news about the JVC ENG (electronic news gathering) HDV camera they were showing off at NAB as a mockup/prototype that theoretically gets released within the year. Shoots 24p at unknown res, 1920x1080 res CMOS chips, capable of overcranking in theory, etc.

Read the Camcorder Info article here.

A 2-pop forum thread about it

JVC's press release on HDV from NAB

Mike's Comments: I read somewhere that this camera was theoretically supposed to start around $20K (lense included? Maybe not). This would be a great midpoint between the sub $5000 cameras and the $65000 Panasonic Varicam.

If you look at the very top of this page, there is a blue bar with a search function. Since I use Google's Blogger, it uses Google's search engine to search the contents of HD For Indies, going through all 100,000+ words, all 254 posts to date (and counting), all the way back to March 9th when I started this thing.

So if you are wondering if I've covered a topic, just plug it into the search bar at the top of the page.

Enjoy.

-mike

Ampede has released their new Ampede PDF plugin for Final Cut Pro HD.

Wait a minute - WTF would I want to deal with PDFs in my video editing application? Isn't that stuff for print work?

Because this plugin allows you to continuously rasterize Illustrator artwork placed in FCP HD. For those of you used to After Effects, you know what this means. For the other 99.999% of the human race, this means that a one inch wide Illustrator file placed in FCP HD can be scaled up to full screen....and still look tack sharp, clean, crisp, as good as possible, as if it had been built that size.

"Ampede PDF renders vector content from Adobe Illustrator and PDF files frame by frame, so the artwork is always sharp regardless of scale, movement, or rotation." they claim

"The US$99 plug-in is optimized for both G4 and G5 processors and offers ColorSync support, Velocity Engine acceleration, multi-threading, sub-pixel rendering, non-square pixel support, field rendering, and a Cocoa-based GUI available inside Final Cut Pro."

For industrial/commercial projects, this sounds quite useful - I used to do this kind of stuff with After Effects all the time.

Medea has introduced a new RAID that interfaces with a Mac/Windows/Linux box via 2 gigabit fibre channel, and uses 5, 10, or 15 SATA drives. Yowza - this sucka is FAST!.

Medea has a patent on something (I forget their name for it, Zone Striping something) that counter stripes the drives while formatting - half are writing from the edge towards the spindle, and the other half write the other way. This gives a steady, consistent data transfer rate, instead of the usual scenario, entailing peak performance when drives are empty and minimal performance when they are almost full. In any case, some stats from their web page on it:

5 drives - up to 125 MB/sec
10 drives - up to 200 MB/sec
15 drives - up to 300 MB/sec

It's not clear whether that is peak performance from the drives, or that is consistently available at RAID 0, or what. I wish they'd state it more clearly. They claim up to 4.8TB of usable space. Dividing that by 15, that's 320GB a drive. The math works out that it could be 400GB Hitachi 7K400 drives in there at RAID 5. RAID 5 is usually a 20% space penalty. Dunno, just guessing as to what they're using in there. Their pricing links aren't working correctly, so only by clicking on the "Buy Online Now" link is pricing...still not revealed, after some hunting (NOTE TO MEDEA: FIX YER DAMN LINKS!). So I have no idea what this sucker costs, but I betcha it's more than the other products in their line, such as $9000 for a 2TB solution.

So how would this work for HD? The 5 drive solution would work for 8 bit 4:2:2 720p60 and 1080p24 work, but that's it. If you wanted to do 10 bit 4:2:2 1080 anything, or 10 bit 4:4:4 anything, you'd need the 10 drive setup. Let's see how it would shake out in terms of space for the 15 drive setup (best $/GB setup I'll bet):

8 bit 4:2:2 720p24: 29.4 hours
8 bit 4:2:2 1080p24: 13 hours
10 bit 4:2:2 1080p24: 10.4 hours
10 bit 4:4:4 1080p24: 6.9 hours

It's late, flag me if my math is wrong.

If the 15 drive solution is really achieving 300MB/sec sustained across the entire array in a RAID 3 or 5 configuration, that would STILL not be enough to do 10 bit 2K (2048x1536) film work. So 20 disks would be enough, if you could set it up that way. Are their RAID sets expandable? Hope so, it's really hard to find a place to stash 6TB of data. It's starting to make good sense as to why 2K DI work is so expensive.

-mike

BTW, they also started shipping their 500GB, dual drive, single FireWire 800 bus, looks like a baby G5 G-RAID, that is the fastest dual drive, single FW800 solution out there right now. It's $700 for the 500GB model. But FireWire800 is still not fast enough for HD work. If took two of these and put each on it's own FW800 bus (the built-in and a PCI card host), it still wouldn't be fast enough for 720p work. Bare Feats has a write up with performance results.

This is the 250th post I've written for HD For Indies since I started this on March 9, 2004.

I think I've written somewhere around 100,000+ words since I started this...6 months ago.

I'd like to thank all those that have written in, and encourage those that have not to please do so - ask questions, share experiences, or call bullshit on my theoretical ramblings. If you disagree with a stance that I have, please tell me - and tell me why.

I'm getting into territory where I'm starting to review equipment and software for magazines, beta test some products, work with some developers on improving their existing and future products, and possibly design some new hardware products in the HD space.

If you have a product you'd like me to review or quietly evaluate, feel free to contact me - mike@hdforindies.com

-mike

I remember reading this in the past, but I don't think I ever posted it on the blog - it's a good article from Digital Cinema Magazine on why HD CRTs don't show all the detail that is really in an HD signal. Worth the read. Check it out.

A good quote to give you the idea of what they're talking about:

"The dirty little secret of HD is that very few people have ever seen a full-resolution HD picture, and the $30,000 to $40,000-plus broadcast monitors that are currently used every day to critically view finished HD product can barely display half the available resolution of a 1920x1080 HD picture."

The article goes on to talk about the pros & cons of LCD, and why LCD is and is not a good solution for certain applications.

If you're considering getting an eCinema box or HDLink, or any kind of an HD monitoring solution, definitely worth the read.

UPDATE MONDAY 11AM CST: Then there is this from the eCinema site on their EPD100, the first device that the HDLink emulated. It talks about zone reference plates, and has photos comparing images on a 24 inch Sony HD studio monitor vs. the EPD100 on a Cinema Display 23HD (the older, plastic one).

I started this posting back in mid July, decided to come back and finish it.

Some of the key differences between the old (plastic frame) and new (aluminum frame) 23 inch Apple Cinema Displays:

Numbers listed are in the order of old (plastic)/new (aluminum)

Brightness: 200cd/m2//270cd/m2
Contrast: 350:1/400:1
Display cable: ADC/DVI-D
Power draw: 70 Watts/90 Watts
Pixels/inch: 85 (?)/100
Pixel Pitch (size): 0.258mm/0.258mm
Unit Weight: 25.3lbs/15.5 lbs

So the new units are brighter, with more contrast, way less, and have more convenenient cabling (in my opinion).

Buy a new one.


This comparison was culled from info from anEveryMac.com page.

Bare Feats is reporting that by flushing all the caches, you can increase the performace of a dual channel La Cie d2 drive FireWire 800 RAID up to 113 MB/sec reads, 88MB/sec writes. Still not enough for any HD work, but better.

The real question then, is how often would you have to flush those caches to get that kind of performance consistently?

If required flushing a regular thing, NOT worth it in my book.

-mike

London based Digital Heaven dumped their Avid gear for Apple G5s running Final Cut Pro HD according to this article in MacWorld UK.

Choice quotes:

We've been watching the development of Final Cut Pro since the beginning. With the release of version 4.0 last year, it was an easy decision to switch."

and

"We strongly feel that Apple is the new leader of post production technology and are 100 per cent committed to its solution."

This article on MacWorld UK discusses a demonstration at a disk production factory - HD-DVDs could be produced in 3.5 seconds each, and a 5 minute retooling allowed the line to produce standard DVDs at one every 3 seconds.

HD-DVD uses better codecs than the competing Blu-Ray, but on smaller capacity discs (15 vs. 23GB). The Blu-Ray camp is considering using the same codecs to augment their current MPEG-2 only option, however.

The article points out that the studios will probably go with whatever is cheapest for them to produce. "If it costs one penny more, you lose." says someone in the article at the production facility.

This echoes what I wrote a while back.

The program was started earlier this summer, but I just heard about this recently - I was in the Apple Store in Austin and the biz sales guy told me about this:

If you are making (and who's to say?) a business purchase, Apple will match a valid/authorized competitor's price up to 10%.

For instance, Amazon.com is selling the 23HD monitor for $1709. If you want to buy one locally (where they might have it in stock), they'd drop their price no more than 10%. From the $1999 price point, they'd drop it to about $1800. Given the size of the Apple 23 HD, the shipping difference might make it worthwhile.

Items on Apple's website claiming to be 3 or more weeks backordered are not available for this offer, however.

So if you are wanting to buy something they carry in the Apple Store, be it Apple or non-Apple branded items, (either online or in the physical stores), price shop around and inform Apple BEFORE you try to make your purchase.

Readers noted some additional advantages of HP's 23-inch L2335 display, a less-expensive alternative to Apple's Cinema Displays, as detailed yesterday:
[Jeff Phillips] The HP unit features one composite video, one s-video, and one set of component video inputs - inputs not listed as being on the Apple display. Does this mean the HP display could also be used with an HDTV receiver?

["Ed S."] Another point about the HP display, it has (15 pin analog) UXGA, DVI-D, component, composite and S-Video inputs. And is compatible with most HDTV formats. And 90 degree pivot feature. Priced around $1599.

Quality of the screen is something to check carefully. Apple in the past has been lauded for the (relatively) low cost of the larger screens.

But analong component HD inputs would be nice

Specs as HP provides:

-500:1 contrast
-1920x1200 resolution
-16ms response
-inputs: "15 pin D-sub analog and DVI-D (digital), video inputs for composite, component, and s-video; this monitor is also compatile with native HDTV video formats" - which makes me wonder if an adaptor or upgrade is necessary for HD signals

Christopher Barry, longtime reader and collaborator, sent in a couple of useful links:

DVX100A footage playing in a Panasonic AJ-HD1200A deck, trying to get it all to work.

 

Getting a consumer HDTV to monitor while capturing via FireWire with Panasonic AJ-HD1200A deck.

 

Interested in how to get the best quality output out of your footage on DVCPRO HD, that you probably shot on a Varicam? The read this thread over at Creative Cow.

It discusses high quality workflows for:

-720p24 DVCPRO HD from and back out to the Panasonic AJ-HD1200A deck
-how to capture 1080p24 out of the deck (it'll do a realtime upsample)
-how the deck works to lay 24p footage back to deck
-how to upsample the footage in FCP to 1080p24 to lay back a 24p master, such as to HDCAM
-how to upsample with better quality than the deck or with FCP, using the Algolith plugins (Mac AE plugs on the way someday)
-how to use Cinema Tools to convert a 1080p24 project to 1080p25 so it can go back out to the Panasonic AJ-HD1200A deck
-and other strategies

If you are planning on a Varicam shoot and posting on the Mac, this is a must read.

Wish I'd read this before staying up until 3am the other day.

Sometimes it's fun to grab a machette and go hacking into the jungle, but it's certainly more efficient to look around for a frickin' TRAIL first!

-mike

Based on published reports and my own inside sources, it looks like the dual 2.5s are chock full of trouble. A goodly number of DOA systems (see Macintouch thread), as well as reports of wierdness during production. As recently as a few weeks ago, Apple was having trouble with the systems leaking, due to use of inferior automotive grade aluminum in the parts dealing with fluid for the newfangled liquid cooling system.

This on top of the fact that Hey Folks - they aren't all that much faster. Yes Yes Yes, it's nice having the fastest, prettiest toy. I have been bitten by that bug, and hard. But if you're looking to do HD stuff, it really isn't buying you all that much. And the darn things are way backordered. The ONLY reason I'd consider getting one right now would be IF the FireWire write speed issue were resolved...and I'm not very optimistic about that happening right now. Nobody has definitively reported as yet.

See this link over at Bare Feats. In After Effects, on a test file, the dual 2.5 was about 21% faster. That's definitely a nice improvement. For FCP, however, the dual 2.5 was only 14% faster rendering 7 effects. Hmm. Not a huge improvement.

For the additional costs, with the additional risks and delays, I just don't see it as worth it if your primary endeavor is going to be HD stuff. I'm not seeing any crucial thresholds involved - maybe it could do one more effect or layer in real time, but is that crucial, as in mission critical, to anything you're planning?

If you're looking to buy in the next 60 days or so, I'd push the dual 2.0 unit, and either pocket the difference in price, or get a better video card, or buy additional 3rd party RAM and hard drives.

I say this NOT casually - I know the addict-like, deep burning desire to have the latest & greatest seconds after Steve takes the wraps off. 3 of the 5 Macs in my house were ordered on the day those models were introduced (G4/450 with DVD-RAM drive, G4/733 with SuperDrive, Rev A Dual 2.0 GHz G5). And, of course, all had 2-3 month delays before I finally laid hands on them.

PS - I NEVER buy additional RAM & HD from Apple - they charge a monster premium for average grade stuff.

As Dennis Miller used to say-

"That's just my opinion, I could be wrong."

-mike

So I stayed up until 3am messing with a bunch of stuff, trying to upsample my footage from 720p24 to 1080p24 and keep it as clean as possible. I was loathe to use the FCP Media Manager route, because I suspected the upsampling algorithm wasn't going to be great. Compressor had done some sloppy working look I'd seen previously.

I tried a bunch of different tools, trying to upsample 720p24 DVCPRO HD footage up to 1080p24 BlackMagic codec 4:2:2, YUV color space. I also played with some other new toys and workflow ideas.

SYNOPSIS:

Discreet Cleaner - makes the footage substantially darker, especially shadow detail. Ick. But marches on through it.

Apple Compressor - makes the footage a bit lighter - I didn't verify, that played through the HD-SDI, that it actually looked different, but I didn't trust it. Plus it looked like it was running pretty damn slow

Sorenson Squeeze - tried this one just for fun. Only writes out Sorenson codec that I could tell. So bleh on them. Egotists.

PopWire's Compression Master - I was really hopeful about this one - I downloaded and installed the full function, time limited demo, but maybe I did something wrong, or perhaps it doesn't recognize the DVCPRO HD codec. In any case I got errors and never got anywhere with it. Drat. Was really hoping to use it. Will play with more.

Adobe After Effects - can get nice clean output from it, but even being clever with it, TONS of handholding involved. But it does offer the possibility of using additional plugins. I was looking forward to using some of Algolith's CAS (Content Adaptive Scaling) plugins to upsample, but I didn't get the chance. YET. I spoke to the developers at NAB, and they weren't in a hurry to provide FCP plugins since they heard everyone wanted realtime performance. But for this kind of upsampling, you're not going to do it but once during a normal project. Then maybe a few little times for pickup bits. But I don't think a long lunch or overnight burn to get a high quality upsample still within FCP is something I'd get too upset about. It's that or a long slow mediocre conversion...and who wants that?

Other stuff - doodled briefly with the BitJazz SheerVideo codec - just enough to recompress a 1080p24 clip from 2.7GB down to 0.9GB. Not sure I had everything set right, but to the naked eye it looked like a good match and still played back in realtime. Their claim to fame is tiny file sizes, mathematically lossless compression, and super fast performance...enough to capture straight to their codec from deck. Much more testing to be done, but looks promising. They handle 4:2:2 and 4:4:4 and even both of those with alpha.

I'm thinking the highest quality way to do this would be to use Media Manager as described at the end of my notes and Recompress To a new location. Those files would then be linked and working. Be sure to base file names on existing media. Then use a Copy function to make a clean set of 720p24 DVCPRO HD footage, then drag those and do a batch type operation using the tool of choice. Render to a third directory, then in FCP mark as Offline the footage in your new project, and relink to your better uprezzed stuff. A lot of work, hopefully it would look better. To be seen.

-mike

Here's all my raw notes:

========================================================

NOTES ON CODEC FLIPPING ATTEMPT:

I duped a bunch of media - about 3GB worth.

I tried to use Compression Master - it bonked, got an error - "Error: Unsupported media format" I guess it doesn't like DVCPRO HD-or did I set it up wrong?

I tried to use Compressor - it said it would take many many hours. Bail. Also, it was lightening the footage.

I tried to use Cleaner just cuz I knew how. Cleaner makes it darker. And it hung on quit. Shit.

So Compressor makes it lighter, Cleaner makes it darker, Compression Master doesn't work. Fuck.

Sorenson Squeeze - nope, it only wants to write Sorenson. A little too "me centric" in codecs

After Effects? How close to a batch process can I create?

-launch AE

-new project


-new folder

-drag in folder contents of "to be converted) source

-drag all folder contents to "make comp" icon

-in "New Comp from Selection" window, select Mutliple Compositions, don't both to click "Add to Render Queue"

-select any comp and Command-M to make movie

-select the desired output directory

-make sure the Output Module and Render Module are set to default to what you want - render to your codec.

-highlight all comps (view by type to make this easier) and under the Composition pulldown menu select Add To Render Queue (can be done with Shift-Command-/ (forward slash, same key as ?)

-batch render GO!

-you can either manually trim back the comp names, or use a batch renamer like ABFR (A Better Finder Rename) to trim the names.

Then use Media Manager in FCP to take the clips offline, relink to these clips using manual selection.

Dupe your prior timeline, and change the Sequence Settings to match your codec.

My switch from 720p24 DVCPRO HD to 720p24 BlackMagic 8 bit 2Vuy was almost successful - it got in there, apparently required no rendering (grey bar at top of timeline) but drops frames!

Only 42.3 MB/sec, but drops frames after about 2-3 seconds consistently.

Tried rendering to the straightforward 8 bit codec, I'll see how that does.

If that doesn't work, I'll run both sets through the "Flatten Only" preset in Cleaner.

The plain 8 bit codec had the exact same problem. Crap. Had this problem before at Martini Shot.

When I reconnected to the new media, it gave me a red bar - clearly didn't like the codec. Went into the sequence settings, that I had loaded from the list of presets, so clearly the 8bit2Vuy is the indicated preferred & correct codec.

Dammit.

So I'm going to try to use cleaner to Flatten both test files - the plain 8 bit codec and the 2VUY codec stuff.

one reason - the 720p24 footage is trying to play back at 60p in the timeline - so it freaks it out, since it thinks it should be playing at 60fps.

Duping Comps in AE, AE tries to advance the # if there is a number at the end of the comp/filename. BAD!

So don't trim the comp names in AE....yet.

Annoyingly, AE 6.5 does implement Shift-Command-S as a Save As... but it strips a keyboard shortcut away for use as Scale settings. Required to use "Size of Comp" for scale


Have to manually adjust comp size - what a pain!

So I run Compressor to make some new 1920x1080 files. They confirm as 1920x1080, 23.976 fps, 8 bit, 95 MB/sec files. But they come in too short, stretched horizontally, too dark, and stuck at DVCPRO HD aspect ratio. CAN'T SEEM TO FIX!

This is soooooooooo not good.

Looks like redig is the way to go...shit.

I'll try some other reconnection action.

A-HA! Right clicking on the aspect ratio lets me change it...but it doesn't seem to do anything. Hmm. Mess with some more.

Changing the aspect ratio on the clips in the bin, not on the timeline, then dragging them in fresh seems to help. But adjust on the timeline itself does NOT, they stay too wide. Hmmm...

So what to do? Replace'em one by one? Ugh.

So lose edit changes.

SYNOPSIS:

Since there is no 720p24 format to tape via DeckLink HD or HD Pro, uncompressed 720p24 footage is kinda worthless.

Compressor makes it too light. Cleaner makes it too dark. Compression Master doesn't work. After Effects is too much work.

However you do it, seems you have to manually reassign the aspect ratio...and if the clip has already been used in a timeline, it DOESN'T WORK!!!

What if I drag it out and drag it back to replace? That's a LOT of handiwork, and wouldn't want to do it for a large editing job. But...

THE KEY MAY BE TO ADJUST THE ASPECT RATIO IN THE MOTION TAB BACK TO ZERO INSTEAD OF -33.33 - play with that

The plain old Duplicate command under Edit makes a copy of the clip - if you rename the copy, the source gets renamed as well.

BUT, unlike I guessed, if you change the aspect of the copy, ONLY IT'S ASPECT IS CHANGED. It isn't linked that deep - only by name.

If you Duplicate As New Master from right clicking, you get to rename it as you wish, and change aspect etc.

DVCPRO HD 720p24 clips with reset aspect ratio drop into timeline just fine and can RT play back with FUBAR'd aspect

Well, duh. Noticed the Recompress option in the Media Manager, so trying that now - taking a 720p24 DVCPRO HD to 1080p24 (23.976) BlackMagic 8 bit 2Vuy codec. 691MB to 10.5GB. Slight file size increse. Yeah. CRAP! It's 2:45am, it was 12:15 last I noticed.

HEY! THAT TOTALLY FUCKING WORKED!

1.) Cool! Really fucking cool!
2.) Doh! I've been trying to re-invent the wheel, and it was in front of me the whole time. Shit. The only thing to verify now is what the quality of the scaling is - what's better:

FCP
AE
Compressor
Cleaner
Compression Master

FCP is clearly the easiest.
AE is probably the best quality (that works so far - Cleaner makes new clips dark, Compressor makes new clips light, Compression Master doesn't take in DVCPRO HD).

Clips that had wrong aspect ratio set came in with Red Line of Render Needed. But as soon as I fixed one aspect ratio to 0 from 33.33, clicked Copy, and used the Paste Attributes command all was made well. BAD ASS!!!

NEXT QUESTION:

RT CC:

Nope, gotta render if using the 3 way color corrector...what happened to my RT goodness? Where'd it go? Do I have to redig?

Testing that theory, I made a new 1080p24 comp from the HDTV 1080p24 8 bit preset, and imported some clips I captured from deck at 1080p24. Nope, looks like render city.

Need to read up on how to get RT CC in FCP using DeckLink RT stuff.

Checking settings, BlackMagic 8 bit 2Vuy and 8 bit 4:2:2 should be handled by Blackmagic for effects. Don't know what's up there.

Read the instructions, messed with it...didn't get RT effects. Will carefully document...later.

Oh yeah -I should install the GOOD card! I've still been working with the DeckLink HD, not the DeckLink HD Pro.

Install LATER. Must...crawl..to bed....[thunk]

Sorry offline for so long, had some wierdness between Blogger/ISP/server. Back to bidness:

That enough acronyms for ya? After having difficulties working with the older Panasonic 130 deck and hardware frame rate converter, we rented a 1200A for the weekend to log and capture a bunch of footage. Very raw notes follow:

OK, below are my raw notes, as I took them, working with the Panasonic 1200A deck (with FireWire option) DVCPRO HD deck and Final Cut Pro HD. The deck also has an HD SDI connector, but I haven't messed with that as much.

Tomorrow I intend to do a bunch of various capture tests.

Here's the scenario: the HD short (15 or so minutes TRT) that I'm post producing was shot on Varicam to the DVCPRO HD format tape. Mostly 720p24, some 720p60 (for slomo). We want to post it at my house. Was unsuccessful trying to capture to 720p24 with BlackMagic DeckLink HD (not to say it can't be done, I just didn't figure it out in a couple of afternoons worth of time).

So we rented the Panasonic 1200A deck for $500 (incl. insurance, but I had to drive to Dallas to fetch it).

Over the weekend, the producers logged all the footage into FCP HD, and with my help we got it all captured into the system. It took about 18 hours total for them (and me some) to get a total of 2 1/2 hours of footage into the system. First timers, give'em a break (me too). But lessons well learned along the way. Read more for the details.

I really dove in blind - didn't read any manuals, just started messing with everything. I could claim I was doing this to simulate an actual working environment, but the truth was I didn't have much time so I just started trying to make it work. Oh yeah, that IS just like an actual working environment.

Here's my notes, exactly as I took'em:

------------------------------------

Got the Pansonic 1200A from MP&E in Dallas (thanks Brian!). OK, rented it.

Pulled it out, got set up & plugged in.

With deck FireWired in and playing footage, G5 seemed to be having trouble booting - possibly the FirWire data stream is confusing it? Get the flashing question mark folder icon - Mac can't find a bootable drive (there are 2 FW drives bootable, but no SATA)

Deck is on it's own bus, a FW400 is on it's own bus, and a FW800 is on it's own bus.

doing a cold shutdown on G5 and drives, disconnecting the deck FireWire, powering up FW drives, deck disconnected from FW, holding down option to select a bootable drive - that worked, got booted up OK.

Reconnected Deck FireWire, turned on deck, NOT playing footage.

Put deck in remote.

Launch FCP. Easy setup - DVCPRO HD

Create new sequence - load setup - DVCPRO HD 720p24

Initialize log & capture....beachball of death.

I think I need to change the Video Preview Out to keep this from shitting the bed.

Force Quit - app still up

Force Quit again - says (not responding) in FQ window

Shut Down.

Interestingly, it does NOT shut down - get a login window.

Click Shut Down

Powered up again - get the ? folder again (deck FW was still connected)

unplug deck firewire

start up again

Fires right up.

Lesson LEARNED - don't power up with FW attached to deck

reconnect FW to deck

Fire up FCP

load prior file

Easy Setup to DVCPRO HD 24p
Sequence Setup to same

Set External Video to NONE

Initiate Log & Capture - it WORKS!

close L&C window, save project

initiate L&C again

deck control - works with JKL

mark in and out

log clip

mark in and out on another clip

batch capture the 2 clips I just defined

test 2 thinks it has a timecode break...shit

Setting Video Playback to DVCPRO HD (720p60) 1280x720

dropping my 60p clips on the timeline

had video just dissapear after about 20 seconds of playback...deck setting?

video came back...but WTF?!?

USING FRC..not needed!

Going into clip properties, the footage shows up as 24p - 5.7 MB/sec....which is correct!

movie FPS is reported as 23.81 in one clip, and 23.95 in another...uh oh...

Analyzing Clip reveals both as 23.98..seems to work OK

NOW GOING TO CAPTURE SOME 720P60 FOR FRC

flipping from TAPE to EE on the deck will hopefully solve this problem

yes, it does - to see tape, set in TAPE

to play through (EE, aka E to E, aka Electrical to Electrical) lets the FireWire signal play throught system as a hardware codec. Just like a DV deck.

Ya know, if I stopped to read, like, ANY manuals I'd probably learn a lot more...

playing back a 720p24 clip runs the CPUS around 75-80% load

woops!

even on EE, it still flakes sometimes...even when on own bus...is it on own bus? Is the front panel FW on same bus as rear panel FW400 on G5? checking...

This all seems to be harder than it neeeds to be...I keep losing video via FireWire, even with NO other FW400 devices attached. About 27-28 seconds into playback, the video drops out.

Also, my original plan had been to capture ALL the footage in big huge swaths, go back in and define subclips, then be able to go back and just back capture the content for the subclips...this does NOT seem to work.

Shit

I need me an FCP guru, big time.

Time to hit the list...

I keep getting dropouts on video and audio when previewing via FireWire from the front panel FireWire port.

There is NOTHTING else on ANY FW400 port, but the machine is booting from a FireWire 800 port.

switching to the back panel FW port: still drops out, just not as frequently

when capturing from tape, we've had a few problems...classic problem of Too Close To Timecode break

a new problem with the DVCPRO HD stuff - you've probably heard you shouldn't mix differing frame rate footage on the same tape....here's why - we had some bars & tone done at 60p, followed by a 24p clip. When we put a 5 second head/tail setting on the capture (or perhaps from the pre-roll), it seems the deck sees that it's in 60p as it starts it's pre-roll or capture, so it assumes the rest is 60p...so even when we set the capture settings to 24p we got 60p footage and it's 2 1/2 times larger frame rate. So THAT'S why not to mix....fuxors your capture abilities.

It doesn't seem to like audio only capture - especially when close to a timecode break - tried 3 times with audio only, reset to aud+vid and it seems to work fine

The deck apparently needs 3 seconds (or so) of pre-roll - so if you have 60p bars (we did), if the pre-roll includes that 60p pre-rolled footage, or a timecode break, you're hosed. Standard capture stuff to know.

Other notes - FCP HD seems to drop footage on whichever of your specified disks has the most room - it does NOT go in the order you assigned them, it simply distributes the footage as it goes. This is NOT the way I'd prefer it - I'd prefer to have disk 1 get full, then move to disk 2, etc.

So, as always, no timecode breaks is VERY important.

It took us about 1 1/2 hours to troubleshoot and recapture the 1/2 dozen shots that didn't properly Batch Capture

OK. Enough for one weekend...

---------------------------------

Digital Producer has this article of a bunch of first hand advice on working with HD.

Just read it.

MacMinute has a report on G5 shortages that may stretch into Q1 2005 - threatening G5 computer availability, and potentially pushing back the release of faster models (that last my conjecture).

MacMegaSite has this link on the release of NitroAV's 23GB FireWire 800 Blu-Ray recordable disks. At this point it is just data backup device, NOT a high definition DVD format (the data standard hasn't been nailed down yet, we are a year out at least from consumer playback devices). But it's the first step, and it's FireWire 800, so that's good.

MacWorld has some benchmarks on the new dual G5s, but not the dual 2.5. Which to me is a "well, who cares!" They compare a new dual 2 and new dual 1.8 to the old dual 1.8.

After my recent trouble with a 23 LCD from Apple, AppleInsider has this article on an Apple document that details guidelines for handling LCD pixel anomalies (aka dead pixels). Key quote: "rarely is the customer eligible for a product replacement." as well as "Apple guidelines list 10 dark subpixels or a combination of 15 bright and dark subpixels to be acceptable, but does not provide acceptability figures for bright subpixel counts on displays of this size." Wow, I got lucky. Or the folks I dealt with were super nice. Both I think.

Macintouch has this report on troublesome G5s that have shipped lately. Key quote: "ly out of the box to ensure they're good, and be especially wary of machines made in late July, apparently. " as well as "my office recently took delivery of 22 dual 2.5GHz G5s. Eight of them had a faulty processor. Symptoms included repeated kernel panics and the windtunnnel effect, where all the fans in the box went to maximum." as well as when running AHT: "A few seconds into the logic board test, an error that appears to have something to do with CPUs. "

There. That ought to hold you for awhile.

I'm debating whether to publish my raw raw raw notes on working with Panasonic AJ-HD1200A deck over the weekend or write a nice, human-readable summary.

One of those will get posted soon.

Been learning lots of neat new stuff.

-mike

This report from a MOSR reader is on the X-SAN demo he saw in Europe. Pretty much just says it looked cool & promising and fast.

Starting on page 24 of the August 2004 issue of Post Magazine (sorry, haven't found it online yet), there is an excellent article about how HD was used to "see into the night, to see everything the naked eye could see."

The shot some 35mm, but a lot of Sony 950 camera footage and Thomson Viper FilmStream camera (recording to a Sony SRW-5000 deck).

There is talk of their entire workflow through Avids, with up and down converts, final color grading done with best quality footage, 4:2:2 950 HDCAM footage with 4:4:4 FilmStream footage with 10-bit Cineon files all living on the same timeline in an Avid.

Cool.

Definite must read for those into HD possibilities. Obviously, they had budget, but there are ways to do it for less...there are ways....

-mike

Mailman brought happiness today - Grant Petty and his crew at BlackMagic Design were kind enough to send me a review unit of two of their coolest new toys - a DeckLink HD Pro card, and an HDLink. I got them installed and started to mess with them today.

To review: the DeckLink HD Pro card has the following features:

-captures 720p, 1080p, 1080i HD
-captures SD PAL & NTSC
-captures are 23.98, 24, 25, 29.97, or 30 fps (progressive or interlaced as applicable)
-has analog monitoring - no longer requires an HD-SDI to analog adaptor (those are $1600 to $2500 alone)
-does realtime hardware downsampling (if you're at the right timebase) to get from 29.97 fps HD to NTSC or 25 fps HD to PAL
-does 4:4:4 full bandwidth RGB captures instead of just 4:2:2
-does 8 & 10 bit now, and the hardware can do 12 bit (when FCP supports it)
-has dual link HD-SDI in and out
-$2500 MSRP
-etc.

HDLink is a one trick pony, but it's a good trick: it converts an HD-SDI (or SDI) signal to a DVI-D signal to work on a flat panel display - the Apple Studio 17 and Cinema Displays 20 and 23 are good candidates to work with it. So I can plug the HD-SDI signal from my DeckLink HD Pro card into the HDLink, and that plugs into the Cinema Display 23, and I see all the pixels of the signal, nice and clear and sharp. It's $1300.

OK, so some comments on my hands on experience:

Yesterday, Apple released a ton of new software: OS X 10.3.5, Compressor 1.2.1, DVD Studio Pro 3.0.1, and Pro Application Support version something (newer). I installed all of that on a FireWire drive I've been booting from.

I'd been having some problems with the regular DeckLink HD I have - in Final Cut Pro HD, with the latest drivers from BlackMagic, in the Audio Video Settings, under Capture Presets, there is no selectable setting to do 23.98 (or 24) fps. I can pick 59.94 or 60 fps, that's it for 720 res. I installed the 4.4.1 drivers from BlackMagic. Still doesn't fix the problem.

Today in the mail I got the HD Pro card and HDLink. I went to the BlackMagic website and they had just released version 4.4.2 of the drivers. I also downloaded the HDLink utility.

After successfully being able to log and capture footage at 60 fps to either 8 bit uncompressed or PhotoJPEG 50 of 66% quality, after installing the new card, installing the new drivers, Final Cut Pro would crash when I tried to edit my capture presets. Dammit.

The HDLink I had working...for awhile. I hooked it up while I still had the older DeckLink HD card installed (not the HD Pro) and it successfully monitored some 720p footage. Looked bright, sharp, & crisp. Very nice. I then opened up a project done with DVCPRO HD footage at 1080i60. It would show stills full screen, but not preview them. So I'm guessing that was due to DeckLink HD can't monitor DVCPRO HD footage (which is actually only 1280x1080 pixels, not 1920x1080).

So I changed the timeline settings to uncompressed 8 bit (the YUV, shown as vuY, codec). Now it showed during playback...but horizontally scrunched.

Uh oh. I'd been planning on doing just this adjustment to take DVCPRO HD footage and make it uncompressed to do better quality color correction.

I probably need to slow down and actually, you know, READ THE MANUAL ON ANY OF THIS, but of course, I'm a kid at Christmas - I'm opening the next box before even seeing what the last one was.

As a temporary "punt" fix, I just used the Distort effect to stretch the corners out to the edge of the screen to make the aspect look correct. Footage looks GOOD. I'm anxious to feed it some better looking source, such as rendered from scratch, to see what it can look like at best possible quality...such as full on 4:4:4 quality.

For those who've been wondering why to spend $1300 for a DeckLink instead of just using a 23HD Cinema Display (or whatever you've got) to monitor your work, here's a couple of reasons why:

The Desktop Preview function in FCP when set to High quality still cheats a bit - it is a bit soft, not the "real" detail of the footage.

And there were also some temporal anomalies when using Desktop Preview - a smooth camera pan in the footage was inconsistent in it's velocity when played back wtih Desktop Preview.

Shown through the HDLink, ON THE EXACT SAME MONITOR WITH THE EXACT SAME FOOTAGE, it was perfectly smooth.

So the Desktop Preview is softer, not as detailed, and not as consistent a playback vehicle (frames have different lengths and/or delays) than the HDLink solution.

I've been having some problems, like the aforementioned lockup (have to Force Quit FCP 4 or 5 times and reboot) while trying to edit the Capture Presets.

The HDLink seems to be picky about being on during reboots, and in general what is plugged in when, in what order. The 23HD shuts itself off if all isn't Just So. So in general, the HDLink is picky about it's own power on status during reboots and boot up. It seems that if you don't get it right the first time, you have to shut down and try again.

1080i footage scaled up: playback is smooth, but the grain is apparent in the shots...and it appears to be assymetrical grain now - since it's been stretched from 1280 out to 1920, it's wider. Harrumph. Colors seem bright and vivid, and the gamma seems to be different, too, so hopefully this is more accurate for video preview than the desktop preview function. . Interlaced shots - you can tell that it is interlaced, but it doesn't seem to be that big of a deal, especially compared to the Desktop Preview function. Just waving the mouse around on the desktop (you can configure the 1920x1080 HD-SDI output as another desktop, how awesome is that) I can see field lines if I wiggle the mouse up and down quickly - makes me think the HDLink is feeding an interlaced signal to the LCD - different from the usual progressive display. The imagery is definitely sharper than Desktop Preview, and the playback speed/consistency is awesome.

(BTW, this test 1080i60 footage is from Apple's DVD-ROM they handed out at NAB of Glacier Bay in Alaska)

Yeah - Desktop Preview is just that - a PREVIEW of what your footage looks like. HDLink allows something closer to true monitoring...but I'll have to do a side by side test with a broadcast monitor to see (that's coming soon).

I'm sure when I look at a regular TV I'll be very dissapointed comparing the quality I've just seen to that stuff.

Processor load is HIGH - both processors running 80-95% when playing back 1920x1080 4:2:2 8 bit footage.

Tomorrow I think I might have to nuke the OS & Apps on that FireWire drive, and re-install 10.3.4, drivers 4.4.1, and whatever was the previous to yesterday versions of the Pro Application Support module.

Carefully step it up one thing at a time until I get it all working.

That's a huge chunk of time.

But then I'll be able to really test all this stuff.

That's enough for now, it's 2am, and I've been fighting technical problems for 2 days trying to post an HD short.

-mike

PS-Oh! And by the way - all of this 1920x1080, 4:2:2, 8 bit uncompressed footage? It's playing off of my cheapie SATA array, the 4x160GB Barracuda. Total cost to have this array that plays back this footage $750. Here's how:

3 more 160 GB Seagate Barracuda 7200.7 drives to supplement the one that came with G5- $110 apiece from zipzoomfly.com (2 day shipping included!)
1 Seritek 1S2 card - $65
SoftRAID - $99
2 PPA, Inc. cases - $50 ea
my old cheapie 120 GB FireWire 400 drive - what are those now, like $150?

So to upgrade a G5 to handle this playback (not include HD card, of course, but that can be as little as $1000 now) - under $750.

So you want to do HD?

G5 dual 2.0 GB - $2500
extra 512MB RAM - $100
RAID as described about - $750
a decent 17" monitor capable of 1600x1200 display - $400
Monitoring solution - HDLink and Apple 23HD - $3300 (ouch!)
DeckLink HD card - still $1000 (I think, still avaiable? Too lazy to check...)
Production Bundle - $1300

So that'd let you do an HD short, uncompressed, 1080i60 for $9350.

Just want to do 720p work? Get a Studio 17 from Apple instead of 23HD - cuts the package to $8350.

Just want to use the FireWire option and rent a Panasonic 1200A deck as needed? ($350/day best rate, $1500ish/wk). Then drop the HD card and monitoring - cuts you to $5350...but no monitoring.

You get the idea - can be done CHEAP!

Apple today announced a new software bundle called Production Suite that includes Final Cut Pro HD 4.5, Motion (their just announced motion graphics package, see last posting), and DVD Studio Pro 3.

This is a very nice kit, and falls in line nicely with what the rest of the industry is doing - offering an intergrated suite that hands off nicely between all programs. This is going to be GREAT for folks working on projects destined for DVD.

It's $1299 to buy it outright, and $699 to upgrade from version 1-4 of Final Cut Pro.

Student? Academic price is $499 for the whole kit.

THIS IS A DEAL. Normally, all of these together would be $1800. So essentially, buy your FCP upgrade to version 4 (price it was yesterday), and for $100 more, get two more software packages.

Have ANY version of FCP and been thinking about buying Motion? For an extra $300, get DVD Studio Pro - that's $200 off.

Have ANY version of FCP and been thinking about buying DVD Studio Pro 3? For an extra $100, get Motion.

Any way you slice it, this is a helluva deal.

Go get it. Run don't walk.

Apple today announced that Motion, their new high speed, real time motion graphics application first demoed at NAB, is shipping as of today.Here's what I had to say about it at NAB.

It's simple, it's easy. It's NOT a replacement for After Effects, Commotion, or combustion. But it is a great way to make quick motion graphics that integrate into the timeline in FCP HD.

Real time particle systems, gestural controls, behaviors (easy scripting!), super fast compositing, all kinds of good stuff.

Apple is going to have special demos of it on August 25th.

I see it as a great tool for creating motion graphics for DVD menus, broadcast graphics, etc. For film? I think it's capabilities will be limited for serious work. For starters, it is strictly 8 bit at this point. But I'll play with it and see.

So I'm up. Here's some stuff:

Apple releases OS 10.3.5 via Software Update. Do you dare install before the reports come in?

Server 10.3.5 was also released.

MovieOut lets you stream a file to a DV source, like a camera or hardware codec. Useful for seamless loops etc.

Alias (maker of 3D software) is buying Kaydara (leading maker of character animation/motion capture software). Not really an NLE kinda thing, but interesting to me (I used to do 3D animation with motion capture data).

Optibase, purchasers of Media 100, will show off a bunch of Mac OS X based MPEG encoding stuff at IBC. I'll check it out.

MacNN is carrying this story about Sonic buying Roxio. Sonic makes a bunch of market leading CD/DVD burning stuff on the PC side, and owns Napster. Roxio makes Toast, the king of Mac based CD & DVD burning.

Busy news day...

There. That oughta hold you for a while...

AppleInsider has this article claiming to have the scoop on the next version of QuickTime, version 6.6.

Finally, more/better audio channel handling, H.264 support (which is a video codec in the HD-DVD spec), and a bunch of other cool/useful stuff for high end workflows, such as, ahem, OURS.

So cool.

checkit.

So, extracted from what I just wrote about desktop DI work, realize this:

It's entirely possible and feasible to do 1920x1080 resolution, full 4:4:4 color bandwidth (is it YUV or truly RGB? YUV I think), at 10 bit color depth, uncompressed (to be more specific, NEVER compressed). You'd use a G5, an array (either SATA, SCSI to ATA bridged, or Apple X-RAID). You'd use Final Cut Pro HD, you'd use either an AJA Kona 2 or a BlackMagic DeckLink HD Pro card. Both can do dual link HD-SDI input for full color bandwidth (4:4:4 RGB).

Where would you get that kind of quality?

-You could get that from an HDCAM SR deck - but it's been compressed pretty heavily. Feh.

-Thompson Viper camera - it's just a camera, not a camcorder. Record directly to the array while on set. Cumbersome, but possible.

-Telecine from film direct to disk via dual link HD-SDI. How much of a hassle would THAT be?

But DAMN, the quality would be PHENOMENAL. Not a bad compromise, cost-wise, from 2K 10 bit log scans. Hmmm....

So anyway, once you've got it, that's one metric assload of footage - I'd consider doing some Cleaner/Compression Master tricks to generate a DVCPRO HD set of footage to do an offline edit with RT effects. (no 10 bit RT effects right now with FCP HD 4.5). That footage can go on a FireWire drive/s. Or perhaps you can set up a new comp and render straight to DVCPRO HD? My concern is that any change would require a re-render from the source 10 bit uncompressed footage, not from the DVCPRO HD footage. That's easy enough to test, I just need to take the time to do so.

So I'm sitting here on hold with the New Mexico Department of Transportation (curses on them for their just-over-the-border Fast Tax!!!!), and was thinking about Digital Intermediate work, and what it would take to do it.

Traditional 2K film scans are 2048x1536 10 bit log 4:4:4 RGB. Yes, this is 4:3 aspect. Anamorphic squish to get 16:9 output. This actually makes sense beyond the aspect ratio of shooting film, because the human eye is more sensitive to vertical resolution than it is to horizontal resolution. "Um, whatever Mike, get to the hard numbers." Ok, uh, yeah, right....

SO:

2048x1536 pixels in an 8 bit Photoshop document is 9 MB. This is 4:4:4 RGB colorspace. (8 bit film-speak, not web-speak, meaning 24 bit color RGB)

Multiply by 1.25 to take 8 bits per channel up to 10 bits makes it 11.25 MB/frame. Those 10 bits are probably logarhythmic, not linear, so a DPX or Cineon file. Log files store the brighter values more like film does.

Multiply by 24 frames per second = 270 MB/sec

Ouch!

But that's just the raw data rate - multiply by 20% to factor in "safety overhead" in your disk system: 270 x 1.2 = 324 MB/sec (let's round it to 325 to keep life easy)

So 325 MB/sec. Damn. No wonder it's so expensive to do DI work!

So what can do that?

Obviously we're talking about a disk array here, and a lot of disks. So let's talk interfaces first:

FireWire: Nope, it's out - even FireWire 800 taps out at 100 MB/sec in theory, and in reality about 85 MB/sec is the best any Mac is doing...and that Mac isn't a G5. So until/unless newer G5s fix this (the dual 2.5s might, maybe....), FireWire, even with multi-port PCI cards, are out.

ATA-nope, never mind. Thank you for playing.

SCSI - Even Ultra 320 would be tapped at or beyond it's theoretical limit of 320 MB/sec. So you'd need TWO Ultra 320 channels. Two Ultra 160 channels wouldn't suffice. The smart way to work with SCSI is to use SCSI to ATA bridging systems, such as are employed in arrays by Huge & Medea. But they are still pretty costly per gigabyte of storage. But SCSI scales up great, so it's a viable possibility.

SATA - my personal favorite these days, at least at the scale of stuff I'm working on...let's see: at present a 6 drive array is the best that's possible. Even with the fastest big drives out there, about 60 MB/sec is as much transfer speed as you'll get per drive, and that's at the fastest (beginning) part of the disk. About 35 MB/sec at the end of the drive. This would imply you'd need 10 drives to always get the throughput, and that's assuming no aggregate overhead slowdown from hooking that many drives together, and assuming that they can all work full speed the way they get hooked up. My sources tell me that there are some PCI-X 4 port SATA cards coming to Mac. If they work perfectly, and you don't saturate the bus, this solution might maybe work. For a partial solution (not fast enough across entire span of array) you'd need 6 drives. 6 drive arrays are possible now (using 2 internal SATA drives and 2 Seritek cards for 4 more drives), but not compatible with another PCI-X card installed. You're going to need some kind of adaptor to get the images in, so that doesn't work right now. But SATA is a dark horse contender. Although you can only get one drive per SATA connector, I've personally assembled a 6 drive (Raptor 10K 73GB) SATA array that delivered 390 MB/sec using the BlackMagic disk testing utility. But that would only offer 410 GB of space - only enough for 26 minutes of storage. No good. SATA doesn't scale well - the best stuff I see coming down the line would only allow for a maximum of 10 drives. Even with RAID zero (no backup/security if a drive fails), using the biggest drives on the market, that's only 4 TB of storage. But then again, that would (in theory, a big fat caveat here) let you do about 4 1/3 hours of 10 bit 2K files. That's a lot. Not nearly enough for all the files in a feature to be online at all times, but still, that's a lot. For starving indies, this MIGHT MAYBE work. More testing to see if feasible. So how do you view these monstrous files? You need a HIGH resolution monitor, and it needs to be calibrated to your target destination. The new Apple 30" LCD might be an interesting choice for this. Lower res proxies have proven to be viable for color correction, so an FCP workflow with a final render out might be viable.

Fibre channel - Fiber channel allows for a LOT of scalability, is wicked fast, and can have incredibly long cable runs. Especially with the Ultra ATA bridging option, it gives a LOT of performance for the buck. a 3.5 TB setup goes for around $12000 with the goodies you'd want. The Apple card is a dual channel 2 GBit card, which has a maximum theoretical speed of 250 MB/sec per channel, so a maximum theoretical throughput of 500 MB/sec. I don't know how much overhead Fibre Channel has, but it doesn't sound unreasonable to get 325 MB/sec in the real world out of a 500 MB/sec theoretically capable bus. So this sound like a good option.

So let's dance through some storage throughput requirements, just for fun, to see how quickly things add up (some of these figures do NOT include audio):

HDV - (less than DV, which is 3.5 MB/sec) - 1280x720p30 - I think something like 2.5 MB/sec

DVCPRO HD for 720p24 8 bit 4:2:2- 1280x720 (really 960x720 stretched) - 5.4 MB/sec

DVCPRO HD for 720p30 8 bit 4:2:2- 1280x720 (really 960x720 stretched) - 6.75 MB/sec

DVCPRO HD for 1080i60 8 bit 4:2:2- 1920x1080 (really 1280x1080 stretched) - about 14 MB/sec

Uncompressed 720p24 8 bit 4:2:2- 1280x720 - 42.3 MB/sec

Uncompressed 720p30 8 bit 4:2:2- 1280x720 - 52.8 MB/sec

Uncompressed 720p60 8 bit 4:2:2- 1280x720 - 105.5 MB/sec (this one I confirmed from FCP captured data)

Uncompressed 720p24 10 bit 4:2:2- 1280x720 - 52.8 MB/sec

Uncompressed 720p30 10 bit 4:2:2- 1280x720 - 66 MB/sec

Uncompressed 720p60 10 bit 4:2:2- 1280x720 - 132 MB/sec

Uncompressed 720p24 8 bit 4:4:4- 1280x720 -63.4 MB/sec

Uncompressed 720p30 8 bit 4:4:4- 1280x720 - 79.2 MB/sec

Uncompressed 720p60 8 bit 4:4:4- 1280x720 - 158.4 MB/sec

Uncompressed 720p24 10 bit 4:4:4- 1280x720 - 79.2 MB/sec

Uncompressed 720p30 10 bit 4:4:4- 1280x720 - 99 MB/sec

Uncompressed 720p60 10 bit 4:4:4- 1280x720 - 198 MB/sec

Uncompressed 1920x1080p24 8 bit 4:2:2- 95 MB/sec

Uncompressed 1920x1080p30/i60 8 bit 4:2:2- 118.6 MB/sec

Uncompressed 1920x1080p24 10 bit 4:2:2- 118.6 MB/sec

Uncompressed 1920x1080p30/i60 10 bit 4:2:2- 148.3 MB/sec

Uncompressed 1920x1080p24 8 bit 4:4:4- 142.3 MB/sec

Uncompressed 1920x1080p30/i60 8 bit 4:4:4- 177.9 MB/sec

Uncompressed 1920x1080p24 10 bit 4:4:4- 177.9 MB/sec

Uncompressed 1920x1080p30/i60 10 bit 4:4:4- 222.4 MB/sec

Uncompressed 2048x1080 12 bit 4:4:4 (the maximum possible dual link HD-SDI frame size and bit depth, SMPTE 372M) - 227.9 MB/sec

True 2K film scan - 2048x1536 10 bit log, 4:4:4: 270 MB/sec

True 4K film scan - 4096x3072 10 bit log 4:4:4: 1080 MB/sec, aka 1.055 GB/sec (!!!)

What the SMPTE DCI (Digital Cinema Initiative) is recommending for D-Cinema in the future: XYZ CIE color space 4:4:4, gamma 2.6, I _THINK_ they were saying it should be 4K:

So: 4096x3072 12 bit log 4:4:4 XYZ, 24fps - 1728 MB/sec, aka 1.6875 GB/sec

...so you can see how storage throughput gets kinda important. And that SATA or other desktop based storage solutions are clearly, massively, out of their league.

Right now, the fastest connection available that I've heard of (used by the Da Vinci color correction system, and them folks have BUDGET) is a quad link 2 Gigabit Fibre Channel link - theoretically capable of about 1 GB/sec if fully saturated with no overhead losses. So it makes sense that they can only get 10-15 frames per second, because the pipe isn't fat enough.

Even IF you could get two Apple Fibre Channel dual link cards, each of the 4 ports talking to it's own X-RAID bank (each X-RAID has 2 7 drive banks), you could get about 500 to 600 MB/sec out of it from what I heard out of the Apple guys. NOT ENOUGH. OK, what if you chain multiple ones together? With 4 X-RAIDs, if it scaled performance linearly, you could theoretically get there. Congrats, you've just spent about $50K...on STORAGE. But it would all be RAID 50, with a total formatted RAID 50 capacity of about 10 1/3 terabytes.

How much footage would that hold, if it were fast enough?

DCI suggested format D-Cinema footage: about an hour and forty-two minutes. Barely enough to hold the final copy of a feature. Yowza.

Traditional 2K 10 bit DI work: 10 hours, 40 minutes. Now we're talking - we're into the range of feasible storage for a feature.

SMPTE 372M - 2048x1080 12 bit 4:4:4 YUV: 12 hours, 38 minutes - again, suitable for a single feature.

1080p24 10 bit HD 4:4:4: 16 hours, 10 minutes - now we're getting into the realm of what's possible on the desktop, by the way. The new cards from AJA (Kona 2) and BlackMagic Design (DeckLink HD Pro) can do this resolution, bit depth, and color depth. Your storage needs are high - about 180 MB/sec, with safety overhead your array needs to deliver a minimum of about 215 MB/sec at all times to do this res.

Not only is this doable, it's doable with as little as a 4 drive SATA array (but not the full capacity of the array, so limited # of minutes of storage) for a movie short. A 6 drive array still wouldn't deliver the necessary data rate across the full span of the array, but using the 7K400 drives would allow for something like 2 hours and 45 minutes of storage that was fast enough. (6*370 GB formatted capacity = 2.22 TB of RAID 0 storage) Then you'd still have about 450 GB of offline storage - enough for more than 23 hours of 720p24 DVCPRO HD footage. That array would cost you something like $3200 when all the parts (the 4 port SATA card and those 7K400 Hitachi 400GB drives) are readily available - something like 30-90 days from now.

-mike

----------------------
HOW I GOT THESE NUMBERS: The vendor websites I don't trust - I got these numbers the brute, simple way - created an 8 bit Photoshop document the right pixel size. To account for 4:2:2 color space (my PShop doc is 4:4:4), I multiplied by 2/3. 4+2+2=8, 4+4+4=12. 8/12=2/3=0.6666.

To account for 10 bit color, I multiplied my 8 bit numbers by 1.25. 10/8=5/4=1.25.

To account for the difference from 24 to 30fps, I multiplied by 1.25. 30/24=5/4=1.25

To account for the difference between 30p and 60p, I multiplied by 2. Duh.

...and so forth.

UPDATED 2:45PM CST MONDAY: I'M GOING TO KEEP ADDING TO THIS THREAD AS WE CONTINUE (New stuff at bottom). CHECK BACK THROUGHPUT THE DAY/S FOR PROGRESS.

Got an email from a reader, of general interest so I'll post the discussion here:

On Aug 9, 2004, at 12:02 PM, Andrew wrote:

Mike-

What kind of Drive system should I get for the Varicam/FCP setup? I am now on a G4, waiting on dual G5 2.5 GHz.

Know any vendors who might sell OEM for PR purposes?

thanks,

-------------------------------

I wrote back:


Depends on a few things:

1.) Are you going to work with the Panasonic 1200A deck and the DVCPRO HD codec for offline work?

2.) How much footage are you going to have?

3.) Do you want to be able to do a final in uncompressed quality?

4.) What framerate do you anticipate working in?

Answer these and I can give more detailed instructions.

The quickie version:

If pure DVCPRO HD workflow, perhaps a single high speed SATA could suffice - such as the Maxtor Maxline III (300 GB max) or the Hitachi 7K400 (400GB max).FireWire drives could suffice for 720p24 work as well.

For uncompressed work, I found that a 4 drive Barracuda 160 GB (the 7200.7 model Apple ships) would work for any 720p work at 8 bits, 4:2:2 just fine, all the way to the end.

So if you're trying to be very efficient, start with an internal SATA drive (or FireWire drive) and do your "offline" DVCPRO HD work there, with RT FX, CC, and transitions. Or consider an external FireWire drive, preferably FireWire 800 with the Oxford 912 chipset.

When ready (if desired) to do uncompressed, use media manager to dupe the timeline and set up for uncompressed. Then you should use an array, moving your footage first somewhere safe (this can be cumbersome/difficult if was on SATA, unless working with FireWire 800 drive), then booting from a FireWire drive (perhaps the one you used for capture before), two internal drives, a Seritek card, two external drives in cases from Addonics or MacGurus. Format with SoftRAID.

Need more space? Consider the following drives as well:

Hitachi 7K250 (250GB)
Maxtor Maxline III (300GB)
Hitachi 7K400 (400GB)

All three of the above drives are even faster than the Barracuda array.

see this link also, it's important:

http://barefeats.com/hard40.html

----------------------

Andrew wrote back:


On Aug 9, 2004, at 1:49 PM, Andrew wrote:

720p, 24fps, captured via firewire on 1200A. About 200+ minutes of captured media. What do you think of the Lacie Bigger Disk Extreme?

Andrew

--------------------

I responded:

I haven't played with one, but they look very interesting and viable.

Might not even need all that space.

If working with 1200A, 720p24 is only 5.4 MB/sec. The 1TB, after formatting, would give you about 170 minutes of captured footage.

The 1 TB model wouldn't quite hold all your footage. The 1.6TB would, with room for FX/CC/etc. rendered files. Check the pricing - what's cheaper, the 1.6 TB model (check availability, too) or a 1TB and a 500GB?

Do you want to finish in uncompressed?

How long is your final piece expected to be?

BTW, I'm mirroring this conversation on the blog since it is of general interest.

-mike

--------------------

Andrew wrote:

On Aug 9, 2004, at 2:01 PM, Andrew wrote:

I spoke with a Panasonic rep who told me that capturing/editing compressed would yield superior results to an uncompressed capture as you are always dealing with native bits. The only way to beat this is to capture direct from the camera via HD SDI which yields a 1080p result (I think). I plan to do my whole project compressed, final out will be either 720p or 1080i. Final piece will be 25 minutes.

Andrew

------------------------------

I wrote back:

That camera is only capable of 1280x720.

You CAN get better quality out of the HD-SDI tap on the camera...but ONLY WHILE YOU'RE SHOOTING. In the process of compressing it to tape, it is downsampled to only 960 pixels wide (from 1280), then color downsampled to 8 bit, 4:2:2 YUV, then compressed with DCT (like JPEG does), then between frame compression is done. That's how it gets 720p24 down to 5.4 MB/sec instead of 42.

I've heard unconfirmed reports that you can get 10 bit out of the HD-SDI tap while shooting, but I need to research/verify that before I swear that works. But that's still 4:2:2 I think - although I want to check that out, too to be sure. I think dual link HD SDI is the only way to get 4:4:4 RGB (or 4:4:4:4) out of a camera.

I'm very curious to compare the codec decompression results between the camera's hardware decompression and Apple's software decompression. They are COMPATIBLE, but are they the SAME is what I want to know.

When capturing via HD SDI uncompressed, it's still digital, you're just relying on the hardware decompression in the deck to decompress the bits on tape to uncompressed bits over HD SDI. There is no analog==>digital conversion.

The D-5 decks can uprez 720p to 1080p, but I don't know the quality of that transformation. Personally, if it were my project, I'd be inclined to do a final version rendered out uncompressed, then use a high quality scaling algorhythm, such as Cleaner or Compression Master, to upsample to 1920x1080 to a new upsampled digital master. I'd send that digital master, in appropriate codec form, on drive/s to a post house for final transfer to tape.

I just noticed you might want 1080i final. Could do that, too, with Cleaner or Compression Master. D-5 decks can do that as well...but they are expensive to buy/rent/use.

A 4xBarracuda array could be had for as little as this:

3 more Barracuda drives (to supplement the 160GB Barracuda it came with) - 3x$110 (from zipzoomfly.com)=$330

Seritek 1S2 twin SATA PCI card - $65 (from lots of online sources)

2 external PPA, inc. SATA cases (from Fry's Electronics) - 2 X $50 =$100

1 external FireWire 800 drive - $200 (lots of choices)

SoftRAID 3.0.3 (or newer)= $100

TOTAL: about $800

...and would have 596 GB RAID 0 formatted capacity

BUT WAIT, YOU'RE DOING 25 MINUTES OF 720P24:

So a 2 X SATA array would work.

So buy one more hard drive, buy SoftRAID, buy a FireWire drive to boot from, and you're spending as little as $400 or so dollars. Use Carbon Copy Cloner (or similar) to dupe your boot drive to the FireWire drive (400 or 800, depending on your budget). So could be spending $110 for new drive and a FireWire400 boot drive.

Install the second drive, use SoftRAID to stripe into an array. (SoftRAID might maybe be skippable, haven't tested to see for sure what speed difference it'll make for FCP HD).

Since your footage is 720p24 8 bit 4:2:2, it's about 42 MB/sec. Using 20% overhead rule, you need 50 MB/sec at least always. Barracuda drives are good for that - they should deliver 2 drive RAID performance around 60 MB/sec even at the tail of the drives.

Two drives should give you right at two hours of 720p24 8 bit 4:2:2 uncompressed storage using the BlackMagic codec.

That's PLENTY to do what you want to do.

So there's your low budget solution.

Post Magazine has this article on DI (Digital Intermediate) process for independent filmmakers.

Note the equipment costs: the new, lower cost systems - $150K for the Nucoda Film Master, another $60K for 4TB of high speed storage.

Ouch.

But it lets him work with 2K (2048x1536) 10 bit log files, either Cineon or DPX.

Some interesting quotes from the article:

For those shooting Super16, Zak says, "The new Kodak Vision II film stocks, which minimize grain, can be further improved using Thomson's Spirit DataCine's Scream grain reducer. In our 4:4:4 uncompressed HD digital intermediate pipeline, we can now take a Super16-originated project, manipulate it and output it to 35mm with remarkable results."

"There's a big misconception with digital intermediates that you just scan in the film frames, do color correction and it all comes together automatically," says Kolbe. "In reality, the project consists of hundreds of thousands of massive digital assets that must be tracked, coordinated and managed. The file nomenclature, color space, file directories and other data must be handled consistently from the outset, or the problems that result can be costly."

"Top facilities such as Technicolor Digital Intermediate and Efilm have invested millions in their color correction theaters, and we're able to help them move more projects through their facilities by providing critical digital conform and asset management services," says Kolbe. The Digital Conform Group works in a complementary, non-competitive fashion with film labs, production companies and major post houses, helping everyone minimize bottlenecks and maintain efficiency for a more streamlined digital intermediate pipeline.

If you're at all interested in how to do great post for independent film, this is a must read article.

ITWorld has this article on Apple's upcoming Storage Area Network software, X-San.

It's not an inexpensive solution (until you look at your other options). It has a nice overview of what's involved, and what pieces are necessary. It's skimpy on details, but a good starter kit.

Why should you care? Because if you want to edit and do effects in a multi-user environment, this will let multiple people edit and work with high data rate content all at the same time from the same server over Fiber Channel at VERY high speeds (over 100 MB/sec).

Got an email from a friend, an NYC based editor (Frank Reynolds) whose comments sent me down a path. Here's an excerpt from my email back to him:

A thought on editing workflow - I wrote about the possibility of Motion/FCP and Premiere Pro/After Effects to do a poor man's digital intermediate process.

But for a "real" feature, the need to do color correction, effects, and editing all roughly concurrently, with the ability to change your mind a LOT all through the process, creates a very strong need and desire to do the whole gob - edit/CC/FX/DI - all in one system...either on one machine, or in a workflow that hands off from one step to the other incredibly easily and well...to the point that it's an import/export, or better yet a magic menu item that says "update" or "sync" or somesuch. AND be able to go BACK to the editing environment EASILY - not import and manually replace & line up. And if you're REALLY going to do it right, the ability to do it in a multi-user environment (OK, that makes it a LOT tougher depending on how you interpret that). The ability for multiple people to edit a film is doable - everybody edits a particular section. (High speed, server based solutions for footage get essential here - hellllllloooooo, X-RAID/X-SAN - you're my new best friend). The ability for multiple individuals to feed revised FX shots into th