HD For Indies

After sitting down with Ted to catch up on the camera and business sides of Red, I thought I'd catch up with Graeme Nattress, the man in charge of the codec and software side of the Red project. Graeme couldn't make it to LA for the screening the other week, so we conducted this Q&A via email. There's one big fat new exclusive piece of info you should be sure to read about: RPL, aka Red Pull List. I've bolded it where it is mentioned below, it is our Secret Sauce to conform projects. OK, on with the show:

Q: So Redcode - tell the uninitiated what it is:

A: Redcode is a compression scheme designed especially for the RED camera. It's a wavelet based compression that works in 10bit or 12bit (the camera is 12bit natively) and with RGB or RAW data. When you're using 10bit mode, you have the choice of either a standard Log curve, or a REC 709 gamma curve, so that you're immediately compatible with whatever NLE or DI program you're using.

Q: ...and now that Redcode has recently been announced to be 12 bit capable, what can that do for us as users?

A: Well, it means less processing has happened to the RAW data - it's as RAW as we can make it, and hence all of it is preserved with no loss of highlight or shadow detail. That means, you can push and pull the image more freely in post without worrying about loosing quality. With 12bit, each RAW pixel uses a range of 0-4095 to represent it's value. (Mike note - most HD tape formats are 8 bit, using a range of only 0-255, or the highest end/most expensive tape formats 10 bit, using a range of 0-1023.) That makes for finer graduations, less possibility of banding, and as it's linear, is already in the best form for compositing.

(Mike addition - I'd previously challenged whether this was of any use in a non-VFX environment - while tools like After Effects 7, Combustion, Shake, fusion, etc. can work with source material with more than 10 bits of precision, most NLEs are limited to either 8 or 10 bits of precision at best. Jim pointed out to me that when you're doing your basic color balancing/correction in REDCINE, it is processed in 32 bit float on the GPU in REDCINE - so you WOULD gain the benefit of that greater precision at this earliest stage when you're making what is most likely your most aggressive color decisions. So 10 or 12 bit source, manipulate in 32 bit float, then deliver the best possible results to generate your working copy for offline or online. As a bonus, if you decide you need to push it in a dramatically different direction later, you can go back to the source and reprocess it again, from source, to get different yet still optimal results - the cleanest possible workflow, starting from the RAW source data)

While proofing the article, Graeme schools me further:

Tape formats use only the 16-235 range for legal values, allowing up to 254 for super white. In 10bit, it's 64 - 960 + superwhite up to 1022.

Q: I noticed that when the announcement was made that Redcode was now 12 bit capable, the target data rates weren't changed. Is that affecting the image quality? Or are you just tweaking the compression figures?

A: We're improving the compression in other areas to compensate. That's working very well, and that, combined with the efficiencies of RAW, and that when there's less processing applied to data, there's less extra noise introduced, it is compressing great. One of the things that people forget, is that all the traditional image processing that gets applied to video detracts from it's ability to be easily compressed. Adding sharpness or edge enhancement, colour processing, or gamma all add noise or make the image harder to compress. By leaving the RAW data in as RAW a state as possible, we can achieve the our quality and compression ratio goals.

4.) Originally, 720p, 1080p, 1080i, 2K, 4K, 4.5K were discussed as camera possibilities - which of these can Redcode record on the Red Flash or Red Drive?

A:

REDCODE Does:
720p, 1-120fps
1080p, 1-60fps
2K, 1-60fps (windowed or scaled, as I understand it - mike)

REDCODE RAW Does:
2k RAW 1-60fps (windowed as I understand it - mike)
4k RAW 1-30fps

FYI - REDCODE is the RGB codec - it isn't RAW, it has been demosaiced/deBayered. But what about 1080i? Did it go away? This broadcast standard will be important to support.

1080i50/60 Interjection:

I got suspicious when I didn't see 1080i60 listed as supported in Graeme's answer, so I followed up with Stuart English, who as always Knows All:

Q: Ah - so using REDCODE RGB  or 1080p60 (windowed or downsampled), in Redcine you could be kicking out a very nice 1080i, right?

But it sounds like there won't be a 1080i60 codec for direct recording, correct? 

A: There really is no point, compression is easier and has less artifacts using 1080p  as its source v's 1080i. At the end of the day, interlace is a transmission path compression, so long as we can create it for a deliverable (including HD-SDI outputs) that's all we should worry about.

Q: So if/when native NLE support comes along, in theory it could just be flagged to deliver its content in this fashion.

A: Yes, when 1080p / 60 NLE's are available, 1080i could be output on the fly - just as 3:2 is added to 24 fps material today. Again, when in a file based workflow, go the legacy tape / broadcast interlace format at the last stage possible, keep it in progressive format(s) as long as possible. 

Q: Or, instead of dropping every other field, just super sample for ultra clean, correct?

A: Perhaps, but dropping odd then even fields in alternate frames easier to do for low power apps.

Q: And 1080i60 could be derived with a crop & process within REDCINE from a 2K 60p as well, correct?

A: Sure, it could be done there. 2K RGB or 1080p RGB to 1080i are both fairly straight forward

Q: Questions questions questions!

A: Think Digital Cinema (film) Negative and Telecine / Lab and most questions get answered ! 

End Stuart Interjection. Also, 1080i50 is easily derived from 50p recording, which of course Red One can do as well same way as 1080i60.

Q: So what is the plan for recording greater than 60fps 2K scaled or windowed?

A: 2K RAW at 120 fps and 4K RAW at 60 fps require external recording. That would be via the RAW Data Port (optical high speed port). Recording solutions for the Raw data port are under development, and there will more than likely be 3rd party partners that will have a number of different options on this front.

(Mike clarification - 2K RAW at >60-120fps and 4K RAW at >30-60fps is another way of thinking of those numbers)

One of the first goals of the 4k RAW compression goal was to be able to record 24p, which we believe is the right choice considering the RED ONE’s goal of becoming a true digital replacement for traditional 35mm film production. We also understand, especially from our friends in Europe, that 4k50p would be, for instance, very desirable. And 60fps 4k on board for slow motion would also have great benefit to our customer base. Although our current specs don't allow us to record higher than 4k30p at the moment, we're actively engaged in research to see how to make it possible. As you know, we're a small team, and have to prioritize our development efforts, and for 4k RAW that means 24fps, 25fps and 30fps, but we've certainly not forgotten about the higher frame rates for either superb slowmo, or for general ultra-high definition television use.

Q: I've heard that with wavelet, it is easy to extract lower resolution during decode, rather than decode whole thing then scale results - true?

A: Yes indeed. It's one of the key benefits of wavelets and how the wavelet transform works.

Q: Redcode on a computer - will it play back in real time?

A: Yes, if your computer is fast enough. How fast is fast enough? Well, we're still in development, and the computer industry just keeps improving things a lower and lower prices.

Q: OK then - Redcode on a computer - will it play back scaled down in real time?

A: Yes indeed. I've played 4k RAW 24p scaled down in real time on my laptop, and it looked great.

OK, switching gears, lets talk about

REDCINE: for the uninitiated, what is it?

A: REDCINE is a fully functioned software application to bring in your REDCODE footage, adjust raw conversion settings, and export to practically any format supported by your system. 

(Mike addition - Graeme has previously also discussed how Redcine will have some basic, one-light style color correction tools - the website mentions white balance, gamma, gain, color, saturation, contrast, & curves. No heavy tools like vignettes or secondary color correction, just the kind of stuff you might do in a one light telecine session - but all done on GPU, in 32 bit float, starting from that 10 or 12 bit source.)

Q: What kinds of file formats will be supported?

A: On import, only RED formats, but export will be to Quicktime or DPX, Cineon, Tiff, BMP etc. for still formats. Additionally, we can take advantage of most extra codecs installed on your system to output to DV or DVCProHD for example.

(Mike addition here - compressed and uncompressed Red formats supported, and for export, if it is a codec that is installed on your system that is normally available for QT aware apps (in the Mac case) to write to, you can use it - that's the caveat. As for still file formats, the website lists TIFF, DPX, JPEG2000, CINEON, PSD, JPEG, and probably others as well to follow. It has been previously discussed, ad nauseum, that any bit depth you have a codec for can be supported - 10 bit 4:2:2, 10 bit 4:4:4, 16 bit 4:4:4, 8 bit 4:2:2, whatever - no artificial boundaries.)

Q: What frame sizes will be supported?

A: It works on all frame sizes supported by the RED camera, so from 720p up to 2540p (4.5k), and can export to whatever size you want, including SD resolutions for PAL and NTSC.

Q: Will it be possible to crop as well as scale?

A: Yes

Q: What kind of conversion times are we looking at?

A: Hard to tell. All depends on what you're doing. We're actively engaged in optimization to get users the best experience we can, and are making full use of GPU and multiple CPUs.

Q: You mentioned single core of a G5 for 1.7 sec 4.9K debayering - isn't Redcine going to be strictly Macintel and Wintel? Why, if you have a running build on a G5, not suppport that platform going forward?

A: Well, development work is on PPC as that's what I have for the most part, but REDCINE is a completely different beast, and people will only really understand when they see it. The costs involved in it's development are significant, and we just don't have the time or the money to develop for, an unfortunately dying platform. However, all the other tools we're developing, like the Quicktime REDCODE and REDCODE RAW codec are cross platform and PPC also.

Remember that demosaic time was only mentioned because people had heard it takes 4 or 8 seconds to demosaic a frame from the (other vendor name omitted - and these 4-8 sec demosaic times were from THEIR year old dev code anyway - mike). The (Red's) demosaic used was development code and will be running much faster by the time it makes it into REDCINE. Another important factor is that you don't want to sacrifice 4k quality by using a very quick and low quality demosaic.

And this brings me to one of the important advantages of using RAW as a digital negative: You can gain the benefit of better conversion algorithms and software developments in the future, and use it on footage you already have. Once the camera is out, it's not like we're going to stop working on improving the RAW conversion, and I expect to have a number of small R&D projects that will address various aspects of the RAW conversion process and bring either speed or quality enhancements to the pipeline

Q: Will conversions times vary based on desired size?

A: Yes indeed. Smaller frame sizes will go quicker.

Q: Will there be a "draft mode" for faster conversions?

A: Yes indeed.

Q: Random question - would there be any use to supporting the HDR OpenEXR format?

A: Perhaps as it might make it easier to handle the 12bit linear dynamic range. If I remember correctly, OpenEXR is supported as an output format from REDCINE.

Q: Will REDCINE be bundled with the cameras?

A: Yes.

Q: Will it be available as a standalone application, and if so how much will that app cost?

A: Yes it will, but the precise details are still to be worked out.

Q: It has been said that with Redcine, you can convert to offline, edit, then come back and convert for online - but other than manually coming up with a list to re-convert (which could be ridiculously daunting on a feature), how will conform be more realistically doable? As in automated? I see this as a HUGE potential stumbling block to the proposed workflow.

A: We're looking to automate this with RED Pull List, which will take a NLE EDL or XML file, and create a new project in REDCINE with just the clips you need to tweak or re-render out to a higher resolution. RPL is still in early stages of development, but I think you can see how this will help enormously.

Q: Great, that's a huge missing piece I was worried about. As a follow up, is RPL a separate app or a function of Redcine? Or TBD?

A: RPL is a function of REDCINE, and the exact details of it's operation and interface are still being worked on.

Q: OK, moving on - you previously said, if I understood correctly, that the 4K RAW could be played in realtime at 1K resolution on a reasonable machine in realtime. Does that imply that a 1K extraction to file could be done in 2x or less realtime, depending on codec used? What guidelines can you give us about time here? It is a potentially large issue in terms of dailies, deadline driven environments, etc.

A: 1k extraction is indeed realtime on even my laptop, so conversion to another codec should be very quick. Remember 1k is 1024x576 which is the square pixel spec for wide PAL, which means it looks like superb SD.

Q: I noticed the sample REDCODE RAW stills posted over on CML are 1.5 MB, but multiplying that by 24 (for 24fps shooting) would generate a 36 MB/sec data rate, well in excess of the 27.5 MB/sec rate Red has quoted as a target in the past.

A: The 27.5 MB/sec number is for 4K, and these frames are 4.9K.

(4900 x 2580) / (4096 x 2304) = 1.333

1.5 / 1.333 = 1.125MB/frame, which is 27MB/sec for 24fps.

-Graeme

(As I understand it, the compression math is slightly fudgier than that - bigger images don't require quite proportionally greater datarates, but it is a pretty reasonable approximation considering it is still developmental code at this time.)

The following snippets I lifted from Stuart's comments on DVXUser.com's excellent and robust Red forums (with permission of Jarred Land, site owner):

------------

DVXuser: Undercranking - slower than 1fps is possible, but needs to be tested to see what happens when the shutter is left open for long periods of time.

DVXuser: Framing, sampling, and aspect ratios:

4.5K is RAW, means sampling is RAW (not 4:4:4) and Super 35 framing
4K is RAW, means sampling is RAW (not 4:4:4) and Academy 35 framing
2K if RAW, means sampling is RAW (not 4:4:4) and Super 16 framing
2K if RGB, means sampling is 4:4:4 and Academy 35 or Super 16 framing
1080p if RGB, means sampling is 4:4:4 and Academy 35 or Super 16 framing
1080p if HD, means sampling is 4:2.2 and Academy 35 or Super 16 framing
720p if RGB, means sampling is 4:4:4 and Academy 35 or Super 16 framing
720p if HD, means sampling is 4:2.2 and Academy 35 or Super 16 framing


DVXuser: Primary recording options - (all specs subject to change):

4K RAW, compressed with REDCODE. Recorded to RED-DRIVE or REDFLASH. Frame rates - variable 1 - 30fps.

4.0/4.5K RAW, uncompressed. Recorded via high speed data port to external custom disk array or RAM disk. Frame rates - variable 1 - 60fps.

2K RAW, compressed with REDCODE. Recorded to RED-DRIVE or REDFLASH. Frame rates - variable 1 - 60fps.

2K RAW, uncompressed. Recorded via high speed data port to external custom disk array or RAM disk. Frame rates - variable 1 - 120fps.

2K RGB or 1080p compressed with REDCODE. Recorded to RED-DRIVE or REDFLASH. Frame rates - variable 1 - 60fps. From 4K RAW or 2K RAW original frame.

2K RGB or 1080p uncompressed. Recorded to external disk array over dual link HD-SDI. Frame rates 24 and / or 25fps. From 4K RAW or 2K RAW original frame.

720p compressed with REDCODE. Recorded to RED-DRIVE or REDFLASH. Frame rates - variable 1 - 60fps from 4K RAW original frame, 120fps from 2K RAW original frame.

DVXuser: 4K to 2K RAW in camera

The answer to the 4K to 2K in camera scaling question is 4K RAW to 2K RGB is possible, but that 4K RAW to 2K RAW is not. However we do understand why people are asking for this, and if it can be worked out how to do it, then we'll let everyone know.

....as an interesting sideline to that, there has been conjecture online about shooting compressed full sensor width, but not full height, in order to get a 'scope aspect ratio (2.35 or 2.40). Just based on the volume of pixels, in THEORY that sounds doable - instead of 4096x2304=9.32 megapixels (current 4K default), shooting 4520x1884=8.5 megapixels. I don't know what other limitations might get in the way, but just based on the number of pixel per second to process, shooting 24p with this size and aspect may be a possible option in the future. I think the team is just focused on fulfilling the existing stated specs right now, this is just conjecture for the moment. But perhaps in time this might be an added capability.

DVXuser: Format Choices
In all of these discussions also don't overlook the possibilities with our 2K RGB. As its scaled in-camera from 4K RAW, it has the same FOV and DOF with the advantage for s/n and dynamic range of being a 2:1 oversampled image. Its also recordable to a RED-DRIVE via internal REDCODE RGB compression at any frame rate between 1 - 60fps - including speed ramps.

Also note: The format options chart at http://red.com/formatoptions.htm has also been upadated today (written 11/7/06).

Graeme chipped in right after with this post on DVXuser: And with a high quality scaling algorithm, 2k RGB scaled to 4k should look good. It helps that scale is the only difference - FOV and colour being the same. And that you're progressive too - interlace is horrible for scaling.

As for Frame Ramping?: Stuart wrote on DVXuser: "Yes, RED can frame ramp - either in a smoth S-linear speed up / slow down fashion or via an instantaneous rate change every frame (where we follow an externally generated command set)." Mike notes: no details of what that system might be, where it might come from, how it may work, etc....but the camera will be capable of ramping. Can't help myself but to say it - COOL!

Somewhat off topic, while pulling notes, I stumbled across this one I'd missed on still lens mounts that Jarred Land wrote:

DVXuser: Red Camera.. first test with Still Lens ( Nikon )Me and Jim put the proto Red-Nikon mount on Franky, mounted a DSLR Nikon Prime, and took some test shots. We were very pleased with the results.

All of you praying that you could use Nikon Still lenses on Red, your day just got alot better.

Lens: Nikon AF Nikkor 50mm 1:1.4 D Manual aperture at f4.0 ( Some Nikon glass, including new models, allow for manual aperture changes.)

here is a still.. 4.9k downrezed to 2k and turned into a medium quality 8 bit JPG - http://www.24puser.com/nikon.jpg

-----end snippets-------

Thanks again to Jarred of DVXuser.com for letting me lift all this good info from his site. If you're interested in, or have a question about the Red, or the DVX100/A/B, or the HVX200, his site is an excellent place to go, if not THE place to go (here isn't bad either for the HD stuff I have to thow in ; ) )

As always, read the rest of those postings and the full threads to get a more comprehensive understanding of what's going on - these were just some nice highlights and summaries and pull quotes.

IN CLOSING

So this is all good news. 12 bit source material, best possible processing space, GPU and multiple CPU acceleration to minimize transcoding times, bundled with the camera, etc. All good stuff.

The matchback/conform issue was the one that had me most concerned - without it, they don't have a viable workflow for larger projects without creating a logistical nightmare.

With a tape based workflow, when it is time for your online, you refer back to the source tapes and just use the same tape names and timecode in/outs to acquire the high quality source material. It is just a matter of feeding in the right tapes when prompted, and waiting for the batch process to capture all the desired footage from each tape.

But in an all digital workflow...there's no batch capturing to be redone, there's only batch copying to be done. And no NLE I'm aware of really has a toolset to help you go find all that media. Let's say you went so far as to assign a removable drive name instead of a reel number in your NLE...the program STILL isn't going to wait for you to hook up drives and copy over the media - NLEs just aren't built to work this way (nor if you'd archived to data tape and wanted to retrieve directly from there, a more convenient yet less likely possibility to work smoothly with current toolsets).

An RPL (Red Pull List) would help this process along. I've shared my own thoughts with the Red team on the nitty gritty of how I'd like this functionality work, starting with an EDL or XML file from your offline edit that is ready to go online, all the way through to an online project with media all linked up. But if it is doing what he says, creating a new Redcine session with your EDL/XML derived selects - then if you're using FCP, it'll be cake to Create Offline using Media Manager and relink to the high res footage that the RPL sets you up to render out.

If you have thoughts on it, or concerns you'd like to see addressed, feel free to use the Comments link at the bottom of this article - the Red guys will read it, trust me.

Next up - analysis of the images posted on CML - 74 MB 16 bit TIFF images right off the Mysterium sensor (after debayering/demosaicing and some minor gamma adjustments), with and without Redcode compression, with and without a matrix applied, etc.

UPDATE: One significant note in that updated chart - clearly, that while the sensor is Super35mm sized, the ONLY way to record using that full size is with the RAW data port...for which Red has no solution that'll ship when the camera does - and it sounds like the REDRAID might be sliding downwards in priority (see Saturday's Ted Schilowitz interview & analysis). In any case, double check the chart before assuming the camera does something you thing it might. The chart is a bit complicated, but study to let it all sink in.

-mike

UPDATE: Oops, I wasn't "teh phreshest" after all - I wrote this over several days and awaited some feedback from the Red team before posting, looks like I wasn't first out of the gate on the Red Pull List after all - after writing the article but not posting it yet, Stuart posted yesterday (Saturday):

A few comments about workflow using REDCINE. - DVXuser.com -- The online community for filmmaking

...and discusses "pull list" in context of Red. Drat, not first at it after all.

He talks about:

-generating pull list from EDL/XML/AAF after editorial complete
-pull list is just the final list of selects for re-export at final online res
-cropping, scaling, changing color values can be done (original color values saved)
-other clever nifty uses of the "look" metadata up and down the production chain
-ongoing discussions with other vendors to use this metadata - sounds promising

Read on, this is only from the first post in the thread which I'm sure will grow.

Other posters immediately pick up on the obvious need for subclips - what if you had an hour long take (interview, doc, etc.) and only need 5 seconds of it? You wouldn't want to reconvert the whole shot, just the selection you used in your timeline.

Ideally, you could generate a new EDL/XML/AAF that would help you rebuild your online file. REALLY ideally, since this is all file based, it could be done via distributed rendering and would create the online file automatically via script assist to control the NLE once all the files were cooked to online res.

I'm posting/commenting/discussing over there as well.

Gratuitous Plug - I've already started doing some consulting for folks interested in small and large scale, Red based workflows - what equipment necessary, what storage and workflow options at different price, speed, and resolution points, etc. based on the preliminary info available and the specific needs of their proposed projects. If you'd like help in these or other high definition related areas, I am indeed available for consulting - mike at hdforindies d0t com.

-mike