In-Three on the Workflow Behind 3D Conversions
All About Tools, Process, and Costs of ‘Dimensionalization’
By Bryant Frazer

James Cameron’s Avatar and Tim Burton’s Alice in Wonderland constituted the one-two punch that convinced Hollywood — and everyone else under the sun — that there’s a lot of money to be made in releasing stereo 3D entertainment to movie theaters. In fact, the case made by Avatar’s box-office gross was so forceful that studios started forcing 3D tentpole releases into their schedules left and right, with Warner Bros. raising more than a few eyebrows by announcing in January that its late March release of Clash of the Titans would be converted to 3D post-haste (and pushed back a mere week to compensate). That decision pointed to a new frontier in Hollywood tech, with its own gold rush for companies equipped to do the 3D conversion work.

In-Three, one of the companies with solid experience in this realm (it converted the guinea-pig action movie G-Force to 3D last year and has demo’d some impressive scenes from the original Star Wars for industry audiences), converted the live-action “real world” material bookending Alice in Wonderland, adding depth to everything except the carriage-ride scene and a dance sequence. Film & Video got In-Three VFX Producer Matthew DeJohn and VP of Business Development Damian Wader to sit for a Q&A addressing the tools, the workflow, and the costs involved in “dimensionalizing” (the term refers to In-Three’s patented Dimensionalization process) everything from cinema commercials and feature films to movie trailers and TV programming.

Film & Video: Tell me how this process works. Is there a little black box with inputs and outputs and a red button that says “Dimensionalize!”?

Damian Wader: We wish it worked like that!

Matthew DeJohn: It breaks down into three broad artistic phases. One is segmenting the image. You break it up in terms of major layers of depth. You cut out a person [in the scene], maybe somebody else behind him, and then you have the background. The next stage is actually generating the depth for that scene. You model the scene, and our system allows you to create a new perspective, or see the scene from a new perspective once you’ve modeled it out in 3D. And once you are finished with that phase, you’ve revealed parts of the background that weren’t visible in the original perspective, so you have to paint those surfaces, inserting image information where there was none.

F&V: The fact that this is happening as a moving picture instead of a still image makes that complicated in terms of tracking and roto work.

MDJ: Absolutely. And there are special cases like transparencies. How do you get that to work in stereo? Smoke particles and sparks are always difficult. [So is] motion blur. The detail and transparent nature of hair has to be captured accurately to fool the brain into thinking it’s real 3D.

F&V: Are you always using the original, flat version of the scene as one eye, and building a second, synthetic eye-view to go with it?

MDJ: That’s one way to do it. Often, especially in cases when you deal with smoke and other transparencies, it’s easier to recreate both perspectives. If you remove all the smoke by painting it out, you would have a new background and you would have to insert new CD smoke. Even for more normal shots, you’re doing a lot of visual effects type work. A live-action 2D-to-3D conversion, even if it’s a drama, is turned into a 100 percent visual-effects show. To get the really nice shaping to people’s faces and to avoid it looking like a cardboard cut-out, you need to model the scene with pretty good detail.

F&V: This sounds like a completely hands-on process. Is there any way to automate, say, a first pass? Or does an artist have to be working on every frame of film?

MDJ: There are some automated techniques and processes out there, but we found they just don’t get to the level of quality necessary when you’re creating the actual depth choices for the scene. At the end of the day, the artist has to analyze and interpret the scene. ‘OK, where is this object in space?’ A computer can’t do that for you. There are some approaches that will assist an artist who is rotoing or keying certain elements. But if a character has bushy hair, we have to pull a key that’s as good as you could pull if it were on a green screen. That’s a significant challenge when you have a character baked into the rest of the scene. There’s not a lot of automation that’s up to snuff.

DW: In terms of the depth itself, those values are propagated fairly automatically [in our system]. Once the choices are made, it’s not as if you’re re-keyframing every frame. Depth-grading is a fairly quick process.

MDJ: That’s a good point. Once we’ve set up the depth of the scene, our adjustment period to tweak that is very quick. For Alice in Wonderland, we submitted an entire sequence that they thought looked great but it wasn’t the artistic direction they wanted to go. We revised the depth on that entire scene in a day or two.

F&V: Is that information kept in metadata attached to the shots? How do you keep it with the files?

MDJ: It’s inside of our internal software, called In3gue [pronounced “intrigue”]. We have our artists working in real time to construct the depth and then tweak it to where [the filmmakers] want to go. When there’s a full shot, in motion, with depth, that’s the first time the director sees it. We submit shots to be approved for depth, and they have rough mattes and rough paintwork. Once the depth is locked down, we final all the fine-tuning stuff, detailed key work and detailed paint work.

F&V: How big is the team for the 2D-to-3D conversion?

MDJ: Generally, for a 100-minute or 120-minute 2D-to-3D conversion, you would need about 300 to 400 artists phasing and out of production over about four to six months.

F&V: Are those 400 artists all working on dimensionalization? That’s not at all a trivial process.

MDJ: It’s very serious visual effects work. We have a lot of ways to attack the process that make it easier for us to get people up to speed and control the quality, but it takes a lot of effort to get a very high quality product.

F&V: How often do those people have to look at shots in 3D, and how do they do that?

MDJ: Internally, we’re working in 3D the entire time, save creating roto splines and stuff like that. For our depth artists, 50 or 75 percent of the day is spent looking through 3D glasses. We’re submitting stuff for review in 3D internally, because we’ll go through three or four internal reviews before we send to the client.

Example of roto work in Mocha for 3D conversion

Not an image from Alice, but an example of what the first phase of roto work for a 3D conversion looks like in Imagineer Systems’ Mocha. (Click to view full-size.)

F&V: I understand a lot of work was done on Alice using Imagineer Systems Mocha. Are there aspects of Mocha that make it especially well-suited to this process, or could any high-end tracking-and-roto software be used?

MDJ: We really like Mocha, and it’s really easy to teach our people how to use it. At the time we made the choice, it was the only one that offered planar tracking. It’s been very powerful and good to us, so that’s what we continue to move forward with. Other companies are implementing forms of planar tracking, but Mocha dovetailed nicely with our internal software.

F&V: And how big is In3gue’s role? Are there more proprietary tools involved in the process?

MDJ: We have other, smaller, tools in our proprietary bag and stuff in development, but In3gue is our main software and where a lot of our IP lies. That really is the depth-creation tool. It’s unique. No other software does what we’re doing. It’s designed to model a scene quickly and provide quick feedback.

F&V: And the big question: how much does this cost?

DW: Typically, it’s about $80,000 to $100,000 a minute. That’s generally speaking. As we move forward through development, we can define things that make it more efficient for us and more cost-effective for the production. A lot of animation houses are coming to us. They want to deal with their 2D story, not stereo or CG. Some projects do come to us as legacy pictures, but when we’re involved in the beginning, it’s the best way to take advantage of the process. It will not affect principal photography as a whole, but it will help our efficiency.

MDJ: Because dimensionalization is a post process, you get more artistic control compared to shooting [in stereo]. We can tailor the depth to match perfectly with the edit. No shots are ever lost if the camera fails. We can go into a stereo show and dimensionalize a [flat] shot to match the rest of a scene. It provides more flexibility in your depth choices. But you are limited by the realities of the world. If you shot with a long lens, you can only get so much shape out of the character’s face without having the scene way too deep. We can go in and make specific choices and use the available depth budget to its best advantage.

DW: Everything we do is to help the creative get his vision out there. It’s not to take away at all. We want to work with creatives who want to use 3D as a storytelling tool.

A still from G-Force.

F&V: Do you ever get a shot from production that’s just a bear to add depth to?

MDJ: We had a shot on G-Force that was shot through clear plastic freezer curtains, and that was unbelievable. It’s distorting the background, and you have to still be able to see through it in stereo! It’s incredibly hard. We wish they didn’t do that. [Laughs.] But we know we’re going to encounter that stuff. In fact, our pipeline was originally set up to deal with legacy work. But it becomes an issue because it’s more expensive to deal with those shots.

F&V: I was curious whether you would be consulted during a shoot, like a VFX supervisor.

DW: We’re sort of looked on as the red-headed stepchild of the industry. But we’re starting to get some play as VFX supervisors. Two years ago, we were an afterthought, but now we’re getting clients saying, ‘We need to get you guys in and utilize your expertise in the development of this.’ We were on set for one picture, and I think we’re going to be on set a lot more.

F&V: Does everything happen in-house at In-Three, including roto and tracking, or just the depth-adding process in In3gue?

DW: The projects so far were done here. We have a joint venture with Reliance Media Works [at a dedicated facility in India], and half of our staff are ramping up our outsourcing capabilities. We’re exclusively using them to outsource, and they’re exclusively using us to dimensionalize. The key artistry will remain here – keyframing, approval and QC — and they will handle the bulk of the manual labor — the roto, the key-pulling, the paint.

F&V: How do you expect your work to evolve in the future across features and television work? What about hybrid projects that were only partly acquired in 3D?

DW: This company was designed to repurpose old libraries, and to date we haven’t done that yet. We’ve been working on pictures where we were an afterthought, and then Alice thought of us and got us involved before the movie started, so that is occurring. We’re going to see shooting and dimensionalization happening inside single projects moving forward. It will be the best of both worlds to use both tools. They have their own domain in terms of broadcast TV, sports — stuff that’s live. We have our domain in terms of legacy pictures and repurposing old libraries, but it comes together on film projects moving forward, repurposed for TV broadcast or direct to DVD [in stereo]. Another one is advertising, especially theatrical advertising. We did the Air Force commercial that ran in front of Avatar on about 1000 screens, and we’re getting a lot of advertising companies coming at us. And there will be trailer work. I get calls every once in a while for 3D trailer work, even for 3D pictures that, for whatever reason, need trailers converted to 3D.

F&V: It was several years ago that I first saw 3D-converted footage from one of the Star Wars movies at an industry gathering. Can you say anything about the status of 3D Star Wars?

DW: You know as much as we do. George [Lucas] has said he wants to do all six films. Five years ago, when we showed our stuff at ShoWest, he said he was going to do it. He hasn’t made any commitment as yet, but we’re as anxious as you are.

F&V: One last question about dimensionalization. Alice in Wonderland got mainly good reviews for its 3D visuals. But as soon as Clash of the Titans started screening for fans, there was a backlash. On the Internet and in some print reviews, a buzz started building that 2D to 3D conversions are no good. [Writing in the Los Angeles Times, Kenneth Turan said, "Consider the possibility that Clash of the Titans is the first film to actually be made worse by being in 3-D."] How do you feel about that?

MDJ: You bring up Alice as a contrast to that, which is good. I would go back even farther, to G-Force. I don’t think anybody knew that was converted. They all assumed it originated in 3D, because it was really clean. In terms of Clash of the Titans and all of the negative press, we look at that situation and see an eight-to-10-week schedule to do an entire movie, and it sounds completely unrealistic to get a high-quality product in that timeframe. If that’s how it turns out [and 3D conversions get a bad reputation], it will be disappointing for us. We’ve been there at the forefront, talking about our dimensionalization process specifically as an extremely viable and artistically competitive approach to generating 3D. If it doesn’t get there, hopefully that doesn’t paint all conversion houses with the same brush. That’s what we’re afraid of. The audience should be able to trust they’ll get a high-quality product, especially on a huge movie.

—

요즘 영화계에서는 3D 입체 영상이 세계적으로 가장 핫이슈이다. 이 시장의 파이를 누가 먼저 가져가느냐로 경쟁이 치열하다. 하지만, 난 아직 불편한 안경을 쓰고 조금은 눈 아프기만 한 이 3D 입체 영화의 매력을 잘 모르겠다.

She barely opens her eyes. She shows facial expression that I can’t understand.

Yet, she is The Beauty.

I know my life will be totally different from before. More responsibility.

It can be a burden but also a joy.

Now I don’t know what to do with this little baby when she cries. I will get better. I will get used to.

I’d like to be a good dad.

I love you my two angels, my wife and daughter.

상현이 자기 자신에게는 물론 신자들에게 가르치며 암송하는 이 기도는 자신의 모든 것을 버리고 세상의 가장 낮은 곳에 임하고자 하는, 모든 개인적 욕망을 버리고자 하는 기도이다. 그리고 영화 는 바로 이 기도문에서 출발한다.
상현은 정말 뱀파이어이긴 한걸까? 상현은 자신이 뱀파이어라고 하지만 누가 그것을 증명할 수 있을까? “박쥐”는 뱀파이어 영화의 장르적 규칙을 하나하나 모두 성실히 따르진 않는다. 관객은 상현의 입에서 돌출된 뾰족한 송곳니를 볼 수 없다. 상현은 십자가가 난무하는 성당을 자유롭게 드나든다. 그러나, 상현은 뱀파이어와 마찬가지로 피에 무척 끌린다. 흥미로운 것은 그가 피의 섭취를 게을리하면 어김없이 그의 몸은 흉측한 수포로 뒤덮히고, 피를 먹는 순간 수포는 씻은 듯이 사라진다. 우리는 아직까지 이러한 뱀파이어의 모습을 어디에서도 보지 못했다. 새로운 모습의 뱀파이어의 탄생일까? 어찌보면, 상현은 뱀파이어가 아니라 (비록, 자신은 자신이 뱀파이어라 확신하지만), ‘이브’라는 바이러스에 의해서 정상적이 아닌 특수한 신체적 능력을 가지게 되었으며, 그 부작용으로 인해 몸이 수포로 뒤덮히는 현상을 겪고 있으며, 혈액만이(혹은 혈액 속에 있는 어떠한 성분) 이 부작용을 치료해주는 특수한 병에 걸린 것일지도 모른다. 다시 한 번 질문을 하자. 상현은 뱀파이어인가? 뱀파이어가 아닐 수 있다. 그러나, 상현이 뱀파이어가 아닐 수 있다는 것에는 동의하더라도 여전히 영화 “박쥐”는 뱀파이어 영화라고 답할 수 있다. 이건 또 무슨 얘기? 뱀파이어 영화는 에로티시즘과 욕망에 관한 영화이다. 아름답고 순결한 여성, 그녀의 하얗고 가녀린 목, 남성의 신체에 위치한 뾰족히 튀어나온 날카로운 송곳니, 그의 거부할 수 없는 매력, 그리고 원색의 빨간 피. 이러한 뱀파이어 영화의 요소는 영화 속에서 에로티시즘을 현상화하고, 남성의 욕망을 구체화한다.
영화의 초반부는 상현의 갈등을 농밀하게 보여준다. 그러나, 상현의 갈등은 자신이 뱀파이어가 되어 피를 마셔야 한다는 사실에 대한 갈등, 즉 뱀파이어로서의 갈등이 아니다. 그의 갈등은 오히려 그동안 성직자로서 억제하고 감추어왔던 인간으로서의 육체적 욕망에 빠져드는 자신의 모습을 보며 느끼는 인간으로서의 도덕적 갈등이다. 병상에 누워있는 오랫동안 알아왔던 사람의 피를 빼내어 마실 때 상현은 아무런 갈등을 느끼지 못한다. 상현은 오히려 그도 이해하고 좋아할 것이라고 말한다. 그의 욕망은 태주를 만나며 폭발한다. 태주의 땀에 젖은 끈적한 목덜미를 보며 그동안 성직자로서 억제해 왔던 성적 욕구가 브레이크 없이 터져나오는 것이다. 그 경계에서 갈등을 하던 상현은 결국 성직자에서 (뱀파이어가 아니라) 욕망에 가득 찬 남자의 모습으로 넘어와 친구의 아내인 태주와 몸을 섞고, 그녀와 함께 친구를 살해하며 원하는 여자를 손에 넣는다. 그리고 영화는 그 이후 원하는 것을 얻었지만 오히려 더 불행해지는 상현을 혼돈 속에서 보여준다.
“박쥐”는 온갖 욕망으로 가득차 있다. 상현의 피를 마심으로써 감각을 회복하여 마지막으로 한 번만이라도 육체적 쾌락을 맛보고자 하는 노신부의 욕망. 상현을 구원이라 믿고 그를 통해 병을 치유하고 고통을 없애고자 하는 신자들의 욕망. 태주를 힐끔힐끔 쳐다보며 그녀와의 하룻밤을 상상하는 남자들의 욕망. 자신의 아들을 죽인 상현과 태주를 죽이고자 하는 엄마의 욕망. 한복집에서 벗어나고자 하는 태주의 욕망. 태주를 갖고자 하는 상현의 욕망. 이 모든 욕망이 살을 섞으며 창조하는 세상이 영화 “박쥐”이다. 상현의 욕망에 대한 내적 사투가 농밀히 그려지는 초반을 지나 태주 역시 상현에 의해 뱀파이어가 된다. 태주는 상현과 달리 자신의 욕구를 거침없고 죄의식 없이 충족시켜 나간다. 상현과 태주. 혹은 갈등 속의 나약한 인간과 욕망의 뱀파이어. 태주를 보며 상현은 그제서야 자신이 더이상 가지 말아야 할 길 위에 있다는 것을 깨닫는 듯하다. 그렇다면, 여기에서 그들 모두의 욕망은 중지되고 억압되어야 하는 것일까? 노신부는 상현의 손에 죽고, 남자들은 상현과 태주에 의해 죽는다. 그리고, 상현과 태주는 아침 해 속에서 목숨을 버린다. 그들은 죽음으로 구원을 받은 것일까? 죽음이 구원이라면 그럴 수 있겠으나, 죽음은 구원이 아니라 고통의 연장일 뿐이다. 상현은 마지막 순간 태주에게 말한다. “태주씨를 사랑했지만, 지옥에서 다시 만나요.” 그들은 죽음을 통해 구원이 아니라 또 다른 고통 속으로 들어갈 뿐이다. 혹은 태주의 말처럼 구원이나 그런 것이 아니라 그냥 거기서 ‘끝’인 것일지도 모른다. 그렇게 그냥 거기에서 욕망이 충족되지 못한 채 끝날 뿐이다. 신자들은 어떠한가. 상현은 마지막 순간 그들에게 자신의 추한 모습을 보임으로써 자신은 그들이 믿는 구세주가 될 수 없음을 강변하여 그들을 구하고자 한다. 과연 그 후로 신자들은 자신들의 무지함을 깨달았을까? “박쥐”에서 유일하게 자신의 욕망을 충족하고 온전히 구원받는 것이 확실한 사람은 바로 엄마이다. 아들을 죽인 상현과 태주가 자신의 눈 앞에서 죽는 모습을 지켜보며 엄마는 바로 자신의 욕망을 충족시키고 동시에 구원받는다.
욕망의 지옥도. “박쥐”는 그렇게 뱀파이어 영화이다.

It’s time to get rid of fear.