The “RealFlex” method for 3D Volume reshaping:

As a Stereographer, I have closely followed and seen the evolution of stereo 3D in the past decade. From anaglyph (red-blue 3D) to autostreoscopic (no glasses needed) and then to the latest incarnation of full color Digital Cinema 3D. Creating content for Autostereoscopic 3D displays was not an easy task. If current 3D cinema uses 2 cameras to shoot 3D and Cinematographers still face challenges…imagine shooting 8 or 9 views as was needed for the existing generation of autostereo 3D screens from the likes of Stereographics (now RealD) Newsight and Philips (who since then closed down that operation).

As a 3D content producer, I had to innovate back then in 2004. Producing 3D CGI content was easy with “virtual cameras” in 3D software, but the real challenge was with Live or Real content, you could not just put 8 cameras side by side and hope to shoot 3D that way.

So without further ado… here’s a closely guarded secret that is being shared, in the hope that it benefits stereographers, VFX artists and 3D movie makers in their search to produce better 3D.

(Select 480p to read the annotation on Youtube: here or if video takes long to load)

Creating a Virtual 3D Interaxial in post:

• Get Re:Vision Effects Re:Flex plugin for Adobe After effects CS5
• Open up After Effects, load in your left and right eye video footage
• Lay out the footage on two layers, and throw on a color corrector on each. Turn down the blue and green channels on the top layer (left eye) and turn down to zero the red channel on the layer below (right eye)
• Set the top layer to 50% opacity. This shows a quick and dirty anaglyph 3D view in the output composition window
• With the rectangle tool draw a simple mask to encapsulate the entire left eye layer (top layer)
• Next, drop in the Revision Effects –>Re:Flex Motion Morph plugin on the top layer.

Settings for Re:Flex morph plugin:

Follow the settings from the image above.

• The ‘Warp “to” Layer’ should be set to the layer below (right eye).
• All blend settings should be set off in the plugin itself.  Do NOT use any blend.
• The opacity setting of 50% as mentioned in the steps above are for anaglyph preview only.
• Now move the “Global Warp” slider to create a new Interaxial for the left eye view. A good starting point is between 15% and 20% warp.
• Drop another version of the Re:flex motion morph plugin on the Right eye view, and do the same
• Select  ”warped and blended” from the drop down menu and render out the new re-converged left eye.
• Do the same previous step, for the Right eye view to render out the new re-converged right eye view. (for right eye set the global warp slider to the same amount as left eye)

The right eye could be the same, or in some scenes if the Global Warp was set to say 15% for a new left eye view, then set the Global Warp to 15% (on the right eye instance of the plugin) and render out the “Warped and Blended” from the drop down as the new right eye. In some cases balancing it this way will even out any visual artifacts in a scene.

Important: This is not a simple H.I.T (horizontal image translation) being performed, as can be seen by the fact that the left and right edges are not sliding. It’s an actual re-compute of pixels, creating the effect of a new view-port onto the scene. There are high end systems such as Mistika and software such as Nuke with the Ocula plugin that can do such re-convergence, each using their own methodology… but at a significantly higher cost.

I would still say that this method produces (in my opinion only) a far superior interaxial re-adjust in post.

Tweaking the slider, along with toggling the “hold edges” button will allow for varying the results depending on the scene. Of course as with most After Effects plugins, almost all parameters can be animated over time and *should* be under the careful supervision and advice of a good Stereographer…from a shot to shot basis, so that the Depth of the scene and the 3D movie is not compromised.

An Example:

Above: The original footage, camera was parallel rig, no intentional toe-in or converge used, and interaxial was approx 3.5 inches.
View on Youtube in 3D and HD quality.

Above: The RealFlex method applied with a 25% warp on both the left eye and right eye view. Watch the Video in anaglyph mode on youtube to see the change in interaxial.
View on Youtube in 3D and HD quality.

Above: The RealFlex method applied with a 35% warp on both the left eye and right eye view. Watch the Video in anaglyph mode on youtube to see the change in interaxial.
View on Youtube in 3D and HD quality.

Observations and Thoughts:

I have tried this approach on many different scenes, and in combination with other tools and methods (HIT, rotation, cropping and floating stereo windows), in my experience it has saved many shots that would otherwise be unusable, or in the case of filming in 3D, would have been impractical, and would need a bulky beam-splitter 3D rig.

You do lose about 0.5% of the top of your frame due to shear / distortion and the amount of warp applied. However, if shooting with 4K cameras such as the RED or even with 2K cameras such as the Si2k Minis, You can simply keep in mind the framing or use the full sensor and crop as needed. In cases such as RED, if using this method, a simple HIT could be employed, if scenes are framed properly to account for stereo window edges.

The Re:Flex plugin’s original use is for something completely different! Even the makers of this excellent application don’t know the potential yet and what it means for the 3D industry! It would be wise to get a copy of this software just in case prices rise!

Other similar approaches could be done with software that use motion vectors, and software that does re-timing or slowmotion effects on footage. Though these are based on motion vector calculations, so one needs to evaluate their effectiveness on a scene by scene basis.

A question: Is re-converging a 3D scene this way akin to producing fake 3D? To a certain extent, yes we are synthesizing new pixel information, but unlike 2D to 3D conversion, we are synthesizing new pixels from information that is already there in the scene. However I admit, I’m not totally convinced that it’s vastly different from 2D to 3D conversion. More thoughts on conversions below.

The reason to Re-shape “3D Volume”:

The most important reason for shooting with a beamsplitter 3D Camera rig, is that it allows for Interaxial (the distance between the two lenses of a 3D camera) to be set to anything from Zero cm to human averages of 6.5cm and more if needed.

Due to the physical profile of today’s professional cameras, if they are placed side by side, they far exceed the typical human eye spacing.  There is also an obsession with today’s 3D movie makers to insist on squeezing the 3D volume down to almost flat 2D like space under the mistaken premise that it makes for “safer” viewing.

They are actually correct to an extent for this way of thinking. We do not want to do harm to people watching 3D, getting headaches and giving up on 3D yet another time. However, simply reducing the interaxial or “converging” cameras on a subject of interest, is not the answer. This would make a nice deep room look like a squashed shoe-box, if interaxial is set to less than human eye spacing for instance.

Good stereoscopic 3D is equal parts “art” and equal part “science”. There is no catch-all safe way to produce good 3D with formulas and calculations alone. That would be the same as producing a painting, using a paint by numbers approach. Careful placement of objects, actor and camera blocking is what is called for. (In Live events, it’s a whole different thing however and thus justified.)

Attempting to faithfully re-create or record the depth and emotion of a real world scene is what should be attempted. After all, 3D movie makers this time around are calling for less use of gimmicks aren’t they? How is this possible if one shoots with 3D cameras with 2D like settings for depth?

Stereoscopic 3D Trade secrets:

Working with a good Stereographer is the need of the hour. Most good stereographers started off their careers / hobby many years before today’s movie industry and current technology caught up. This is not to say that older stereographers are better, in fact they may not be! Just as some Cinematographers find it hard to un-learn 2D film school lessons, some older stereographers themselves may be tempted to hold on to set ways of producing 3D content.

The trick is to find Stereographers and Cinematographers who are willing to take their rich past experience and marry it to new technology, both hardware and software to advance the art of stereoscopic 3D moviemaking.

There are many tips, tricks and trade secrets in making of 3D movies that are not readily shared. For instance, 2D to 3D conversion houses boast of “patented” methods of converting 2D footage. This is questionable, to the extent that, there is no secret algorithm for unlocking 3D depth from 2D footage (yet). The usage of “macros” to semi-automate some of the workflow is possibly as far as “patented” goes. All good conversions for instance, are based on depth map creation, extraction, rotoscopy and then there are some artifact ridden approaches such as off-set painting as seen in a few movies where foreground pixels get stuck to the background. For the record I am for 2D to 3D conversion as a VFX tool that’s much needed for 3D movies, but not as an excuse to blanket convert a 2d movie.

** Update 21st March 2012**

A new software Emotion3D, promises better. Read about it here: http://bit.ly/GDb6Cx

** Older Updates **

I’ve had the chance to test the RealFlex method on more scenes, in full HD. There are some scenes where additional distortion processing is needed. Some scenes will need a mixture of H.I.T and a max setting of no more than 20 to 25% warp. Some kind of barrel distortion is visible on vertical lines, but not in all scenes. Other scenes for example with huge “gap” between foreground and background with no mid-ground depth, may need masking to avoid artefacts.

Conclusion: The RealFlex method is not for every scene imaginable, as is the case with most workflows. However for a majority of shots it should work very well with variation of “warp” settings, and in combination with H.I.T to allow for quite a bit of side-by-side rig shooting.

Note to Hardware and Software manufacturers of Stereo 3D Systems:

If you have produced or are about to introduce new systems for furthering the art of stereoscopic 3D and I believe in it, I would be happy to evaluate, and evangelize it at the seminars, conferences and workshops I do. Forward thinking organizations such as Frame Forge already lead the way and are contributing to an important aspect of 3D film making – previsualization in 3D