Made of shapes, colours and a bit of story, dERIVATIVE is a short
film I’ve directed for a wonderful Mixpoint Studio in Prague. It follows
through a row of visual transformations and is likely to be more a work
of motion design than classical CG animation. The project was a
months-long effort and this time I had a chance to personally craft
every single pixel of the final film – what has really helped me is a
compositing-centered workflow which I’d like to talk about in this
It’s not uncommon among the film and animation 3D artists to overlook compositing as a mere slapping of few effects’ nodes to fancy up the picture or hide some mistakes in the end of the work. And sadly enough, game development seems to be almost ignorant of compositing as a discipline – at least that’s the impression I’ve been getting. Yet you can do almost anything to the picture in comp – build the whole animations from the ground up, do shading and look development and completely redesign the already rendered scenes – all at the much more interactive speeds than traditional 3D animation pipelines would allow.
The key is breaking up the renders into the passes, also known as Render Elements or Arbitrary Output Variables (AOVs). While there’s only that much you can do with a single beauty pass, the full set of AOVs unlocks the whole new range of options. Here’s my approach.
Shuffling the Elements In
The first step is to import the 3D renders into Fusion. I typically render to multi-channel EXR file format and instead of using multiple Loaders often prefer to shuffle the extra AOVs into the technical slots at the EXR Format tab. (Remember that channels are just numerical data and you can generally put any pass into any slot and easily extract or rearrange them later!) Then I’d use the Channel Booleans node to move these AOVs into the RGB channels and treat them as regular images further on. Minimizing the number of loaders makes it easier to update the source renders with new versions.
Stocking Up on the Masks
There are two things to generally look for when preparing the AOVs setup for heavy compositing. One is to make sure we can isolate any image area we might possibly need, either with a dedicated AOV matte or with some combination of channels. You might either use an automated solution like Cryptomatte or set up the mattes manually in my favourite old-school way – note the variety of masks that can be created by merging, multiplying or subtracting the individual Red, Green and Blue channels of these passes.
Recreating the Beauty Pass
The second crucial thing is to make sure the lighting passes assemble – when merged together they should match the beauty pass perfectly. The exact setup differs for every render engine and pipeline, but generally the lighting components are merged additively in linear-to-light colour space, while unshaded texture/colour passes are applied with “Multiply” blending mode. This is in fact quite logical since light does behave additively in the real world, while the colour of objects filters (or multiplies) the incoming spectrum. To achieve the additive mode in Fusion – set the “Alpha Gain” slider of the Merge node to zero.
Assembling the Look in Compositing
Now we can not only adjust the final picture, but directly manipulate (i.e. grade/recolour) individual lighting components like Diffuse, Reflection, Refraction or SSS before they get merged together – meaning we can actually move large part of the shading job into compositing! This provides incredible freedom as many big look decisions can be made interactively on the final image. Top left: Source 3D render before any manipulations. Bottom left and top right: Two shading options assembled from that initial render in Fusion. Bottom right: The final look – a mix of the two versions.
A Different Take
And here is a totally different look assembled from all the same 3D render. This one largely relies on Diffuse pass, while mixing in much smaller fraction of specular components like Reflection and Refraction. Light Wrap effect accents the softness of shadows, and the Primitive IDs pass makes for a great mask introducing some variation into the colouring of individual cells.
Pretty Endless Variations
Yet another example of the technique and of what could be built from our source render in the matter of minutes in compositing. As opposed to the previous look, this time it’s mainly the Reflection AOV. On top of it a small amount of Refraction pass and strong green colour applied to only one part of the object (thanks to the comprehensive set of masks exported) while introducing a contrast through keeping the rest sleek black.
Fighting the Artefacts
Such heavy manipulations of the source elements can often be pushing them to the limits. It goes without saying that 3D renders should have at least 16 bits per channel depth, floating point formats strongly preferred. The sampling is important too – any jitter and aliasing artefacts can become much worse after heavy compositing. Velocity Blur can help fighting some noise, but my personal favourite is the Reduce Noise plug-in from Neat Video – one of the frequency domain denoisers the film compositors have been using for years before AI denoising entered the 3D rendering scene. You can apply it at any stage – to either clean the sources or the final composite, and to fight the compression artefacts too.
Depth Of Field in Compositing
It’s much faster to apply depth of field to a scene as a post effect rather then raytracing it straight through. However, achieving clean Z-defocus in compositing is quite challenging and I’ve been experimenting with different solutions for years. The technique I eventually came up with relies on Fusion’s GPU-accelerated 3D environment. 1) Place an Image Plane in front and exactly matching any static 3D camera; 2) Increase the number of subdivisions and apply 3D Displacement to the plane based on the Z-Depth channel; 3) Set up the focal plane in the Camera and use the Render 3D node in OpenGL mode to render the defocused version of the scene.
The Finishing Touches
As the last step merge all the elements together with the background and complete the shot by applying the final effects. These may include vignetting (which is a powerful composition tool!); lens flares, distortions and chromatic aberrations; camera shake used to accentuate any particularly strong impacts within the picture and, importantly, film grain. While it’s best to keep the picture clean during the work, the touch of grain on the output adds in extra details and texture, as well as helps against the banding artefacts and excessive video compression. The word “touch” is crucial too – the bells and whistles work best when used sparingly and not overdone!
Advanced Masking Techniques
Not only Object IDs and designated matte renders can be used for masking – UVs, Normals, Rest and World Position data, Ambient Occlusion and pretty much any technical pass imaginable make for great matte tools too. (For instance, Normals could be used to mask all the surfaces looking up in the picture). Render elements can be keyed, graded, inverted, added, subtracted or multiplied just to name a few common adjustments – all to arrive at the perfect matte for the exact area we want to manipulate. I’ve covered these and other related techniques in the previous posts: