VFX artists often need to composite the CG objects over a live action film plate. To convince the audience that the CG objects are real and belong to the scene, holdouts are used to collect the CG objects' shadow and reflection that match very closely to the shadow and reflection in the live action plate.
Holdout objects are used in place of real world object to catch light interactions from 3D objects. In RenderMan, holdout objects are specified by enabling the trace holdout attribute.
Integrators and AOVs
Holdouts are an illumination feature implemented at the integrator level. They are able to catch a variety of integrator-specific illumination effects such as diffuse and specular reflections, transmissions, and caustics. Available integrator that support holdouts are: PxrPathTracer, PxrVCM, and PxrUPBP.
LPE workflow remains unchanged for holdout geometry.
Because holdouts workflow involves multiple AOVs, rendering with adapt all enabled can reduce noise.
By default, light sources are considered holdouts and do not illuminate other holdout objects directly. This prevents double contribution between lights used to illuminate non-holdout geometry and the illumination already captured on the plate image.
To create a light source that does directly illuminate holdout objects, use "__illumholdout" as the light group suffix. This tells integrator plug-ins to consider this light as a non-holdout light source when determining illumination.
Alpha, Shadows, and Compositing
Holdout contribution can be composited over a film plate using an alpha and shadow pass as masks. The shadow pass is generated using the PxrShadowDisplayFilter plug-in and can be optionally written to its own AOV or added to the alpha. The occluded and unoccluded input AOVs should be rendered using the holdouts LPE prefix so that they only contain the shadowing term for holdout geometry.
Compositing can be interactively previewed in RenderMan by using the PxrImageDisplayFilter plug-in. This will composite the film plate with the beauty channel, using the alpha channel and holdout shadow AOV to mask the plate.