VFX artists often need to composite the CG objects over a live action film plate. To convince the audience that the CG objects are real and belong to the scene, holdouts are used to collect the CG objects' shadow and reflection that match very closely to the shadow and reflection in the live action plate.
Holdout objects are used in place of real world object to catch light interactions from 3D objects. In RenderMan, holdout objects are specified by enabling the trace holdout attribute.
Integrators and AOVs
Holdouts are an illumination feature implemented at the integrator level. They are able to catch a variety of integrator-specific illumination effects such as diffuse and specular reflections, transmissions, and caustics. Available integrator that support holdouts are: PxrPathTracer, PxrVCM, and PxrUnified.
LPE workflow remains unchanged for holdout geometry.
Because holdouts workflow involves multiple AOVs, rendering with adapt all enabled can reduce noise at the cost of slower speed as it may trigger more samples based on noisier AOVs.
By default, light sources are considered holdouts and do not illuminate other holdout objects directly. This prevents double contribution between lights used to illuminate non-holdout geometry and the illumination already captured on the plate image.
To create a light source that does directly illuminate holdout objects, use "__illumholdout" as the light group suffix. This tells integrator plug-ins to consider this light as a non-holdout light source when determining illumination.
Alpha, Shadows, and Compositing
Holdout contribution can be composited over a film plate using an alpha and shadow pass as masks. The shadow pass is generated using the PxrShadowDisplayFilter plug-in and can be optionally written to its own AOV or added to the alpha. The occluded and unoccluded input AOVs should be rendered using the holdouts LPE prefix so that they only contain the shadowing term for holdout geometry.
Compositing can be interactively previewed in RenderMan by using the PxrImageDisplayFilter plug-in. This will composite the film plate with the beauty channel, using the alpha channel and holdout shadow AOV to mask the plate.
When compositing the image if the shadow matte is part of the beauty, it is a simple over operation. This will provide you with the correct shadow density and look based on the render settings.
However, this means adjusting the shadow density is tied to your beauty alpha, you will need a mask to prevent shadow adjustments from ruining your CG object composite. Cryptomatte may handle this for you as part of your usual arsenal.
If you output the shadow matte separately then you can adjust the shadow density separately, however you may have a coverage issue, the solution is to merge your alphas together in comp from the beauty and the shadow matte. An example using Nuke is here.