Defining group memberships allows RenderMan to associate different objects and lights for special effects. For example, you can group a light with objects to prevent them from casting shadows from that specific light or even avoid reflecting something for artistic purposes.
You can achieve some of these effects in post compositing using Light Path Expressions (LPE), but you can achieve some of these effects globally and during the render. In doing so, your LPE will conform to these choices globally.
Trace sets are valuable for tweaking images artistically. Many images you see have been altered for the public. Blemishes get removed, distracting patterns get painted out, and even bad highlights and composition gets "fixed" based on art direction. When rendering these same rules apply. You may have had an art director tell you to remove a distracting shadow from off screen or erase a reflection from something too bright or obscuring. In some cases this is simple, but other times you need to remove the offending effect from just a single object or maybe a collection. Trace Sets are where you go to make this happen!
Trace Sets can vary per-instance and apply to single objects or groups. There are also a few ways to identify a trace set. For example, in Maya you can use a Maya selection set or material shading group to define a trace set for RenderMan.
You can set these groupings to exclude or include.
These are set on the object you wish to affect, not the object doing the affecting. To be more clear, in the examples below, let's say you wanted to keep the ball from reflecting in the mirror. I would select the mirror object and say not to trace the ball. Since the mirror is being affected, this is where I set the trace group to exclude the red ball, not on the red ball itself. Trace sets can be shared by multiple objects or transforms. Materials take the material name and would be shared by objects with the same material.
Below are some examples.
Standard beauty render – a couple objects on a plane and a mirror next to them
Here the mirror object has been told not to reflect the sphere or accept its shadow. Here it's using the exclude option on the mirror set to redSphere.
This is more subtle but you'll notice here that the green post now lacks the shadow cast from the sphere. The reflection also correctly omits this shadow. The green post was told to exclude the shadows subset of redSphere
Here I tell the red ball not to reflect the green post. Perfect example of art direction, "Can you get rid of that green reflection?"
This is a combination that shows you can be more flexible and creative. Inside the red sphere is a smaller blue diffuse ball. I've told the mirror not to reflect the red ball or its shadow. The blue ball is also told not to trace the red sphere (since it's inside, it would be totally black since the red ball would completely shadow it.) This lets you be creative in non-physical ways inside the render.
Caveat: Light sources may be blocked by hidden objects. Below we said not to reflect the red ball, and it doesn't but instead we get a black cutout against the dome light. To avoid this we set both the reflection and shadow sets to exclude the red ball on the mirror. This is because rays that don't reflect the ball are still blocked by the transmission/shadow rays.
Without telling it to exclude the shadow subset, transmission rays block light sources. This is only an issue if seen against a light indirectly, in this case the dome light.