We'll have a section in the documentation for it. For now I can explain here. It makes animation clean up preserve more keys.
Normally animation clean up assumes the animation is applied on top of the setup pose
in other words, that the animation is the first one to be applied. Clean up will delete keys at the start of the animation that are the same as the setup pose.
When an animation is applied on top of another animation, those keys that are the same as the setup pose may be needed to override the animation below. Without this Layered check box, you can't use clean up because it will delete those keys. Now you can check the box and still use clean up.
