Shadow masking/layers not working as expected?

Alcalientre

So I have a scene with a tile based grid with a cursor that you move around, and based on the relative position of the cursor and the camera certain obstacle mesh instances get turned semi-transparent so you can see through them to where the cursor is. This all works great.

My issue is that I want to also have the objects continue to cast shadows while they are turned transparent, so I am dynamically creating a shadow_mesh duplicate of the original MeshInstance3D, putting it on layer 2, and setting the camera to not see layer 1. This all works perfectly. My transparency continues to work, and the duplicate mesh is not seen by the camera.

However, no matter what I do with the layers, caster cull layers, etc. in my DirectionalLight3D, the shadow meshes will not cast shadows.

I know the duplicate shadow mesh instances are there in the right place, because when I set the camera to see layer 2, they are properly there in the same place when the original mesh is turned transparent. And when I set the camera to see layer 2, they cast shadows.

Why would they cast shadows when the camera can see them, and not when the camera can't? I thought you were supposed to be able to use the visibility layers to have an invisible object cast a shadow? What am I misundertanding/doing wrong?

xyz

Alcalientre Why use additional instances though. You can use shadow_mesh property in the mesh resource.

Alcalientre

xyz

I didn't know this existed, I'll try this out. I'd still appreciate understanding why the other approach doesn't work.

Is it just a case of if the object isn't visible to the camera, it gets ignored for rendering and shadow mapping entirely no matter what? I though I remembered doing something similar to this before in a different project, so I really though it was possible.

Alcalientre

xyz

I don't know if I'm using it wrong, but setting the shadow_mesh property on the ArrayMesh of the MeshInstance3D to be a duplicate of the ArrayMesh doesn't do anything. It says shadow mesh should be the same as the arraymesh it's applied to, so what am I supposed to do to get it to cast shadows other than duplicate the mesh I already have? The documentation doesn't explain at all how to actually use this property. Any help?

xyz

Alcalientre If visual instance is invisible/culled it doesn't go down the rendering pipeline so it won't be rendered for shadow map too. Instead you can try setting visual instance transparency property to 1

xyz

Alcalientre How do you make your primary objects transparent? Is material assigned to the mesh resource or to the mesh instances?

Alcalientre

xyz

Ahhh thank you, I think this does it. from the documentation: However, unlike using a transparent material, setting transparency to a value greater than 0.0 (exclusive) will not disable shadow rendering.

This is what I want. I guess the other approaches just don't work. The shadow_mesh property isn't doing what I want.

Alcalientre

xyz

So I understand for the future, I had material override on the meshinstance3D, yes. So not just on the mesh resource. Is that why the shadow_mesh property wouldn't do anything? Does surface material override on the meshinstance3d apply to both the mesh and the shadow_mesh so setting it to transparent affected the shadow_mesh too and made it not cast shadows?

xyz

Alcalientre setting transparency to a value greater than 0.0 (exclusive) will not disable shadow rendering.

Well that's the whole point, I meant to use it on duplicate shadow meshes to hide them.

Alcalientre

Well that's the whole point, I meant to use it on duplicate shadow meshes to hide them.

Yes, thank you, I didn't know that property existed. I thought you could only affect transparency through materials. Now that I know this exists I don't need duplicate shadow meshes, since I can just make the original meshes the transparency I want and they still cast shadows.