- Edited
I'm writing a vertex displacement shader that needs to do some operations in view space. The problem is, if I just use MODELVIEW_MATRIX or VIEW_MATRIX then the shadows get messed up (I'm guessing it uses a different transform matrix when rendering shadows or something). As a work around, I thought I'd make a mat4 global shader variable to hold the transform of my main camera and update it in my camera script whenever the camera moved like so:
RenderingServer.GlobalShaderParameterSet("CameraTransform", GetCameraTransform());
This code works, but the matrix it's passing to the shader doesn't. If I try just replacing "VIEW_MATRIX" with "CameraTransform" in the shader code, the result is messed up. One problem is I'm not totally sure what the VIEW_MATRIX is supposed to look like. What I get from my camera script looks right from what I can tell, but clearly it isn't the same matrix since I'm getting different results. I also tried using the transpose and inverse of the matrix, but neither looked any better. I have some screenshots below of the camera transform and how it's being applied to the global shader variable, as well as a snippet of the relevant shader code.
//The code below lives in a shader include file and is called from various shaders
//vec3 vertex is the vertex passed in from the actual shader
//model is MODELVIEW_MATRIX
//view is VIEW_MATRIX
global uniform float ScaleFactor;
global uniform mat4 CameraTransform;
global uniform vec3 CameraPos;
global uniform vec3 CameraDir;
vec3 ScaleOffsetVector(vec3 vertex, mat4 model, mat4 view) {
//convert to World space
vec3 wv = (model * vec4(vertex,1)).xyz;
//convert to View space
//vec3 vv = (view * vec4(wv,1)).xyz;
vec3 vv = (CameraTransform * vec4(wv,1)).xyz;
//Scale the vertex positions in view space
float d = dot((wv.xz - CameraPos.xz), CameraDir.xz);
float s = max(d*ScaleFactor,1.f);
s = 1.f + (d*ScaleFactor -1.f ) * smoothstep(1.f/ScaleFactor,1.f/ScaleFactor+10.f,d);
vv.xy /= s;
//Convert back to model space
//mat4 w = inverse(view);
mat4 w = inverse(CameraTransform);
vv = (w * vec4(vv,1.f)).xyz;
vv = (inverse(model) * vec4(vv,1.f)).xyz;
return vv;
}
