Make a shader ignore specific objects

Devkure

Hello everyone! 😊

I'm encountering a specific issue in Godot, and I'd love some guidance. I followed this tutorial: to set up an outline shader, and it works as intended. However, a problem arises when I add foliage, which uses a separate shader from another developer (thank you, kind shader wizards!). Here’s what happens:

Whenever I view the scene from the ViewportContainer or from a certain distance within the scene with the "Post Processing" outline shader applied and visible, the objects with the foliage shader appear unusually dark. I suspect this might be due to the way the outline shader operates, but I’m not certain.

My Setup:
Outline Shader: Attached to a 2x2 quad mesh with the “Flip Faces” option on. This quad is a child of the camera.
Foliage Shader: Applied to a material, which is then used on a mesh that imitates a bush in the scene.

I’m still new to Godot and shaders, and I know I’m probably combining things in an unconventional way, but I'm eager to learn. I do not claim that these shaders are mine - i openly admit, this is other peoples work that i simply try to stitch together to form something... My knowledge of both godot and shaders is very limited, and the topic seems really scary to me overall.

Is there any way to prevent the outline shader from affecting the foliage shader, or a workaround to fix the darkening issue? Any advice on what could be causing this interaction or how to separate the effects would be extremely helpful.

Thank you so much in advance to anyone who takes the time to help – I really appreciate any non-judgmental guidance! 😊

Code of shaders and screenshots below:

OUTLINE

shader_type spatial;
render_mode unshaded;

uniform sampler2D screen_texture : source_color, hint_screen_texture, filter_nearest;
uniform sampler2D depth_texture : source_color, hint_depth_texture, filter_nearest;
uniform sampler2D normal_texture : source_color, hint_normal_roughness_texture, filter_nearest;

uniform float depth_threshold : hint_range(0, 1) = 0.05;
uniform float reverse_depth_threshold : hint_range(0, 1) = 0.25;
uniform float normal_threshold : hint_range(0, 1) = 0.6;

uniform float darken_amount : hint_range(0, 1, 0.01) = 0.3;
uniform float lighten_amount : hint_range(0, 10, 0.01) = 1.5;

uniform vec3 normal_edge_bias = vec3(1, 1, 1);
uniform vec3 light_direction = vec3(-0.96, -0.18, 0.2);

float get_depth(vec2 screen_uv, mat4 inv_projection_matrix) {
	float depth = texture(depth_texture, screen_uv).r;
	vec3 ndc = vec3(screen_uv * 2.0 - 1.0, depth);
	vec4 view = inv_projection_matrix * vec4(ndc, 1.0);
	view.xyz /= view.w;
	return -view.z;
}

void vertex() {
	POSITION = vec4(VERTEX.xy, 1.0, 1.0);
}

void fragment() {
	float depth = get_depth(SCREEN_UV, INV_PROJECTION_MATRIX);
	vec3 normal = texture(normal_texture, SCREEN_UV).xyz * 2.0 - 1.0;
	vec2 texel_size = 1.0 / VIEWPORT_SIZE.xy;

	vec2 uvs[4];
	uvs[0] = vec2(SCREEN_UV.x, min(1.0 - 0.001, SCREEN_UV.y + texel_size.y));
	uvs[1] = vec2(SCREEN_UV.x, max(0.0, SCREEN_UV.y - texel_size.y));
	uvs[2] = vec2(min(1.0 - 0.001, SCREEN_UV.x + texel_size.x), SCREEN_UV.y);
	uvs[3] = vec2(max(0.0, SCREEN_UV.x - texel_size.x), SCREEN_UV.y);

	float depth_diff = 0.0;
	float depth_diff_reversed = 0.0;
	float nearest_depth = depth;
	vec2 nearest_uv = SCREEN_UV;

	float normal_sum = 0.0;
	for (int i = 0; i < 4; i++) {
		float d = get_depth(uvs[i], INV_PROJECTION_MATRIX);
		depth_diff += depth - d;
		depth_diff_reversed += d - depth;

		if (d < nearest_depth) {
			nearest_depth = d;
			nearest_uv = uvs[i];
		}

		vec3 n = texture(normal_texture, uvs[i]).xyz * 2.0 - 1.0;
		vec3 normal_diff = normal - n;

		// Edge pixels should yield to the normal closest to the bias direction
		float normal_bias_diff = dot(normal_diff, normal_edge_bias);
		float normal_indicator = smoothstep(-0.01, 0.01, normal_bias_diff);

		normal_sum += dot(normal_diff, normal_diff) * normal_indicator;
	}
	float depth_edge = step(depth_threshold, depth_diff);

	// The reverse depth sum produces depth lines inside of the object, but they don't look as nice as the normal depth_diff
	// Instead, we can use this value to mask the normal edge along the outside of the object
	float reverse_depth_edge = step(reverse_depth_threshold, depth_diff_reversed);

	float indicator = sqrt(normal_sum);
	float normal_edge = step(normal_threshold, indicator - reverse_depth_edge);

	vec3 original = texture(screen_texture, SCREEN_UV).rgb;
	vec3 nearest = texture(screen_texture, nearest_uv).rgb;

	mat3 view_to_world_normal_mat = mat3(
            INV_VIEW_MATRIX[0].xyz,
            INV_VIEW_MATRIX[1].xyz,
            INV_VIEW_MATRIX[2].xyz
	);
	float ld = dot((view_to_world_normal_mat * normal), normalize(light_direction));

	vec3 depth_col = nearest * darken_amount;
	vec3 normal_col = original * (ld > 0.0 ? darken_amount : lighten_amount);
	vec3 edge_mix = mix(normal_col, depth_col, depth_edge);

	ALBEDO = mix(original, edge_mix, (depth_edge > 0.0 ? depth_edge : normal_edge));
}

FOLIAGE

shader_type spatial;
render_mode depth_draw_opaque, specular_schlick_ggx, depth_prepass_alpha ;
//render_mode blend_mix, cull_disabled, depth_draw_opaque, specular_disabled;

uniform vec4 TopColor : source_color = vec4(0.24, 0.47, 0.27, 1.0);
uniform vec4 BottomColor : source_color = vec4(0.13, 0.33, 0.25, 1.0);
uniform sampler2D Alpha;
uniform vec4 FresnelColor : source_color = vec4(0.58, 0.65, 0.33, 1.0);

uniform float WindScale : hint_range(1.0, 20.0) = 1.0;
uniform float WindSpeed : hint_range(0.0, 20.0) = 4.0;
uniform float WindStrength : hint_range(1.0, 20.0) = 5.0;
uniform float WindDensity : hint_range(1.0, 20.0) = 5.0;
uniform float ClampTop : hint_range(0.0, 1.0) = 1.0;
uniform float ClampBtm : hint_range(-1.0, 0.0) = 0.0;
uniform float MeshScale : hint_range(-5.0, 5.0) = -0.333;
uniform float ColorRamp : hint_range(0.05, 5.0) = 0.3;

uniform float FaceRoationVariation : hint_range(-3.0, 3.0) = 1.0;

uniform float FresnelStrength : hint_range(-2.0, 2.0) = 0.5;
uniform float FresnelBlend : hint_range(-1.0, 1.0) = 1.0;
uniform bool DeactivateGlobalVariation;
// Uniforms for wiggling
uniform sampler2D WiggleNoise : hint_default_black;
uniform float WiggleFrequency = 3.0;
uniform float WiggleStrength = 0.1;
uniform float WiggleSpeed = 1.0;
uniform float WiggleScale = 3.0;

uniform float DistanceScale : hint_range(0.0, 5.0) = 0.5;
uniform float DistanceStart = 0;
uniform float DistanceScaleRange = 70;

vec2 rotateUV(vec2 uv, float rotation, vec2 mid)
{
	float cosAngle = cos(rotation);
	float sinAngle = sin(rotation);
	return vec2(
		cosAngle * (uv.x - mid.x) + sinAngle * (uv.y - mid.y) + mid.x,
		cosAngle * (uv.y - mid.y) - sinAngle * (uv.x - mid.x) + mid.y
	);
}

varying vec3 obj_vertex;
void vertex()
{
	float distanceScale = 1.0;
	vec3 world_pos = (MODEL_MATRIX * vec4(VERTEX, 1.0)).xyz;	//Generates world coordinates for vertecies
	vec3 distance_vector = world_pos - (INV_VIEW_MATRIX * vec4(0.0, 0.0, 0.0, 1.0)).xyz;
	float square_distance = distance_vector.x * distance_vector.x + distance_vector.y * distance_vector.y + distance_vector.z * distance_vector.z;
	float square_end = (DistanceScaleRange + DistanceStart) * (DistanceScaleRange + DistanceStart);
	float square_start = DistanceStart * DistanceStart;
	float square_range = square_end - square_start;
	
	float distance_influence = clamp((square_distance - square_start) / square_range, 0.0, 1.0);
	//Camera-Orientation based on https://www.youtube.com/watch?v=iASMFba7GeI
	vec3 orient_2d = vec3(1.0, 1.0, 0.0) - vec3(UV.x, UV.y, 0.0);
	orient_2d *= 2.0;
	orient_2d -= vec3(1.0, 1.0, 0.0);
	orient_2d *= -1.0;
	orient_2d *= MeshScale;
	orient_2d *= (1.0 + distance_influence * DistanceScale);
	
	//random tilt
	float angle = 6.248 * UV2.x * FaceRoationVariation;
	float cos_ang = cos(angle);
	float sin_ang = sin(angle);
	mat3 rotation = mat3(vec3(cos_ang, -sin_ang, 0.0),vec3(sin_ang, cos_ang, 0.0),vec3(0.0, 0.0, 0.0));
	
	orient_2d *= rotation;
	
	vec3 oriented_offset = reflect((INV_VIEW_MATRIX * vec4(orient_2d, 0.0)).xyz,INV_VIEW_MATRIX[0].xyz);
	//vec3 oriented_offset = (INV_VIEW_MATRIX * vec4(orient_2d, 0.0)).xyz;
	vec3 obj_oriented_offset = (vec4(oriented_offset, 0.0) * MODEL_MATRIX).xyz;
	
	//Wind-Effect
	//adapted from: https://github.com/ruffiely/windshader_godot
	float contribution = 1.0 * (1.0 - float(DeactivateGlobalVariation));
	vec3 world_pos_eff = world_pos * contribution;	//Generates world coordinates for vertecies
	// Removed using world_position due to dragging bug
	float positional_influence = -VERTEX.x + VERTEX.z -world_pos_eff.x + world_pos_eff.z;
	float offset = fract(positional_influence * (1.0 / WindScale) + (TIME * WindScale/1000.0));	//Generates linear curve that slides along vertecies in world space
	offset = min(1.0 - offset, offset);														//Makes generated curve a smooth gradient
	offset = (1.0 - offset) * offset * 2.0;													//Smoothes gradient further
	
	float t = TIME + sin(TIME + offset + cos(TIME + offset * WindStrength * 2.0) * WindStrength); //Generates noise in world space value
	
	//float mask = fract(v.y * wind_density) * v.y; //Generates vertical mask, so leaves on top move further than leaves on bottom
	//mask = clamp(mask, 0.0, 1.0);                 //Clamps mask
	
	float mask = clamp(VERTEX.y* WindDensity, 0.0, 1.0) * (ClampTop - ClampBtm) + ClampBtm;
	
	
	float si = sin(t) / 20.0 * WindStrength * offset;	//Generates clamped noise, adds strength, applies gradient mask
	float csi = cos(t)/ 20.0 * WindStrength * offset;	//Generates clamped noise with offset, adds strength, applies gradient mask
		
	vec3 wind_offset = vec3(VERTEX.x * si * mask, VERTEX.y * si * mask, VERTEX.z * csi * mask);
	
	float col = VERTEX.y * ColorRamp;
	COLOR = vec4(col, positional_influence, distance_influence, 1.0);
	VERTEX += obj_oriented_offset + wind_offset;
	
	obj_vertex = VERTEX;
}

void fragment()
{
	float rate_col1 = clamp(COLOR.r,0.0, 1.0);
	float rate_col2 = 1.0 - rate_col1;
	
	float fresnel = pow(1.0 - clamp(dot(NORMAL, VIEW), 0.0, 1.0), 3.0);
	float fresnel_rate = clamp(rate_col1,0.1,1);
	
	vec3 albedo = TopColor.rgb* rate_col1 + BottomColor.rgb * rate_col2;
	
	vec3 fres_col = albedo *(1.0 - FresnelStrength);
	fres_col += FresnelColor.rgb * FresnelStrength;
	fres_col *= fresnel;
	fres_col *= fresnel_rate;
	fres_col *= FresnelBlend;
	//fres_col *= (1.0 - COLOR.b);
	
	vec2 wiggle_uv = normalize(obj_vertex.xz) / WiggleScale;
	float wiggle = texture(WiggleNoise, wiggle_uv + TIME * WiggleSpeed).r;
	float wiggle_final_strength = wiggle * WiggleStrength;
	wiggle_final_strength *= clamp(sin(TIME * WiggleFrequency + COLOR.g * 0.2), 0.0, 1.0);
	vec2 uv = UV;
	uv = rotateUV(uv, wiggle_final_strength, vec2(0.5));
	uv = clamp(uv, 0.0, 1.0);
	vec3 tex = texture(Alpha, uv.xy).rgb;
	float x = COLOR.b;
	float alpha = clamp(tex.r + tex.g * 2.0 * COLOR.b ,0.0, 1.0);
	alpha = clamp((clamp(tex.g * 1.0 , 1.0 - x, 1.0) - (1.0 - x)) * 10.0 + tex.r, 0.0, 1.0);
	//albedo = vec3(COLOR.b,COLOR.b,COLOR.b);
	ALBEDO = albedo;
	ALPHA = alpha;
	EMISSION = fres_col;
}

Viewport container
![

Viewport container - outline disabled
![

3D Scene
![

When you get really close the issue doesnt appear
![

Post process quad
![

xyz

Devkure Use less detailed meshes for foliage.

Devkure

xyz I will try using something with smaller polycount later and let you know if that helped, thank you!

xyz

Devkure It may not be polycount per se but in fact the amount of details in general (i.e. the transparency texture). Hard to tell from your screenshots. It's not really clear what that bush is made of.

In general, you have two options. Either adjust the design/geometry of your meshes so they lend themselves to visually pleasing outline rendering, or render outlined and non-outlined things into separate viewports and do a depth-composite. The former is technically much less troublesome with potentially much better looking end result.

Devkure

xyz

The scene is also populated with grass, a lot of instances. Dont remember the exact number, but perhaps around over 30k (multi-mesh component).

The Bush in itself is, as much as i remember - composed of only quads. I will check the exact count when i lay my hands on pc (sorry and thank you for your patience 🙏) but i'm pretty sure its not a ridiculously huge polycount.

"Amount of details in general" - i understand it as, if i have a ton of objects on the scene, this issue may appear?

I will post more screenshots and info soon

xyz

Devkure My point is that it may not be a technical issue, but purely visual. If you have many leaves it the bush, and look at it from the distance, each leaf will be tiny and get a pixel-wide outline, creating a visual mess.

There still may be technical issues with normals though as the outline shader uses them.

So first make sure that your geometry is clean and adequate for this post processing effect.

Devkure

xyz AFAIK - this shader is of "billboarding" type. Its a texture of leafs slapped on some mesh. I will check out if it works on Simple shapes like a cube.

Devkure

xyz Okay, so i have recorded how it looks in the engine:

I will further try to fiddle around with the foliage mesh and such, or maybe even try my luck with the outline shader. The current foliage mesh is this blob in the end.

Thats the 3d object as seen in blender:
![

The polycount is a bit high, but as i said - i dont know much about shaders, thats why i dont wanna make any assumptions...

xyz

Devkure Yeah the post processing shader might not play nice with billboards. For debugging, you can try to render the normal texture directly onto the screen to see how billboards appear there. Also try to billboard a simple shape texture, e.g. a single circle, to see if it gets outlined correctly.

Devkure

xyz Update. I have tried another outline shader, by reducing normal and depth thresholds respectively i was able to replicate the effect. Seems like the best course of actionwould be to separate these effects. You have mentioned previously that it can be achieved with separating it into two different subviewports? Correct? "render outlined and non-outlined things into separate viewports and do a depth-composite". Can you please expand on that idea? As of right now i have everything placed in a subviewport, that is placed in SubViewportContainer with "Stretch" on and "Stretch Shrink" set to three (to imitate pixelart)

xyz

Devkure What happens if you don't put a transparent texture on the bush?

Devkure

xyz Pretty much the same effect. It goes black at certain angles and distance. I think im sure its because of the outline shader, because i tried putting a different one in its place - and it worked fine (although, the effect isnt as nice as this one...)

xyz

Devkure by reducing normal and depth thresholds respectively

Why not just adjust those thresholds to give acceptable results?

Devkure

xyz i tried fiddling around with that, unfortunately with no effect... The outline is messing the bushes up making them look weird, unfortunately - and if i reduce too much, then outline isnt looking too well on objects i actually want outlined. Is rendering in two different viewports so the effects dont overlap too hard to do?

Devkure

xyz Update!
The issue was connected to transparency.

I fixed it by setting up the render priority on the material to 1.

Now the only question is, how can i separate other potential objects, like grass, to not be outlined. ![

xyz

Devkure I'd avoid rendering this in two viewports. The way to go is to adjust the bush asset and/or its shader so that outline shader renders it nicely.

Devkure

xyz Thing is i dont necessarily want outline to render on everything... Imho the foliage would look far better with no visual border (outline), the same goes for any kind of grass and whatnot. My question is, if its troublesomesome to render it separately. If not viewports - is there any other way to separate the two, for instance - having the outline only attached to the building or any other kind of object (maybe attached to the material of said object?)

xyz

Devkure Render them after the post filter is applied and use the depth texture to discard obscured pixels. You'll have to maintain the camera clone for the second render.