Anime Shader in Godot 4.0 (WIP)

Been working on this the past few days. Trying to get an anime look similar to the Netflix Stand Alone Complex series. It looks pretty close to what I was imagining. The character is basically working like I wanted, still need to do a lot of work on the background.

There are two parts to it. The character is using a custom ramp shader for the highlights. Then I am using a sobel filter with a custom multiply operation and tweaks to the color. This gets a subtle outline without the jarring look of the traditional sobel filter. The background does not have the ramp shader, but the outline filter is full screen. I plan to write a NPR type of shader for the background to make it similar to a painting or water color. Probably work on that over the next few days.

Let me know what you think.

This is how it looks without the shaders. Obviously the lighting was tuned for how the shader works, so the character is too dark without it, but it should give you an idea.

The cel shading or ramp shading is essentially free. It's just a modification to the standard material. You can easily convert the standard material to a shader material with one button in Godot, then edit it as you like. In this case I just altered the light() function to sample from a ramp texture (sort of like a palette map) rather than doing the normal dot product of the light angle. This essentially give you steps, so you get large areas of a single color rather than a smooth gradient. So this part is free.

The second part is the outline. I am using a 3x3 sobel filter. Not the one from the Godot docs, cause that one looks too dated, but I found some other open source code that fit my needs better. This has a thinner line and also better anti-aliasing. The result of the sobel is a black image with a white outline. Then I change the color to a brownish red, and multiply that by the screen buffer. This allow the color to better match the environment. Additionally, on the full screen ColorRect (that the shader is attached to) I set the alpha to about 60% so it is not as noticeable. This gets a subtle effect but it actually does quite a lot without being in your face. This is not free, but also not that expensive either. It takes 9 texture samples and some simple math. My other shaders (like Godot Super Scaling) were also doing 9 samples (sometimes more) and the cost is low. About 1 ms on a good computer. For this, it should be even faster as I'm not doing any heavy math, mostly addition and subtraction, which are cheap. So my guess it would be even less than 1 ms.

As far as I can tell, I'm happy with the look of the character, so I would consider that done. The background was setup really quickly this morning just for the render, so it needs a lot of work. I want it to look more like a painting, so I will explore NPR (non-photorealistic rendering) to make it look more like visual novel or anime backdrops. This shouldn't be too hard, but I haven't wrote one before, so it might take be a couple days. I also have to remodel parts of the background because the walls are too thin and the shadows don't work. But this is fairly easy.

Thanks. I still have to work on this more. I've just been distracted with other stuff.

Started work on the background shader. This uses a Kuwahara filter. Not exactly what I was going for, but it looks close enough. Will have to check performance on older machines, since the shader needs 200 texture samples, it might be too intensive. But it works on my machine.

My idea was to use this just for the background, and then a cel shader with sobel filter just for the character. I'm not using photos in the game, this was just to test the shader because I haven't finished modeling yet. The toon shading and sobel filter are finished and very fast. However, this Kuwahara filter is way too slow. I tried a few different algorithms, but they take around 200 texture samples per pixel, which is nuts. I'm only getting 60 fps and I have a top end machine. So I'll have to look into alternatives.

The next thing I have to do, is figure out a layer system. The transparent viewports in Godot 4.0 don't currently work, so that limits the possibilities. I have to look into the source code this week and see if I can fix this. I tried doing the compositing with a full screen shader and a mask. It does work, however, only one viewport can have environment lighting, which makes it look incorrect. And my sobel filter requires the full viewport, which messes up the outline. So even that hack won't work (though I'll have to figure something out, cause I may have a similar problem even if I fix the transparent viewport bug).

I'll give it about a week to try to figure this out. I really wanted an anime look, sort of like a visual novel but fully in 3D. I know it's possible, but I have to get it working fast on old computers and fix the bugs in Godot 4.0. If not, I would have to compromise and maybe use unshaded lighting for the background (which is what a lot of older anime games did) but I think it looks dated and not the style I was going for.

You can see the toon shading and outline are basically final on the character. Ignore the background because it's just placeholder and I have to remodel most of it.

So I got the Kuwahara filter mostly working. This uses a new method that is much faster and actually looks better. Still takes a big portion of the performance, but I think I can live with it and make some compromises elsewhere. Note, the character won't be affected by the filter, she will look like the above image once I get the compositing working.

I got the compositing working. It's not perfect, but I think it looks good enough to move onto other things.

Did my first animation test. This is not with the original model I bought (I couldn't get that working), but I used VRoid and DEEPMOTION instead. Only spent about 2 hours on it, and it's pretty rough, but I got to see that the tools sort of work. Will need a lot of cleanup on the animation, but at least I made some progress today.

Yeah, it got the general pose, but it's really jumpy. This was using a video to convert to animation with AI. I mean, it's not horrible, but will need manual clean-up.