I can't get my ShaderMaterial to load correctly

Shadowphile

@ xyz: You have been helping me move away from relying on the shader but the project is getting klunky and has performance problems and I realized that the ONLY reason I had to totally revamp my graphics pipeline was to speed up animations, otherwise my project is running great. So I decided to just streamline how I apply patches by sending a 2D image of frames to the shader and letting it pick out the frames.
In this demo though, it's just one frame.
Attached is an example.
I am pasting a full background onto a sphere then overlaying a patch. The background is a closed toilet, the patch is an open toilet.
When I manually load the correct values into the Shader the patch works great.
But when I try to load them dynamically I see nothing. I wasted the whole day yesterday hoping ChatGPT could help but it just led me down a rabbit hole, but it did give me this basic structure to learn from.
Anyway, run the demo to see the default. I put in my mouse-steering code to make it easy to navigate by just dragging with the middle mouse button.
Then in Pano.gd, uncomment the indicated section and run again. The patch disappears.
For some reason when in Remote viewing, the Shader parameters don't seem to be updating, although I have seen it do so in the past.
I can't tell whether the problem is mine or the IDE just doesn't update correctly. I've seen it work in the past at runtime.
Man I hate debugging shader code :/

***EDIT: I found that my both of images and one float uniform are passing through ok but the uniform vec4 sheet_rect0 is not receiving numbers like it should. I wired a simple if statement to color the whole display for whether the first value in rect0 is higher than a value that I manually selected then tediously walked through a range of numbers (did I already say how I hate debugging shaders?) and recorded this behavior.
When gdscript sends 1165.0, it turns into about 20,110,600 in the shader.
Similarly, sending 1.0 = 1.0 inside. Sending 0.5 = 0.215 inside. I don't know if these number are meaningful except maybe the equivalency at 1.0.
Godot 3.6 supposedly will send unrestricted floats using Color(x,x,x,x) when its passed to a uniform vect4 but that sure looks broken to me.
Please confirm this before I decide to try passing vect3. Since I plan to have 6 or more overlays I need to keep these parameters caged up in bundles, not sent one parameter per uniform. I haven't looked at sending arrays yet. This has been an exhausting non-adventure.

dynamicpatchestest.zip

2MB

xyz

Shadowphile Don't composite this in shader. It's just too messy and complicated.
Here's how you should approach it instead:

Set up a viewport that's the size of your main texture. And assign this viewport's texture to the shader.
Under that viewport put a texture rect node with the main texture.
Add smaller texture rects that overlap the big texture rect and contain your patches. Put them on their proper positions. Those rects are initially invisible.
Set viewport's update mode to "once"

Now, whenever a patch needs to be toggled, set its texture rect visibility accordingly and set viewport's update mode to "once". It will re-draw the viewport in the new state, and the ball texture will be automatically updated.

The shader code will be trivial (just a texture lookup), and you'll have easy control over toggling patches from script code. It's also the best approach in respect to performance and scalability. You don't need to do complicated in-shader compositing every frame and you can handle large number of patches without any per-frame performance cost.

This is extremely easy to debug as well. In debugging phase you can keep viewport update mode at "always" and then position and toggle every patch rect in the editor with immediate visual feedback. Easy peasy 😉

Shadowphile

xyz
You describe exactly how I ended up redesigning my architecture before I changed my mind.
What I haven't passed on very clearly is that I'm building a scriptable game engine, not just porting a game, based on the architecture of the original game engine I was using and perfectly designed for this kind of point and click game. Each 2D or 3D game location can request to overlay other locations (mainly 2D on top of 2D or 3D), arbitrarily, nesting endlessly if they want. To implement I had to make a system that recursively loads based on what the scripts request then load the whole thing into the scenetree.
However, I found that loading a panoramic subviewport took 6 seconds to load, which happens every time the player changes location. I decided to try just hard coding a limited set of subviewports into the scenetree and got THAT working but I didn't like the loss of flexibility and I still had to redesign a lot of other elements and when the game starting having FPS problems I finally realized that animating patches was my only real issue so I decided to just optimize that using the Shader.
I did finally get the demo I uploaded to work passing vec3 groups of floats instead of using Color which I was badly informed would work in 3.6 so right now I'm sticking with the Shader.
A professional game designer got me into the idea of my own point and click when he released his game engine used for a popular game called Scratches, almost 20 years ago. He has released dozens of games, mainly points and clicks, and teaches Unreal courses. He just released his magnum opus on Steam, Asylum, after almost 18 years, based on Unreal engine. It's a huge game and he recently told me this kind of game really needs it's own engine because Unreal was unwieldy handling an image-based system that has huge pre-rendered images. Since nobody is going to bother making a modern engine for a some-what obsolete kind of game I'm doing the best I can with Godot.
I've never been able to ask anybody proficient in Godot how they would try to implement something like this and I'm really curious what they would say if I pointed to the original game engine, Pipmak, and said 'How would you implement this in Godot?' But its hard to find anybody interested enough to look into how that game is structured since most game makers aren't interested in this kind of old-school point-and-click anymore.

xyz

Shadowphile Not sure how using shaders the way you described optimizes things here. What exactly do you think is getting optimized? If anything - it makes things worse. You still need to load the same amount of bitmap data while the number of pixels you're processing per frame goes considerably up. So it can't be a solution for optimizing loading times.

The performance of this scales really poorly. Not a good choice for a generalized "engine" solution.

The problem with shader mixing is that for every additional patch, you'll process a whole additional screen resolution worth of pixels, no matter how small the patch is. It makes your rendering time needlessly dependent on the number of patches in a very bad way. On top of that, you'll do this every frame just to compose the same static image over and over again. It's extremely wasteful.

Viewport is the way to go here. By drawing rects into a viewport you don't render/process any excess pixels. You draw the exact amount of pixels that are in each patch. And you don't need to do it every frame, only once when the state changes.

Godot's viewports are nothing more than off-screen rendering. It is a "natural" solution for real time compositing.

If loading is causing bottlenecks, stream the scenes using threaded resource loading. As soon as the player arrives on location X, start threaded loading of all adjacent locations.

Shadowphile

xyz
I hear you on this. I'm just really struggling in my life right now, personally, and the game-engine side is not enjoyable whereas all of the story-building and asset creation IS relaxing. It took me six months to decide to abandon the old game engine and another six months to wrap my head around Godot enough to get it working in a similar fashion; going through that again is very depressing. I feel like a novel writer full of ideas who is told he can't write anymore unless he invents a new printing press. And then told his first attempt wasn't good enough. And all he wants to do is write...

If I throw some basic starting points at you, would you be willing to suggest how you would organize? Trying to explain the whole thing at once would be too much to ask of anybody.

xyz

Shadowphile Well if you already have a good enough working system - go with that. Things don't need to be perfect. Just accept its limitations and adapt to the constraints. If the constraints are not acceptable, you'll need to re-engineer one way or another. As always - precisely identify the bottlenecks and work to minimize their impact.

Be willing to compromise. As they say - perfect is the enemy of done 🙂
If you're burned out on the technical side, I'd just lower the resolution of all images to a quarter, and finish to a playable demo, focusing on gameplay, visuals and story. Even the worst kind of brute force may be good enough here. Start thinking about optimizations and system redesign only if/when the game as a whole becomes interesting enough to impartial audience.

That said, basing the system on viewports + background loading should perform well for this type of game in most situations, especially if you don't unload locations that are nearby but not directly adjacent. The pace of play in games like this is rather slow. If you cache enough locations you should be able to get away without displaying any loaders.

Even with location caching, there could always be unexpected situations where a location is not ready if the player moves quickly through a sequence of locations to a yet unseen location. In that case be ready to display a loader disguised as a time-stretchable transition animation.

You can ask anything. If I know the answer I'll do my best to help you.

Shadowphile

xyz
You are reading my mind. I understand the need to optimize when the conditions require it but since I don't plan to run this on mobile devices, power consumption (from inefficient cpu/gpu activity) and tightly managed resources just aren't a high priority. As you say, it's a slow-paced game and only the response to changing scenes needs to not lag. My 'minimum' required platform is my laptop, which doesn't have a GPU so if I can get away with my current approach on that, then I'm satisfied.
(Now I have to find somebody to lend me a mac so I can see how broken my setup might be lol)