Proper parallax mapping over geometry edges
I was looking at the Unreal 5 unveiling video last night. It got me thinking about pushing conventional shaders to obtain such fine detail on tiny areas... something that looks similar to Nanite as far as the viewer can perceive, but of course without billions of vertices for every little crack. To achieve such accuracy you need two things: Normal mapping but also parallax mapping. I got curious to check what parallax implementation Godot has, and sure enough it's looking great with the right textures! But I'm seeing the same issue I've always noticed with almost all parallax implementations: Edge cutoff.
The depth effect will only act within the geometry of the mesh the surface is on. If you're looking sideways at a face, the effect won't pop either in or out; You still see the outline of the object's model, the illusion of displacement will always be contained within that. This can easily be noticed in the default material demo with large gaps:
I've always considered this the deal-breaker of displacement mapping: You can use it to improve all these fine dimples and scratches, but if a larger detail falls on the rim of a mesh you're going to notice the effect is fake which ruins the whole thing. Yet I'm fairly certain there are implementations that worked around this issue; I don't remember if it was the Darkplaces engine (Nexuiz / Xonotic) but I'm sure I once saw parallax mapping on a brick wall where the bricks popped out of the plane geometry the wall was on and correctly acted on the edges.
Thinking of it technically from my very limited experience with shader code: I can tell some wizardry is likely needed to draw parts of a texture out of the triangle it's drawn on... don't know if GLSL would even allow this easily and with acceptable performance too. The opposite might work however: Couldn't we reverse the effect and displace the texture inward, then use transparency to mask out parts displaced to below the containment of the mesh?