Loving Godot, but I have a requirement that seems
currently difficult to impossible to do:
Rendering a 3D Stereoscopic 360 degree view of the scene.
Cubemap
We can do a monocular projection by fixing six square 90-degree cameras, one for each axis, rending them each into a texture. Then use a shader loaded with those 6 source-textures to figure out which pixel should be mapped where in the final output.
That's the approach taken by Cykrios with his Godot 360 system to emulate many other kinds of lens too. Amazingly fast considering all the intermediate steps, no obvious lack of detail.
You might think that you could just do two of those: left eye top, right eye bottom,
bish bosh there's your stereo 3d map.
But no.
As explained very well in this amazing page by shubhmehta, what you'll get if you do that is a view from two eyes swiveling in their sockets:
when what we need is a map from a head rotating two eye-cams:
When you look backwards, your eyes are the wrong way round! When you look left your right eye is pointed directly at your left eye.
We definitely need a different kind of projection than
just rotation about the origin, it also needs a position
translation that's different for each column of pixels
in the output image.
Monsterous Camera Rig.
Okay, we can think of extending the method though. Instead of six cameras pointed each along one axis, we can have one camera for every vertical column of pixels on the output image.
Four thousand and ninety six cameras.
Well, for each eye. So.. Eight thousand one hundred and ninety two cameras!
Well, actually you can't cover 180 degrees with one camera, so each column will need two, one pointed up at 45 degrees and one down. So... Sixteen thousand three hundred and eighty four cameras!
Monsterous!
It can be built though. I built a mono-scopic one
and a 4096-pixel wide image filled from 2048 cameras
and it doesn't even crash Godot.
It takes about five minutes for the script to add
though, so you really do think it's crashed.
And then it's like half a minute a frame.
For a 1/16th size model.
Too slow by far. I don't mind a minute per frame in
a batch-render but this is not gonna come close to that,
even with a nearly empty scene.
And also the resulting render is only good for maybe
half the frame. The rest just streaks, I guess memory
issues?
So that ain't gonna work.
Would have been surprised if it did.
Edit Godot!?
I looked at the Godot source-code.
It builds pretty easily.
Camera projection code seems to be mostly in
scene/3d/camera_3d.cpp
And there's a function there called project_ray_origin
that seems like it could maybe be altered to give
that 360 view?
Though that'd still just be a socket-spinning projection
through 360 degrees, not a head-turning projection.
I changed all these files to try and invent a new
camera projection mode PROJECTION_PANORAMA, just as
a copy/paste from the "perspective" functions for now,
to maybe do some experiments on it at least.
modified: core/math/projection.cpp
modified: core/math/projection.h
modified: doc/classes/Camera3D.xml
modified: doc/classes/RenderingServer.xml
modified: editor/plugins/gizmos/camera_3d_gizmo_plugin.cpp
modified: editor/plugins/node_3d_editor_plugin.cpp
modified: scene/3d/camera_3d.cpp
modified: scene/3d/camera_3d.h
modified: scene/main/viewport.h
modified: servers/rendering/renderer_scene_cull.cpp
modified: servers/rendering/renderer_scene_cull.h
modified: servers/rendering/rendering_method.h
modified: servers/rendering/rendering_server_default.h
modified: servers/rendering_server.cpp
modified: servers/rendering_server.h
And while the new projection mode's name does show up
in the editor's panel for the camera, pressing it
makes no actual change.
I guess I missed some place that needs to know about
the extended range of the ProjectionType enum?
I'm very lost, in a very big codebase,
and seems like there's a good chance it'd need more
than just changes to the project_ray functions
anyway because the camera-origin needs to be translated
differently for each screen-X, as well as rotated.
So not even sure it'd be possible even if I got it
to notice my new projection mode.
Forks
@BastiaanOlij off of this forum right here did
a few posts in 2016
about the topic and code for V3 but it's not clear if that
really went anywhere and I can't tell if
their fork
still has any changes like that or what other things
it might have.
I saw someone with a
system to add a 4x4 transform matrix
to the camera's projection properties but I
don't know enough to know if the kind of
mapping needed will fit into a single
transform matrix or not.
Or indeed if that code is ever going to make it into main?
Questions
Is there anyone working on enabling that kind of
camera projection with a plugin or a patch or
something?
If not, is there somewhere I can chip in on a bounty?
If not, is there anyone who can at least reassure
me it's actually possible to do if I keep bashing
my head against the Godot code trying to add a
new projection mode myself?
Or if I just chill for a few months is it coming
in like version 4.5 anyway?
Or will that 4x4 transform matrix actually let you do it?
I suspect it'd have to be a different matrix for
each pixel column but I don't really understand
how the matrix multiplication magic works.
Image
Top-left: Desired stereo projection
Top-Right: Monocular 360 via Cykrios method
Bottom-Left: Monster-Rig, 10 columsn
Bottom-Right: Monster-Rig, 2048 columns