Question for the cracks: pointers or handles ?

    unsigned int m_x;
unsigned int m_z;
unsigned int m_level;
unsigned int m_size;
omath::vec2 m_min_max_height;

node *m_tl{nullptr};
node *m_tr{nullptr};
node *m_bl{nullptr};
node *m_br{nullptr};

So this is the current node struct I use to build the quad tree for my terrain renderer. X, z and level are vital, size, min and max values are there for sanity checks, will be eliminated.

So that leaves me with 44 bytes for a node. Not much, but with a huge terrain can still be unwieldy, possibly gigabytes of memory for the quad tree of a continent sized terrain (example 1.4 Gb for a terrain of size 1048576 by 1048576, leaf node size 256*256, 7 lod levels, nodes have 60 bytes with size and min/max values).

Biggest block are the four pointers, would I reduce them to 32 bit int values, that would reduce the structure size (and thus improve performance) considerably and I may end up with "just" 650mb for the same structure.

I stumbled upon this:
https://floooh.github.io/2018/06/17/handles-vs-pointers.html

and several discussions around the topic.

My question asks for an opinion (inadequate for stackexchange), has someone actually realized a project with handles, like written a class "handle" based on 32 bit integers that replicate the behaviour of classic raw pointers, and did it have the hoped-for impact on performance ?

Is/does this all have to be loaded in memory all at once or could some lazy/JIT loading be done? Basically runtime streaming of the data into/out-of memory when needed? Memory might be relatively cheap these days but disk space is cheaper still I would think. I would imagine this would likely have to involve some multiprocessing/threading for all I know.

Note: not a programmer, a "script kiddie" at best far as I myself am concerned. Not to say that what you are asking about is not a worthwhile optimization/consideration. I just can't offer much(well, anything) in that regard.

Thanks, no worries.

The spatial data structure (quad tree) is an element distinct from the data, used for selection of the visible parts without defining what they actually contain. It is in this case and for real time best kept in memory for optimal performance, while the data at the patches selected for display, height map and other textures, other surface elements, models of things, can be streamed in and out as needed. It doesn't take long to build the quad tree (around 2 seconds for the really big one), which doesn't beg parallelization as it only happens once, and couldn't be done lock free.

So it is best to keep that as small as possible. The question "handles or pointers" arises with really large terrains. Such environments may be nice to behold but would be pretty dull when not filled with things. So I asked myself is it worth the effort.

Maybe I wait 'till the first reactions.

    Having a vulkan implementation of this is quite impressive ^^
    Does this allow for non planar terrain ? Like if I put 6 of these can I have a planet terrain based of a heightmap ?

    It is not Vulkan (yet), I'm doing it in OpenGL 4.5 first. This isn't for mobiles anyway.

    And yes, my plan is to expand this to a spheroid or even ellipsoid cube map with a quad tree per face. For this to be doable with reasonable effort I need double precision on the graphics card in the vertex stage, which I hope will be there when I am there :-)

    Just to make that clear, I will not do the data part because it is a bit too much for a part time hobbyist. If you want fractal or noisy or even terrain based on physical processes like erosion and tectonics you'd have to help out ;-)

    I will ponder an interface to place models of things, buildings, caves, and scatterings like plants and rocks. Wouldn't make much sense without.

    But first the basis, sorting out the parts to display and finding the visible surface.

    Pixophir So it is best to keep that as small as possible. The question "handles or pointers" arises with really large terrains. Such environments may be nice to behold but would be pretty dull when not filled with things. So I asked myself is it worth the effort.

    Well, in that case may I raise the question: if you were considering supporting really large terrains in a future version, say "2.0", can this be also handled then or is it better to deal with this now even if the actual/official support for really largetm terrains would come at a later TBD date?

      Size of the terrain is a bother when streaming data, filling the buffers for drawing, and wrapping them around I think. So if you turn around too quickly you might realize, for a fleeting glimpse, that the Langoliers have already been at work 😰

      But that, I fear, cannot be delayed much ... would be like selling a car and charging extra for ... oh, bad example 😎

      ... will stay with pointers.

      I may have a very early alpha by the end of the next week.

      May I be so bold as to ask a colleague if they could lend me a hand and try to compile it under windows ? I plan to push the source code to sourceforge.net. It is stand alone for now (no Godot extension yet), just flat shading and no streaming of data yet. I'd just be interested in hearing if it compiles on Windows, maybe with the MS compiler if that's what most people use.

      Dependencies are GLFW3, ImGui, stb_image, libpng, glad and OpenGL development/header files. I can add glad, so that's one low hurdle less, but the rest came from the repositories of my OS (Debian Bookworm), so on Windows there may be some prior configuration efforts necessary.

      I don't expect many problems some GNU extension to C++ (like sincos()), but GLFW should take care of the windowing/input handling in a transparent manner.

        Pixophir May I be so bold as to ask a colleague if they could lend me a hand and try to compile it under windows ?

        What does it take to do that? In my whole life I have only compiled ArmorPaint from source. 😊

          Hi,

          let's see, can you follow this tutorial until the hello-triangle (and further on if you wish ofc.) ?

          You would additionally have to install the other mentioned dependencies for my cdlod, without the loader glad. Then you're set, at least you could forward me the MS compiler's error messages.

          But I cannot help you debugging your environment, should problems arise there. So, you either can compile and run the triangle, or you can not, I am as sorry as possible :-)

            Pixophir It might be a good tutorial... but it's obviously not for a quick check… you have to dive in, figure it out and install the missing stuff… it's time consuming. Is there a difference — Win 7 or Win 11 — which one is needed? I'll try to make time in the next few days.

              I certainly get it, no worries. I mean, we'd obviously need some deeper C++ understanding here that I did not mention. Nevertheless, if you like to dive in, there you go. Have fun :-)

              @Tomcat

              The Wicked Engine is a very cool project with lots of high tech, but still over my head 🙄

                Pixophir To the greatest regret, I can't help in the near future. It turns out that you have to register with M$ to work with Visual Studio… If you can't find a helper, I may do it later, perhaps in time, but for now I'm not morally ready… I'm extremely sorry.

                  Pixophir I can do that (VS2017, 2019, or 2022), but not this week, as I'm away at a conference. But I can do it next week.

                    7 days later

                    Refactoring. Pls. stand by :-/

                    Am currently simplifying the conversions between raster space, the space represented by the raster of the spacial data structure which are basically integers for the positions in heightmap textures, and world space, which is the usual floating point stuff one deals with in any 3d application.

                    Before there was the possibility to extend a terrain in all directions, but that lead to conversion problems I could not really wrap my brain around. Si si, somos aficionados ...

                    So, in future I will start any quad tree's raster at 0,0,0 and extend it out to max, which is limited to power of 2 sizes. That makes things considerably easier, also when I think of presenting everything to people who probably just want to slab in a bunch of textures and expect it to just work.

                    Since I also have a new home to build, this may all take a bit longer.

                    Lesson learnt: Do not say it will be done until it is done.

                    4 days later

                    The last overworking got me one step forward. That's from the rim of the canyon where I stood :-)

                    Never mind the flat shading. Shown is one terrain tile made from a 16bit heightmap 4096*4096 pixels, but it works well up to a tile size of the maximum the graphics card can digest. My test data is SRTM (Shuttle Radar Topographic Mission) data, shown is a part of the Himalaya, distance between two posts is about 90m. Purposefully exaggerated to better show the boxes that cover the terrain for view frustum selection.

                    The white lines are bounding box of the whole tile, the light blue ones are all the boxes of each node of the spatial data structure, the coloured ones are the per frame selected lod levels as seen from camera position.

                    Sending two 0s away gives us:

                    with the bounding boxes of the lod levels.

                    A close-up of a valley gives an impression of what the terrain lod is planned for (no planetary curvature yet, or any sophisticated shading, that's future music for now):

                    11 days later

                    So, the base algorithm is finally working.

                    https://github.com/BaronVerde/cdlod

                    Tagging @Haystack , whenever you like.

                    A 4k heightmap texture comes with it. Ask me if you want a 16k texture, or search SRTM 90m data download, select the region you want, choose ARC ASCII as format, and see my other rep srtm-converter for how to generate .png tiles from the SRTM ASCII data. Also, try noise textures you may have somewhere. Eventually tweak the HEIGHT_FACTOR in settings.h.

                    Loading of a large heightmaps takes a few seconds, though that's going to change. Please mark the b* and krams directories under src as exclude from compile or you get multiple definition errors. These are just backups from working states.

                    It is only a single heightmap texture for now, and only flat vertex shading. No cosmetics. If you use an own heightmap, make sure it is 16bit-grayscale png of power-of-2, edit RASTER_MIN and MAX values in src/applications/cdlod/settings.h, and add a .bb text-file next to the heightmap texture. Look into src/renderer/renderer.cpp and change the filename there. Don't add .png. Attention, though there is a vector defined there, only one texture is loaded.

                    Next steps:
                    Comb the code. There are quite a few implicit conversions I must get rid off. - done -
                    Texture compression. - scrubbed, using half floats and uint16_t -
                    Parametrize and correctly calculate raster-to-world sizes and distances. My poor brain ...
                    Abstract texture types and bit depths and quite generally heightmap sources.
                    Build a data structure for heightmaps, and load/unload them asynchronously.
                    Render relative to eye. That means an own terrain camera.

                    Then: try to get it into the Godot scene tree.

                    Then proper texturing, rendering, evtl. cascaded shadow maps that fit the lod levels and ranges, collisions, placing of game objects.

                    Need a beer.