Hi,

How do I detect what object has the mouse clicked in a 3D scene?

I understand there are 2 ways: 1 ray casting 2 collisionObject

I tested with ray casting added this code when the mouse clicking event is fireing

		if new_mouse_click:
			
			var from = camera.project_ray_origin(click_pos)
			var to = from + camera.project_ray_normal(click_pos) * ray_length
			var space_state = get_world().direct_space_state
			var result = space_state.intersect_ray(from, to)
			print (result)
			new_mouse_click = false

didn't work collisionObject I didn't see a clear example of how to use it, I have a group of objects, I want the children to be selcteble, some are spatials and some meshinstances, what do I do?

I've always used raycasting for this kind of thing, as I find it to be the most reliable overall. The code you posted above should work, but it will only return physics nodes. This means StaticBody, KinematicBody, and RigidBody nodes. Area node detection being an optional parameter if I recall correctly.

Then you can do some additional processing to get the node you want. Personally, I prefer to have the collision body as the root node of whatever I am wanting to select, as it makes things easier, but you can also have it as a child node of the Spatial/MeshInstance nodes if necessary.

I don't have an example handy, unfortunately. You are right about there not being much material out about object picking in 3D, despite it being a fairly common mechanic. After several Google searches, I did find a few posts, but that was about it:

  • Get clicked object in 3d (Reddit link)
  • Move a 3D object with the mouse pos (Godot QA Link)
  • Godot CollisionObject input_event on Godot documentation (Link)

There are other ways to go about it, but none of them are particularly easy. Color mask picking is one and it is very precise, but it can be a pain to setup and use. If you can, I would highly suggest using raycasting.

Hopefully this helps!

Thank you, so basically, I have to add a collision body to each and every spatial or MeshInstance in the scene, otherwise the ray will not detect them. And I have to set set_ray_pickable(true) for each and every one of them. is there a way to do it with code at run time?

Is it possible to put all my spatials/meshinstances into one Collision body? will it then detect the specific child I clicked or should each mesh have it's own Collision body?

and what if I group all items in one group and assign a collision body to that group, will it detect on what child I have clicked? will the ray result object give me a list of meshes or only the name of the collision body?

@DanielKotzer said: ... I have to add a collision body to each and every spatial or MeshInstance in the scene, otherwise the ray will not detect them. And I have to set set_ray_pickable(true) for each and every one of them.

Unfortunately that is pretty much the only way to do it, at least that I know of. You only have to create a collision body for each of the things you want to be uniquely select.

For example, if you have a bunch of tomatoes and you want to select the group of tomatoes, then you can make a single collision body with a collision shape covering all of those tomatoes. Then when the raycast hits that collision body, you can determine that you have hit a group of tomatoes. If you want to select an individual tomato, then you will need to create a collision body for each of the tomatoes in the scene, with the collision shape covering a single tomato.

Is it possible to put all my spatials/meshinstances into one Collision body? will it then detect the specific child I clicked or should each mesh have it's own Collision body?

If you want to detect the individual spatials/meshinstances, then they will each need their own collision body.

and what if I group all items in one group and assign a collision body to that group, will it detect on what child I have clicked? will the ray result object give me a list of meshes or only the name of the collision body?

The result array will return only the collision object. To the best of my knowledge, there is no way to retrieve the individual child. The collision object returned will be the StaticBody/RigidBody/KinematicBody node the raycast collided with, if there was a collision.

If you are using the PhysicsDirectSpace for raycasting (documentation), then there might be some additional processing you can do because of the additional information returned from the raycast. You might be able to figure out which collision shape the raycast collided with using the collider_id or rid, but I personally do not have any experience with that so I cannot say if it is possible or how feasible it is.

There might be other ways though that I am unaware of, but to the best of my knowledge, every item you want to individually select will need its own collision body.

How about color mask picking, in Stage3D I had access to low level programing, I was making my own vertex shader and fragment shader and render the objects in a loop. The vertex shader is basically giving the geometry of the object, and the fragment shader is coloring it, if you make the fragment shader so that it is only coloring the object with a flat solid color, ignoring all light effects, you can give each object a unique color, then render the scene and check the color of the pixel underneath the position where the mouse has clicked, this is also very accurate, as a collision body is covering the mesh outline approximately at the most, and takes a lot of computing power. while if you render the scene, you are in fact projecting a 3D problem on a 2D plain and deal with it as a 2D problem, makes much more sense, and gives you the answer you really need - to what the object the pixel you clicked on belongs. I thing this should be build in the Godot engine.

Any way, I saw this : https://godot-es-docs.readthedocs.io/en/latest/tutorials/shading/migrating_to_godot_shader_language.html

I thought maybe I could play with the fragment shader in Godot, as I did in Stage3D.

Do you know of any other way of doing this? -- coloring all the objects with a solid color, no lighting effects, and render the scene to a bitmap the size of the screen, and then check the pixel in the position of the mouse click on that image, this should not be complicated.

Using a color mask is certainly one way to work around the issue. I’ve written a tutorial on color masking, though for shaders rather than color picking. Because of this, I know it is theoretically possible, but that’s about it.

You are on the right track for color masking. You need to take the geometry and shade it with a solid color that is not effected by lighting. You can do this using Godot’s built in SpatialMaterial, or through a custom shader, just set the unshaded flag to true. If you are using an Albedo texture, you will need to remove it for the instance you are using for the color mask.

For my personal projects, I’ve written a simple script that copies the MeshInstance node and assigns an unshaded material to it. Then I put this copied MeshInstance node on a render layer that the main camera cannot see, so it doesn’t show up in the window the player sees. I can share this script if you want.

Rendering the color mask is fairly easy if you use a Viewport node. You just need to make a Viewport node with the same size as the main/root viewport. As a child of the Viewport, you need to add a camera that has the same position as the camera rendering the normal scene. Then in the camera for the color mask (a child of the Viewport node), you need to set it’s rendering mask to the same mask as the MeshInstance nodes you are using for the color mask. Then you can access the color mask texture using the Viewport get_texture function.

The hardest part, and I honestly don’t totally know how to do this, is get the pixel in the color mask texture underneath the mouse. I suspect there is probably a way to do this, but I personally have not had the need so I am not sure what it is. You might be able to use viewport_node.get_texture().get_data().get_pixel(mouse_pos.x, mouse_pos.y), though I have no idea if it would work, nor what the performance would be.

Then when the mouse is clicked, you just check to see what color the pixel at the mouse has, and which color represents which object. This limits how many objects you can have by how precise you can detect colors, but it shouldn’t be an issue unless you have hundreds of objects.

In summery: It should be totally doable to use color masking.

Hopefully this helps!

@TwistedTwigleg said: I can share this script if you want.

Sure I would, anything that might lead me in the right direction. Thanks

No problem! Here is the helper script:

extends MeshInstance

export (bool) var create_masking_mesh_on_runtime = true;
export (Vector3) var masking_mesh_scale = Vector3.ONE;
export (Material) var masking_material;

func _ready():
	if (create_masking_mesh_on_runtime == true):
		var new_mesh_node = MeshInstance.new();
		
		new_mesh_node.mesh = self.mesh;
		new_mesh_node.set_surface_material(0, masking_material);
		
		# NOTE: unfortunately, we cannot export a layer/mask value like in the Godot editor
		# so we'll just have to set it manually through script.
		# This value, 524288 corresponds to layer 20 on the cull/rendering mask of a Camera node
		# (The farthest bottom right mask)
		new_mesh_node.layers = 524288;
		
		add_child(new_mesh_node);
		
		new_mesh_node.scale = masking_mesh_scale;

Hopefully it is of some help :smile:

Well, because I was curious, I decided to take a shot at implementing color based object picking. Turns out it wasn't quite as hard as I thought it would be, though I've been working a lot with Viewport nodes as of late so that might have helped.

Here is a link to the project on my Google Drive. I'll keep the file there for a couple days before removing it. I plan to use the project for a tutorial explaining how the project works on RandomMomentania. Hopefully the project is helpful :smiley:

Saw the project, very nice, but I'm fairly new to Godot so I can't say I understood all of the code, I'll work on it. I hope you make a clear tutorial so even beginners can understand, and I hope it will be a video tutorial.

So basically, even though you say there is something you call "viewport noise" which I've never heard of before (maybe you can explain), still we have 100X100X100 variations of color, which means we can have a 1000000 selectable objects in one scene, that is more then enough for sure.

But Why did you make a dictionary with a color for each object. The less work you have to do by hand is better.

I started working on a script myself, first I did a recursive function that searches all the meshinstances in the scene and put them in an array, like so:

func collect_MeshInstance(root_node):
	for node in root_node.get_children():
		if node.get_class() == "MeshInstance":
			rendered_objects_list.append(node)
		elif node.get_class() == "Spatial":
			collect_MeshInstance(node)

now I want to make a function that takes the index number of the mesh instance in the array and turn it into a color, had such a function in AS3:

		public function getColorVector(color:int):Vector.<Number>
		{
			var _red:Number = (color >> 16 & 0xFF)/255;
			var _green:Number = (color >> 8 & 0xFF)/255;
			var _blue:Number = (color & 0xFF)/255;
			return Vector.<Number>([_red,_green,_blue,1]);
		}

and then one that is converting the color back to a number and this way I can tell which mesh it is in the mesh array.

		public function ColorVectorToInt(c:Vector.<Number>):uint
		{
			var integer:int = (c[0]*255) << 16 | (c[1]*255) << 8 | (c[2]*255);
			return integer;
		}

I'll have to see how I translate these functions to GDScript, seems simple enough.

Also, to make it work automatically, I would prefer to run through the array of meshes, and assign the script you made (MeshToColorMaskMesh.gd) to each and every one of them at run time, not to have to do it manually, is there a way to do so?

Also, why do you create a new mesh instance just so you can assign a new mask material?

new_mesh_node.set_surface_material(0, masking_material);

can't I just assign it to the old mesh?

And why do I have to do it from a script which is assigned to each specific mesh instance? I crated a script/class and want everything to be handled from within the class, once I get the root spatial and collect all mesh instances in an array within this class, can this be done? or is there a specific reason why you had to assign a script to each mesh?

@DanielKotzer said: Saw the project, very nice, but I'm fairly new to Godot so I can't say I understood all of the code, I'll work on it. I hope you make a clear tutorial so even beginners can understand, and I hope it will be a video tutorial.

I will do my best! Though it will be a text based tutorial.

I currently do not have a setup for making video tutorials and my current development environment is a tad noisy for video making. One day I'd like to pursue making video tutorials, but right now it just isn't possible to make video tutorials with the quality I want.

So basically, even though you say there is something you call "viewport noise" which I've never heard of before (maybe you can explain), still we have 100X100X100 variations of color, which means we can have a 1000000 selectable objects in one scene, that is more then enough for sure.

Sure. If you print the color retrieved from the Viewport node before it has been rounded, you will find that the color isn't perfect, that the values don't quite match up to what it is supposed to.

For example, selecting a red pixel will give you something like (0.98512, 0.0, 0.0, 1.0) instead of (1.0, 0.0, 0.0, 1.0). This makes it where you cannot directly compare colors with any accuracy, so the "viewport noise" has to be removed through some simple rounding.

But Why did you make a dictionary with a color for each object. The less work you have to do by hand is better.

It was just for the example. Obviously this is not idea if you have dozens of objects or more, but I didn't want to bloat the example with more than what was necessary. The project is only intended to showcase how object picking through a color mask, not how to implement it within a game.

(For something more specific, I'd either need it myself or have to be paid to create it :tongue: )

I started working on a script myself, first I did a recursive function that searches all the meshinstances in the scene and put them in an array, like so: (Code removed to save space) now I want to make a function that takes the index number of the mesh instance in the array and turn it into a color, had such a function in AS3: (Code removed to save space) and then one that is converting the color back to a number and this way I can tell which mesh it is in the mesh array. (Code removed to save space) I'll have to see how I translate these functions to GDScript, seems simple enough.

Seems doable, especially if each object is unique and/or you are assigning a unique color to each object in the scene.

Also, to make it work automatically, I would prefer to run through the array of meshes, and assign the script you made (MeshToColorMaskMesh.gd) to each and every one of them at run time, not to have to do it manually, is there a way to do so?

Sure, just move the code in MeshToColorMaskMesh to a function and call that function for each of the meshes. It would only require a few minor modifications to have it work as an independent function.

Also, why do you create a new mesh instance just so you can assign a new mask material? 

new_mesh_node.set_surface_material(0, masking_material);

can't I just assign it to the old mesh?

You can, but then your old MeshInstance node will have to be a single uniform color and will have to be unlit. This means no textures, lights, shadows, etc, as that will mess up the color picking part of the color mask. This means your game view will look just like the color mask preview texture. If this works for your game, then go for it!

The reason I wrote the example the way I did is because it is more flexible to use a separate MeshInstance node. That way you can use textures, have shadows, etc, and still be able to accurately select the MeshInstance nodes.

And why do I have to do it from a script which is assigned to each specific mesh instance? I crated a script/class and want everything to be handled from within the class, once I get the root spatial and collect all mesh instances in an array within this class, can this be done? or is there a specific reason why you had to assign a script to each mesh?

It just made it easier as an example, there is no specific reason.

As far as doing it all at once, it is entirely doable if you take the code in MeshToColorMaskMesh and make it a single function. Everything in the _ready function should be doable in a single function, you will just need to pass in some variables to the function so it knows what to operate on and what data to use.

Hopefully this helps answer your questions! :smile:

@TwistedTwigleg said: The reason I wrote the example the way I did is because it is more flexible to use a separate MeshInstance node. That way you can use textures, have shadows, etc, and still be able to accurately select the MeshInstance nodes.

Yes but then you double the amount of geometry in the scene, which is taking a lot of computing power, can't I just save the old shaders in an array/dictionary, assign the masking shaders, take the image I need from the viewport, and then assign the old shader back?

Um, both of you, we have material overrides, and material passes... while I haven't tried to use either for something like this, I have a sneaking suspicion setting and clearing them from code might be worth a closer examination.

@DanielKotzer said: Yes but then you double the amount of geometry in the scene, which is taking a lot of computing power, can't I just save the old shaders in an array/dictionary, assign the masking shaders, take the image I need from the viewport, and then assign the old shader back?

Well, it is true that it doubles the amount of the geometry in the scene, which depending on much geometry you have in the scene, this could be a deal breaker. Are you having performance issues?

The problem with changing materials/shaders is that in Godot, there is no way to tell a Viewport to render an image, whether it be a custom Viewport or the main game Viewport. If there is a way, I have not been able to figure out what it is, despite having kept an eye out and done many Google searches on the topic.

Without the ability to tell the Viewport to render, switching the materials/shaders and caching the old materials/shaders will not work. You can indeed cache the materials/shaders and switch them during run time, but without telling the Viewport to render, you will have to wait a single frame before the color mask texture will be updated, which means the color mask shaders will be visible to the player every other frame. Even at high frame rates, this would give a flicking appearance as the materials constantly change back and forth.

I totally agree with you that changing the shaders would be the idea solution, but as far as I know this cannot work in Godot due to the inability to tell the Viewport to render an image.

That said, I am by no means a Godot expert, so it is quite possible that there is a way, but I am just unaware of it. If you can find a way to workaround/fix the issue, go for it! I just don't personally know what it would be.

(Though if you do find a way, I would be curious to how it works! I have some projects that could benefit from a workaround)

Also, on an aside, I'm not sure it would actually improve computing power, though it would reduce the amount of memory used for a scene.

Even if you could switch the shaders out and tell Godot to render the scene again, you would still be rendering the scene twice, one for the normal/player-viewable image and another for the color mask. The GPU would still have to render 120 frames a second for a 60 FPS game (60 for normal rendering + 60 for color mask rendering).

However, it would reduce the memory requirements, making the game run better on computers with limited VRAM where the amount of VRAM is the bottleneck. It would be interesting to test and see if there is a performance benefit.

@Megalomaniak said: Um, both of you, we have material overrides, and material passes... while I haven't tried to use either for something like this, I have a sneaking suspicion setting and clearing them from code might be worth a closer examination.

True, that is something I have not played around with. If there is a way to specify which render pass gets rendered to which Viewport and/or override materials only for specific Viewports, then that would be a good solution.

Otherwise, as I mentioned above, changing the material wouldn't really help. I do not think it is possible to force a Viewport to render an image, which is necessary if you want to change the materials without the player seeing the material swap in-game.

There very well could be a way, but from what research I have done online and my own personal testing, it doesn't appear to be possible currently in Godot. That said, I am by no means opposed to being wrong and figuring out there is a way! I would definitely be interested in knowing what it is, as I have my own projects that could benefit from a solution to this limitation.

Problem is you don't have enough control over the rendering process in Godot. You can't tell the engine - render now, or don't render, and the image will stay static on the screen, while you can render to an image. Unless we miss something.

Unless you can cover the screen with an image of the last rendering, and dedicated this frame, for mouse picking, hidden behind the static image.

Is there a way to tell the Godot engine to draw a frame? I was doing my own script of color mask mouse picking, and was changing the materials of all the mesh instances like this:

	for i in range(0, rendered_objects_list.size()):
		rendered_objects_list[i].set_surface_material(0, mask_material_list[i]);

then did the mouse picking from the image

	var color_mask_image = viewport.get_texture().get_data()
	color_mask_image.lock()
	var pixel_at_mouse_position = color_mask_image.get_pixelv(mouse_position)
	color_mask_image.unlock()

then returned the original materials

	for i in range(0, rendered_objects_list.size()):
		rendered_objects_list[i].set_surface_material(0, original_material_list[i]);

it works, but you don't get the color of the mask material, only the original color, why? because as TwistedTwigleg said, there was no frame change for the new colors to take effect.

So what do we need to finish this: 1. a way to tell the engine to draw a frame, like you have in other programming language where you write ValidateNow() of Refresh() or something, in between likes of code to tell the engine, don't wait for all the code and only then refresh the screen, draw the frame now. 2. we need to have a way to put the image in front of the camera just for 1-2 frame so we can pick a color , this seems simple, you just put a quad in front of the camera, with the texture from the main viewport, even just a canvaslayer or some 2D element to cover the screen for a moment. 3. what TwistedTwigleg called "viewport noise" seems to me as if we did not fully succeeded to neutralize all environmental effects on the color of the mask material. it is not the end of the world but would be good to find out how to make a completely flat rendering. 4. I would like to find a simple way to make the background black, even if I change the object materials I still have to find a way to change the background for this mouse picking frame rendering.

got a solution for the first problem: no frame draw before mouse picking to solve this I added yield:

	viewport.debug_draw = 1
	for i in range(0, rendered_objects_list.size()):
		rendered_objects_list[i].set_surface_material(0, mask_material_list[i]); 
	
	#yield(scene.get_tree(), "physics_frame")
	yield(scene.get_tree(), "idle_frame")
	yield(scene.get_tree(), "idle_frame")
``` 
it works either if I wait for a physics_frame or for 2 idle_frame, don't know what is better
also I added viewport.debug_draw = 1 for a more flat rendering, it improved the viewport noise a bit. putting it back to viewport.debug_draw = 0 when I return to normal rendering.

Now I need to put the last normal frame image, in front of the camera just until the mask material rendering and mouse picking will end.

Finally, it is working, finished the job, my project can be downloaded from google drive, here is the link:

https://drive.google.com/file/d/1gdUyoeq5EHYKMf97W3XGfECws5ikJU9x/view?usp=sharing

I'll leave it there for some time rotate and move the object by dragging it with the mouse, click right mouse button to toggle between moving and rotating, click left mouse button on object to select a part of the object to rotate or move it

Maybe one of the Godot developers, will put it in the Godot engine so it will run faster, a script is much slower and you don't have access to all the internal code of the engine.

No other way to do proper mouse picking!

One thing I didn't solve is: when I click on the scene to select an object, I put all the mask materials into the scene, then when the click ends, I want to put back all the original materials I thought I saved using .get_surface_material(0), but for some reason this function returns null, so nothing was saved, but when I use .set_surface_material(0,null) my mesh instances get back their original look meaning their original material, how could this be?

Found the answer for my last question myself.

A mesh instance has a property called mesh which is a resource, many mesh instances can be instances of the same mesh resource, the mesh has it's own material property. if you change the mesh material, all of its instances will also have a different material, but sometimes you don't want to change the resource mesh, but change only the specific mesh-instance material, that is when you use mesh-instance .get_surface_material(0) or mesh-instance .set_surface_material(0), but if you want to get or set the mesh resource material, you do:

	var m = node.mesh.surface_get_material(0)
	m.albedo_color = Color(1,0,0,1)

but then all other objects that are using this material will also change, because a material is also a resource that can be shared by many objects, so if you don't want all objects that share this material to change, you can use this code instead:

	var m = node.mesh.surface_get_material(0).duplicate()
	m.albedo_color = Color(1,0,0,1)
	node.set_surface_material(0, m);

this way you created a duplication of the original material resource, and changed it only at the mesh-instance level.

Just realized this was a discussion and not a question, so I changed it. This way you can mark replies(including your own) that have been useful as answers so that anyone else searching the net can(hopefully) easily find the thread and answers for a potential solution to their problems too. :)