How do I make a custom 3D collider shape?

Okay, I did some further testing with ConcavePolygon.

As far as I can tell, setting the triangles in ConcavePolygon does work. The visual debug shapes for ConcavePolygon shapes does not work though, which makes it hard to tell the shape of the ConcavePolygon.

I'm not sure why the debug visual doesn't work, but I'm guessing like @Megalomaniak said, the mesh block/input being empty might have something to do with it.

I found a simple workaround that uses the SurfaceTool to generate a line mesh like the visible collision mesh debug option. After writing the code below, I can now see the collision mesh and everything seems to work as expected.

Here is the code I used:

extends CollisionShape

export (bool) var show_debug_shape = false;
export (int, "lines", "triangles") var debug_shape = 0;

var triangles = PoolVector3Array();

func _ready():
	triangles = PoolVector3Array();
	
	generate_collision_cube();
	
	if (show_debug_shape == true):
		if (debug_shape == 0):
			generate_concave_visual(triangles, "lines");
		else:
			generate_concave_visual(triangles, "triangles");


func generate_collision_cube():
	
	var CES = 1; # Cube Extends Size
	
	# Top
	generate_quad(Vector3(-CES, CES, -CES), Vector3(-CES, CES, CES), Vector3(CES, CES, CES), Vector3(CES, CES, -CES));
	# Bottom
	generate_quad(Vector3(-CES, -CES, -CES), Vector3(-CES, -CES, CES), Vector3(CES, -CES, CES), Vector3(CES, -CES, -CES));
	
	# Forward
	generate_quad(Vector3(-CES, -CES, CES), Vector3(-CES, CES, CES), Vector3(CES, CES, CES), Vector3(CES, -CES, CES));
	# Backward
	generate_quad(Vector3(-CES, -CES, -CES), Vector3(-CES, CES, -CES), Vector3(CES, CES, -CES), Vector3(CES, -CES, -CES));
	
	# West
	generate_quad(Vector3(CES, -CES, -CES), Vector3(CES, -CES, CES), Vector3(CES, CES, CES), Vector3(CES, CES, -CES));
	# East
	generate_quad(Vector3(-CES, -CES, -CES), Vector3(-CES, -CES, CES), Vector3(-CES, CES, CES), Vector3(-CES, CES, -CES));
	
	if (shape is ConcavePolygonShape):
		shape.set_faces(triangles);



func generate_quad(position_one, position_two, position_three, position_four):
	# First triangle
	triangles.append(position_one);
	triangles.append(position_two);
	triangles.append(position_three);
	
	# Second triangle
	triangles.append(position_one);
	triangles.append(position_three);
	triangles.append(position_four);



func generate_concave_visual(triangles_array, mode = "lines"):
	var surface_tool = SurfaceTool.new();
	
	if (mode == "lines"):
		# Line mesh:
		surface_tool.begin(Mesh.PRIMITIVE_LINES);
		
		# Go through the array adding lines.
		# NOTE: This code should add two lines per vertex, which makes them a little easier to see.
		var i = 0;
		while (i < triangles_array.size()):
			surface_tool.add_vertex(triangles_array[i]);
			if (i > 1):
				surface_tool.add_vertex(triangles_array[i-1]);
			i += 1;
		
		# Add the last line:
		surface_tool.add_vertex(triangles_array[triangles_array.size()-3]);
		surface_tool.add_vertex(triangles_array[triangles_array.size()-1]);
	
	
	elif (mode == "triangles"):
		# Solid mesh:
		surface_tool.begin(Mesh.PRIMITIVE_TRIANGLES);
		
		for vertex in triangles_array:
			surface_tool.add_vertex(vertex);
	
	
	var collision_mat = SpatialMaterial.new();
	collision_mat.albedo_color = Color.green;
	collision_mat.flags_unshaded = true;
	# NOTE: Disable culling because triangle order isn't quite right in the add_quad function.
	# however, this only effects normals and shouldn't matter for collision geometry.
	collision_mat.params_cull_mode = SpatialMaterial.CULL_DISABLED;
	
	surface_tool.set_material(collision_mat);
	
	var mesh_node = MeshInstance.new();
	mesh_node.mesh = surface_tool.commit();
	
	add_child(mesh_node);

I tried to write generate_concave_visual so that it is independent of the rest of the code so it can be more easily used in other projects.

Also, I should note that the generate_quad function doesn't take the order of the inputted vertices into account, which is fine for collision mesh generation, but for drawing the mesh it requires the cull mode being disabled.

Hopefully this helps!