My voxel game supports 8 viewing angles. Each voxel consists of 8 billboards, each facing one of the 8 supported camera angles. In the shader I show only the billboard that is facing the camera. My problem is that this is killing performance when I have too many voxels. The reason that the voxel engine is as performant as it is, is that it is able to do lots of culling, but when each voxel has 8 faces and none can be culled its slow. As you guys mentioned before, discarding the unused billboards in the shader is too slow. But I know culling is fast. So how can I cull these faces instead of discarding them?
How to cull specific visible faces of a mesh?
Yes, you can use discard in a fragment shader, but it will be too slow for a large world. A better method is to use degenerate triangles in a vertex shader. These are triangles where 2 or 3 of the vertices are the same point, so they are not rendered or visible on the screen. Since this in on the vertex level, it will be dramatically faster than anything on a pixel level. You can also do it without branching if you code it correctly.
However, even with the degenerate triangles, in a large world (like a Minecraft game) there may still too too many polygons. So you will want to do some culling on the CPU before the meshes are sent to the shader. BSP (binary space partitioning) or quad tree would probably be useful here. This will allow you to hide vast areas of the world (for example, voxels behind the camera, voxels that are hidden under the floor, inside buildings or caves, etc.). So that the shader only receives the most likely voxels that will be visible, and only renders them.
Doing the degenerate polygon thing can still help, on top of the CPU culling, if you really want to optimize performance. And you might still need a fragment shader for transparent voxels (water, glass, etc.) but, at that point, you know for sure the polygon is visible so performance should not be a problem.