Antialiasing (smoothing) a texture

Use a viewport container if you can, or create a separate scene to experiment with it in there, the viewport container should help you get more familiar with viewports. Once you understand it better it shouldn't be hard to figure this out. I recon that's probably better than me painting it red for you, though if you still can't figure it out I guess I can do that too, it's mostly that I'm currently not behind my workstation where I have godot installed.

Thinking about it some more I recalled that I have already produced and shared a demo making use of a viewport container, so I can just share that link here.

https://drive.google.com/file/d/1KY-Rn4OW295R2p8Tvx_e1sdtsWQOC7T-/view?usp=sharing

It's a demo for a custom brightness, contrast, saturation control on a viewport. Not perhaps exactly what you need, but might give some guidance. It's not creating a viewport from code IIRC but it certainly should give you essentially an idea of how it works.

Right, the AA setting is what I was thinking of too, if you need mipmaps that should be a project setting.

@DanielKotzer Now the question is, what level of MSAA should I select? You have different levels there - 2x2 4x4 8x8 16x16, I believe it is all a matter of computing power vs image quality, so what would you recommend would not be too much of a burden on the average computer?

From what I can gather, it looks like MSAA performance depends on scene complexity and the size of the screen. This StackExchange post gives a rough explanation of how MSAA works and the performance cost.

There is also this comparison by SapphireNation, that gives charts showing the performance difference for a bunch of different anti-aliasing methods, including comparing MSAA 2x2 and MSAA 4x4.

I would say, that probably 2x2 or 4x4 is a good bet if you want decent performance on most consumer machines while still having MSAA. On mobile or lower end devices, I would try 2x2 or disabling MSAA altogether.

Another option is having a default of something like 2x2 and allow the user to raise or lower the value through an options menu or config file.

I wonder - iPhone and Android devices are supporting GLES3? and if so, from what version of iPhone and Android is it supported?

Last I knew, as long as the iOS or Android devices supports OpenGL ES 3, then it should be able to run GLES3 exported Godot projects. This page on Wikipedia lists a bunch of different devices that support OpenGL ES 3.

The best way to know would be to test on a device though, and see what the performance looks like. I imagine that most of the newer phones/tablets should be able to run most Godot projects with little difficulty, so long as there is not too much post-processing or scene complexity.

FXAA is basically a fragment shader process, so something you could potentially implement as a shader yourself. Essentially you'd do some edge detection and selectively blur by sampling the depth pass per each detected edge pixel/fragment. It's perhaps not as useful for textures.

FXAA is a image effect, it is not aware of the 3D scene, put rather just the pixels in the texture. Because it works with the pixels of the image, it has to rely on some shader tricks to detect what is an edge and what is not. This is unlike MSAA, which instead samples the scene multiple times to provide anti-aliasing.

Depending on the implementation used to detect edges, FXAA can look pretty good, but depending on the textures used, the geometry used, and other factors (like transparency), FXAA can get confused and apply anti-aliasing to the wrong things or not apply it to the right things.

FXAA can have better performance in comparison to MSAA, as the performance of FXAA is largely dependent on the size of the screen, not the complexity of the scene like MSAA. However, performance varies depending on the complexity of the FXAA edge detection shader code.

If you are curious about FXAA, I would highly suggest searching things like "FXAA vs MSAA" and "FXAA how it works" in a search engine (like Google) and explore the results. Doing research and investigating what is available will help give you an idea on what is best for your project.

You can find a FXAA effect in Godot, written by the team developing Hight Hat with Godot: Link to GitHub Gist.

SMAA is pretty simple and gives the best results, but is extremely heavy on performance. The idea behind SMAA, as far as I understand it, is to render the game at a higher resolution that the display, and then resize it down to the size of the display. This will interpolate the colors and reduce jagged edges.

The huge issue with SMAA is that you have to render the game at a much higher resolution than what will be outputted to the player for it to look nice. This means the GPU will have to work extra hard to render the scene at a higher resolution. For example, to use SMAA on an 1920x1080 (HD) display, you would have to render the screen at 3840x2160 resolution to have SMAA x2.

SMAA is a brute force approach to fix jagged edges, and because of that is one of the most inefficient solutions (though visually it is probably the nicest looking). As far as I can tell, there is no SMAA for Godot, with good reason because for many real time applications SMAA is simply too performance hungry.

Again, if you are interested to learn more, you can always search something like "SMAA vs MSAA" and "SMAA how it works" in a search engine and explore the results.

@DanielKotzer said:

But my texture is a rendered scene. can you be more specific? show me how it is done or refer me to a tutorial.

I was avoiding making assumptions, you could have easily been rendering a procedural texture into that viewport.

@Calinou said:

SMAA is pretty simple and gives the best results, but is extremely heavy on performance. The idea behind SMAA, as far as I understand it, is to render the game at a higher resolution that the display, and then resize it down to the size of the display. This will interpolate the colors and reduce jagged edges.

You're talking about SSAA here (super-sample antialiasing), not SMAA (subpixel morphological antialiasing) ;)

My bad! Yeah, I was talking about SSAA :sweat_smile: