housatic I'm not 100% sure but this checkbox looks like a kind of deprecated remnant that used old school OpenGL smoothing hints, which according to gl standard itself come without any guarantees. They may do something or do nothing, depending on gl implementation/driver.
Rendering with Vulkan and enabling either FXAA or 2D MSAA in project settings should anti alias those edges.
