If the problem is that when they bunch up is making them act silly then I suggest adding a collision shape as a child of the root node that your agent is on, then allocate a collision layer specifically for that specific unit type's collision with each other, and have the layer and mask for that layer set to true on the agent's parent.
This makes it so they don't overlap. they bunch up. make the collision shape a circle and half the diameter of the width of the agent's sprite. this makes them half overlap each other and actually looks pretty cool.
As far as path finding being jittery due to multiple agents interacting with each other I'd remove their ability to consider eachother's positions and bodies in your pathfinding. just have them barrel at their target. the collision layer implementation I suggested makes them bunch up nice like a little hoard all respectfully trying to reach their destinations.
after that the focus would be fixing the wall clipping. I have no fixed this yet.
I still haven't found how to make nav agents work properly in Godot 4 yet and I too am making a game with potential for MANY agents chasing the player through winding environments.
Ill post any findings I have in the future.
hope this is helpful.
Im using TileMap nav painting for my navmesh.
Mackbook air m1
16 GB ram