I ran into something similar, and what ended up working pretty well for me was setting up a basic backend using a cloud call center. It gave me enough flexibility to connect voice between clients without having to host everything myself. Took some fiddling to get audio levels and echo right, but once it clicked, it was smooth. Still fiddling with better NAT traversal though.
