I think you can just change the collision mask to turn off collisions for the objects once they are in place. This page on the documentation explains how the collision layer/mask system in Godot works. To be fair, the collision layer/mask system is rather confusing, and I have been using Godot for quite awhile now.
That way you could either move the objects when you no longer want them to collide to a different collision layer, and/or you could change the collision mask so that it no longer detects collisions.
Another thing you can potentially do is disable the collision shape(s) of the objects once you no longer need them. You just need to set the disabled property to true in the collision shape nodes and then they will no longer be used to process collisions.
Hopefully one of those methods will stop them from colliding with each other :smile: