Update 2:
Ran into the same bloating issue when reconstructing the new scene, but I also managed to find the specific action which causes the file size to multiply by hundreds.
In the scene each button has three label objects using the same root LabelSettings, however to adjust the font sizes etc. I used Make Unique on two of them. I apparently picked the recursive option on one, which caused the massive jump in memory usage.
Before using Make Unique (Recursive):
And after:
Using recursive generates a PackedByteArray containing over 500 thousand characters, as well as hundreds of rows of cache data