Well, just tried updating the colors on the sRGB profile in Aseprite and exporting that and now it looks the same between Aseprite and Godot, but looks different in Affinity, which has color profile already set as sRGB... So Im not sure this is the best way of doing this.
I also found this issue in github, which is probably the exact same issue, without solution unfortunately. And in that case, I do think the issue is with Godot.
Im not sure how to proceed, using the sRGB profile in Aseprite so it looks ok in Godot but wrong everywhere else is not ideal solution.
I also found some more discussions where people suggested the same thing as @SnapCracklins where setting the sRGB to enabled in the import settings helped, but again. that was in 3.X version and is not present in the 4.X version.