Related discussion:
https://godotforums.org/d/35504-script-of-camera-and-movements-limits
You can try to convert deg_to_rad and see if rotation improve,
I'm using this for mine Camera 3D but on different project because involve also the movements.
First i am capture the mouse inside the window.
Then i add a spring/camera attach to the player on another node called "mount".
This fix the rotation on the model.
I guess you can do the same by rotation the scenario only
Not sure if this will help, hope still it gives you a direction on the solution.
(mini related discussion have other formula and infos)
func _unhandled_input(event: InputEvent) -> void:
if event is InputEventMouseMotion and Input.get_mouse_mode() == Input.MOUSE_MODE_CAPTURED:
mount.rotate_y(deg_to_rad( - event.relative.x * mouse_sensitivity))
spring.rotate_x(deg_to_rad( - event.relative.y * mouse_sensitivity))