Alright.. with LLM consultation figured this out. Kinda buggy, maybe need to play with the impulse settings etc etc.. but atleast it works the way i imagined it to work without this baboonary.. shesh
So the fix is this.
First match the Rigid body layer to Character collision mask


This will detect collision but wont move the object.
Now in character script i did this:
func _physics_process(delta: float) -> void:
detec_collision()
func detec_collision():
for index in get_slide_collision_count():
var collision = get_slide_collision(index)
var body = collision.get_collider()
if body is RigidBody3D:
var rel_vel = (velocity - body.linear_velocity).length()
print("Rel vel: %.2f" % rel_vel)
if rel_vel < 0.05:
return
print("Collided: ", body.name)
var push = -collision.get_normal() * 1.0
body.apply_central_impulse(push)
Another way is to use Area3D and still need to apply impulse force to move the object..