Moving Separate Parts of Human Model

mecmec Posts: 17Member
in 3D

Hey , this is Mason
I'm recently working on a motion capture project on Godot,
I have done all the electronic parts for IMU sensors.
but there is problem with moving the body parts on Godot, IMU sensors sends Quaternion data which I send it over serial port and it's ready to use in Godot,
1.Which format should export my human model? dea , gltf or ...
2.What exactly should I do for moving different body parts?
and also all the project will be published for all users for free
at last thanks for your attention also i will be grateful for any help
Mason

Comments

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    @mec said:
    1.Which format should export my human model? dea , gltf or ...

    GLTF is probably the best choice, as it has the most support in Godot currently, and should be reasonably future proof as well.

    2.What exactly should I do for moving different body parts?

    If the model is rigged to a skeleton, than you will need to use the set_global_pose_override function (documentation) to move the bones. The one gotcha with global poses is that they are not world transforms, like global_transform, but instead are the transform relative to the Skeleton node, so more like transform on a node that is a child of the Skeleton.

  • mecmec Posts: 17Member
    edited May 31

    @TwistedTwigleg thanks for your attention and time
    for better understanding the final goal I will attach a video file, also there is not enough information about the function you mentioned.(I mean there is nothing)
    Kind regards
    Mason

  • newmodelsnewmodels Posts: 479Member

    You might find the function, under the Skeleton section of the programming manual online. 'Godot Skeleton' in google.com.

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    @mec said:
    @TwistedTwigleg thanks for your attention and time
    for better understanding the final goal I will attach a video file, also there is not enough information about the function you mentioned.(I mean there is nothing)
    Kind regards
    Mason

    For the function, as @newmodels mentioned, it is under the Skeleton node's API. To use it, you will need to get a Transform, either from a node or elsewhere, and then call the function using something like $Skeleton.set_bone_global_pose_override(bone_id, your_transform_here, 1.0, true).

    Here's a better example, which assumes the script is attached to the Skeleton node, there is a child node of the Skeleton called bone_node whose transform you want to use, and that you want to move the first bone:

    extends Skeleton
    
    var bone_node : Spatial
    var bone_id_to_manipulate = 0
    
    var is_bone_overridden = false
    
    func _ready():
        bone_node = get_node("bone_node")
        start_bone_override()
    
    func start_bone_override():
        is_bone_overridden = true
    
    func update_bone_override():
        # if we are overriding the bone pose...
        if is_bone_overridden == true:
            # set's the bone override transform
            set_bone_global_pose_override(bone_id_to_manipulate, bone_node.transform, 1.0, true);
    
    func stop_bone_override():
        # stop overriding the bone pose every _process call
        is_bone_overridden = false
        # reset the bone pose
        set_bone_global_pose_override(bone_id_to_manipulate, Transform(), 0.0, false)
    
    
    func _process(delta):
        update_bone_override()
    

    For motion capture, it is a bit more complicated, as you probably have bone chains you want to update and manipulate, but it's certainly doable! What I would recommend doing is making a Bone3D node/class to help make the bone manipulation easier and more straightforward. It is pretty much what I wrote above, but you'll want to make the script on a Spatial node instead of a Skeleton (so you can have multiple) and probably will want to add checks and exported variables, for easier customization.

    If you do not mind using C# and a plugin, I have written a IK plugin for Godot that already has a Bone3D node that makes this process a bit easier. It is free for non-commercial use, so as long as you are not sharing the source code for the IK plugin and are not using it commercially, the free version should be usable for your project.

    If you are looking to make your project commercial or release the source code, then the C# IK plugin will not work. However, this PR has modifications to the BoneAttachment node that basically makes it possible to optionally be a Bone3D node, so that may be worth looking at. It is in C++ though, but it may helpful as a reference.

  • mecmec Posts: 17Member
    edited June 2

    @TwistedTwigleg
    Thanks again for your attention , It totally works.
    but yet I have another question :
    If we consider W,X,Y,Z as Quaternion data from Sensors how should I relate them as rotation data to individual bone?
    in this command if I'm not wrong the transform data is related to "bone_node.transform" : set_bone_global_pose_override(bone_id_to_manipulate, bone_node.transform, 1.0, true);
    what should I use instead of "bone_node.transform" for quaternion data
    I already wrote a function to convert quaternion data to Transfer data type but the results are not valid at all :*

    func quat_transform(x, y, z, w):
    var t = $"Physical Bone Bone".get_transform() #Physical Bone Bone is the exact bone that I want to move
    var myRot = Quat(x, y, z, w)
    var m = Basis(myRot)
    var n = Transform(m.x.normalized(), m.y.normalized(), m.z.normalized(), t.origin)

    also @newmodels thanks for guiding.
    Thanks Again
    Mason

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    @mec said:
    @TwistedTwigleg
    Thanks again for your attention , It totally works.
    but yet I have another question :
    If we consider W,X,Y,Z as Quaternion data from Sensors how should I relate them as rotation data to individual bone?

    You will need to construct a Transform with the desired rotation basis. The trickier part here is making sure the position is correct, especially if all you have is rotation data from a Quaternion.

    To construct the Transform, you just need to do something like this:

    func quat_transform(x, y, z, w):
        var bone_trans = $"Physical Bone Bone".get_transform()
        var myRot = Quat(x, y, z, w)
        var myBasis = Basis(myRot)
    
        # then you can just set the basis property of the bone trans
        bone_trans.basis = myBasis;
        # or return a transform with the desired rotation instead
        # var return_trans = bone_trans.duplicate()
        # return_trans.basis = myBasis;
        # return return_trans;
    

    However, this does not take bone child relationships into account, which is something to be aware of if you have several bones in a row and you want to move only one of them (like the upper arm but not the lower arm, for example). The easiest way to keep bones in the correct parent-child relationship is to setup the Bone3D-like nodes in your scene where child bones are children of the Bone3D-like node that represents the parent. For example:

    • UpperArm_Bone3D
      • LowerArm_Bone3D
      • Hand_Bone3D

    That way, when you rotate something like UpperArm_Bone3D, the child bones will be rotated as well, giving a more realistic (and likely expected) result.

  • mecmec Posts: 17Member

    Hi again @TwistedTwigleg ,
    Sorry for delay, but I was testing all the possible ways to make it work and it took so much time.
    Anyways, there are some real problems with location(origin) of rotation and parenting that I couldn't solve it.

    In this video I tried to move Arm and Forearm separately, as you can see rotation origin is (0,0,0) and when the arm is moving the forearm should move with that but it's not.

    Here is the script, q variables are the Quaternion data which I'm getting from IMU sensors:

    extends Skeleton
    
    var loadedScript = load("res://Model/metarig.py")
    var loadedScriptInstance = loadedScript.new()
    
    var bone_node : Spatial
    var bone_node_2 : Spatial
    
    var bone_id_to_manipulate = 8
    var bone_id_to_manipulate_2 = 9
    
    var is_bone_overridden = false
    
    var q0=0
    var q1=0
    var q2=0 
    var q3=0
    var q4=0
    var q5=0
    var q6=0
    var q7=0
    
    var trans
    var trans2
    
    func _ready():
    #   bone_node = get_node("bone_node")
    #   bone_node_2 = get_node("bone_node_2")
        start_bone_override()
    
    
    func start_bone_override():
        is_bone_overridden = true
    
    
    func update_bone_override():
        # if we are overriding the bone pose...
        if is_bone_overridden == true:
            # set's the bone override transform
            set_bone_global_pose_override(bone_id_to_manipulate, trans, 1.0, true);
            set_bone_global_pose_override(bone_id_to_manipulate_2, trans2, 1.0, true);
    
    
    func stop_bone_override():
        # stop overriding the bone pose every _process call
        is_bone_overridden = false
        # reset the bone pose
        set_bone_global_pose_override(bone_id_to_manipulate, Transform(), 0.0, false)
        set_bone_global_pose_override(bone_id_to_manipulate_2, Transform(), 0.0, false)
    
    
    func _process(_delta):
        var dataPacket = loadedScriptInstance.myProcess()
        var splitPacket=dataPacket.split(',')
    
        q0 = splitPacket[0]
        q1 = splitPacket[1]
        q2 = splitPacket[2]
        q3 = splitPacket[3]
        q4 = splitPacket[4]
        q5 = splitPacket[5]
        q6 = splitPacket[6]
        q7 = splitPacket[7]
    
        q0 = float(q0)
        q1 = float(q1)
        q2 = float(q2)
        q3 = float(q3)
        q4 = float(q4)
        q5 = float(q5)
        q6 = float(q6)
        q7 = float(q7)
    
        quat_transform(q1, q2, q3, q0)
        quat_transform2(q5, q6, q7, q4)
    
        update_bone_override()
    
    
    func quat_transform(x, y, z, w):
        #var bone_trans = $"Physical Bone Bone".get_transform()
        var bone_trans = $"Physical Bone shoulderL/Physical Bone upper_armL".get_transform()
        var myRot = Quat(x, y, z, w)
        var myBasis = Basis(myRot)
        # then you can just set the basis property of the bone trans
        bone_trans.basis = myBasis;
        trans = Transform ( myBasis )
        # or return a transform with the desired rotation instead
        # var return_trans = bone_trans.duplicate()
        # return_trans.basis = myBasis;
        # return return_trans;
    
    func quat_transform2(x, y, z, w):
        #var bone_trans = $"Physical Bone Bone".get_transform()
        var bone_trans2 = $"Physical Bone shoulderL/Physical Bone upper_armL/Physical Bone forearmL".get_transform()
        var myRot2 = Quat(x, y, z, w)
        var myBasis2 = Basis(myRot2)
        # then you can just set the basis property of the bone trans
        bone_trans2.basis = myBasis2;
        trans2 = Transform ( myBasis2 )
        # or return a transform with the desired rotation instead
        # var return_trans = bone_trans.duplicate()
        # return_trans.basis = myBasis;
        # return return_trans;
    

    also I did tried the same process for a simple cube and here is the result:

    The length of the cube is enough to cover the space between two black blocks, but it gets short when I'm trying to move it also as you can see deformation of mesh is not good at all because it seems a part of a cube is shifted inside.
    Thanks again for your attention.
    Kind Regards,
    Mason

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    It is probably because the child bone's transform isn't in global_pose space, so it's origin is offset incorrectly.

    Try modifying the global transforms for the bones and using this function to convert it to a global pose:

    func global_transform_to_global_pose(p_skeleton, p_transform):
        p_skeleton.global_transform.affine_inverse() * p_transform;
    

    Then you just need to adjust a few functions. Adjustments below:

    func update_bone_override():
        # if we are overriding the bone pose...
        if is_bone_overridden == true:
            # set's the bone override transform
            set_bone_global_pose_override(bone_id_to_manipulate, global_transform_to_global_pose(self, trans), 1.0, true);
            set_bone_global_pose_override(bone_id_to_manipulate_2, global_transform_to_global_pose(self, trans2), 1.0, true);
    
    func quat_transform(x, y, z, w):
        var bone_trans = $"Physical Bone shoulderL/Physical Bone upper_armL".get_transform()
        var myRot = Quat(x, y, z, w)
        var myBasis = Basis(myRot)
        bone_trans.basis = myBasis;
        # just assign to bone_trans, as we want to keep the origin (position) of the bone the same
        trans = bone_trans;
    
    func quat_transform2(x, y, z, w):
        var bone_trans2 = $"Physical Bone shoulderL/Physical Bone upper_armL/Physical Bone forearmL".global_transform
        var myRot2 = Quat(x, y, z, w)
        var myBasis2 = Basis(myRot2)
        bone_trans2.basis = myBasis2;
        # just assign to bone_trans2, as we want to keep the origin (position) of the bone the same
        trans2 = bone_trans2;
    
  • mecmec Posts: 17Member
    edited June 12

    Hi again,
    Dear @TwistedTwigleg I really appreciate for your help, there is a real progress with your guides.
    As you said I did some adjustments in code and there is some improvements , also some new issues :
    1. When I'm moving just Left Arm everything is ok except scale (You will notice in first gif)
    2. When I'm moving the Left Arm and Forearm the parenting concept problem is still remaining, also the scaling problem too.
    I will post the script and both condition gifs for better understanding the problem:

    extends Skeleton
    
    var loadedScript = load("res://Model/metarig.py")
    var loadedScriptInstance = loadedScript.new()
    
    
    var bone_id_to_manipulate = 8
    var bone_id_to_manipulate_2 = 9
    
    var is_bone_overridden = false
    
    var q0=0
    var q1=0
    var q2=0 
    var q3=0
    var q4=0
    var q5=0
    var q6=0
    var q7=0
    
    var trans
    var trans2
    
    var x
    
    
    func _ready():
        start_bone_override()
    
    
    func global_transform_to_global_pose(p_skeleton, p_transform):
        x = p_skeleton.global_transform.affine_inverse() * p_transform;
    
    
    func start_bone_override():
        is_bone_overridden = true
    
    
    func update_bone_override():
        # if we are overriding the bone pose...
        if is_bone_overridden == true:
            # set's the bone override transform
            global_transform_to_global_pose(self, trans)
            set_bone_global_pose_override(bone_id_to_manipulate, x, 1.0, true);
    
    
            set_bone_global_pose_override(bone_id_to_manipulate_2, global_transform_to_global_pose(self, trans2), 1.0, true);
            global_transform_to_global_pose(self,trans2)
            set_bone_global_pose_override(bone_id_to_manipulate_2, x, 1.0, true);
    
    
    
    func stop_bone_override():
        # stop overriding the bone pose every _process call
        is_bone_overridden = false
        # reset the bone pose
        set_bone_global_pose_override(bone_id_to_manipulate, Transform(), 0.0, false)
        set_bone_global_pose_override(bone_id_to_manipulate_2, Transform(), 0.0, false)
    
    
    
    
    
    func _process(_delta):
        var dataPacket = loadedScriptInstance.myProcess()
        var splitPacket=dataPacket.split(',')
        q0 = splitPacket[0]
        q1 = splitPacket[1]
        q2 = splitPacket[2]
        q3 = splitPacket[3]
        q4 = splitPacket[4]
        q5 = splitPacket[5]
        q6 = splitPacket[6]
        q7 = splitPacket[7]
        q0 = float(q0)
        q1 = float(q1)
        q2 = float(q2)
        q3 = float(q3)
        q4 = float(q4)
        q5 = float(q5)
        q6 = float(q6)
        q7 = float(q7)
    
        quat_transform(q1, q2, q3, q0)
        quat_transform2(q5, q6, q7, q4)
    
        update_bone_override()
    
    
    func quat_transform(x, y, z, w):
        var bone_trans = $"Physical Bone shoulderL/Physical Bone upper_armL".get_transform()
        var myRot = Quat(x, y, z, w)
        var myBasis = Basis(myRot)
        bone_trans.basis = myBasis;
        trans = bone_trans
    
    
    
    func quat_transform2(x, y, z, w):
        var bone_trans2 = $"Physical Bone shoulderL/Physical Bone upper_armL/Physical Bone forearmL".global_transform
        var myRot2 = Quat(x, y, z, w)
        var myBasis2 = Basis(myRot2)
        bone_trans2.basis = myBasis2;
        trans2 = bone_trans2
    

    and here is the gifs:
    First one is only for Left Arm which everything is ok but scale

    This one is for moving two bones, interesting thing about this one is location of forearm is kind of correct but kind of

    Kind Regards,
    Mason

  • mecmec Posts: 17Member
    edited June 12

    I Hope it's all clear

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    Right, that makes sense with the scale. Scale data is stored in the Basis along with rotation, so when you construct a new Basis you will need to preserve the scale as well.

    Try adding this to the global_transform_to_global_pose function:

    func global_transform_to_global_pose(p_skeleton, p_transform):
        x = p_skeleton.global_transform.affine_inverse() * p_transform;
        x.basis = x.basis.orthonormalized()
    

    This will set the global pose's scale to be (1, 1, 1), which I think should fix the scale issue you are having. If not, then you'll want to get the global pose scale for each bone, and then apply that, using something like this:

    x.basis = x.basis.orthnormalized().scaled(global_pose_bone_scale_here)
    
  • mecmec Posts: 17Member
    edited June 13

    Hey @TwistedTwigleg , R U a Orthopedic doctor?
    Cause you know how to deal with bones ;) appreciation
    As you said I just added the following line of code and the scaling problem is settled.

    x.basis = x.basis.orthonormalized()
    

    and this is the result for a single bone (Left_Arm)

    but when I'm trying to move the forearm too , here is what happens

    as you can see mesh deformation is not logical and there is no parenting concept too
    thanks again
    kind regards
    Mason

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    @mec said:
    Hey @TwistedTwigleg , R U a Orthopedic doctor?
    Cause you know how to deal with bones ;) appreciation
    As you said I just added the following line of code and the scaling problem is settled.

    x.basis = x.basis.orthonormalized()
    

    and this is the result for a single bone (Left_Arm)

    but when I'm trying to move the forearm too , here is what happens

    as you can see mesh deformation is not logical and there is no parenting concept too
    thanks again
    kind regards
    Mason

    LOL, no, I'm not a doctor of any type! I've just written too many IK algorithms :lol:

    What does your node tree look like? Also, I would double check to make sure that the code is getting the global transform for both bones, rather than the transform, since the global_transform_to_global_pose needs global transforms now.

  • mecmec Posts: 17Member

    Hi @TwistedTwigleg , thanks again for your attention
    Here is my node tree:

    I forgot to say, I exported the model in blender with glTF Embedded format.
    I hope it helps to understand the problem cause I'm disturbing you so often.
    Kind Regards,
    Mason.

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    It looks like the scene is setup fine for the shoulder, upper arm, and fore arm. I wonder if the issue is the use of PhysicalBones then, perhaps their physics update is causing the issue? Looking at the GIF, it seems the issue is that when the upper arm rotates, the fore arm stays in the same position even though it's parent node rotated. This might be because it is a physics node and so it ignores the parent's transform.

    If you add a MeshInstance as a child of the upper arm and position it roughly where the fore arm is (like make it a scaled cube that roughly encompasses the size of the fore arm) does it rotate correctly? If it rotates correctly with the upper arm, then it is probably the use of PhysicalBone nodes that is causing the issue since it's a physics node (and therefore doesn't take parent transforms into account).

  • mecmec Posts: 17Member
    edited June 26

    Hi @TwistedTwigleg
    Sorry for late reply,
    I tried adding Mesh Instance as a child for Upper Arm (I didn't write any scripts for that) but it didn't work.
    I don't know if its a good news or a bad one.
    Here is what I did:

    thanks again for your attention.
    Kind Regards,
    Mason

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    Great, that should help with figuring out the issue.

    Does the added MeshInstance node position and rotate itself correctly when the upper arm bone rotates? If it is rotated and positioned correctly, then that would pretty much confirm that the physics in the PhysicalBone3D is causing the issue. If it doesn't work, then it may still be the physics, but it could be something else as well.

    The hard part with debugging this kind of thing is that it's very fiddly! Sorry for having to ask so many questions, just trying to figure out where the problem might be :smile:

  • mecmec Posts: 17Member
    edited June 27

    Thanks for fast respond, @TwistedTwigleg
    The added MeshInstance is not moving with upper arm,
    Don't be sorry at all cause It's nice of you
    Thanks again for your attention
    Mason

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    Well unfortunate that it's not moving since it doesn't get us any closer to a solution.

    I'll try to see if I can get a simple scene working using a similar 3D model and rotate it using the code a few posts back and see if I can figure out what is going on :+1:

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    Okay, I think I got an example working! It is a bit rough, but it uses a Quat set with individual values (four sliders in this case) and the lower arm correctly follows the upper arm's rotation.

    I've attached the project to this comment :smile:

  • mecmec Posts: 17Member

    @TwistedTwigleg I don't know how to appreciate you, a simple thank you won't cut it but huge thanks for everything.
    I will test it with my own model and sensor data and I will come back with results.
    Kind Regards
    Mason

  • mecmec Posts: 17Member

    Hey @TwistedTwigleg
    Recently I implemented IMU data on Skeleton Project which you kindly built, In this gif I attached two IMU sensors to my arm and forearm and its done. I'm gonna attach the results down here:

    Appreciated
    yet I have some question about the project
    You added two spatial nodes Bone_UpperArm and Bone_LowerArm and attached a script for both of it. I want to know the reason
    Regards
    Mason

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    Awesome, I’m glad it’s working!

    yet I have some question about the project
    You added two spatial nodes Bone_UpperArm and Bone_LowerArm and attached a script for both of it. I want to know the reason

    Sure. The script on each of the Bone_ nodes is a script that simply takes the GlobalTransform of the Spatial and applies it to the bone. I wrote the script so it could work with any bone, and simply by changing the exported variables you could change which bone the node effects.

    The reason I did it this way is that I find it’s more flexible. The same script can be used for as many bones as you want to influence, you just need to make a Spatial node per bone and assign the exported variables. The other reason for doing it this way is that it keeps the code that changes the GlobalTransform separate from the code that applies it to the bones, which also makes the code a little cleaner in my opinion.

    That’s the reason why though! It is totally possible to control both bones through a single script, it’s just not the route I took for the example project.

  • mecmec Posts: 17Member
    edited September 14

    Hi again @TwistedTwigleg ,
    I hope your well,
    Would you please say some more information about BoneAttachment node that your using in projects.
    What should I do for moving whole left and right arm in the same time?

    Regards
    Mason

  • TwistedTwiglegTwistedTwigleg Posts: 4,771Admin

    @mec said:
    Hi again @TwistedTwigleg ,
    I hope your well,
    Would you please say some more information about BoneAttachment node that your using in projects.

    Sure!

    I am using a custom Bone3D like node for my projects, specifically for my IK plugin Twisted IK 2. The bone setting part of the node is pretty simple, it attaches a signal to the idle_frame or physics_frame signal in the SceneTree, and then uses that to apply and then get the global pose transform.

    There is a bit more to it for IK related stuff, like getting the bone length and correctly rotating the bone directions so forward for the bone is -Z even if the base model uses a different bone forward axis, but that’s really mostly just extras. The important parts is converting the transforms from a GlobalTransform to a GlobalPose (which I think my example had code to do?) and setting up the scene correctly so child bones nodes are correctly modified by their parent bones nodes.

    What should I do for moving whole left and right arm in the same time?

    The easiest solution is to recreate the bone hierarchy exactly as it is in the 3D file. For example, if you want to move the whole left arm, then you’ll want something like this as your Bone3D-like nodes:

    = Shoulder_Left
    == Forearm_Left
    === Lowerarm_Left
    ==== Hand_Left
    ===== Etc
    = Shoulder_Right
    == Forearm_Right
    ==== Lowerarm_Right
    ===== Hand_Right
    ====== Etc

    Then you want to follow and apply the bones in the correct order. If you have it like the example scene tree setup above, then it’s normal process order should be fine, as it will do the children and then the parents as expected without any issue. The reason it has to be like this is because the global_pose_override only affects the bone it’s applied to, so child bones use the non override pose and then become detached and look visually strange.

    In Godot 4.0 this won’t be totally necessary, as you can override the local pose using set_local_pose_override, but this function isn’t in Godot 3.X and so recreating the bone hierarchy in the scene is sadly the only way to replicate the functionality where you can modify bones and have all the children bones correctly update.


    Hopefully that helps and makes sense! If anything isn’t clear, do not hesitate to ask and I’ll do my best to answer when I have the time :smile:

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file