Trying to implement the "pick up" and "put down" object actions

[basteran]

Hi, thanks for the great work. This tool looks very interesting and I was trying to implement the "pick up" and "put down" object actions as in your documentation and tutorials (btw they are great!) you mention that these are not built-in.

I am facing some issues when executing the "pick up" action. This is the error from Python.

The build quit due to an error. Check the build log for more info.
If the build is on the same machine as this controller, the log path is probably C:\Users\user\AppData\Local\Unity\Editor\Editor.log
If the build is on a remote Linux server, the log path is probably ~/.config/unity3d/MIT/TDW/Player.log (where ~ is your home directory)        

Here I paste my code, maybe you can give me some feedback or ideas.

from tdw.controller import Controller
from tdw.add_ons.floorplan import Floorplan
from tdw.add_ons.first_person_avatar import FirstPersonAvatar
from tdw.output_data import OutputData, Transforms


"""
Generate a floorplan scene and populate it with a layout of objects.
User implementation of "pick up" and "put down" actions.
"""

c = Controller()
a = FirstPersonAvatar(position={"x": 0, "y": 0, "z": 4})

# Initialize the floorplan add-on.
floorplan = Floorplan()
# Scene 5, visual variant c, object layout 2.
floorplan.init_scene(scene="5c", layout=2)

c.add_ons.extend([floorplan, a])
# Initialize the scene.
c.communicate([])
# Make the image 1080p and hide the roof.
# c.communicate([{"$type": "set_screen_size",
#                 "width": 1920,
#                 "height": 1080}])
c.communicate([{"$type": "set_screen_size",
                "width": 1280,
                "height": 720}])


def calculate_distance(position1, position2):
    return ((position1['x'] - position2['x'])**2 + (position1['y'] - position2['y'])**2 + (position1['z'] - position2['z'])**2) ** 0.5

def get_object_position(object_id):
    resp = c.communicate({"$type": "send_transforms", "ids": [object_id]})
    for r in resp:
        # Check if this part of the response is a "Transforms" message
        if OutputData.get_data_type_id(r) == "tran":
            transforms = Transforms(r)
            for i in range(transforms.get_num()):
                object_id_from_response = transforms.get_id(i)
                if object_id_from_response == object_id:
                    position_list = transforms.get_position(i).tolist()
                    return {
                        "x": position_list[0],
                        "y": position_list[1],
                        "z": position_list[2]
                    }


def get_avatar_current_position():
    global a
    return {"x": a.transform.position.tolist()[0], "y": a.transform.position.tolist()[1], "z": a.transform.position.tolist()[2]}


done = False
grasp_range = 2.0  # Define grasp range
held_object_id = None  # Variable to store the object ID if holding one

while not done:
    c.communicate([])

    if a.left_button_pressed:
        if held_object_id is not None:
            # Drop the currently held object
            c.communicate({"$type": "drop_object", "id": held_object_id})
            held_object_id = None  # Reset the held object
        elif a.mouse_is_over_object:
            object_id = a.mouse_over_object_id
            object_position = get_object_position(object_id)  # function to get object's position
            # Calculate distance between avatar and object
            avatar_position = get_avatar_current_position()
            distance = calculate_distance(avatar_position, object_position)
            
            # Only grasp the object if it's within range
            if distance <= grasp_range:
                # Grasp the object
                c.communicate({"$type": "grasp_object", "id": object_id, "grasping_avatar": a.avatar_id})
                held_object_id = object_id  # Store the held object ID
                print(f"Grasped object '{held_object_id}'")
            else:
                print("Object is out of range!")
    
    if a.right_button_pressed:
        done = True

        
c.communicate({"$type": "terminate"})

Now the error says to check the Editor.log file and it's empty! Do you have any ideas or feedback? Is this feasible? Or do you know any implementation of it?

Thank you once again!

[alters-mit]

Hi, thanks for sending this.

Now the error says to check the Editor.log file and it's empty!

Check the Player log, not the Editor log. The error message is making some assumptions about how you ran your controller that I guess aren't good assumptions.

The fundamental problem in your code is that you're trying to define new commands. The logic behind commands is stored in the build, not Python. Thus, drop_object and grasp_object will throw an error. Instead, you must use a list of existing commands to achieve the same behavior. grasp_object might be equivalent to (in pseudo-code)

[
    set_kinematic_state, // Temporarily disables physics on this object
    parent_object_to_avatar, 
    teleport_object, // Snap the object to a "hand" position, using the "absolute": False to set a local position.
    rotate_object_to // Optional if you want to set the object to a canonical rotation
]

And drop_object would be something like:

[
    set_kinematic_state, // Enable physics
    unparent_object
]

Note that I'm defining lists of commands because you can send a list of multiple commands that will all execute (sequentially) on the same physics step. Under the hood, this is how TDW achieves most of its complex behavior (especially agent actions).

Hope that helps.

[basteran]

Thank you for your response!

Check the Player log, not the Editor log. The error message is making some assumptions about how you ran your controller that I guess aren't good assumptions.

I found it, thanks! It was a bit tricky to locate, though.

The fundamental problem in your code is that you're trying to define new commands. The logic behind commands is stored in the build, not Python. Thus, drop_object and grasp_object will throw an error. Instead, you must use a list of existing commands to achieve the same behavior. grasp_object might be equivalent to (in pseudo-code)

So, just to clarify, the drop_object and grasp_object commands are not predefined? And I should implement their behavior using the existing list of commands you mentioned? I’ll give that a try.

One more question: Is there any documentation or a list of all the commands available? I was under the impression that grasp_object was a valid command!

[alters-mit]

So, just to clarify, the drop_object and grasp_object commands are not predefined?

Correct.

And I should implement their behavior using the existing list of commands you mentioned? I’ll give that a try.

Yes, or something like the list of commands I provided.

Is there any documentation or a list of all the commands available?

This is the command AP documentation: https://github.com/threedworld-mit/tdw/blob/master/Documentation/api/command_api.md

You will find commands called replicant_grasp_object and replicant_drop_object but those commands are only applicable to the Replicant agent.

[basteran]

Thanks, found it! I still have some questions if I may:

1. Displacement of Objects After FPA Spawning
   When I spawn the FPA, some of the objects appear to have been displaced, as if something has collided with them upon spawning. This causes them to shift from their intended position. I’ve attached a top-view of the scene BEFORE the agent spawns and the scene looks ok to me.

As you can see, in the red area all the furniture is well placed. After I spawn the FPA, from the in-game view you can notice the difference.

Do you have any ideas why this is happening? Could you please clarify how I can prevent it?

2. Saving and Loading Scenes
   Is there a way to save the current scene to a file and reload it later? I would like to be able to load a scene from a file, make modifications, and save it again for future changes. Could you please let me know if this is possible and, if so, how it can be done?

3. Opening and Closing Objects
   How can I interact with objects like refrigerators or drawers to open and close them? Is there a specific command or procedure for this?

4. Positioning Objects
   I’m trying to place objects in specific locations (e.g., moving a pillow from the bed to the couch), but when I drop them, they often end up in strange positions, floating or moving unexpectedly. How can I ensure that an object is placed exactly where I want it to be? I am sorry but it seems that I am not able to find the exact commands. Maybe if you can suggest me the correct list, I can implement this behavior and, if you want, I can contribute to the repository!

Thank you once again.
CDH

[alters-mit]

Do you have any ideas why this is happening? Could you please clarify how I can prevent it?

The avatar is probably spawning on or nearby the objects. Try using the OccupancyMap add-on to get a free floor position.

Is there a way to save the current scene to a file and reload it later?

Read these documents (go through each "Next" link) and let me know if they answer your question: https://github.com/threedworld-mit/tdw/blob/master/Documentation/lessons/read_write/overview.md

How can I interact with objects like refrigerators or drawers to open and close them? Is there a specific command or procedure for this?

See the documentation for composite objects: https://github.com/threedworld-mit/tdw/blob/master/Documentation/lessons/composite_objects/overview.md

You could interact with them via a first person avatar by sending apply_force and apply_torque commands to the sub-objects.

How can I ensure that an object is placed exactly where I want it to be?

Once the object is in the scene, the physics engine may cause it to move. The best way to avoid perturbation is a) make the object kinematic with set_kinematic_state (this will cause the object to never move) and/or set the positions of the objects only on the first communicate() call (Unity sometimes interprets teleports as very fast changes in velocity)

[basteran]

Thank you, @alters-mit! Your guidance is invaluable during these times.

Using the FPA, I successfully managed to move objects around, place them where needed, remove them from the scene, and even add new objects via a simple script that opens a pop-up input field for entering the "model type." Now, I’m working on saving the scene data to a JSON file, and overall, it’s going well. However, I’ve encountered some issues when trying to load that file and instantiate the environment from it.

Here’s the process I’m following:

• I load scene scene5c2 and use the FPA to adjust some objects.
• I save the object data (including id, model_name, URL, position, and rotation) to a file.
• I close the Controller and Unity.
• When I reload the same scene (scene5c2), I remove all the objects, clear the assets cache, and attempt to place the saved objects one by one in their specified positions and rotations from the file.

The problem arises here: it seems the object ID is always set to 0, even though I specify it in the file. This prevents me from fully recreating the scene with the modifications I saved and the process crashes.

Do you have any suggestions on how to resolve this? I currently can’t restore the scene with the intended changes from the file.

Thanks again for your help!

[basteran]

Hi @alters-mit , any news about this Issue?