[Question] Pre-training phase action execution with Franka using manager-based RL framework

[PenglinCai]

Question

Pre-training phase action execution:
Before each training episode starts, how can I execute a predefined sequence of actions or motions (e.g., having the arm grasp an object or move the end effector to a specific position) as a “pre-training phase”? During this phase, observations should not be passed to the RL agent's network, and the network should not output actions. This is especially important for tasks that require the robot to move to a certain position as the initial condition of an episode, not simply resetting the robot joint position to that position, but actually moving it there before training begins.

[RandomOakForest]

Thank you for posting this. To execute a predefined sequence of actions with the Franka arm before each RL episode in the manager-based RL framework, it's recommended to programmatically step through the environment using a scripted motion policy (such as inverse kinematics, joint position targets, or operational space controllers) prior to activating the RL agent for training. This approach is standard for tasks that require reaching a physical initial state (e.g. grasping or placing the end effector with dynamic contact), rather than simple joint resets.¹²

How to Implement Pre-Training Motions

Before starting RL training in each episode, follow these steps:

• Scripted Action Sequence: Implement a loop that steps through the environment using explicit, hand-crafted action values (either joint positions, Cartesian targets, or other control primitives) to accomplish the preparatory motion. You can use action manager classes like JointPositionAction, OperationalSpaceControllerAction, or differential inverse kinematics DifferentialInverseKinematicsAction defined in Isaac Lab for this purpose.¹
• Disable Agent Observations: During this phase, do not pass observations to the RL agent or query its policy; instead, drive the robot using your scripted policy and wait until the sequence is finished.¹
• Observation and Action Management: Ensure your code separates this phase (using e.g., a pre-training flag or state variable), so RL agent networks are only called after completion of the scripted phase. This may involve managing episode state within your environment or using a custom wrapper.

Example Code Outline

Here's a concise outline with code snippets using Isaac Lab's Gym-compatible API. This follows best practices and uses only documented classes:

# Launch Isaac Sim
from isaaclab.app import AppLauncher
app_launcher = AppLauncher(headless=True)
simulation_app = app_launcher.app

import gymnasium as gym
from isaaclab_tasks.utils import load_cfg_from_registry

cfg = load_cfg_from_registry("Isaac-Reach-Franka-v0", "env_cfg_entry_point")
env = gym.make("Isaac-Reach-Franka-v0", cfg=cfg)

def scripted_init_motion(env, target_pose):
    done = False
    while not done:
        # Compute action towards target (e.g., via inverse kinematics)
        action = compute_action_to_pose(target_pose)
        obs, _, _, _ = env.step(action)
        done = check_if_target_reached(obs)
    return obs

while True:
    obs = env.reset()
    # Pre-training phase: run scripted motion to desired start state
    obs = scripted_init_motion(env, desired_pose)
    # Now begin episode proper with RL agent policy
    while not terminated:
        action = agent.compute_action(obs)
        obs, reward, terminated, info = env.step(action)

Use DifferentialInverseKinematicsActionCfg or similar classes to produce the correct control signals for the Franka's required movements.¹

Best Practices and Documentation

• Carefully separate episode phases so the RL network is not triggered until pre-training completes.¹
• Use native environment managers and action configurations from the Isaac Lab API, which provide robust controls for sophisticated robot motions (including for Franka).¹
• Refer to the latest Isaac Lab documentation for all manager and action types available for scripting these motions.¹
• If writing a custom environment or manager, introduce a “pre-training” state variable and skip RL agent invocations while it is active.¹

Footnotes

1. https://isaac-sim.github.io/IsaacLab/main/source/api/lab/isaaclab.envs.mdp.html ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

2. https://isaac-sim.github.io/IsaacLab/main/source/how-to/wrap_rl_env.html ↩

[PenglinCai]

Thank you for your kind guidance. Where should I put this ''Example Code Outline" you provided? Becuase manager-based RL use a train.py file to train the registered environment. Could you point it out? Thank you.

[RandomOakForest]

Following up, I will move this post to our Discussions. Let us know if you have made any inroads and you need further help.