LeRobot: Sim-to-Real Tutorial

Record in sim · Deploy on real

Manual leader-arm teleop.
One human, one demo at a time.

Scripted in Lucky Engine.
Hands-free, repeatable, same dataset format.

by hand, one at a time  →  by script, as many as you need

Trained on those demos ↓

The same checkpoint, autonomous on the real SO-100. Two-camera DICE-IMLE @ 30 Hz.

72% success on the real SO-100, trained only on sim recordings.

This guide isn't tied to one task or robot: the commands and API snippets adapt to any Lucky Engine scene with a robot agent. It assumes you're comfortable training imitation-learning policies. If LeRobot is new to you, start with the LeRobot docs first.

In short

A C# scene script drives the robot and the Observer records each demo as a LeRobot 3.0 dataset. Train an ACT policy on it with lerobot-train, evaluate it in Lucky Engine over gRPC with the luckyrobots SDK, probe it in Genesis, then run it on the real SO-100 at 30 Hz. Record with Recorder Mode set to Robot Arm so the recorded state matches what the real robot exposes.

What you'll build:

• A LeRobot 3.0 dataset recorded entirely inside Lucky Engine
• An ACT policy trained on that dataset (~2–6 hours on a consumer GPU)
• A 30 Hz inference loop running on the physical SO-100

---

Prerequisites

• Lucky Engine scene already built with a robot agent in it
• Python 3.12+ (current lerobot requires ≥ 3.12)
• Any CUDA GPU for training; Lucky Engine renders on the host GPU
• pip install "lerobot[training]" luckyrobots torch grpcio numpy opencv-python. The [training] extra is required as of lerobot 0.6.0: a plain pip install lerobot no longer pulls the training and dataset dependencies, so lerobot-train in §2 will not run without it
• The lerobot package is 0.6.x. "3.0" is the on-disk dataset format version, not the package version
• For §4 (Genesis sim-to-sim): pip install genesis-world
• A Lucky Robots account, signed in to LuckyHub with Profile at the bottom of the side bar (see Signing in). Observer.StartRecording() (§1) refuses to start while you are signed out. Recording counts against your monthly episode quota, which is checked both when the session opens and per episode while it runs

---

1 · Record demos in Lucky Engine

Lucky Engine (LE) demos are produced by scripted C# scene scripts, with no human teleop. The script drives the robot through waypoints; the engine writes a LeRobot 3.0 dataset to disk in parallel.

Start a recording

See Assets/ContentVault/Examples/SO100 Pick And Place/SO100PickAndPlace.cs for a complete example. The minimum:

Observer.RegisterTask(0, "Pick up the lego block and place it on the target");
Observer.StartRecording();

foreach (var episode in episodes) {
    DriveSO100ToWaypoints();
    bool ok = CheckTaskSuccess();
    Observer.EndCurrentEpisode(ok);
    ResetSceneForNextEpisode();
}

Observer.StopRecording();

DriveSO100ToWaypoints(), CheckTaskSuccess(), and ResetSceneForNextEpisode() are illustrative names. The shipped scene implements this as BuildTaskSequence() + VerifyPlacement() driven by the Observer. Read the .cs above for the real code rather than searching for these names.

What ends up on disk

DataSessions/session_<date>_<time>/
├── data/chunk-NNN/file-NNN.parquet
├── videos/observation.images.<cam>/chunk-NNN/file-NNN.mp4
└── meta/info.json + stats.json + tasks.parquet + episodes/

Field	Description
action	float64, one value per actuator in the model
observation.state	float64, and dependent on the Recorder Mode you pick. See the note below.
observation.images.<cam>	H.264 .mp4 chunks, stored as yuv420p, with the default Video Format (In-Process GPU (H.264 / Vulkan Video) in new scenes on Windows and Linux) and with the FFmpeg formats. In-Process GPU (HEVC / Vulkan Video) writes HEVC instead. <cam> is the camera binding's Output Name.

Cameras are recorded through camera bindings, added with Add Camera Binding under Camera Configuration on the Video tab of the Scene → Recorder settings page (Edit → Settings). When you pick a binding's Source Entity, an empty Output Name is filled in with the camera entity's name, and you can rename it. Cameras that also record depth write extra streams; see Dataset Format and Video Encoding and FFmpeg.

Pick the Recorder Mode before you record

Recorder Mode is on the Data tab of the Scene → Recorder settings page (Edit → Settings), and it decides what observation.state contains.

Robot Arm records only actuated joints. For the SO-100 that is the 6 joints, which is exactly what the real robot exposes in §5. This is the mode you want for a policy you intend to deploy.

MuJoCo Full Scene records every joint's qpos in the scene. A free-floating object contributes 7 values, so the pick-and-place scene yields 6 arm joints plus the block's 7 = 13. Those extra 7 are the block's exact pose, which the policy will happily learn from and which does not exist on real hardware. If you record this way, trim the state to the first 6 before training, or the policy will not transfer.

→ With episodes on disk, train a policy to imitate them.

---

2 · Train an ACT policy

To train an ACT policy on your recorded session, run:

lerobot-train \
  --dataset.root=path/to/your_session \
  --dataset.repo_id=local/your_session \
  --policy.type=act \
  --policy.chunk_size=100 \
  --batch_size=8 \
  --steps=100000 \
  --output_dir=outputs/act_my_task \
  --wandb.enable=true

Parameter	Notes
policy.chunk_size	Actions per forward pass. 100 is a good default.
policy.kl_weight	CVAE KL term. Default 10; lower if the latent collapses.
steps	~100k for ~200 demos on a tabletop task.

Wall-clock: 2–6 hours on an RTX 3090/4070/4090. Keep at least the last few checkpoints. Best performance is rarely the final step.

→ With a checkpoint in hand, close the loop in the same simulator you trained in.

---

3 · Evaluate in-domain (Lucky Engine)

Close the loop in the same simulator you trained in using the luckyrobots SDK. step() returns a synchronous ObservationResponse: state and camera frames together, with no async stream to race against.

Full in-domain eval loop (Python)

Reset, let the scene settle, then step the policy until it succeeds or times out:

from luckyrobots import Session
import numpy as np

with Session(host="127.0.0.1", port=50051) as session:
    session.start(scene="my_scene", robot="my_robot", task="my_task")
    session.configure_cameras([
        {"name": "TripodCamera", "width": 96, "height": 96},
        {"name": "LaptopCamera", "width": 96, "height": 96},
    ])

    policy, pre, post = load_policy(checkpoint_dir)
    state_dim = 6   # arm joints; assumes they lead the observation vector

    for trial in range(N_TRIALS):
        obs_resp = session.reset()
        for _ in range(SETTLE_STEPS):
            obs_resp = session.step(HOME_ACTION)
        for step in range(MAX_STEPS):
            state  = obs_resp.observation[:state_dim]
            frames = {cf.name: np.frombuffer(cf.data, np.uint8).reshape(cf.height, cf.width, cf.channels)
                      for cf in obs_resp.camera_frames}
            action = predict(policy, pre, post, build_obs(state, frames))
            obs_resp = session.step(action.tolist())
            if success(obs_resp):
                break

Helpers & constants you provide

load_policy/load_checkpoint, predict/build_obs, success, ActionMapper, precise_sleep, and abort are thin adapters between your ACT checkpoint and this loop. They aren't part of lerobot or luckyrobots. Implement them once: load the checkpoint and its normalization stats, shape the observation the policy expects, map dataset↔robot units, hold cadence, and safety-stop. The CAPS names (N_TRIALS, SETTLE_STEPS, HOME_ACTION, MAX_STEPS, CKPT_DIR, STATS, CALIB, SERIAL_A/SERIAL_B) are values you supply.

Common gotchas

• task is required in session.start(...).
• State is at obs_resp.observation (a flat list[float]). Slice the first state_dim entries.
• Camera frames arrive in the same response after configure_cameras(...), with no separate stream.
• Frames come back as RGBA, so cf.channels is 4. Slice [:, :, :3] if your policy expects RGB.
• step() is rejected outright if the scene has no registered external agent.
• A camera name that does not resolve does not stop the step: the step runs and the observation comes back, but without that camera's frame (the engine reports the step as unsuccessful). Check that every camera you configured is in camera_frames.

→ Passing in-domain? Probe whether it generalizes before risking hardware.

---

4 · Probe generalization in Genesis (sim-to-sim)

Run the same checkpoint in Genesis, a different physics solver and renderer, as a cheap generalization probe before touching real hardware.

1. Convert axes. LE is Y-up, Genesis is Z-up: le_to_gs(p) = (p[0], -p[2], p[1])
2. Reuse the MJCF. Pass the same so_arm100.xml and match the home-pose keyframe.
3. Place cameras by intrinsics. Same position, lookat, FOV, and 96×96 size.
4. Rebuild task objects. Close enough. You're testing generalization, not pixel parity.

→ Survives a different renderer and solver? Time for the real robot.

---

5 · Deploy to the real SO-100

Stage latency breakdown

Stage	Latency
Camera async_read + joint encoder	~5–8 ms
Preprocess (resize 96×96, state units)	~2–3 ms
policy.predict (EMA + AMP)	9–17 ms
Action mapper + safety scan	<1 ms
follower.send_action @ 30 Hz	~2–4 ms
Total budget	33 ms

1. Match cameras & rate. Same resolution, mounting, and Hz as your LE scene. If you mirrored a camera in LE, apply cv2.flip(..., 1). Without it the policy sees a mirror-image arm and fails silently.
2. Match action units. Verify radians↔degrees conversion and dataset min/max against calibrated joint range.
3. Calibrate joint zeros. Re-run so "0 rad in dataset" maps to physical home consistently.
4. Add a safety scan. Abort (don't clip silently) if the policy drifts off-distribution.

Use async_read camera backends so I/O overlaps the previous step's policy call. If you can't hold 30 Hz, drop to 20 Hz uniformly. Jitter hurts more than a lower steady rate.

Full real-robot inference loop (Python)

The full 30 Hz loop. It observes, maps units, runs the policy, sends the action, then sleeps to hold cadence:

import time
import numpy as np
import cv2
from lerobot.robots.so_follower import SOFollower, SOFollowerRobotConfig
from lerobot.cameras.realsense import RealSenseCamera, RealSenseCameraConfig

# The 6 SO-100 joints in dataset order (matches the sim recording after the joint rename).
JOINT_NAMES = ["shoulder_pan", "shoulder_lift", "elbow_flex", "wrist_flex", "wrist_roll", "gripper"]

follower = SOFollower(SOFollowerRobotConfig(port="COM5", id="my_follower", use_degrees=True))
follower.connect(calibrate=True)

cam_a = RealSenseCamera(RealSenseCameraConfig(serial_number_or_name=SERIAL_A, fps=30, width=640, height=480))
cam_b = RealSenseCamera(RealSenseCameraConfig(serial_number_or_name=SERIAL_B, fps=30, width=640, height=480))
cam_a.connect(); cam_b.connect()

policy, pre, post = load_checkpoint(CKPT_DIR)
mapper = ActionMapper(stats_path=STATS, calib_path=CALIB)

dt = 1.0 / 30.0
max_steps = 5000                       # ~2.7 min at 30 Hz; size to your episode length
for step in range(max_steps):
    t0 = time.perf_counter()
    obs = follower.get_observation()
    state_real  = np.array([obs[f"{j}.pos"] for j in JOINT_NAMES], dtype=np.float32)
    state_train = mapper.state_to_training(state_real)
    img_a = cv2.flip(cam_a.async_read(), 1)
    img_b = cam_b.async_read()
    frames = {
        "LaptopCamera": cv2.resize(img_a, (96, 96), interpolation=cv2.INTER_LINEAR),
        "TripodCamera": cv2.resize(img_b, (96, 96), interpolation=cv2.INTER_LINEAR),
    }
    action_deg, _, outside = mapper.action_to_real(policy.predict(state_train, frames))
    if outside: abort("action outside calibrated range")
    follower.send_action({f"{n}.pos": float(v) for n, v in zip(JOINT_NAMES, action_deg)})
    precise_sleep(dt - (time.perf_counter() - t0))

→ Here's what this pipeline actually produced on hardware.

---

6 · Case study: what we shipped

Takeaway: a policy trained only on LE recordings reached ~84% in-sim success, then fell off a ~17 pp transfer cliff in a different renderer. A DICE-pretrained encoder recovered most of the gap, reaching 72% on the real SO-100, with no teleop demos anywhere in the pipeline.

DICE is optional / advanced

§1–§5 form a complete sim-to-real pipeline on their own. DICE is an advanced add-on that closed a sim-to-real appearance gap; the ~21k paired LE↔Real frames it was trained on aren't published, so treat this section as a reported case study, not a reproducible step. A vanilla ACT/IMLE policy (no DICE) lands lower on the real robot.

Results breakdown

200 LE-recorded SO-100 episodes, 30 Hz, 96×96, two cameras. Vanilla IMLE policy with ResNet18 + SpatialSoftmax encoder, 6-DOF joint state, 1D U-Net generator (~66 M params).

Vanilla IMLE: LE finalist eval (25 trials × 3 checkpoints)

Checkpoint	Success	Lift	Grasp
020160	52%	72%	68%
022400	60%	92%	92%
044800	84%	96%	96%

The transfer cliff: checkpoint 044800 dropped from 84% in LE to 67% in Genesis zero-shot, a 17 pp gap from renderer differences (colour cast, micro-shading, edge sharpness).

DICE v3 closed the gap: a frozen encoder pretrained on ~21k paired LE↔Real frames, producing L2-normalized (B, 128, 6, 6) grids with dense InfoNCE + VICReg + DANN losses.

DICE-IMLE: Genesis sim2sim (25 trials)

Checkpoint	Success	Lift	Grasp
006280	76%	80%	84%
009430	64%	68%	72%

DICE-IMLE: Real SO-100 (25 trials each)

Checkpoint step	Success
18 000	60%
19 500	72%

---

Where to go next

• gRPC API: the step loop, action gate and camera frames that the luckyrobots SDK is built on
• luckyrobots SDK: reference for Session, step(), and ObservationResponse
• LeRobot docs: policy training reference
• Genesis: sim-to-sim transfer
• ACT paper: Zhao et al. 2023