LeRobot: Sim-to-Real Tutorial

Record in sim · Deploy on real

Manual leader-arm teleop.
One human, one demo at a time.

Scripted in LuckyEngine.
Hands-free, repeatable, same dataset format.

by hand, one at a time  →  by script, as many as you need

Trained on those demos ↓

The same checkpoint, autonomous on the real SO-100. Two-camera DICE-IMLE @ 30 Hz.

72% success on the real SO-100 — trained only on sim recordings.

This guide isn't tied to one task or robot: the commands and API snippets adapt to any LuckyEngine scene with a robot agent. It assumes you're comfortable training imitation-learning policies — if LeRobot is new to you, start with the LeRobot docs first.

What you'll build:

• A LeRobot 3.0 dataset recorded entirely inside LuckyEngine
• An ACT policy trained on that dataset (~2–6 hours on a consumer GPU)
• A 30 Hz inference loop running on the physical SO-100

---

Prerequisites

• LuckyEngine scene already built with a robot agent in it
• Python 3.10–3.12
• Any CUDA GPU for training; LE renders on the host GPU
• pip install lerobot luckyrobots torch grpcio numpy (LeRobot ≥ 3.0)
• For §4 (Genesis sim-to-sim): pip install genesis-world

---

1 · Record demos in LuckyEngine

LuckyEngine (LE) demos are produced by scripted C# scene scripts — no human teleop. The script drives the robot through waypoints; the engine writes a LeRobot 3.0 dataset to disk in parallel.

Start a recording

See Assets/ContentVault/Examples/SO100 Pick And Place/SO100PickAndPlace.cs for a complete example. The minimum:

Observer.RegisterTask(0, "Pick up the lego block and place it on the target");
Observer.StartRecording();

foreach (var episode in episodes) {
    DriveSO100ToWaypoints();
    bool ok = CheckTaskSuccess();
    Observer.EndCurrentEpisode(ok);
    ResetSceneForNextEpisode();
}

Observer.StopRecording();

What ends up on disk

session_<ts>/
├── data/chunk-NNN/file-NNN.parquet
├── videos/observation.images.<cam>/
└── meta/info.json + stats.json + tasks.parquet + episodes/

Field	Description
action	float32 per actuator (m->nu)
observation.state	All actuated joint qpos — trim to what your policy needs
observation.images.<cam>	uint8 RGB as h264 mp4 chunks

→ With episodes on disk, train a policy to imitate them.

---

2 · Train an ACT policy

To train an ACT policy on your recorded session, run:

lerobot-train \
  --dataset.root=path/to/your_session \
  --dataset.repo_id=local/your_session \
  --policy.type=act \
  --policy.chunk_size=100 \
  --batch_size=8 \
  --steps=100000 \
  --output_dir=outputs/act_my_task \
  --wandb.enable=true

Parameter	Notes
policy.chunk_size	Actions per forward pass. 100 is a good default.
policy.kl_weight	CVAE KL term. Default 10; lower if the latent collapses.
steps	~100k for ~200 demos on a tabletop task.

Wall-clock: 2–6 hours on an RTX 3090/4070/4090. Keep at least the last few checkpoints — best performance is rarely the final step.

→ With a checkpoint in hand, close the loop in the same simulator you trained in.

---

3 · Evaluate in-domain (LuckyEngine)

Close the loop in the same simulator you trained in using the luckyrobots SDK. step() returns a synchronous ObservationResponse — state and camera frames together, no async stream to race against.

Full in-domain eval loop (Python)

Reset, let the scene settle, then step the policy until it succeeds or times out:

from luckyrobots import Session

with Session(host="127.0.0.1", port=50051) as session:
    session.start(scene="my_scene", robot="my_robot", task="my_task")
    session.configure_cameras([
        {"name": "CameraFront",  "width": 96, "height": 96},
        {"name": "LaptopCamera", "width": 96, "height": 96},
    ])

    policy, pre, post = load_policy(checkpoint_dir)
    state_dim = 6

    for trial in range(N_TRIALS):
        obs_resp = session.reset()
        for _ in range(SETTLE_STEPS):
            obs_resp = session.step(HOME_ACTION)
        for step in range(MAX_STEPS):
            state  = obs_resp.observation[:state_dim]
            frames = {cf.name: cf.image for cf in obs_resp.camera_frames}
            action = predict(policy, pre, post, build_obs(state, frames))
            obs_resp = session.step(action.tolist())
            if success(obs_resp):
                break

Common gotchas

• task is required in session.start(...).
• State is at obs_resp.observation (a flat list[float]). Slice the first state_dim entries.
• Camera frames are in the same response after configure_cameras(...) — no separate stream.

→ Passing in-domain? Probe whether it generalizes before risking hardware.

---

4 · Probe generalization in Genesis (sim-to-sim)

Run the same checkpoint in Genesis — a different physics solver and renderer — as a cheap generalization probe before touching real hardware.

1. Convert axes. LE is Y-up, Genesis is Z-up: hz_to_gs(p) = (p[0], -p[2], p[1])
2. Reuse the MJCF. Pass the same so_arm100.xml and match the home-pose keyframe.
3. Place cameras by intrinsics. Same position, lookat, FOV, and 96×96 size.
4. Rebuild task objects. Close enough — you're testing generalization, not pixel parity.

→ Survives a different renderer and solver? Time for the real robot.

---

5 · Deploy to the real SO-100

Stage latency breakdown

Stage	Latency
Camera async_read + joint encoder	~5–8 ms
Preprocess (resize 96×96, state units)	~2–3 ms
policy.predict (EMA + AMP)	9–17 ms
Action mapper + safety scan	<1 ms
robot.send_position @ 30 Hz	~2–4 ms
Total budget	33 ms

1. Match cameras & rate. Same resolution, mounting, and Hz as your LE scene. If you mirrored a camera in LE, apply cv2.flip(..., 1) — without it the policy sees a mirror-image arm and fails silently.
2. Match action units. Verify radians↔degrees conversion and dataset min/max against calibrated joint range.
3. Calibrate joint zeros. Re-run so "0 rad in dataset" maps to physical home consistently.
4. Add a safety scan. Abort (don't clip silently) if the policy drifts off-distribution.

Use async_read camera backends so I/O overlaps the previous step's policy call. If you can't hold 30 Hz, drop to 20 Hz uniformly — jitter hurts more than a lower steady rate.

Full real-robot inference loop (Python)

The full 30 Hz loop — observe, map units, run the policy, send the action, then sleep to hold cadence:

from lerobot.robots.so_follower.so_follower import SOFollower
from lerobot.robots.so_follower.config_so_follower import SOFollowerRobotConfig
from lerobot.cameras.realsense.camera_realsense import RealSenseCamera
from lerobot.cameras.realsense.configuration_realsense import RealSenseCameraConfig

follower = SOFollower(SOFollowerRobotConfig(port="COM5", id="my_follower", use_degrees=True))
follower.connect(calibrate=True)

cam_a = RealSenseCamera(RealSenseCameraConfig(serial_number_or_name=SERIAL_A, fps=30, width=640, height=480))
cam_b = RealSenseCamera(RealSenseCameraConfig(serial_number_or_name=SERIAL_B, fps=30, width=640, height=480))
cam_a.connect(); cam_b.connect()

policy, pre, post = load_checkpoint(CKPT_DIR)
mapper = ActionMapper(stats_path=STATS, calib_path=CALIB)

dt = 1.0 / 30.0
while step < max_steps:
    t0 = time.perf_counter()
    obs = follower.get_observation()
    state_real  = np.array([obs[f"{j}.pos"] for j in JOINT_NAMES], dtype=np.float32)
    state_train = mapper.state_to_training(state_real)
    img_a = cv2.flip(cam_a.async_read(), 1)
    img_b = cam_b.async_read()
    frames = {
        "LaptopCamera": cv2.resize(img_a, (96, 96), interpolation=cv2.INTER_LINEAR),
        "CameraFront":  cv2.resize(img_b, (96, 96), interpolation=cv2.INTER_LINEAR),
    }
    action_rad, _, outside = mapper.action_to_real(policy.predict(state_train, frames))
    if outside: abort("action outside calibrated range")
    follower.send_action({f"{n}.pos": float(v) for n, v in zip(JOINT_NAMES, action_rad)})
    precise_sleep(dt - (time.perf_counter() - t0))

→ Here's what this pipeline actually produced on hardware.

---

6 · Case study — what we shipped

Takeaway: a policy trained only on LE recordings reached ~84% in-sim success, then fell off a ~17 pp transfer cliff in a different renderer. A DICE-pretrained encoder recovered most of the gap — 72% on the real SO-100, with no teleop demos anywhere in the pipeline.

Results breakdown

200 LE-recorded SO-100 episodes, 30 Hz, 96×96, two cameras. Vanilla IMLE policy with ResNet18 + SpatialSoftmax encoder, 6-DOF joint state, 1D U-Net generator (~66 M params).

Vanilla IMLE — LE finalist eval (25 trials × 3 checkpoints)

Checkpoint	Success	Lift	Grasp
020160	52%	72%	68%
022400	60%	92%	92%
044800	84%	96%	96%

The transfer cliff: checkpoint 044800 dropped from 84% in LE to 67% in Genesis zero-shot — 17 pp gap from renderer differences (colour cast, micro-shading, edge sharpness).

DICE v3 closed the gap: a frozen encoder pretrained on ~21k paired LE↔Real frames, producing L2-normalized (B, 128, 6, 6) grids with dense InfoNCE + VICReg + DANN losses.

DICE-IMLE — Genesis sim2sim (25 trials)

Checkpoint	Success	Lift	Grasp
006280	76%	80%	84%
009430	64%	68%	72%

DICE-IMLE — Real SO-100 (25 trials each)

Checkpoint step	Success
18 000	60%
19 500	72%

---

Where to go next

• gRPC API — full SDK reference for Session, step(), and ObservationResponse
• LeRobot docs — policy training reference
• Genesis — sim-to-sim transfer
• ACT paper — Zhao et al. 2023