Skip to content

Dataset Format

Every recording session writes one self-contained folder of Parquet tables and video files. This page describes that folder: where it goes, how it is split into chunks, what each data column holds for each recorder mode, and what the metadata files under meta/ contain. Use it when you load a session into pandas, LeRobot or your own tools.

The layout is modelled on the LeRobot v3.0 dataset format: chunked Parquet data, one video file per camera stream per episode, and the same meta/ files. The LeRobot SO-100 tutorial trains a policy directly on a session folder.

In short

Each session is a DataSessions/session_<date>_<time>/ folder. data/ holds one Parquet row per MuJoCo robot per data tick, videos/ holds one file per camera stream per episode, and meta/ describes both. Read meta/episodes/ to find an episode's rows and video files, and to see whether it succeeded. Recorded files stay on your computer.

Where sessions are written

Each recording session gets its own folder:

<Output Directory>/DataSessions/session_YYYY-MM-DD_HH-MM-SS/
  • Output Directory is set on the General tab of the scene's Recorder settings (Settings → Scene → Recorder). It defaults to [Project Root], the project folder.
  • The date and time are the local time, to the second, at which the session was set up.

The Data Sessions list at the bottom of the Observer panel shows these folders. See Recording for the panel, and Viewing Files for the Parquet Table and Video Player tabs that open recorded files in the editor.

Folder layout

A session that records data, one camera with depth, and one lidar sensor looks like this:

session_2026-10-08_14-30-05/
├── data/
│   └── chunk-000/
│       ├── file-000.parquet                  state and action rows
│       ├── lidar-<Name>-000.parquet          one lidar sensor, one episode
│       └── lidar-<Name>-000.json             that file's sensor description
├── meta/
│   ├── info.json                             dataset description and features
│   ├── stats.json                            per-column statistics
│   ├── tasks.parquet                         task labels
│   └── episodes/
│       └── chunk-000/
│           └── file-000.parquet              one row per episode
└── videos/
    ├── observation.images.<Name>/chunk-000/file-000.mp4
    ├── observation.images.<Name>_depth/chunk-000/file-000.mkv
    ├── observation.images.<Name>_rgbd/chunk-000/file-000.mkv
    └── observation.masks.<Name>/chunk-000/file-000.mp4

Folders appear only for the streams the session records. meta/ and the file-NNN.parquet data files are written only while Record Data is on.

Video streams

<Name> is the camera binding's Output Name. Streams are written only while Record Cameras is on, and only for bindings that are Enabled. Colour and depth also need Record Color Video to be on.

Folder under videos/ Written when Contents
observation.images.<Name> Depth Mode is Off, RGBD or Separate Colour, .mp4, 8-bit 4:2:0. H.264, or HEVC with In-Process GPU (HEVC / Vulkan Video)
observation.images.<Name>_depth Depth Mode is RGBD, Separate or DepthOnly 16-bit depth, lossless FFV1 in .mkv
observation.images.<Name>_rgbd Depth Mode is RGBD The colour and depth tracks copied into one .mkv after each episode, without re-encoding
observation.masks.<Name> Record Masks is on, both globally and on the binding Mask video, .mp4

In RGBD mode the separate colour .mp4 and depth .mkv are kept next to the combined file. A depth code converts to metres as code × depth_scale, where depth_scale is the binding's Depth Scale (m/code), also written into meta/info.json. See RGBD and Depth for depth modes and sensor models, and Video Encoding and FFmpeg for the Video Format choices.

Masks are experimental

Mask recording is limited to specific objects. Mask streams and the RGBD combined file have no entry in meta/info.json or meta/episodes/.

Lidar files

Each lidar sensor writes one Parquet file per episode into data/, named after the binding's Output Name: data/chunk-NNN/lidar-<Name>-NNN.parquet. A JSON file with the same name records the sensor configuration that produced it, and Record Video Proxy adds an .mp4 preview with the same name. The column layout is covered in Lidar.

Lidar files are described by their JSON files, not by meta/.

Chunks and files

Two settings on the Data tab of the Recorder settings control how files are grouped:

Setting Default Range What it does
Files Per Chunk 40 1 to 1000 How many files go into each chunk-NNN folder
Episodes Per File 1 1 to 1000 How many episodes each data Parquet file holds
Capacity Per Chunk read-only Episodes Per File × Files Per Chunk: the number of episodes per data chunk

The streams are split differently:

  • Data files hold Episodes Per File episodes each. File numbers restart at 000 in every chunk.
  • Video files always hold one episode. Episode N goes into chunk N / Files Per Chunk (rounded down), and its file number is the remainder.
  • Lidar files hold one episode and go into the same chunk as its video, but their file number is the episode index itself. Episode 45 with the default 40 files per chunk is data/chunk-001/lidar-<Name>-045.parquet.

With the defaults, episode 45's data is in data/chunk-001/file-005.parquet and its colour video in videos/observation.images.<Name>/chunk-001/file-005.mp4. With a larger Episodes Per File, data and video numbering no longer line up, so look up each episode's files in meta/episodes/ rather than computing them.

Data columns

Each data file has one row per MuJoCo robot (each entity with a MujocoSceneComponent) per tick of the data recorder. Every row starts with the same columns:

Column Type Description
index int64 Row number within the session, starting at 0
robot_id, sim_id or entity_id int64 ID of the robot entity this row belongs to. The column name depends on the recorder mode (see below)
timestamp float64 Simulation time in seconds since the episode began. 0 on the first frame
episode_index int64 Episode number within the session, starting at 0
frame_index int64 Data tick within the episode, starting at 0. All robots in one tick share it
task_index int64 The task set with Observer.SetCurrentTaskIndex, 0 if none was set
frame_flags uint8 Per-frame flags, see below
action list of float64 Actuator commands. Not written in Custom Schema mode
observation.state list of float64 Joint positions. Not written in Custom Schema mode

In Custom Schema mode, your own columns follow frame_flags in the order the schema lists them. See Custom Recording Columns.

The entity ID is the entity's 64-bit ID stored as a signed integer, so large IDs can appear negative. timestamp advances by one data-runner step per frame: 0.02 s at the 50 Hz Robot runner that new scenes record on. The data rate is the frequency of the runner chosen as MuJoCo Observer [Data] TimeRunner on the Recorder settings' General tab.

action and observation.state are fixed-length lists: every row in a session has the same length. In a scene with several robots, every robot must therefore have the same number of actuators and the same joint layout.

Frame flags

Bit Value Name Set when
0 1 new_policy_step A policy on this robot ran inference on this step, rather than holding its previous action
1 2 post_reset This is the first recorded frame after the robot was reset to its initial state, for example with MujocoSceneComponent.ResetToInitialState()

Drop rows without new_policy_step to train at the policy's own rate. Use post_reset to find the jump in joint positions that a reset causes.

What each recorder mode records

The Recorder Mode on the Data tab decides the entity column and what goes into action and observation.state:

Recorder Mode Entity column action observation.state
Robot Arm (default) robot_id One value per actuator Positions of the actuated joints only
MuJoCo Full Scene sim_id One value per actuator Positions of every joint in the model, including passive joints and the free joints of loose objects
Custom Schema entity_id Not written Not written. Your declared columns instead

Joint positions are MuJoCo's position coordinates, in radians for hinge joints and metres for slide joints (see Units). Hinge and slide joints contribute one value each. A ball joint contributes four (quaternion qw, qx, qy, qz), and a free joint seven (x, y, z followed by qw, qx, qy, qz).

action holds each actuator's command. If an actuator has a limited control range and drives a single hinge or slide joint with a limited range, its command is clamped to the control range and mapped linearly onto the joint's range. Otherwise the raw command is recorded. Neither Robot Arm nor MuJoCo Full Scene records joint velocities, contact forces or end-effector poses. To record values such as an end-effector pose or a MuJoCo sensor reading, use Custom Recording Columns.

Drone mode

The Drone recorder mode does not produce a usable dataset in this release. Use one of the modes above.

Metadata

All metadata lives in meta/. It is written every 25 episodes and when the session ends.

info.json

meta/info.json describes the dataset as a whole:

Field Description
codebase_version The Lucky Engine version that recorded the session
neural_denoise true if any colour camera recorded with Path Traced and NVIDIA DLSS Ray Reconstruction
bit_reproducible false when neural_denoise is true. Skip such sessions if you need byte-identical re-runs
robot_type, sim_type or entity_type Named after the entity column. A fixed label for the recorder mode: ur5e for Robot Arm, mujoco_scene for MuJoCo Full Scene, custom for Custom Schema. It does not identify the robot model
total_tasks Number of task labels, or 1 if none were registered
total_episodes Number of episodes listed in meta/episodes/
total_frames Total number of data rows
fps A session-wide video rate kept for LeRobot compatibility. Use each video feature's own video.fps instead
hz The data recording rate
chunks_size Files Per Chunk
data_files_size_in_mb, video_files_size_in_mb Size limits expected by LeRobot tooling. The recorder does not split files by size
data_path Template for data file paths
video_path Template for video file paths. Present when video is recorded. Depth files use .mkv, although the template ends in .mp4
splits {"train": "0:<total_episodes>"}
features One entry per data column and per colour or depth stream, described below

Each data column's feature gives its dtype and shape. frame_flags lists its bit names, ["new_policy_step", "post_reset"]. For action, names lists the actuator names. For observation.state, names gives a readable name per value (the driving actuator's name for an actuated joint) and lucky_names the joint names from the MuJoCo model, with suffixes such as _x or _qw for multi-value joints. In a scene with several robots, the name lists cover every robot in turn. Custom columns also carry their unit, if the schema set one.

Each colour stream has a feature named observation.images.<Name>, and each depth stream one named observation.images.<Name>_depth. Both have dtype "video", a shape of [height, width, channels] (3 channels for colour, 1 for depth), and an info block:

Key in info Colour Depth
video.fps The binding's Frame Rate The binding's Frame Rate
video.codec h264 or hevc, whichever was actually used ffv1
video.pix_fmt yuv420p gray16le
video.is_depth_map false true
depth_scale Metres per depth code
depth_dtype, depth_units uint16, meters
sensor_model The binding's Depth Sensor Model, for example None or ActiveStereo
zero_means What a depth code of 0 means for this stream
sensor_params The sensor model's calibration. Only when a sensor model is set
debug_channel, NOTE Only when a diagnostic channel is on. The values are then not depth
has_audio false false

video.fps is the rate stamped into the file, not necessarily the rate frames were captured at. See Video Encoding and FFmpeg for keeping the two in step, and Diagnostic channels for depth diagnostic output.

stats.json

meta/stats.json has an entry for action and observation.state (except in Custom Schema mode) and one per custom column. Each entry holds min, max, mean and std arrays with one value per element, and count, the number of rows as a one-element list. Statistics cover every row written, including rows of episodes that ended unsuccessfully.

tasks.parquet

meta/tasks.parquet maps task indices to the descriptions given to Observer.RegisterTask. It has a task_index column and stores the description as the table's index, so pandas.read_parquet returns a table indexed by description. It lists the tasks registered when the recorder was armed, by Arm Recorder or by Observer.StartRecording(): call RegisterTask before either. With no tasks registered, it holds a single task, index 0, Default Task.

Episode metadata

meta/episodes/chunk-NNN/file-NNN.parquet has one row per episode:

Column Type Description
episode_index int64 The episode number
chunk_index, file_index int64 The data file that holds the episode's rows
data/chunk_index, data/file_index int64 The same values, under LeRobot's names
length int64 Number of data rows in the episode: one per robot per data tick
success bool Whether the episode succeeded (see Failed episodes)
dataset_from_index, dataset_to_index int64 The episode's rows: index from dataset_from_index up to, but not including, dataset_to_index

Each recorded colour and depth stream adds four columns that locate its files. Their names start with the stream's folder, videos/observation.images.<Name>/ for colour and videos/observation.images.<Name>_depth/ for depth:

Column suffix Type Description
chunk_index, file_index int64 The episode's video file
from_timestamp float64 Always 0.0, because each video file holds one episode
to_timestamp float64 The number of video frames divided by the binding's Frame Rate

The values are empty if the stream wrote no file for that episode.

Episode metadata is appended rather than rewritten: each metadata write adds a new file to the chunk folder, listing the episodes finished since the previous write. Read every file under meta/episodes/ to get the full list.

Loading an episode

This Python example uses pandas to find episode 3 and load its rows:

from pathlib import Path
import pandas as pd

session = Path("DataSessions/session_2026-10-08_14-30-05")

episodes = pd.concat(
    pd.read_parquet(p) for p in sorted(session.glob("meta/episodes/chunk-*/file-*.parquet"))
)
ep = episodes[episodes["episode_index"] == 3].iloc[0]

chunk, file = int(ep["data/chunk_index"]), int(ep["data/file_index"])
data = pd.read_parquet(session / f"data/chunk-{chunk:03d}/file-{file:03d}.parquet")
rows = data[data["episode_index"] == 3]

key = "videos/observation.images.Front"   # "Front" is the binding's Output Name
video = (session / key
         / f"chunk-{int(ep[key + '/chunk_index']):03d}"
         / f"file-{int(ep[key + '/file_index']):03d}.mp4")

Filter on the success column to leave out unsuccessful episodes.

Failed and discarded episodes

An episode is recorded as unsuccessful (success is false) when:

  • the script ends it with Observer.EndCurrentEpisode(false) or Observer.EndCurrentEpisodeBeginNext(false);
  • it reaches Max Episode Length (Seconds) on the Session tab;
  • the session is stopped while it is still open;
  • any of its video, depth, mask or lidar streams failed, or its data could not be written, even if the script reported success.

What happens next depends on Discard & Retry Failed on the Session tab:

  • Off (default): the episode is kept, listed in meta/episodes/ with success set to false, and the next episode gets the next index. Filter on success when you load the dataset.
  • On: the episode's data rows are never written and it gets no entry in meta/episodes/. The next episode reuses its index, so its video and lidar files replace the failed attempt's.

With Discard & Retry Failed on, a session that ends on a failed attempt can leave that attempt's video and lidar files on disk with no matching episode. Treat meta/episodes/ as the list of valid episodes.

A camera that fails to write stops the whole session.

Interrupted sessions

Metadata files are first written under a temporary name and then renamed, so a session that is cut short, for example by a crash, keeps the metadata from its last complete write rather than a half-written file. As metadata is written every 25 episodes, the episodes finished since the last write are missing from meta/, and a session interrupted before its 25th episode has no metadata at all.

A data Parquet file becomes readable only once it is closed, which happens when it reaches Episodes Per File episodes or the session ends. Keep Episodes Per File at 1 if completed episodes must survive an interrupted session. Files with .tmp in their names, left behind by an interrupted session, are incomplete.

Where to go next