Skip to content

Recording

Lucky Engine records simulation runs as datasets for training and analysis. The recorder, called the Observer, captures robot state and actions, camera video (colour, and optionally depth) and lidar scans, and writes them to disk as a session made of episodes: Parquet tables for the data, video files for the cameras.

The work is split in two. In the editor you decide what to capture, on the scene's Recorder settings page and in the Observer panel. A C# script decides when: it starts the session, marks where each episode ends and stops the session.

In short

Sign in to LuckyHub, bind a camera under Settings → Scene → Recorder → Video, and call Observer.StartRecording() from a script while the scene plays. Follow progress in Recorder Status, on the viewport toolbar or in the Observer panel, and open the results from the panel's Data Sessions list. The script side is covered in Recording with the Observer.

Quick start

  1. Sign in with the Profile button at the bottom of the side bar (see Signing in). Recording is not available while you are signed out.
  2. Open Settings → Scene → Recorder with the gear button on the side bar or Edit → Settings. Check that Enable MuJoCo Observer is ticked. It is on in new scenes.
  3. On the General tab, leave Record Data and Record Cameras on.
  4. On the Video tab, under Camera Configuration, click Add Camera Binding and set Source Entity to a camera in the scene.
  5. Check how colour video will be encoded. On Windows and Linux, new scenes use In-Process GPU (H.264 / Vulkan Video), which needs no other software on a GPU that supports it. On macOS, on a GPU without that support, and for depth or masks, you also need the FFmpeg program, which Tools → FFmpeg Settings can install. See Video Encoding and FFmpeg.

  6. Add a script that starts the recording and marks the episodes, and attach it to an entity. A minimal one, driven by buttons in the Inspector:

    using Hazel;
using Hazel.Data;

public class QuickRecorder : Entity
{
    protected override void OnCreate()
    {
        Observer.StartRecording();
    }

    [Button("Next episode")]
    public void NextEpisode() => Observer.EndCurrentEpisodeBeginNext(true);

    [Button("Stop recording")]
    public void Stop()
    {
        Observer.EndCurrentEpisode(true);
        Observer.StopRecording();
    }
}

  7. Click Play. The script starts the session. Click Recorder Status on the viewport toolbar to check that it reads Recording episode 1 of 300.

  8. Select the entity that has the script. In the Inspector, below the Scene Hierarchy, click Next episode at each episode boundary and Stop recording when you are done.
  9. Stop Play and open the Observer panel (View → Observer, or Observe on the side bar). The new session is at the top of Data Sessions.

There is no button in the editor that starts a recording, and no gRPC call either: a session always starts from a script. Creating and attaching scripts is covered in Creating and Building Scripts.

Stopping ends the open episode as failed

Stop Recording in Recorder Status, stopping Play and switching scenes all stop the session and finalise its files. An episode that is still open at that moment is ended as a failed episode. End it with Observer.EndCurrentEpisode(true) first if it should count as a success.

The recording pages

  • Video Encoding and FFmpeg

    The Video Format choices, what still needs FFmpeg and how to install it, frame rates, and path-traced camera recording.

  • Dataset Format

    What a session writes to disk: folders, chunks, data columns per recorder mode, metadata files and failed episodes.

  • Custom Recording Columns

    The Custom Schema recorder mode: choose exactly which MuJoCo values become columns with a YAML schema.

  • Viewing Files

    The Parquet Table and Video Player tabs that open when you click a recorded file.

Before you record

  • A LuckyHub sign-in. Starting a session creates a recording task in LuckyHub for your Lucky Robots account, and a short summary of every finished episode (such as its number, success, duration and frame count) is reported to it. The recorded files stay on your machine. If LuckyHub refuses the session, for example because the sign-in expired or the account's episode quota is used up, recording does not start and Recorder Status shows the reason as LuckyHub: …. If LuckyHub stops accepting episodes during a session, the session ends after the current episode.
  • Enable MuJoCo Observer on. The switch is at the top of the Recorder page, and while it is off Recorder Status offers an Enable Observer button. A change takes effect the next time you press Play.
  • Something to record. The data recorder records the MuJoCo robots in the scene, and cameras are recorded only when they have a camera binding.
  • FFmpeg, only for what the built-in GPU encoder does not cover: depth, masks, lidar video proxies, the two FFmpeg video formats, and colour video on macOS or on a GPU that cannot encode it. See Video Encoding and FFmpeg.

The Observer panel

The Observer panel shows Recorder Status, the state of the recorder with controls to prepare, stop and adjust a session, above the Data Sessions list of the sessions already on disk.

Open it in any of these ways:

  • View → Observer. It docks on the left by default, and can also be docked on the right.
  • The Observe button on the side bar switches to the built-in layout with the Observer on the left, the Scene viewport in the centre and the Log at the bottom.
  • The Recorder Status button in the centre of the viewport toolbar, beside Play, opens Recorder Status on its own as a popup over the viewport, without the session list.
  • In the HIGH PERFORMANCE MODE panel during Play, Open Data Observer (or D with the cursor over the viewport) opens that popup too.

The panel has Recorder Status at the top and the Data Sessions list at the bottom, with a splitter between them that you can drag.

If Enable MuJoCo Observer is off, Recorder Status only shows The MuJoCo Observer is off in this scene's settings., a line explaining that recording needs it, and an Enable Observer button. The button works in Edit mode, and the change applies on the next Play.

Recorder Status

The Recorder Status header has a Settings button, which opens the Recorder settings page, and a docs button that opens Recording with the Observer.

The line below the header says what the recorder is doing:

Status line When
Not recording. Recording starts when the scene's script calls Observer.StartRecording() during Play. Edit mode, no session.
Waiting for the scene's script to start recording. During Play, before a session starts.
Ready. Recording starts when the scene's script calls Observer.StartRecording(). A session is armed during Play, but nothing is being captured yet.
Armed in the edited scene. Play runs a copy of the scene, which sets up its own recording when its script calls Observer.StartRecording(). A session is armed in Edit mode. It does not carry over into Play.
Recording episode N of M A session is recording episode N, out of Total Episodes.
Recording, waiting for the scene's script to begin the next episode. A session is recording, but no episode is open, so nothing is captured.
Finishing the recording... The session is stopping and its files are being finalised. A spinner shows beside it.
Recorded N episodes. The session reached Total Episodes and finished.

While a session is recording, and after it finishes, a bar under the line counts the episodes against Total Episodes, for example 12 / 300 episodes.

While you are signed out, Recorder Status says Recording sessions are registered with your LuckyHub account. and shows a Sign In to LuckyHub to record button, which starts signing in. It reads Finish signing in in your browser... until you complete the sign-in. If LuckyHub refused the last session, the reason appears as LuckyHub: ….

The buttons under the status depend on the state:

Button When What it does
Arm Recorder No session, or the last one finished Prepares a session ahead of time. Optional: StartRecording() arms the recorder itself. Disabled while you are signed out.
Disarm Armed Cancels the prepared session.
Recording... Recording Disabled; shows that a session is in progress.
Stop Recording Armed or recording While recording, stops the session and finishes saving what was recorded. While armed, cancels the session without recording anything. Reads Stopping... until it is done.
Open Output Folder The session finished Opens the session's folder.

Recorder Status controls the scene that is currently running, so arm the recorder while the scene is playing.

After a session reaches Total Episodes, a script cannot start another one in the same Play run until you click Arm Recorder again (or stop and restart Play).

Data, camera and lidar recorders

Below the session controls are three sections, Data, Cameras and Lidar, one for each recorder. Each section heading has:

  • An Edit button, which opens that recorder's settings: the Recorder page's Data tab, the Camera Configuration part of the Video tab, or the Lidar tab.
  • A switch that turns the recorder on or off for the next session. It is the same setting as Record Data, Record Cameras or Record Lidar on the Recorder page's General tab.

The switches work only in Edit mode with no session armed or recording. Otherwise they are disabled, with the tooltip Stop the scene and disarm the recorder to change this.

Each section lists rows with a status on the right. A row reads Off while its recorder is off, and otherwise follows the session: Ready, Armed, Recording, or Waiting for episode while a session is recording with no episode open.

  • Data has one row, Data Log.
  • Cameras has one row per camera binding, described in Bound cameras. With no binding, it says No cameras are bound. Click Edit and bind a scene camera for the recorder to record from.
  • Lidar has one row per lidar binding, named by its Output Name, or (unnamed) without one. A row reads Missing lidar entity if the binding's lidar no longer exists. With no binding, the section says No lidars are bound. Click Edit and bind a lidar sensor from the scene for the recorder to record. Lidar bindings are covered in Lidar.

When the cameras would record something that needs the FFmpeg program and it is not installed, the Cameras section says what needs it and offers Install FFmpeg, plus Use In-Process GPU (H.264) where that takes colour video off FFmpeg. See When FFmpeg is missing.

While a session is armed or recording, Recorder Status shows the settings that session started with. Otherwise it shows the scene's current settings, which the next session will use.

Bound cameras

The Cameras section lists every camera binding by its Output Name, with a status and a switch. The status says why that camera will or will not write anything:

Status Meaning
Disabled The binding's switch is off.
Off The camera recorder is off.
No streams This camera has nothing to record: Record Color Video is off, and the camera does not record masks.
Missing camera entity The binding's camera entity no longer exists.
Needs FFmpeg One of this camera's streams needs the FFmpeg program, which is not installed.
Ready, Armed, Recording, Waiting for episode The camera follows the session.

Hover a row to see a summary of that camera: resolution and frame rate, Rasterized or Path traced (with the quality, denoiser, bounces and samples or sub-frames), the configured streams (Color only, Color + depth (muxed), Color + separate depth or Depth only) and whether masks are on.

Each row's switch includes or excludes that camera's colour, depth and masks in the next session. It is the same setting as the binding's Enabled field. Like the recorder switches, it works only in Edit mode with the recorder disarmed.

To add or change camera bindings, click Edit on the Cameras heading. It opens the Recorder page's Video tab at Camera Configuration.

Live analytics

While a session is recording, and after it finishes, Recorder Status adds a Live Analytics section:

Field Shows
Thin bar The current episode's progress towards Max Episode Length, with a tick where the last successful episode ended.
N / M frames Frames in the current episode, against the length of the last successful episode (or Max Episode Length before there is one).
Successful Episodes that ended successfully.
Avg Episode Average real (wall-clock) duration of an episode, in seconds.
Speed Simulated time recorded per second of real time, smoothed, for example 1.02x (Real-time). Below 0.98x it reads (Slow). Hover it for the speed of this episode, the last episode and the whole session.
Est. Completion Estimated time left until Total Episodes is reached.

Data Sessions

Data Sessions lists the session folders in the DataSessions folder of the Output Directory, or of the project folder when no output directory is set. Sessions are named session_<date>_<time> and listed newest first. Inside each session, only .parquet, .mp4 and .mkv files are shown.

  • The drop-down at the top filters by age: Last 5 Min, Last 15 Min, Last 30 Min, Last Hour, Last 12 Hours, Last Week or All (the default).
  • The refresh button (Refresh data sessions) scans the folder again. The list also refreshes by itself when Play stops.
  • Below them are a session count and a search box.
  • Click a file to open it in its own tab: .parquet files open in a Parquet Table, .mp4 and .mkv files in the Video Player. See Viewing Files.
  • Right-click a session or a file and choose Show in Explorer to open its folder.

Cleanup Data deletes old sessions. In the Cleanup Data Sessions dialog, choose how much to keep under Keep data from: (default Last Hour). The dialog shows how many sessions will be deleted. Click Delete, then Yes, Delete to confirm. Deleted sessions cannot be recovered.

Recorder settings

The capture settings are saved with the scene, on the Recorder page of the Scene group in Settings. At the top of the page, Enable MuJoCo Observer switches the whole recorder on or off for the scene. It applies the next time you press Play, and the tabs below it are shown only while it is on.

When settings are locked

The capture settings can be changed only while the scene is in Edit mode and no session is armed or recording. Otherwise the page is read-only and says why:

  • Stop the scene before changing capture settings. appears during Play.
  • Disarm the recorder or stop the session before changing capture settings. appears while a session is armed or recording.

The switches in Recorder Status follow the same rule.

General tab

Field Default Description
Output Directory [Project Root] Folder that receives the DataSessions folder. ... browses for a folder, R resets to the project root.
Record Data On Records robot state and actions to Parquet.
MuJoCo Observer [Data] TimeRunner Robot [50.00Hz] The time runner that sets the data rate: data is recorded once per tick. Shown while Record Data is on.
Record Cameras On Records the camera bindings: colour video, depth and masks. Called Record Video in 2026.2.
Record Color Video On Shown while Record Cameras is on. Records each enabled camera's colour video, and its depth where the camera's Depth Mode asks for it. Masks are set on the Video tab.
MuJoCo Observer [Video] TimeRunner Robot [50.00Hz] The time runner that sets how often camera frames are captured. Shown while Record Cameras is on.
Record Lidar Off Records the lidar bindings.
Lidar Recorder TimeRunner Robot [50.00Hz] The time runner that drives the lidar recorder. Shown while Record Lidar is on.

Record Data, Record Cameras and Record Lidar are the same settings as the recorder switches in Recorder Status. The camera recorder is on while Record Cameras is ticked and it has something to record, colour video or masks. Ticking Record Cameras when neither is on also turns Record Color Video on.

When the cameras would record something that needs the FFmpeg program and it is not installed, a warning below the video settings says what needs it, with a Use In-Process GPU (H.264) button where that takes colour video off FFmpeg. The settings themselves stay as they are. See When FFmpeg is missing.

The runner lists show each runner as its name and frequency, such as Robot [50.00Hz] or Capture [30.00Hz]. Rates count simulated time, so the recording keeps the same cadence however fast the simulation runs. Each runner picker also has an Execution Phase (Acquisition for data and video, Validation for lidar by default) and an Internal Phase Priority, which orders it among the systems in that phase; see Phases.

Session tab

Field Default Description
Total Episodes 300 1 to 1000. When this many episodes have been kept, the session finishes and Recorder Status shows Recorded N episodes.
Max Episode Length (Seconds) 90 1 to 86400 seconds of simulated time. An episode that reaches it is ended as failed, and no new episode is opened.
Discard & Retry Failed Off When on, an episode that ends as failed is thrown away and the next attempt reuses its episode number. When off, failed episodes are kept and marked as unsuccessful.

The tab is greyed out while Record Data is off. Max Episode Length is checked by the data recorder, so it has no effect while Record Data is off. Failed episodes are described in more detail in Dataset Format.

Data tab

Field Default Description
Recorder Mode Robot Arm Which values each data row holds. See below.
Data Format Parquet (Apache Arrow) Fixed; the drop-down cannot be changed.
Column Schema [None] Custom Schema mode only. The YAML file listing the columns to record. ... browses, R clears. See Custom Recording Columns.
Files Per Chunk 40 1 to 1000. How many files go into each chunk-NNN folder.
Episodes Per File 1 1 to 1000. How many episodes each data file holds.
Capacity Per Chunk 40 Read-only: Episodes Per File × Files Per Chunk.

The tab is greyed out while Record Data is off. The four recorder modes are:

  • Robot Arm: the commands sent to each actuator, and the positions of the actuated joints.
  • MuJoCo Full Scene: the same, but with the positions of every joint, including passive ones.
  • Drone: does not produce a usable dataset in this release. Use one of the other modes.
  • Custom Schema: only the columns declared in the Column Schema file.

No preset mode records contact forces or end-effector poses. To record values such as an end-effector pose, use Custom Schema. The exact columns of each mode are listed in Dataset Format.

Video tab

Field Default Description
Video Format In-Process GPU (H.264 / Vulkan Video) on Windows and Linux, FFmpeg (Generic / CPU) on macOS The encoder for colour video. Scenes saved before 2026.3 keep FFmpeg (Generic / CPU). See Video Encoding and FFmpeg.
Record Masks Off Experimental and not intended for general use. Writes an extra mask video for each camera. Needs FFmpeg.
Jitter Masks On Part of the experimental mask recording.
Default Depth Mode Off The Depth Mode given to new camera bindings.
Default Depth Scale 0.001 The Depth Scale (m/code) given to new camera bindings. 0.00001 to 0.1.

The two defaults are copied into a binding only when you click Add Camera Binding. Changing them later does not change existing bindings. Below them, Camera Configuration holds the camera bindings.

Lidar tab

The Lidar tab holds the lidar bindings. It is greyed out until Record Lidar is on. Every field is described in Lidar.

Camera bindings

A camera is recorded only if it has a camera binding: an entry in the Recorder settings that points at a camera entity and says how to record it. The camera's own component provides the position and projection; the binding provides everything else. The Sensors page explains this binding model.

Adding a camera

  1. Open the Recorder page's Video tab. Under Camera Configuration, click Add Camera Binding. The Edit button on the Cameras section of Recorder Status leads to the same place.
  2. Set Source Entity. The picker lists only entities with a Camera component. When you pick one while Output Name is empty, it is filled in with the entity's name.
  3. Set the resolution, Frame Rate, Depth Mode and Render Mode described below.

Each binding appears as a section headed by its output name, or Camera #N if the name is empty. The X button on the header removes it. Some Content Vault packs add bindings for their own cameras when you place them in a scene.

Binding fields

Field Default Description
Source Entity None The camera entity to record.
Output Name Entity name Used in folder and column names, as observation.images.<Output Name>. Give each camera a different name.
Width, Height 256, 256 Recorded resolution in pixels.
Frame Rate 30 fps 24, 25, 30, 50, 60, 100 or 120 fps. See Frame Rate.
Enabled On Off skips this camera entirely: no colour, depth or masks.
Record Masks On Shown only while the Video tab's Record Masks is on. Part of the experimental mask recording.
Depth Mode Default Depth Mode Which streams this camera writes. See Depth Mode.
Depth Scale (m/code) Default Depth Scale Metres per 16-bit depth code for this camera. Shown when Depth Mode is not Off.
Depth Sensor Model None Clean depth, or a simulated depth camera with dropout and noise. Shown when Depth Mode is not Off.
Render Mode Rasterized Rasterized or Path Traced. See Render Mode.

When a Depth Sensor Model other than None is chosen, its calibration fields (Max Range (m), Min Range (m), Detect Floor, Stereo Baseline (m), Disparity Noise (px)) and a Diagnostic channel drop-down appear. They are covered in RGBD and Depth, including Diagnostic channels.

Depth Mode

Each camera chooses its own streams:

Depth Mode Writes
Off Colour only, as an .mp4.
RGBD Colour and depth, plus a copy combining both in one .mkv after each episode.
Separate A colour .mp4 and a separate depth .mkv.
DepthOnly Depth only, no colour.

Depth is lossless 16-bit video in an .mkv file, and recording it always needs the FFmpeg program. Depth is recorded only while Record Cameras and Record Color Video are on and the binding is enabled. It always comes from the rasterized scene, even when the camera's Render Mode is Path Traced. Depth encoding, scales and sensor models are covered in RGBD and Depth.

Upgrading from 2026.2

In 2026.2, depth was one scene-wide Depth Mode plus a Record Depth checkbox on each camera. Scenes saved by 2026.2 are converted when they open: each camera that had Record Depth on gets the old scene-wide mode, the others get Off, and the old scene-wide mode becomes the Default Depth Mode.

Frame Rate

Frame Rate is the rate written into the camera's video file and its entry in the dataset metadata. It does not change how often frames are captured: that is the frequency of the MuJoCo Observer [Video] TimeRunner on the General tab. If the two differ, the video's timestamps do not line up with the data.

Match the frame rate to the video runner

The defaults differ. New scenes capture video on the Robot [50.00Hz] runner, while a new binding's Frame Rate is 30 fps. Either set each camera's Frame Rate to 50 fps, or choose the Capture [30.00Hz] runner for video.

Render Mode

Rasterized records the camera with the same renderer as the editor's Rasterized view. Path Traced records its colour stream with the path tracer instead, and adds these fields to the binding:

  • Denoiser: None (reference) (the default), ReBLUR, ReLAX, OIDN or NVIDIA DLSS Ray Reconstruction (NVIDIA RTX GPUs only).
  • DLSS Mode: shown with NVIDIA DLSS Ray Reconstruction.
  • Quality: Draft, Medium (the default), High, Reference, or Custom once you edit the values below.
  • Bounces, and either Samples (with None (reference) and OIDN) or Sub-frames (with the other denoisers).

Path-traced recording needs a GPU that can path trace and is not available on macOS. If the camera cannot be path traced, recording fails rather than falling back to a rasterized image. The presets, the denoisers and their effect on reproducibility are covered in Path-traced camera recording.

Where to go next