Recording with the Observer

The Observer is the engine's data recorder. It captures robot state and actions, camera video and lidar scans while a scene plays, and writes them to disk as a dataset. Scripts control it through Hazel.Data.Observer, a small static class: they start and stop recording sessions, mark where episodes begin and end, and label the sub-tasks inside an episode.

What gets captured (the cameras, the recorder mode, the rates, the output folder) is set in the editor, not from script, apart from adjusting existing camera bindings. That side is covered in Recording.

In short

Observer.StartRecording() starts a session and opens its first episode. Observer.EndCurrentEpisode(success) closes an episode, and Observer.BeginEpisode() opens the next one: episodes never start by themselves. EndCurrentEpisodeBeginNext does both in one call. Observer.StopRecording() finishes the session. Read IsRecording and IsEpisodeOpen to follow the recorder's state.

Setting up recording in the editor?

Signing in, camera bindings, FFmpeg, the Observer panel and every Recorder setting are described in Recording. Start there, then come back here for the script side.

The session lifecycle

A recording is one session that contains one or more episodes. A session writes a single folder, DataSessions/session_<date>_<time>/, under the Recorder's output directory. Each episode is one trajectory within that session.

stateDiagram-v2
    [*] --> Idle
    Idle --> Armed : Arm Recorder
    Armed --> Idle : Disarm
    Idle --> Recording : StartRecording
    Armed --> Recording : StartRecording
    Recording --> Idle : StopRecording
    Recording --> Complete : Total Episodes reached
    Complete --> Armed : Arm Recorder

    state Recording {
        [*] --> EpisodeOpen
        EpisodeOpen --> BetweenEpisodes : EndCurrentEpisode
        BetweenEpisodes --> EpisodeOpen : BeginEpisode
    }

State	Recorder Status during Play	IsRecording	IsEpisodeOpen
Idle	Waiting for the scene's script to start recording.	false	false
Armed	Ready. Recording starts when the scene's script calls Observer.StartRecording().	false	false
Recording, episode open	Recording episode N of M	true	true
Recording, between episodes	Recording, waiting for the scene's script to begin the next episode.	true	false
Complete	Recorded N episodes.	false	false

• Idle: no session.
• Armed: a session has been prepared with Arm Recorder in Recorder Status. Arming is optional, because StartRecording() arms the recorder itself when needed.
• Recording: StartRecording() opens the session and episode 0 in one call. Data rows and camera frames are captured only while an episode is open.
• Complete: the session reached the Total Episodes setting and finished on its own. StartRecording() is ignored in this state until Arm Recorder is clicked in Recorder Status, or Play is restarted.

Episodes

EndCurrentEpisode(success) closes the current episode and leaves the session recording with no episode open. No data rows or camera frames are captured until the script calls BeginEpisode(). During that gap the recorder rows in Recorder Status read Waiting for episode. The gap is a good place to reset the scene, so that the reset is not part of either episode.

EndCurrentEpisodeBeginNext(success) ends the current episode and opens the next one in the same call, with no gap. If the episode it ends reaches Total Episodes, the session completes instead and no new episode opens.

Stopping a session

StopRecording() is a request. Capture stops immediately (IsEpisodeOpen becomes false), and the files are finalised a moment later. Until then IsRecording can still read true, and a StartRecording() call is ignored.

An episode that is still open when the stop happens is closed as failed. To keep the last episode as a success, call EndCurrentEpisode(true) before StopRecording().

A session also ends without a script call:

Cause	Result
Stop Recording in Recorder Status	The same as StopRecording().
Stopping Play, or switching scenes	The same as StopRecording().
Total Episodes reached	The session finishes in the Complete state.
LuckyHub stops accepting episodes	The session ends after the current episode, and the Log shows the error LuckyHub returned.
A camera's video stream fails	The session stops, and the episode in progress is marked as failed.

Episodes the recorder ends by itself

• When an episode reaches Max Episode Length (Seconds), the recorder ends it as failed and does not open a new one. A script that runs many episodes should check IsEpisodeOpen and call BeginEpisode() when it finds no episode open.
• If any video, depth or lidar stream fails during an episode, that episode is marked as failed even when the script passes true.

Pausing

Pausing the simulation pauses the recorder. The open episode stays open, and capture continues when the simulation resumes.

The Observer API

Every member is static on Hazel.Data.Observer; add using Hazel.Data; to the script. This page is the reference for the class, which has no page in the generated API reference.

Member	Description
void StartRecording()	Starts a session and opens episode 0. See StartRecording.
void StopRecording()	Requests the end of the session. An open episode is closed as failed. See Stopping a session.
void BeginEpisode()	Opens the next episode. Ignored if no session is recording or an episode is already open.
void EndCurrentEpisode(bool success)	Closes the current episode as a success or a failure. Does not open the next one. Ignored if no session is recording or no episode is open.
void EndCurrentEpisodeBeginNext(bool success)	Closes the current episode and opens the next one in one call, unless the session completes. Ignored if no session is recording or no episode is open.
void RegisterTask(int index, string description)	Adds a task label, or updates the description of an existing index. See Task labels.
void SetCurrentTaskIndex(int index)	Tags every following data row with this task index.
bool IsRecording { get; }	true while a session is recording, between a successful start and the end of the session.
bool IsEpisodeOpen { get; }	true while a session is recording and an episode is open for capture.

Calls that do not fit the current state are ignored. For example, BeginEpisode() straight after StartRecording() does nothing, because StartRecording() already opened episode 0.

Upgrading from 2026.2

IsRecording used to follow only the script's own StartRecording() and StopRecording() calls. It now reports the recorder's real state, so it also becomes false when a session ends on its own: Total Episodes reached, Stop Recording clicked, or LuckyHub refusing further episodes. IsEpisodeOpen is new in 2026.3.

StartRecording

StartRecording() does the following, in order:

1. Arms the recorder if it is Idle, loading the scene's Recorder settings.
2. Checks that you are signed in to LuckyHub and creates a recording task there. The call blocks the script briefly while LuckyHub responds.
3. Creates the session folder and opens episode 0.

It does not start a session when:

• you are not signed in, in which case Recorder Status offers a Sign In to LuckyHub to record button;
• LuckyHub refuses the session, for example because the sign-in expired or the episode quota is used up. Recorder Status shows the reason as LuckyHub: …;
• Enable MuJoCo Observer is off in the scene's Recorder settings;
• the recorder mode is Custom Schema and the column schema cannot be loaded or does not match the robot (see Custom Recording Columns);
• a session is already recording, is still stopping, or is in the Complete state.

If a camera's video stream cannot start, the session is stopped straight away. After the call, Observer.IsEpisodeOpen tells you whether recording actually began.

Task labels

Task labels name the phases or goals of the work being recorded. Each data row carries a task_index, and the session's meta/tasks.parquet file maps each index to its description.

• RegisterTask(index, description) adds a label. Calling it again with the same index replaces the description.
• SetCurrentTaskIndex(index) tags every data row from now on with that index. It can be called at any time, including in the middle of an episode.

Register every label before StartRecording(). The session's list of labels is fixed when the recorder arms, so labels registered later are not written to that session's tasks.parquet. A session with no registered labels gets a single Default Task at index 0.

Common patterns

Bracket a single task

The simplest pattern: buttons in the Inspector that drive one session by hand. [Button] methods appear in the Inspector of the entity that has the script, and run while the scene is playing.

using Hazel;
using Hazel.Data;

public class TaskRecorder : Entity
{
    [Button("Start recording")]
    public void StartSession()
    {
        Observer.StartRecording();
        if (Observer.IsEpisodeOpen)
            Log.Info("Recording started");
        else
            Log.Warn("Recording did not start. Check the Observer panel.");
    }

    [Button("End episode (success)")]
    public void Succeed() => Observer.EndCurrentEpisode(true);

    [Button("End episode (failure)")]
    public void Fail() => Observer.EndCurrentEpisode(false);

    [Button("Begin next episode")]
    public void Next() => Observer.BeginEpisode();

    [Button("Stop recording")]
    public void StopSession()
    {
        // Close the open episode as a success, or the stop records it as failed.
        if (Observer.IsEpisodeOpen)
            Observer.EndCurrentEpisode(true);
        Observer.StopRecording();
    }
}

After either End episode button, click Begin next episode to record another one. Buttons and other Inspector attributes are covered in Editor attributes.

Run a batch of episodes

For data collection, a script starts the session once, then ends each episode, resets the scene and begins the next one. This example records fixed-length episodes of a robot. Attach it to the robot's root entity, the one with the MuJoCo Scene component, and replace the success check with your task's own.

using Hazel;
using Hazel.Data;

public class EpisodeRunner : Entity
{
    [Tooltip("Episodes this script ends before it stops the session.")]
    public int EpisodeCount = 10;

    [Tooltip("Length of each episode, in seconds of simulated time.")]
    public float EpisodeSeconds = 5.0f;

    private MujocoSceneComponent? m_Mujoco;
    private int m_Ended = 0;
    private float m_Elapsed = 0.0f;
    private bool m_Running = false;

    protected override void OnCreate()
    {
        m_Mujoco = GetComponent<MujocoSceneComponent>();

        // Labels must be registered before the session starts.
        Observer.RegisterTask(0, "reach");
        Observer.SetCurrentTaskIndex(0);

        Observer.StartRecording();
        m_Running = Observer.IsEpisodeOpen;
        if (!m_Running)
            Log.Warn("Recording did not start. Check the Observer panel.");
    }

    protected override void OnUpdate(float ts)
    {
        if (!m_Running)
            return;

        // The session can end without this script: Total Episodes reached,
        // Stop Recording clicked, or LuckyHub no longer accepting episodes.
        if (!Observer.IsRecording)
        {
            m_Running = false;
            return;
        }

        // The recorder ends an episode by itself at Max Episode Length.
        if (!Observer.IsEpisodeOpen)
        {
            StartNextEpisode();
            return;
        }

        m_Elapsed += ts;
        if (m_Elapsed < EpisodeSeconds)
            return;

        bool success = true; // Replace with your task's success check.
        Observer.EndCurrentEpisode(success);
        m_Ended++;

        if (m_Ended >= EpisodeCount)
        {
            Observer.StopRecording();
            m_Running = false;
        }
        else if (Observer.IsRecording) // false if that episode completed the session
        {
            StartNextEpisode();
        }
    }

    private void StartNextEpisode()
    {
        // Reset between episodes, while no episode is open.
        m_Mujoco?.ResetToInitialState();
        m_Elapsed = 0.0f;
        Observer.BeginEpisode();
    }
}

The first data row after ResetToInitialState() carries a post-reset flag, described in Dataset Format.

Two episode limits

The scene's Total Episodes setting (Settings → Scene → Recorder, Session tab) defaults to 300, with a maximum of 1000. When the session has kept that many episodes it completes on its own, even if the script's own count is higher.

For episodes that must start on exactly the same engine step in every run, see Deterministic recording with sync points.

Label sub-tasks within an episode

For episodes with several phases (approach, grasp, place), register one label per phase before the session starts and switch the index as the phase changes. Every data row is tagged with the current index, so downstream tools can split the dataset by phase.

using Hazel;
using Hazel.Data;

public class PickAndPlaceRecorder : Entity
{
    private const int k_TaskApproach = 0;
    private const int k_TaskGrasp    = 1;
    private const int k_TaskPlace    = 2;

    protected override void OnCreate()
    {
        // Register every label first: the session's labels are fixed when it starts.
        Observer.RegisterTask(k_TaskApproach, "approach");
        Observer.RegisterTask(k_TaskGrasp,    "grasp");
        Observer.RegisterTask(k_TaskPlace,    "place");

        Observer.SetCurrentTaskIndex(k_TaskApproach);
        Observer.StartRecording();
    }

    // Call these as the task moves between phases.
    public void EnterGraspPhase() => Observer.SetCurrentTaskIndex(k_TaskGrasp);
    public void EnterPlacePhase() => Observer.SetCurrentTaskIndex(k_TaskPlace);

    // Call this when the episode ends; the next one starts in the approach phase.
    public void FinishEpisode(bool success)
    {
        Observer.EndCurrentEpisodeBeginNext(success);
        Observer.SetCurrentTaskIndex(k_TaskApproach);
    }
}

Failed and discarded episodes

An episode is failed when the script ends it with success = false, and also when:

• it reached Max Episode Length (Seconds);
• one of its video, depth or lidar streams failed;
• it was still open when the session stopped.

What happens next depends on Discard & Retry Failed on the Recorder's Session tab:

• Off (the default): the episode is kept, and marked as unsuccessful in the episode metadata.
• On: the episode's data is thrown away, and the next episode reuses its episode number and file names. Only successful episodes end up in the dataset's data and metadata.

Total Episodes counts the episodes that are kept. Every episode that ends, failed or not, is reported to LuckyHub as a short summary. How failed episodes appear on disk is described in Dataset Format.

Configuring cameras from a script

MujocoSceneComponent can read and change a camera binding's resolution and depth settings. Both calls identify the camera by its own entity ID, and work only on a camera that already has a binding in the Recorder settings.

public void SetCameraConfig(ulong cameraEntityId, int width, int height, bool recordMask,
    int depthMode, bool enabled, int sensorModel, float depthScale, float maxRange,
    float minRange, float detectFloor, float baseline, float subpixelSigma)

public bool GetCameraConfig(ulong cameraEntityId, int[] outInts, float[] outFloats)

Parameter	Values
depthMode	0 Off (colour only), 1 RGBD, 2 Separate, 3 DepthOnly
sensorModel	0 None (clean depth), 1 ActiveStereo, 2 iToF, 3 StructuredLight, 4 dToFSparse, 5 All. 2, 4 and 5 currently record the ActiveStereo model. The models are described in RGBD and Depth.
maxRange, minRange, detectFloor, baseline, subpixelSigma	The Max Range (m), Min Range (m), Detect Floor, Stereo Baseline (m) and Disparity Noise (px) fields

GetCameraConfig returns false if the camera has no binding, or if either array has fewer than 6 elements. It fills:

• outInts: [width, height, recordMask, depthMode, enabled, sensorModel, diagnosticChannel]. Give the array 7 elements to also read the diagnostic channel.
• outFloats: [depthScale, maxRange, minRange, detectFloor, baseline, subpixelSigma].

SetCameraConfig is ignored while a session is recording, and when depthMode or sensorModel is out of range. A width or height below 1 becomes 1, and a depth scale of 0 or less becomes 0.001. Changes last for the rest of the Play run and are not saved to the scene file.

When changes take effect

The recorder loads its camera list from the Recorder settings when it arms, with Arm Recorder in Recorder Status or StartRecording(). Until it has armed once in the current Play run, GetCameraConfig returns false and SetCameraConfig changes nothing, even for a camera that has a binding. A change applies to the next session that is armed after the call, not to one that is already armed. To record with different settings from the first session, change the binding in the Recorder settings before pressing Play.

Upgrading from 2026.2

The fifth parameter of SetCameraConfig was bool recordDepth. It is now int depthMode, because depth is chosen per camera. Pass 0 where you passed false, and the depth mode you want (for example 1 for RGBD) where you passed true. Index 3 of GetCameraConfig's outInts is now the depth mode rather than 0 or 1.

The same settings are available over gRPC through CameraService, described in gRPC API. The full component reference is MujocoSceneComponent.

What gets recorded

The Recorder Mode on the Recorder's Data tab decides what each data row holds; it cannot be changed from script. Robot Arm, the default, records the commands sent to the actuators and the positions of the actuated joints. MuJoCo Full Scene records the same, but with the positions of every joint, passive ones included. Custom Schema records only the columns you declare in a YAML file, described in Custom Recording Columns. Drone does not produce a usable dataset in this release.

Alongside the data, each camera binding records colour video and, depending on its Depth Mode, depth video. Lidar bindings record lidar scans.

The recording rate is independent of the control loop

Data rows and camera frames are captured on the time runners chosen on the Recorder's General tab: MuJoCo Observer [Data] TimeRunner and MuJoCo Observer [Video] TimeRunner. New scenes use the 50 Hz Robot runner for both. These rates count simulated time, so the recording keeps the same cadence however fast the simulation is stepped: playing on its own, or driven flat out by a gRPC client.

This is different from the camera frames a gRPC client requests on each Step. Those are observations for a policy, rendered once per step, not a recording; see gRPC API. Use the Observer for datasets and video.

Output

A session writes one folder under DataSessions/ in the Recorder's Output Directory, or in the project folder when none is set:

DataSessions/session_<date>_<time>/
├── meta/      info.json, stats.json, tasks.parquet, episodes/
├── data/      data rows as Parquet, in chunk-NNN/file-NNN.parquet
└── videos/    one folder per camera stream, one video file per episode

Every data row carries episode_index, frame_index, timestamp (seconds since the episode began) and task_index. The full layout, every column and the metadata files are described in Dataset Format.

Caveats

Things to know

• Recording does not start automatically. A script must call Observer.StartRecording(). The Observer panel has no start button, and there is no gRPC call to start recording.
• A LuckyHub sign-in is required, and Enable MuJoCo Observer must be on. A change to Enable MuJoCo Observer applies on the next Play.
• StartRecording() blocks briefly while LuckyHub creates the recording task.
• Episodes do not advance by themselves. After EndCurrentEpisode, call BeginEpisode, or use EndCurrentEpisodeBeginNext.
• Stopping closes the open episode as failed. End it with success = true first to keep it.
• Reaching Total Episodes completes the session. StartRecording() is then ignored until the recorder is armed again or Play restarts.
• Recording pauses with the simulation. The open episode stays open across a pause.
• Capture settings are locked during Play. Change the Recorder settings in Edit mode.

Where to go next

• Recording covers the Observer panel and every Recorder setting.
• Dataset Format describes the files a session writes.
• Deterministic recording with sync points locks episode boundaries to a fixed cadence.
• Video Encoding and FFmpeg covers video formats and path-traced camera recording.