Skip to content

Custom Recording Columns

The Custom Schema recorder mode records exactly the values you list in a YAML file, each as its own named, typed Parquet column. Use it to record joint velocities, MuJoCo sensor readings or end-effector poses, which the built-in recorder modes leave out.

In this mode the recorder writes a fixed set of index columns plus your columns, and nothing else: there are no action or observation.state columns. Camera, depth and lidar recording work as usual.

In short

Write a column schema in YAML, then on the Data tab of the Recorder settings set Recorder Mode to Custom Schema and pick the file under Column Schema. Each column names a source, such as mujoco.qpos or mujoco.body_pose, and the joint, actuator, sensor or body to read. If anything in the schema does not match the robot, the recording is refused and the Log panel says why.

When to use Custom Schema

Choose Custom Schema when:

  • You need values the other modes do not record, such as joint velocities, MuJoCo sensor data or a body's pose.
  • You want one clearly named column per quantity instead of the packed action and observation.state vectors.
  • You want a smaller dataset that holds only what your pipeline reads.

Stay with Robot Arm or MuJoCo Full Scene if your training pipeline expects action and observation.state columns. Custom Schema does not write them. The Dataset Format page compares the modes.

Turning it on

Recorder settings can only be changed in Edit mode, while the recorder is not armed.

  1. Write a schema file (see The schema file) and save it inside the project folder.
  2. Open Settings → Scene → Recorder and make sure Record Data is on, on the General tab. The Data tab is greyed out without it.
  3. On the Data tab, set Recorder Mode to Custom Schema. A Custom Columns section appears.
  4. Next to Column Schema, click ... and choose your .yaml or .yml file. R clears the selection, which shows as [None].

The choice is saved with the scene. A schema file inside the project folder is stored relative to the project, so the scene still finds it after the project is moved or opened on another computer. A file outside the project is stored as an absolute path.

With no schema selected, Custom Schema records only the index columns. Other recorder modes ignore the schema.

No scripting API

Custom columns are defined only by the schema file chosen in the Recorder settings. There is no C# or gRPC call that adds or changes them.

The schema file

A schema is a YAML file with an optional version and a list of columns:

version: 1
columns:
  - name: joint1_vel
    source: mujoco.qvel
    select: { joint: joint1 }
    unit: rad/s
  • version is optional. If present, it must be 1.
  • columns is required and must list at least one column.

Each column is a map with these keys. Other keys are ignored.

Key Required Description
name Yes The Parquet column name. Must be unique within the schema
source Yes What to read. See Sources
select Depends on the source A map of selector values, such as { joint: joint1 }. A selector can also be a list, such as fields: [pos, quat]
dtype No The stored type. Defaults to the source's own type
shape No The number of values per row, as a number (7) or a one-element list ([7]). If given, it must equal the source's width
unit No Free text, copied into meta/info.json

shape is a check rather than a setting: a column always has its source's width, and a shape that disagrees refuses the recording. Use it to catch a joint whose type is not what you expected.

Data types

dtype Also accepted Stored as
f64 float64, double 64-bit float
f32 float32, float 32-bit float
i64 int64 64-bit integer
u8 uint8 8-bit unsigned integer

Values are converted to the chosen type as they are recorded. Use f32 to halve the size of a float column. Choose an integer type only for values that are whole numbers within the type's range: fractional parts are dropped.

Sources

Names in select must match the robot's MuJoCo model exactly, including case. The joint, actuator, sensor and body names are the ones in the robot's MJCF (.xml) file. To read it in the editor, right-click the file in the Content Browser and choose Open in Text Editor (see Viewing Files).

MuJoCo values

Source Selector Width Default dtype Value
mujoco.qpos joint 1, 4 or 7 f64 The joint's position
mujoco.qvel joint 1, 3 or 6 f64 The joint's velocity
mujoco.ctrl actuator 1 f64 The actuator's control value
mujoco.sensor sensor The sensor's dimension f64 The sensor's output
mujoco.body_pose body, optional fields 3, 4 or 7 f64 The body's position and orientation

mujoco.qpos and mujoco.qvel depend on the joint type:

Joint type mujoco.qpos mujoco.qvel
Hinge 1 value, radians 1 value, radians per second
Slide 1 value, metres 1 value, metres per second
Ball 4 values, quaternion w, x, y, z 3 values, angular velocity
Free 7 values, position x, y, z then quaternion w, x, y, z 6 values, linear then angular velocity

mujoco.ctrl records the actuator's raw control value. Unlike the action column of the other modes, it is never clamped or rescaled to the joint's range.

mujoco.sensor reads a sensor declared in the robot's MJCF <sensor> section, such as a touch, force or accelerometer sensor. It does not read the engine's lidar or depth sensors, which record into their own files.

mujoco.body_pose reads the body frame's position (3 values, metres) and orientation (4 values, quaternion w, x, y, z) in the MuJoCo model's world coordinates. Set fields to [pos], [quat] or [pos, quat] to choose. Without fields, both are recorded: 7 values, position first.

See Units for the engine's unit conventions.

Engine values

Source Selector Width Default dtype Value
engine.sim_time 1 f64 Simulation time in seconds since the episode began
engine.frame_flag optional flag 1 u8 One frame flag as 0 or 1, or the whole flags byte
engine.episode 1 i64 The episode index
engine.frame 1 i64 The frame index within the episode
engine.task 1 i64 The current task index

engine.sim_time holds the same value as the timestamp column: it restarts at 0 for each episode and advances by one data-runner step per frame. It is not the time since Play started.

engine.frame_flag takes flag: NewPolicyStep (a policy ran inference on this step) or flag: PostReset (the first recorded frame after the robot was reset to its initial state). Without flag, it records the whole flags byte, the same value as the frame_flags column.

engine.episode, engine.frame and engine.task repeat values the index columns already hold, under a name of your choosing.

Derived values

Each derived value is one f64 per row, computed over the whole robot model:

Source Value
compute.l2_ctrl Sum of the squares of every actuator's control value
compute.l2_qvel Sum of the squares of every joint velocity value
compute.qpos_l2 Sum of the squares of every joint position value, including the quaternion values of ball and free joints

These are squared magnitudes: no square root is taken. They are useful as control-effort and motion measures.

Example schemas

For any robot

This schema uses only sources that do not name anything in the model, so it is accepted for any scene with a MuJoCo robot:

version: 1
columns:
  - name: sim_time
    source: engine.sim_time
    unit: s
  - name: is_policy_step
    source: engine.frame_flag
    select: { flag: NewPolicyStep }
  - name: ctrl_effort
    source: compute.l2_ctrl
  - name: qvel_energy
    source: compute.l2_qvel

For the Franka Panda

The Panda's MJCF names its arm joints joint1 to joint7, its finger joints finger_joint1 and finger_joint2, its actuators actuator1 to actuator8, and its gripper body hand. This schema records one joint's position, velocity and command, one gripper finger's position, and the end-effector pose:

version: 1
columns:
  - name: joint1_pos
    source: mujoco.qpos
    select: { joint: joint1 }
    unit: rad
  - name: joint1_vel
    source: mujoco.qvel
    select: { joint: joint1 }
    unit: rad/s
  - name: joint1_cmd
    source: mujoco.ctrl
    select: { actuator: actuator1 }
  - name: gripper_pos
    source: mujoco.qpos
    select: { joint: finger_joint1 }
    unit: m
  - name: ee_pos
    source: mujoco.body_pose
    select: { body: hand, fields: [pos] }
    shape: [3]
    unit: m
  - name: ee_quat
    source: mujoco.body_pose
    select: { body: hand, fields: [quat] }
    dtype: f32
  - name: is_policy_step
    source: engine.frame_flag
    select: { flag: NewPolicyStep }

For another robot, replace the joint, actuator and body names with the ones in its MJCF.

What lands in the dataset

The data files under data/ hold the index columns followed by your columns, in schema order:

Column Type
index int64
entity_id int64
timestamp float64
episode_index int64
frame_index int64
task_index int64
frame_flags uint8
Your columns As declared

A column one value wide is a plain Parquet column of its type. A wider column is a fixed-size list of that type: ee_pos above is a list of 3 float64 values per row.

The metadata describes your columns too:

  • meta/info.json has a feature per column, with its dtype, shape and, if the schema set one, its unit. It also sets "entity_type": "custom".
  • meta/stats.json has min, max, mean, std and count for each column.

There are no action or observation.state features or statistics. Everything else about the session, including video and episode metadata, is as described in Dataset Format.

Scenes with several robots

Each MuJoCo robot in the scene (each entity with a MujocoSceneComponent) writes its own row on every data tick, told apart by entity_id. The schema is applied to every robot, so:

  • every joint, actuator, sensor and body named in the schema must exist in every robot's model;
  • each column must resolve to the same width and type on every robot.

If one robot lacks a name, or a column has different widths on two robots, the recording is refused.

When recording is refused

The schema is checked when the recorder is armed, either by Arm Recorder in Recorder Status or by Observer.StartRecording(). If any check fails, the session does not start, and the Log panel shows one of these messages:

Record columns: schema '<path>' rejected — <reason>. Recording refused.
Record columns: <reason> — recording refused.

The first form reports a problem with the file itself, the second a problem matching it to the scene. When the schema is accepted, the Log shows:

Record columns: <N> custom column(s) registered from <path>.
Reason (excerpt) Fix
schema file not found The file was moved or renamed. Pick it again under Column Schema
YAML parse error Fix the YAML syntax, often indentation or a missing colon
unsupported schema version Set version: 1 or remove the line
schema has no 'columns' sequence Add a columns: list with at least one entry
missing 'name', missing 'source' Give every column a name and a source
duplicate column name Make every name unique
unknown dtype Use f64, f32, i64 or u8
only scalar or 1-D 'shape' is supported, 'shape' must be positive Write shape as a positive number or a one-element list
unknown source Check the spelling against Sources
requires select.joint (or actuator, sensor, body) Add the selector the source needs
joint '<name>' not found in model (or actuator, sensor, body) Use the exact name from the robot's MJCF. Names are case-sensitive
body_pose fields must include 'pos' and/or 'quat' Set fields to [pos], [quat] or [pos, quat]
unknown frame flag Use NewPolicyStep or PostReset
declared shape [N] != source width M Remove shape, or correct it to the source's width
resolves to a different shape/type across robots Make the column read a joint, sensor or body with the same layout on every robot
no MuJoCo robot in the scene Add a MuJoCo robot, or clear the Column Schema

Where to go next