RGBD and Depth¶
RGBD is metric depth alongside a camera's colour image. Depth is rendered from the same viewpoint as the colour frame, encoded as 16-bit metric distance, and either written with the colour video, written instead of it, or returned live over gRPC. An optional depth sensor model replaces the clean depth buffer with the artefacts a real depth camera produces.
Depth is configured per camera, on the camera's recorder binding. Each camera decides for itself whether it records depth, at what scale, and through which sensor model.
In short
Add a camera binding on the Video tab of the Recorder settings and set its Depth
Mode. Depth is gray16le: one unsigned 16-bit code per pixel, converted with
metres = code * depth_scale. Each binding has its own Depth Scale (m/code) and
Depth Sensor Model. Under a depth sensor model, a code of 0 marks a pixel with no
return.
The camera model¶
Three separate things are involved, and only the first is an Inspector component:
- The Camera component is the entity's camera: projection, vertical FOV, near and far clip, whether it is the main camera, exposure, the Basic or Physical camera mode, and its viewport icon. It has no depth fields at all.
- A camera binding is a recorder setting that references that camera entity and carries every recording option: resolution, frame rate, Depth Mode, depth scale, the depth sensor model, and the render mode.
- Depth Video Defaults are scene-wide values that are copied into each new binding when it is created. They do not affect existing bindings.
There is no depth camera component to add. The engine's gRPC schema puts it plainly: RGBD is a per-camera option, not a separate pipeline.
Depth ignores path tracing and lens models
Depth is always rendered from the rasterised scene with the camera's basic projection, whatever the binding's Render Mode and whatever the camera's Physical lens settings. The Physical camera's lens model affects only path-traced colour, so with a non-pinhole lens and Path Traced colour, the colour and depth images do not line up pixel for pixel. See Cameras and exposure.
Camera vertical FOV is in degrees, both in the Inspector and from script.
Enabling depth¶
- Stop the scene if it is playing, then open Edit → Settings and select the Recorder page. Enable MuJoCo Observer must be on for the page's tabs to appear.
- On the General tab, turn on Record Cameras, and leave Record Color Video below it on. Record Cameras is the master switch for all camera recording, and Record Color Video covers depth as well as colour: with either off, no camera records colour or depth.
- On the Video tab, under Camera Configuration, click Add Camera Binding and set Source Entity to the camera.
- Set the binding's Depth Mode.
| Depth Mode | What is written for this camera |
|---|---|
| Off | Colour only |
| RGBD | Colour and depth as two aligned files, plus a muxed .mkv holding both, written after each episode |
| Separate | Colour and depth as two aligned files |
| DepthOnly | Depth only, no colour |
A camera records depth when Record Cameras and Record Color Video are on, its binding is Enabled, and its Depth Mode is not Off. To capture depth without colour, set that camera's Depth Mode to DepthOnly.
Depth needs the external FFmpeg program
The editor can encode colour video on the GPU without any other program, but depth is
always encoded by the external FFmpeg program, whatever the Video Format. The muxed
file of the RGBD mode is made with it too. While a camera records depth and FFmpeg
is not installed, the Recorder page says so below Record Cameras, and the camera
shows Needs FFmpeg in the Observer panel. Install FFmpeg from Tools → FFmpeg
Settings, or add it to your system PATH. See
Video encoding and FFmpeg.
The Observer panel (View → Observer) lists each binding under Cameras, and so does the Recorder Status popup on the viewport toolbar. Each row shows the camera's status and has a switch, the same as the binding's Enabled field. Hovering a row shows the camera's resolution, frame rate, render mode and Configured streams. The switch works only in Edit mode with the recorder disarmed.
Depth Video Defaults¶
The Depth Video Defaults block on the Video tab seeds new bindings:
| Field | Default | What it does |
|---|---|---|
| Default Depth Mode | Off | Copied into a binding's Depth Mode when you click Add Camera Binding |
| Default Depth Scale | 0.001 |
Copied into a binding's Depth Scale (m/code) the same way. Range 0.00001 to 0.1 |
Changing a default later does not touch existing bindings. Each camera then owns its own mode and scale.
Upgrading from 2026.2
In 2026.2, Depth Mode was a single scene-wide setting and each camera had a Record
Depth checkbox. Scenes saved by older versions are converted when opened: the old
scene-wide mode becomes Default Depth Mode, and each camera gets that mode if its
Record Depth was on, or Off if it was off. A camera that recorded nothing under
the old combination of DepthOnly and Record Depth off becomes colour-only.
Scripts that call SetCameraConfig need updating, as described under
Reading depth configuration from C#.
Camera bindings¶
Each binding is a collapsible entry headed by its Output Name, or Camera #N while the
name is empty. The X button on the header removes it.
| Field | Default | What it does |
|---|---|---|
| Source Entity | The camera entity. The picker lists only entities with a Camera component | |
| Output Name | Entity name | Name used in output folders and dataset keys. Filled from the entity's name when empty |
| Width | 256 |
Recorded width in pixels |
| Height | 256 |
Recorded height in pixels |
| Frame Rate | 30 fps | Rate written into this camera's video files and metadata: 24, 25, 30, 50, 60, 100 or 120 fps |
| Enabled | on | Per-camera master. Off means this camera records nothing: no colour, depth or masks |
| Record Masks | on | Per-camera opt-out for the mask stream. Shown only while Record Masks under Masked Video on the same tab is on |
| Depth Mode | Default Depth Mode | Which streams this camera writes; see Enabling depth |
| Depth Scale (m/code) | Default Depth Scale | Metres per 16-bit code for this camera. Range 0.00001 to 0.1. Shown when Depth Mode is not Off |
| Depth Sensor Model | None | Clean depth, or a simulated sensor; see Depth sensor models. Shown when Depth Mode is not Off |
| Render Mode | Rasterized | Rasterized or Path Traced, for the colour stream only |
Frame Rate does not set the capture rate
Frame Rate is only stamped into the camera's video files and its info.json
entry, including the depth stream's. How often frames are captured is set by the runner
in MuJoCo Observer [Video] TimeRunner on the General tab. Set Frame Rate to the
rate frames are actually produced at, or video timestamps will not line up with the
data. In a new scene that runner ticks at 50 Hz while Frame Rate defaults to 30 fps, so
check both.
Pick a depth scale per camera
A close-range wrist camera wants a finer scale than a room camera, so its depth fills
the 16-bit range instead of the bottom fraction of it. A useful rule of thumb is
scale = max_depth_m / 65535.
Render Mode set to Path Traced records the camera's colour stream with the path tracer and adds the Denoiser, Quality, Bounces and Samples or Sub-frames fields. Depth and masks are unaffected and stay rasterised. The path-traced options are covered in Recording and Video encoding and FFmpeg.
Depth encoding¶
Depth is gray16le: single-channel, 16-bit, little-endian, 2 bytes per pixel, with a
row stride of width * 2. Colour frames returned over gRPC are RGBA8 at 4 bytes per pixel;
recorded colour video is a standard .mp4.
Each pixel is an unsigned code, converted to metres with:
The distance is depth along the camera's viewing direction (planar depth), not the length
of the ray through the pixel. At the default scale of 0.001 that is 1 mm per code, giving
a range of 0 to 65.535 m. Codes are rounded to the nearest value and clamped, so distances
beyond the scale's reach read as 65535 rather than wrapping.
What 0 means depends on the sensor model
The two depth paths use different conventions.
Under a sensor model (ActiveStereo, StructuredLight), 0 is a no-return
marker. It marks a pixel with no measurement: background with no surface, a distance
outside Min Range to Max Range, or a dropout on a dark, glossy, or grazing
surface. Treating it as a valid depth places phantom geometry at the camera's origin.
With None (clean depth) there is no dropout and no marker. Every pixel carries a
quantised distance, and empty background reads as the far-clip distance
(round(far / scale), clamped to 65535), never 0. Masking on code == 0 finds
nothing.
Neither convention matches lidar, which reports a missing beam as -1.
Depth video is written with FFV1, a lossless codec, in a .mkv file. Depth codes
survive encoding exactly, which is why depth is never written as H.264.
Depth sensor models¶
By default depth is a clean depth buffer: every pixel in range returns an exact distance. Real depth cameras do not behave that way. A sensor model reproduces their artefacts.
| Model | Behaviour |
|---|---|
| None | Clean depth. The default |
| ActiveStereo | RealSense-style active infrared stereo: noise that grows with the square of distance, dropout on dark and glossy surfaces, and occlusion shadows beside near objects |
| StructuredLight | Kinect v1 and TrueDepth style: coarser, blocky depth and more aggressive dropout |
| iToF | Experimental. Greyed out in the list, and records the ActiveStereo model |
| dToFSparse | Experimental. Greyed out in the list, and records the ActiveStereo model |
| All (disabled) | Temporarily disabled. Records a single ActiveStereo stream |
Treat None, ActiveStereo and StructuredLight as the usable set. The greyed-out entries can still be selected, but they produce active-stereo artefacts under a different name. On the per-step gRPC path, All gives clean depth.
When a model other than None is selected, five calibration fields appear:
| Field | Default | Range | What it does |
|---|---|---|---|
| Max Range (m) | 10.0 |
0.1 to 100 | Beyond this the sensor returns nothing. Raise it for large scenes |
| Min Range (m) | 0.1 |
0 to 5 | Near blind zone; closer returns are dropped |
| Detect Floor | 0.03 |
0 to 0.5 | Return strength below which a pixel drops out. Lower means fewer dropouts |
| Stereo Baseline (m) | 0.05 |
0.005 to 0.5 | Virtual stereo or projector baseline. Drives the distance-dependent noise and the width of occlusion shadows |
| Disparity Noise (px) | 0.08 |
0 to 1 | One-sigma disparity noise. Higher means grainier depth |
The baseline is what makes active-stereo depth degrade quadratically with distance, and what produces the occlusion shadow beside near objects.
The model and its calibration can be changed at runtime, not just in the editor: from
script with SetCameraConfig, or over gRPC. See
Setting the depth sensor model over gRPC.
Materials and the depth model¶
The depth sensor models read the same LiDAR (NIR) material channels as the lidar, and need no bake. Authoring a material for lidar improves depth at the same time. The channels and their one-click presets are covered under Authoring a material.
| Channel | Effect on depth |
|---|---|
| NIR reflectivity | Drives return strength directly. Unauthored materials fall back to the brightness of the visible albedo |
| Continuation | Glass and water reflect differently from opaque surfaces |
| NIR transmission | Suppresses the surface's own return, so a surface the beam passes through drops out |
The visible material still matters: roughness and metalness decide how glossy a surface is. A glossy metal returns only near its specular peak and drops out elsewhere, a Porous material such as foliage thins out, and a dark surface drops out in the same places lidar says it should.
Seeing through glass and water¶
With None (clean depth), transparent surfaces do not appear in the depth image at all: the camera reports the opaque surface behind a window as if the glass were not there.
Under a sensor model, the camera still reports the opaque surface behind a transparent material, which is what real hardware does, but not at full strength:
- Glass dims what is behind it. The pulse crosses the glass twice, in and back out, so the surface behind is weakened by the glass's NIR transmission both ways.
- Water absorbs over depth. For a Medium material, the Absorption (1/m) applies along the real distance between the water surface and the surface below, both ways. A pool floor fades out with depth and disappears once the water is deep enough. With the Still water preset that happens within roughly 15 cm.
- Unauthored glass blocks. A transparent surface lets the sensor through only when
its Continuation is Slab, Medium or Porous and its NIR transmission
is above
0. Otherwise it hides what is behind it. Both default to opaque, so glass that has not been authored for the sensor blocks it.
A real depth camera does not simply see past glass, though. Head-on, a window throws back a bright glint, and the camera locks onto that, reporting the pane's own range rather than the room behind it. Both surfaces compete for the pixel:
- Head-on, the pane's glint wins and the camera reports the glass.
- Off-normal, the glint fades quickly and the surface behind takes over. This is the angle-dependent flicker real hardware shows across a pane of glazing.
- If neither is strong enough, the pixel drops out. A dark or strongly absorbing pane therefore blocks rather than revealing what is behind it.
Diagnostic channels¶
Each depth camera has a Diagnostic channel on its binding, shown when the Depth
Sensor Model is not None. It replaces the recorded range with one internal term of
the sensor model, reported for every pixel, including the ones being dropped, which
are usually the ones worth looking at. It is also available as debug_channel over gRPC
and through SetCameraDepthDebugChannel in C#, and it applies to per-step gRPC depth
frames too.
| Value | Option | Reports |
|---|---|---|
0 |
Off (record depth) | Normal depth |
1 |
Backscatter B (x1000) | The return strength compared against Detect Floor |
2 |
Incidence cos (x1000) | The cosine of the incidence angle at the surface |
3 |
NIR reflectivity (x1000) | The reflectivity actually used, after the albedo fallback |
4 |
Transparent surface depth (m) | Depth of any transparent surface covering the pixel, in metres |
5 |
Metalness (x1000) | Surface metalness. Approaching 1 removes the matte part of the return |
6 |
Roughness (x1000) | Surface roughness. Sets how narrow the glint is |
7 |
Transmission suppression (x1000) | How much a transmissive surface suppresses its own return |
8 |
Continuation class (x100) | The surface class, from 0 (Opaque) to 4 (Specular) |
9 |
Stereo occlusion reject (x1000) | 1 where the stereo occlusion test rejected the pixel |
For the scaled channels the raw code is the value multiplied by 1000 or 100, so divide the
code rather than multiplying by depth_scale. Transparent surface depth (m) is decoded
with depth_scale like normal depth. Pixels with no surface, or outside Min Range to
Max Range, read 0 on every channel.
A diagnostic stream is not a depth map
While a channel is selected, the binding shows Diagnostic channel is ON - this camera
will not record usable depth. The setting is saved with the scene, and a recording
made with it announces itself: the stream's meta/info.json entry carries
debug_channel and "NOTE": "diagnostic capture - values are NOT depth". Set it back
to Off (record depth) for any real capture.
A diagnostic capture is how a dropout is traced to its cause. A metal sphere, for example, reads high metalness and low roughness, which is why it returns only at its specular peak.
Reading depth configuration from C#¶
The Camera component covers geometry only:
using Hazel;
public class CameraSetup : Entity
{
protected override void OnCreate()
{
var cam = GetComponent<CameraComponent>();
cam.SetPerspective(verticalFov: 60.0f, nearClip: 0.05f, farClip: 50.0f);
}
}
The RGBD configuration is on MujocoSceneComponent, keyed by the camera's entity id:
var scene = GetComponent<MujocoSceneComponent>();
var camera = Scene.FindEntityByTag("CameraFront");
var ints = new int[7];
var floats = new float[6];
if (camera != null && scene.GetCameraConfig(camera.ID, ints, floats))
{
int width = ints[0];
int depthMode = ints[3]; // 0 Off, 1 RGBD, 2 Separate, 3 DepthOnly
int sensorModel = ints[5];
float depthScale = floats[0];
}
| Member | What it does |
|---|---|
GetCameraConfig(cameraEntityId, outInts, outFloats) |
Read a camera's configuration. false when that camera has no binding, or before the recorder has armed (see below) |
SetCameraConfig(cameraEntityId, width, height, recordMask, depthMode, enabled, sensorModel, depthScale, maxRange, minRange, detectFloor, baseline, subpixelSigma) |
Rewrite a camera's configuration |
SetCameraDepthDebugChannel(cameraEntityId, channel) |
Set the diagnostic channel. 0 records depth as normal |
The arrays are positional:
outInts = [width, height, recordMask, depthMode, enabled, sensorModel, depthDebugChannel]
outFloats = [depthScale, maxRange, minRange, detectFloor, baseline, subpixelSigma]
Both arrays must have at least 6 elements, or the call returns false. Size the int array
for 7 to also read the diagnostic channel. depthMode is 0 Off, 1 RGBD, 2 Separate,
3 DepthOnly, and sensorModel is 0 None, 1 ActiveStereo, 2 iToF, 3
StructuredLight, 4 dToFSparse, 5 All.
SetCameraConfig and SetCameraDepthDebugChannel write their changes back to the running
scene's Recorder settings, so they survive the next recording session. Like other changes
made during Play, they are discarded when the scene stops. SetCameraConfig raises a
width or height below 1 to 1, and replaces a depth scale of 0 or less with 0.001.
When SetCameraConfig does nothing
It does nothing while recording, for a camera that has no binding, and when
depthMode or sensorModel is out of range. A camera entity alone is not enough; the
binding is what holds the configuration being written. The diagnostic channel is also
ignored while recording, because a stream's meaning is fixed when its file opens.
The recorder also loads the camera bindings only when it arms, with Arm Recorder in
the Observer panel or Observer.StartRecording(). Until it has armed once in the
current Play run, GetCameraConfig returns false and both setters change nothing,
even for a camera that has a binding.
Upgrading from 2026.2
SetCameraConfig now takes int depthMode in fifth position where it used to take
bool recordDepth, and index 3 of GetCameraConfig's int array is now the depth mode
rather than a true or false value. Update scripts that call either method.
The full member list is in the MujocoSceneComponent API reference.
Recorded output¶
Output follows a LeRobot-style layout, one folder per stream, with one file per episode:
DataSessions/session_<date>_<time>/videos/
├── observation.images.<Name>/ colour, chunk-NNN/file-NNN.mp4
├── observation.images.<Name>_depth/ depth, chunk-NNN/file-NNN.mkv (FFV1, gray16le)
└── observation.images.<Name>_rgbd/ colour + depth muxed, chunk-NNN/file-NNN.mkv
Colour is H.264, or HEVC with the in-process HEVC Video Format. In RGBD mode the separate colour and depth files are kept, and the muxed file is an extra copy made after the episode by copying both streams without re-encoding, so its tracks are identical to the separate files.
Each depth stream has its own entry in meta/info.json, keyed
observation.images.<Name>_depth. Besides the codec and frame rate, it records:
| Key | Meaning |
|---|---|
depth_scale |
Metres per code for this camera |
depth_dtype, depth_units |
uint16 and meters |
sensor_model |
The depth sensor model the stream was recorded with |
zero_means |
What a code of 0 means for this stream: no return under a sensor model |
sensor_params |
The five calibration values, under a sensor model |
debug_channel, NOTE |
Present only for a diagnostic capture |
The rest of the session layout is described in Dataset format.
Over gRPC¶
Two gRPC surfaces touch a camera: the per-step frame path, which can carry depth, and
CameraService, which configures cameras and streams colour.
Per step, as an observation. Each StepRequest.camera_requests entry names a camera
and a stream kind, and the frames come back on StepResponse.camera_frames, rendered at
that tick after physics. Colour and depth are separate entries: one camera producing
both streams needs two requests.
resp = agent.Step(pb.StepRequest(
actions=action,
camera_requests=[
pb.GetCameraFrameRequest(name="CameraFront", width=96, height=96,
kind=pb.CAMERA_STREAM_COLOR),
pb.GetCameraFrameRequest(name="CameraFront", width=96, height=96,
kind=pb.CAMERA_STREAM_DEPTH),
],
))
for cf in resp.camera_frames:
frame = cf.frame
if frame.depth_scale > 0.0:
codes = np.frombuffer(frame.data, dtype="<u2").reshape(frame.height, frame.width)
metres = codes * frame.depth_scale # under a sensor model, 0 is a no-return
GetCameraFrameRequest takes either an entity id or a name, plus width and height
and kind. A width or height of 0 uses the scene's current viewport size. The
format field is ignored: colour frames always come back as raw RGBA with 4 channels, and
depth frames as gray16 with 1 channel. ImageFrame returns data, width, height,
channels, format, timestamp_ms (wall-clock time), frame_number, and depth_scale,
which is 0 for colour frames.
Per-step depth uses the camera binding's Depth Sensor Model, its calibration and its Diagnostic channel, read from the same configuration the recorder uses. There is no per-request override, so a model is selected by configuring the camera, not by asking for it on a frame. A camera with no binding returns clean depth. The recorder loads the bindings when it arms, so until it has armed once in the current Play run, every camera returns clean depth.
Per-step depth is always 1 mm per code
Per-step depth frames are always encoded at 0.001 m per code, so 0 to 65.535 m, and
depth_scale reports 0.001. The binding's Depth Scale (m/code) applies to
recorded depth only. Always convert with the frame's own depth_scale.
A missing camera does not stop the step
Camera names resolve by entity tag, and the entity must have a Camera component. If a
requested camera cannot be captured, the step still runs and the response comes back
with success=false and the message Step observation is available, but one or more
requested camera captures are missing or invalid. The observation and any frames that
were captured are still returned, so check success before using the frames. Only a
request with neither a name nor a non-zero id is rejected, before the tick runs.
If a camera's binding has Render Mode set to Path Traced and its Depth Mode is not
DepthOnly, its live colour frames, from Step and from StreamCamera, are path traced
too, whether or not the binding is Enabled. They use the binding's samples and bounces
without a denoiser, so more samples mean more latency per frame. Depth frames are
unaffected.
Live configuration, via CameraService.
| RPC | Purpose |
|---|---|
ListCameras |
Discover cameras: id and name |
GetCameraConfig / SetCameraConfig |
Read and write a camera's configuration |
StreamCamera |
A continuous colour stream |
Depth comes per step, not from a stream
Depth over gRPC is available only through Step.camera_requests with
kind = CAMERA_STREAM_DEPTH, one frame per Step. The continuous
CameraService.StreamCamera RPC is colour only: StreamCameraRequest has no kind
field, and the server always sends raw RGBA. There is no standalone single-frame RPC.
Setting the depth sensor model over gRPC¶
A camera's depth settings live in its binding, and CameraService.SetCameraConfig sets
them. The request carries a camera id or name and a CameraConfig:
| Field | Meaning |
|---|---|
width, height |
Recorded resolution in pixels |
record_mask, enabled |
Per-camera mask opt-out and recording master switch |
depth_mode |
0 Off (colour only), 1 RGBD, 2 Separate, 3 DepthOnly. The way to choose a camera's streams |
record_depth |
Legacy, for older clients. Derived from depth_mode on read, and used on write only when depth_mode is absent |
sensor_model |
0 None (clean depth), 1 ActiveStereo, 3 StructuredLight. 2 iToF, 4 dToFSparse and 5 All record ActiveStereo |
debug_channel |
0 off. Anything else replaces the recorded range with a diagnostic channel. Applied only when sent |
depth_scale |
Metres per 16-bit code, for recorded depth |
max_range, min_range, detect_floor, baseline, subpixel_sigma |
Sensor-model calibration, used when sensor_model is not 0 |
When depth_mode is absent, record_depth=false sets the camera to Off, and
record_depth=true keeps the camera's current mode, or sets RGBD if it was Off.
SetCameraConfig replaces the whole configuration. Read the current one, change the field,
and write it back, rather than building a CameraConfig from scratch, which would shrink
the resolution to 1 x 1 and zero the ranges.
cam = rpc.CameraServiceStub(channel)
cfg = cam.GetCameraConfig(pb.GetCameraConfigRequest(name="CameraFront")).config
cfg.sensor_model = 1 # ActiveStereo
cfg.depth_mode = 2 # Separate
cam.SetCameraConfig(pb.SetCameraConfigRequest(name="CameraFront", config=cfg))
From then on, every Step depth frame from that camera carries the model's artefacts. The
change is written to the running scene's Recorder settings, so it is kept when a recording
session is armed, and is discarded when the scene stops.
The camera must already be bound
SetCameraConfig tunes a camera that already has a binding in the scene's Recorder
settings; it does not create one. It answers ok=false with one of these messages:
camera not found, camera has no binding in the scene's Recorder settings,
request carried no config, no MuJoCo scene in the active world, or an
invalid depth_mode or invalid sensor_model message listing the valid values.
While a recording is in progress, changes are ignored, although the reply is still
ok=true, so reconfigure before starting one.
Until the recorder has armed once in the current Play run, GetCameraConfig answers
found=false and SetCameraConfig answers camera has no binding in the scene's
Recorder settings, even for a bound camera.
See Sensors over gRPC for how this fits the wider API.
Where to go next¶
- Lidar is the other simulated sensor, and where the NIR material channels are authored.
- Recording covers the Observer panel and the rest of the Recorder settings.
- Cameras and exposure covers the Camera component's exposure and Physical camera settings.
- gRPC API covers the server, the step loop, and the action gate.