Sensors¶

Lucky Engine simulates two robotics sensors beyond the plain rendered camera: lidar, which casts ranging beams against the MuJoCo scene, and RGBD, which adds metric depth to a camera. Both are written to the dataset the recorder writes to disk and can be read live over gRPC. Lidar scans can also be read live from C# script, while for cameras a script configures depth but does not read frames.

Both sensors are configured on the scene's recorder settings rather than in the Inspector. This page explains that model, compares the two sensors, and shows when each one is sampled during a simulation step.

In short

Neither sensor is a component. Both are bindings on the Recorder page of the Settings panel (Edit → Settings), saved with the scene. A lidar is added on the Lidar tab. Depth is chosen per camera with the binding's Depth Mode on the Video tab. Both expose their scripting API on MujocoSceneComponent, and both are readable over gRPC while a scene plays.

Sensors are bindings, not components¶

Adding a lidar is not an Add Component step, and there is no depth camera component to add.

A sensor binding is a configuration entry stored on the scene's recorder settings. Each binding names an entity that already exists, then describes how to sample it:

A lidar binding points at an entity carrying a MujocoBodyComponent or a MujocoSiteComponent. That entity supplies the sensor's pose. The binding supplies the beam pattern, field of view, range limits, mount offset, what the beams ignore, and whether secondary returns are captured.
A camera binding points at an entity carrying a CameraComponent. That component supplies pose and projection only. The binding supplies resolution, frame rate, the Depth Mode, depth scale, the depth sensor model, and the render mode used for recording.

The lidar's return model, which adds range noise and dropout, is not part of a binding. It is a single calibration for the whole scene, set from script or gRPC, and is covered under The return model.

graph LR
    E([Entity in the scene]):::ent --> B([Binding in Recorder settings]):::bind
    B --> R([Recorder: Parquet / video]):::out
    B --> S([C# via MujocoSceneComponent]):::out
    B --> G([gRPC: LidarService / CameraService / Step]):::out
    classDef ent fill:#2d3a4f,stroke:#4a5e7e,color:#ffffff;
    classDef bind fill:#4f2d2d,stroke:#7e4a4a,color:#ffffff;
    classDef out fill:#2d4f3a,stroke:#4a7e5e,color:#ffffff;

Where bindings live¶

Open Edit → Settings (or the gear button on the side bar) and select the Recorder page in the Scene group. The page shows its tabs only while Enable MuJoCo Observer is on. Lidar bindings are on the Lidar tab, and camera bindings are under Camera Configuration on the Video tab.

Capture settings can be changed only while the scene is in Edit mode and no recording session is armed. Otherwise the page is read-only and says why: Stop the scene before changing capture settings., or Disarm the recorder or stop the session before changing capture settings.

There are two shortcuts to the same place:

The Observer panel (View → Observer), and the Recorder Status popup on the viewport toolbar, list the bound cameras and lidars under Cameras and Lidar. The Edit button on each heading opens the Recorder page at the right tab, and the switch beside it turns that recorder on or off.
Placing a robot pack that ships sensors in a scene (see Adding a robot to a scene) adds its lidar and camera bindings for you, and turns on the matching recording switches. Installing the pack from the Content Vault alone does not.

What follows from the binding model¶

Bindings are saved with the scene. They live in the scene file, not on the entity.
Deleting a binding does not delete the entity, and the reverse. A camera entity with no binding still renders. It records nothing and has no depth settings of its own, although a gRPC Step request can still return clean depth from it. A binding whose entity was deleted shows Missing camera entity or Missing lidar entity in the Observer panel.
The scripting API is on MujocoSceneComponent, not on the camera or the body. The target sensor is passed as an argument: a 0-based index for lidar, in binding order, and an entity id for a camera.

The two sensors¶

Lidar

Ranging beams cast against the MuJoCo scene. Spinning and non-repetitive scan patterns, presets for common units, a calibrated return model with dropout and noise, material authoring for glass, water and mirrors, and secondary returns.
RGBD and depth

Metric depth alongside a camera's colour image. Four per-camera depth modes, per-camera depth scale, and depth sensor models that reproduce the artefacts of real depth cameras instead of a clean depth buffer.

Recorded lidar sessions are reviewed in the Lidar Replay panel, which plays them back in the viewport, shows per-beam metrics, and exports material-informed variants of a recording.

How they compare¶

	Lidar	RGBD
Configured in	Recorder → Lidar tab	Recorder → Video tab → Camera Configuration
Turned on by	Record Lidar on the General tab	Record Cameras and Record Color Video on the General tab, plus the camera's Depth Mode
Binds to	A MuJoCo body or site entity	A camera entity
Addressed by	0-based sensor index	Camera entity id (C#), entity id or name (gRPC)
Live value	`ranges[]`, one float per beam	`ImageFrame`, 16-bit depth pixels
No return is	`-1`	`0`, under a depth sensor model only
gRPC	`LidarService`, polled	`Step.camera_requests` for depth, `CameraService` for configuration
Recorded as	Parquet, one row per beam	Video, lossless FFV1 `.mkv`

The no-return conventions do not line up, which is a common source of bugs. Lidar reports a missing beam as -1. Depth reports a missing pixel as 0, but only under a sensor model: clean depth has no dropout at all and encodes empty background as the far-clip distance, never 0. Code that carries one convention across to the other reads a dropout as a hit at zero range.

When sensors run¶

Both sensors are sampled during the simulation step. By default they run in different phases:

Sampler	Default phase	Relative to physics
Camera recorder (colour, depth, masks)	Acquisition	Before
Lidar recorder	Validation	After
Per-step gRPC camera frames	Export	After

The recorders run on time runners chosen on the Recorder page's General tab: MuJoCo Observer [Video] TimeRunner for cameras and Lidar Recorder TimeRunner for lidar. Each selector also has an Execution Phase and an Internal Phase Priority, so the rate, the phase, and the order among the systems in that phase can all be changed. A lower priority runs earlier in the phase.

A lidar does not cast its whole pattern on every tick. Each tick casts the part of the pattern the sensor sweeps during that tick, so a live scan always holds the newest value for each beam. The details are under Scan timing.

A frame pulled per step over gRPC and a recorded dataset are separate mechanisms. The frame pulled per step is what a policy sees on that tick, while the recorder's output is a dataset on disk. The gRPC API page covers the step loop, and Recording covers the recorder.

Where to go next¶

Lidar covers scan patterns, presets, the return model, materials and bakes.
RGBD and depth covers depth modes, encoding, and depth sensor models.
Recording covers the Observer panel and the Recorder settings.
The Time Manager explains phases and runner rates.