Video Encoding and FFmpeg¶

Camera bindings record colour as compressed video and depth as lossless video. This page explains the Video Format setting that picks the colour encoder, which recordings still need the separate FFmpeg program and how to install it, how frame rates work, and how to record a camera with the path tracer.

All the settings here are on the Recorder page: Settings → Scene → Recorder, Video tab. The page and its camera bindings are introduced in Recording.

In short

On Windows and Linux, new scenes encode colour video on the GPU with In-Process GPU (H.264 / Vulkan Video), which needs nothing else installed. The separate FFmpeg program is still needed for depth, masks, lidar video proxies, the two FFmpeg formats, GPUs that cannot encode video themselves, macOS, and playing videos in the editor. Tools → FFmpeg Settings can download it for you. Set each camera's Frame Rate to the rate of the video time runner.

Choosing a video format¶

Video Format applies to the colour stream of every camera binding in the scene.

Video Format	Encodes on	Needs the FFmpeg program	Codec
In-Process GPU (H.264 / Vulkan Video)	The GPU's hardware video encoder, inside Lucky Engine. The default for new scenes on Windows and Linux.	No	H.264
In-Process GPU (HEVC / Vulkan Video)	The GPU's hardware video encoder, inside Lucky Engine.	No	HEVC (H.265)
FFmpeg (Generic / CPU)	The CPU, through the FFmpeg program. The default for new scenes on macOS.	Yes	H.264
FFmpeg (NVIDIA NVENC)	The hardware encoder of an NVIDIA GPU, through the FFmpeg program.	Yes	H.264

Whichever you choose, colour video is written as an .mp4 file with 8-bit 4:2:0 colour (yuv420p). The dataset metadata records the codec that was actually used for each camera, as described in Dataset Format.

Some guidance on picking one:

In-Process GPU (H.264 / Vulkan Video) keeps encoding on the GPU and needs no extra software. H.264 is the most widely supported codec, so it is the safe choice for training pipelines.
In-Process GPU (HEVC / Vulkan Video) works the same way. Choose it only if the tools that read your datasets can decode HEVC.
FFmpeg (Generic / CPU) works on any machine that has the FFmpeg program, at the cost of CPU time. Use it where the in-process formats are not available.
FFmpeg (NVIDIA NVENC) moves the encoding work to an NVIDIA GPU, through the FFmpeg program.

Scenes saved by earlier versions

A scene keeps the Video Format it was saved with. Scenes saved before 2026.3 used FFmpeg (Generic / CPU), the default at the time, so they still need FFmpeg for colour video. Switch them to an In-Process GPU format to record colour without it.

In-process GPU encoding¶

The two In-Process GPU formats encode on the same GPU that renders the scene, with an encoder built into Lucky Engine and no separate program. They are available on Windows and Linux, but whether they work depends on the GPU and its driver supporting hardware video encoding through Vulkan:

A format this machine cannot use is greyed out in the Video Format list. Hovering it explains that the GPU or driver cannot encode that codec, and that a scene set to it records with FFmpeg (Generic) on this machine.
A scene set to a format this machine cannot use keeps that setting, so it records with it again on a machine that can. The Video tab warns In-process H.264 encoding isn't available on this device. Cameras will record with FFmpeg (Generic) instead. (or HEVC). This is what you see when a new scene is created on such a GPU.
Each session started on that machine then records colour with FFmpeg (Generic / CPU) and logs a warning that in-process encoding is not available. The FFmpeg program then needs to be installed.

On macOS the in-process formats are never available, and new scenes use FFmpeg (Generic / CPU).

When FFmpeg is required¶

FFmpeg is a separate program, and Lucky Engine does not include it. It is needed for:

The two FFmpeg video formats, and an In-Process GPU format that this machine cannot use, because colour then records with FFmpeg (Generic / CPU).
Depth from any camera, that is any Depth Mode other than Off. Depth is always written as lossless 16-bit .mkv video through FFmpeg, whatever the Video Format. This includes the combined file that RGBD writes.
Record Masks, which is experimental.
Record Video Proxy on a lidar binding, described in Lidar.
Playing recorded videos in the Video Player, which also needs FFprobe, the companion tool that comes with FFmpeg.

Only what the scene actually records counts: a disabled camera binding, or a recorder that is switched off, does not need FFmpeg. With an In-Process GPU format this machine can use, and no depth, masks or lidar video proxies, recording needs no FFmpeg at all.

When FFmpeg is missing¶

The recorder settings stay as they are. Instead, the editor says what needs FFmpeg:

On the Recorder page's General tab, a warning below the video recording settings names the streams, for example Recording depth needs the external FFmpeg program, which isn't installed. Install it from Tools > FFmpeg Settings.
The Cameras section of Recorder Status shows the same message with an Install FFmpeg button, and each camera that needs FFmpeg reads Needs FFmpeg.
When the colour video is what needs FFmpeg and this GPU can encode H.264 itself, both places also offer Use In-Process GPU (H.264). It switches the scene's Video Format to In-Process GPU (H.264 / Vulkan Video), after which only depth and masks, if the scene records them, still need FFmpeg.
The Log also warns, once per run, when you switch the camera recorder on or open a scene that needs FFmpeg.

If a session starts anyway, a camera stream that needs FFmpeg cannot open, and the session stops as described in Recording with the Observer.

Installing FFmpeg¶

From the editor¶

Open Tools → FFmpeg Settings. The dialog explains what FFmpeg is used for and shows External FFmpeg program (ffmpeg) and Video playback (ffprobe), each Available or Not found. When FFmpeg is found, it also shows the Version and the path of each tool.
If either tool is missing, click Download FFmpeg + FFprobe. The editor downloads both into its own cache folder and shows Downloading FFmpeg... while it works.
When the download finishes, both lines read Available. No restart is needed.

If the download fails, the dialog says Download failed. Please install FFmpeg manually and ensure it is in your PATH. and offers Retry Download.

The Install FFmpeg button in Recorder Status runs the same download, and so does the Download FFmpeg button in the Video Player.

Installing it yourself¶

You can also install FFmpeg, including FFprobe, with your system's usual tools and make sure both are on the system PATH. The editor looks for FFmpeg in this order:

The Lucky Engine installation folder.
The editor's download cache, used by Download FFmpeg + FFprobe.
The system PATH.
On macOS only: /opt/homebrew/bin, /usr/local/bin and /opt/local/bin, where Homebrew and MacPorts install it. Apps started from the Dock do not see the shell's PATH, so these folders are checked directly.

The editor checks for FFmpeg once per run. If you install it yourself while the editor is open, restart the editor so it is found.

Frame rate and capture rate¶

Two settings control timing, and they are independent:

The capture rate is how often camera frames are taken. It is the frequency of the MuJoCo Observer [Video] TimeRunner on the General tab, and it is the same for every camera.
Each binding's Frame Rate (24, 25, 30, 50, 60, 100 or 120 fps) is the rate written into that camera's video file and into its video.fps entry in the dataset metadata. It does not change how often frames are captured.

If a camera's Frame Rate differs from the capture rate, its video plays too fast or too slow, and its timestamps do not line up with the data.

Match each camera's Frame Rate to the video runner

The defaults do not match. New scenes capture video on the Robot [50.00Hz] runner, but a new binding's Frame Rate is 30 fps. Either set every camera to 50 fps, or choose the Capture [30.00Hz] runner as the MuJoCo Observer [Video] TimeRunner. If you use a runner of your own, give it one of the frequencies the Frame Rate list offers.

Time runners and their frequencies are explained in The Time Manager.

Path-traced camera recording¶

A camera binding can record its colour stream with the path tracer instead of the rasterizer. Set the binding's Render Mode to Path Traced and its extra fields appear.

Things to know before you use it:

Only the colour stream is path traced. Depth and masks never are: depth always comes from the rasterized scene.
It needs a GPU that can path trace, as described in Rendering. Path tracing is not available on macOS.
Every recorded frame takes several path-tracing passes (its Samples or Sub-frames), so path-traced recording is much slower than rasterized recording. The Speed row in the live analytics of Recorder Status shows how it compares with real time.
It never falls back silently. If the camera cannot be path traced, or the chosen denoiser cannot run, recording fails rather than writing a rasterized or undenoised image. If this happens as the session starts, the session is stopped straight away. If it happens during a session, the session stops and the episode in progress is marked as failed.

Denoiser¶

Denoiser decides how each recorded frame is cleaned up, and which per-frame setting applies:

Denoiser	How it works	Per-frame setting	`bit_reproducible`
None (reference)	The default. Accumulates samples into a clean frame. Slowest, highest quality.	Samples	`true`
ReBLUR	Fast temporal denoiser that reuses earlier frames. Can ghost on motion.	Sub-frames	`true`
ReLAX	Fast temporal denoiser that reuses earlier frames. Can ghost on motion.	Sub-frames	`true`
OIDN	Accumulates samples, then denoises each frame on the CPU. No ghosting, but slower.	Samples	`true`
NVIDIA DLSS Ray Reconstruction	NVIDIA's AI denoiser and upscaler. Temporal.	Sub-frames	`false`

NVIDIA DLSS Ray Reconstruction is greyed out, with the hover text Requires an NVIDIA RTX GPU, on machines that cannot run it. Its results can differ between GPUs, drivers and versions, which is why datasets recorded with it are flagged as not reproducible (see Reproducibility flags). The viewport's denoisers, which do not include OIDN, are described in Path Tracing.

Quality presets¶

Quality is a shortcut for three values. Picking a preset fills them in:

Quality	Samples	Bounces	Sub-frames
Draft	16	3	2
Medium (default)	64	4	4
High	256	6	4
Reference	1024	8	6

Editing any of the three values switches Quality to Custom.

Field	Range	Shown with	Description
Bounces	1 to 8	Every denoiser	Indirect light bounces. More gives better indirect lighting and is slower.
Samples	1 to 4096	None (reference), OIDN	Samples accumulated for each recorded frame. More is cleaner and slower.
Sub-frames	1 to 32	ReBLUR, ReLAX, NVIDIA DLSS Ray Reconstruction	Motion-smoothing steps within each recorded frame, not samples. About 2 looks as smooth as a 60 fps viewport, 4 as smooth as 120 fps. Raise it for fast motion, lower it for speed.

DLSS Mode¶

With NVIDIA DLSS Ray Reconstruction, DLSS Mode chooses the internal render resolution: Quality, Balanced, Performance, Ultra Performance or DLAA (the default for recording). DLAA renders at the camera's resolution. The other modes render at a lower resolution and upscale to the camera's size, which is faster at some cost in quality. For those modes, a read-only Render resolution row shows the approximate internal size, for example ~171 x 171 (upscaled to 256 x 256). Recording with DLSS stays non-reproducible in every mode.

Reproducibility flags¶

The dataset's meta/info.json file contains two flags about the colour streams of the session:

Field	Value
`neural_denoise`	`true` if any recorded colour camera used Path Traced with NVIDIA DLSS Ray Reconstruction, otherwise `false`.
`bit_reproducible`	`false` in that case, otherwise `true`.

Pipelines that need byte-identical re-runs of a scene can check bit_reproducible and skip datasets where it is false. The rest of the file is described in Dataset Format.

Live camera frames over gRPC¶

A camera binding's Render Mode also affects that camera's live colour frames over gRPC. When it is Path Traced (and Depth Mode is not DepthOnly), colour frames requested from that camera are path traced with the binding's Samples and Bounces values and no denoiser, even when the binding's Enabled is off. Depth frames are unaffected. See gRPC API.

Where to go next¶

Recording covers Recorder Status, the Recorder settings and camera bindings.
Dataset Format lists the video folders and metadata a session writes.
Viewing Files covers the Video Player for recorded videos.
RGBD and Depth covers depth encoding and depth sensor models.