gRPC API¶

Lucky Engine runs an in-process gRPC server. It is the main programmatic interface to the engine: an external process steps the simulation, reads observations and camera frames, sends actions and policy commands, and resets episodes over gRPC. The interface is language agnostic, so a client can be written in Python, C++, Go, Rust, or anything else with a gRPC stack.

This page covers starting the server, connecting a client, the External Action Gate that keeps the simulation in lock-step with the client, and each part of the API. Python also has a convenience package, covered at the end.

In short

Open the gRPC Server panel, click Start Server (this also turns on the External Action Gate for the open scene) and press Play. The server listens on 127.0.0.1:50051 without TLS. A scene script registers an external agent with RobotManager; the client then calls GetAgentSchema and ResetAgent, and Step on repeat. Each Step advances exactly one tick, so pace the loop by step count or by the simulation clock, never by a wall clock.

Starting the server¶

Open the gRPC Server panel in one of these ways:
- Click the gRPC Server button (server icon) in the title bar.
- Choose the Server layout in the side bar. It shows the gRPC Server panel with the Scene viewport and the Log.
- Choose View → gRPC Server. The View menu lists panels once a project is open.
Check Address and Port under Connection Settings. The defaults are 127.0.0.1 and 50051.
Click Start Server. This also turns on the External Action Gate for the open scene (see Enabling it).
Press Play. Until the scene plays, the panel reminds you that the client cannot exchange messages yet.

The server belongs to the editor session, not to a scene: it keeps running when you open another scene. With Persist across play checked (the default), it also stays up when you stop Play. Stopping Play restarts it briefly on the same port, so a client can see a short disconnection. Clear the checkbox to have the server stop with the scene instead.

To start the editor with the server already running, launch it with --server. The server starts once the project and its startup scene have loaded; the option does not press Play. See Command Line.

The gRPC Server panel¶

Section	What it shows
gRPC Status	Action Gate: Enabled (Waiting) while the gate is holding the simulation for a client, Enabled (Idle), or Disabled. Server: Running or Stopped. Scene: Playing or Stopped.
Connection Settings	Address and Port (1–65535). Both are locked while the server runs; stop it to change them.
Persist across play	Keeps the server running across Play and Stop. On by default.
Action Gate Timeouts	Timeout (s) and Warn @ (s), shown when the open scene has the gate enabled. They edit the same values as the scene settings described in Gate settings.
Start Server / Stop Server	Starts or stops the server.
Services	One checkbox per service group: Agent, Scene, MuJoCo, Telemetry, Camera, Viewport. All are on by default and can only be changed while the server is stopped. See Services.

The panel also shows messages while a client works with the scene:

External Action Gate is DISABLED in Scene Settings. with a Fix: Enable Action Gate button, when the open scene has the gate off.
Agent not ready: followed by the reason, when a client has connected while the scene plays but no external agent is ready (see Registering an agent).
Waiting for client action... with a timer, when the gate has waited longer than the warning threshold.
Client disconnected — watchdog deactivated the action gate..., after the watchdog has stopped the scene (see How it behaves).
Evaluation Progress and Evaluation Results, when the client reports its progress (see Reporting progress).

The help button in the top-right corner of the panel opens this page.

Network access¶

The server has no TLS and no authentication. With the default address 127.0.0.1 it accepts connections from the same computer only. To train from another machine, set Address to 0.0.0.0 before starting the server, which accepts connections on every network interface.

Only open the server on a trusted network

With Address set to 0.0.0.0, anyone who can reach the port can call every enabled service, unencrypted. Do not expose the port to the internet.

Connecting a client¶

There are two ways to build a client: generate stubs from the .proto files, or discover the API at runtime with reflection.

Generate stubs from the protos¶

The API schema ships with the editor. In the install folder on Windows and Linux, the files are in Data/Resources/Scripts/Grpc/Proto/; on macOS they are inside the app bundle, under Contents/Resources/Data/Resources/Scripts/Grpc/Proto/. There are three files, all in the hazel.rpc package: hazel_rpc.proto, mujoco_scene.proto and task_contract.proto. Generate them together. A short README.md quickstart sits one folder up, in Scripts/Grpc/; run the commands below from that folder.

pip install grpcio grpcio-tools
python -m grpc_tools.protoc -I Proto --python_out=. --grpc_python_out=. Proto/*.proto

import grpc, hazel_rpc_pb2 as pb, hazel_rpc_pb2_grpc as rpc

channel = grpc.insecure_channel("127.0.0.1:50051")
scene = rpc.SceneServiceStub(channel)
print(scene.GetSceneInfo(pb.GetSceneInfoRequest()))

Other languages use the matching protoc plugin (--cpp_out, --go_out, and so on) with the same -I Proto Proto/*.proto arguments.

Regenerate stubs after upgrading

Some fields are optional, so the server can tell "not sent" from "sent as 0". Stubs generated from an earlier release's protos cannot set or clear the newer ones. Regenerate your stubs from the files that ship with the editor you run.

Explore with reflection¶

The server has gRPC reflection enabled, so tools can discover the API without the proto files:

grpcurl -plaintext 127.0.0.1:50051 list
grpcurl -plaintext 127.0.0.1:50051 describe hazel.rpc.AgentService

grpcui and Postman work the same way. Reflection lists only the services that were enabled when the server started.

Services¶

All services are in the hazel.rpc package. The Panel checkbox column shows which Services checkbox in the gRPC Server panel enables each one.

Service	Panel checkbox	Purpose
`AgentService`	Agent	The step loop (`Step`, `ResetAgent`, `GetAgentSchema`), progress reports, policy and motion-graph control, and task contracts
`SceneService`	Scene	Scene info, entities and transforms, play mode, and simulation mode
`MujocoService`	MuJoCo	Agent-scoped joint state and model info, with a streaming variant
`MujocoSceneService`	Always on	Full MuJoCo model and state: model info, full state, `SetControl`, `SetQpos`, `ResetScene`, actuator gains, applied forces, and body, site and geom poses
`TelemetryService`	Telemetry	A best-effort stream of `qpos` and the last applied actions
`CameraService`	Camera	List cameras, stream camera frames, and read or write a camera's recording and depth configuration
`LidarService`	Camera	Lidar scans, beam counts and angles, scan and return-model configuration, and material bakes
`ViewportService`	Viewport	Stream frames from the scene's primary camera
`DebugService`	Always on	Draw debug primitives in the viewport

Server reflection (grpc.reflection.v1alpha.ServerReflection) is registered as well.

The Camera checkbox only affects CameraService and LidarService. Camera frames that you request inside Step work either way.

The action gate¶

The External Action Gate keeps the simulation in lock-step with the client. Once the gate is armed, each tick of the gate's time runner waits until the client sends a Step, so one Step advances exactly one tick. The gate is what makes episodes reproducible regardless of how fast or slow the client is.

Registering an agent¶

Step, ResetAgent and GetAgentSchema drive an external agent that a scene script registers. A C# script registers one by constructing a RobotManager whose agents are RobotAgent subclasses created without a policyPath, then calling Setup with the robot's MujocoSceneComponent. Until such an agent is registered and ready:

Step fails with "Step requires a registered external agent (none is active in this scene)."
ResetAgent returns success=false with "External agents not ready yet:" and the reason.
GetAgentSchema returns observation_size=0 and action_size=0.

The reason is also written to the Log, and the gRPC Server panel shows it in an Agent not ready: message once a client has connected while the scene plays. There is no agent-free Step. A tick-only Step (empty actions) also needs the registered agent; to drive actuators without an agent, use MujocoSceneService.SetControl instead.

Enabling it¶

The gate is a per-scene setting, and new scenes start with it off. It is turned on in any of these ways:

Start Server turns it on for the open scene, and for any scene you open while the server runs. Stop Server turns it off again, but only where the server turned it on; a gate you enabled yourself stays on. Launching with --server does the same as Start Server.
In the gRPC Server panel, Fix: Enable Action Gate appears while the gate is off.
In Settings (Edit → Settings), on the Scene → Subsystems page, tick Enable External Action Gate (gRPC Sync) under External Action Gate (gRPC Sync) Configuration.

The setting takes effect the next time the scene enters Play. All three routes change the scene in memory only; save the scene (File → Save, Ctrl+S) to keep the setting. There is no RPC to enable the gate: a client cannot turn it on.

Keep the gate on while a client drives an agent

With the gate off, the simulation runs freely and never waits for the client, and the external agent does not receive the actions sent with Step. One Step no longer maps to one tick, so episodes are not reproducible.

Gate settings¶

These settings appear on the Scene → Subsystems settings page once the gate is enabled. They are saved with the scene.

Setting	Default	Description
External Action Gate TimeRunner	Robot (50 Hz)	The time runner the gate holds. One `Step` advances one tick of this runner.
Watchdog Timeout (s)	10	Seconds the gate waits for a `Step` before it gives up and the editor stops the scene. 1–600.
Warning Threshold (s)	3	Seconds without a `Step` before the gRPC Server panel shows the waiting message. 0–60.
Reset Settle Steps	20	Ticks the robot holds its default pose after each reset. 1–1000, applied at the next reset.

Timeout (s) and Warn @ (s) in the gRPC Server panel edit the watchdog timeout and the warning threshold too. For time runners in general, see Time Manager.

How it behaves¶

Aspect	Behaviour
Arming	The gate arms only once a client has connected (see below), so an enabled gate never freezes an editor that has no client. After Play, it arms at the end of the first settle period if a client has connected by then; otherwise it arms when the client calls `ResetAgent`. Call `ResetAgent` before the first `Step`.
Lock-step	Once armed, every tick waits for the client. Each `Step` releases exactly one tick. A `Step` with `actions` (or `action_groups`) imposes that action for the tick. A tick-only `Step` (no `actions` and no `action_groups`) releases one tick without imposing an action: the in-engine policy, or an earlier `SetControl`, drives the actuators that tick.
Settle period	After every reset, and when Play starts, the robot holds its default joint positions for Reset Settle Steps ticks while it settles. While a client is driving, the gate stays armed during the settle, so each settle tick still takes one `Step`, and the action in that `Step` is ignored. The first `Step` after `ResetAgent` returns the observation of the reset pose. When Play starts with no client connected, the settle runs on its own.
Watchdog	If the armed gate waits longer than Watchdog Timeout (s) for a `Step`, the gate releases and the editor stops the scene.
Finishing	`ReportProgress` with `finished=true` releases the gate and keeps it released, through later resets and settle periods, so the scene runs on its own. A later `Step` or `ResetAgent` makes the client active again; call `ResetAgent` to arm the gate straight away. Send `finished=true` when a run ends instead of letting the watchdog stop the scene.
Stopping	The gate also releases when the scene stops and when the server stops.

The gate governs whether the simulation waits. It does not change how fast simulation time advances; for that, see Scene and simulation mode.

Observe-only clients also arm the gate

A client counts as connected as soon as it makes most AgentService, MujocoService or MujocoSceneService calls, or SceneService.EnterPlayMode / ExitPlayMode. That includes read-only calls such as MujocoSceneService.GetFullState. The other SceneService calls, ReportProgress, and the CameraService, LidarService, ViewportService, TelemetryService and DebugService calls do not count.

In a scene with an external agent, a connected client that never sends Step freezes the simulation once the gate arms, until the watchdog stops the scene. A client that only watches the simulation should send ReportProgress(finished=true) each time the scene starts playing, or use only the calls that do not count.

The simulation loop¶

Actions in, one tick, observation out: the whole loop is AgentService.Step, and the action gate is what makes each Step advance exactly one tick. Setup is three calls. GetAgentSchema reports the observation and action sizes, SetSimulationMode takes the simulation off real time (see Scene and simulation mode), and ResetAgent starts a fresh episode and arms the gate. GetAgentSchema, ResetAgent and Step need a registered external agent; SetSimulationMode does not.

After entering Play, poll GetAgentSchema until action_size is non-zero before driving the agent (see Scene and simulation mode).

agent = rpc.AgentServiceStub(channel)
scene = rpc.SceneServiceStub(channel)

# Needs a registered external agent in the scene and the External Action Gate
# enabled (Start Server turns it on).
scene.SetSimulationMode(pb.SetSimulationModeRequest(
    mode=pb.SIMULATION_MODE_DETERMINISTIC_REALTIME))   # reproducible; use _HIGH_PERF to train faster
schema = agent.GetAgentSchema(pb.GetAgentSchemaRequest()).schema   # observation_size, action_size
agent.ResetAgent(pb.ResetAgentRequest())   # new episode; the next ~20 Steps hold the default pose

for _ in range(1000):
    # Each Step releases exactly one gated tick. This one imposes an action; a Step with
    # empty actions would release a tick without imposing one (a "tick-only" Step).
    resp = agent.Step(pb.StepRequest(actions=[0.0] * schema.action_size))
    if not resp.success:
        raise RuntimeError(resp.message)
    observation = resp.observation.observations      # repeated float
    if resp.terminated or resp.truncated:
        agent.ResetAgent(pb.ResetAgentRequest())

agent.ReportProgress(pb.ProgressReport(finished=True))   # releases the gate cleanly

StepRequest fields:

Field	Description
`agent_name`	The agent to step. Empty means the default agent, `agent_0`. Agents are named `agent_0`, `agent_1`, and so on. An unknown name falls back to `agent_0` in `Step`; `ResetAgent` instead fails with "Agent '' not found."
`actions`	The action vector, sized to `action_size`. Leave it empty for a tick-only `Step`.
`timeout_s`	How long to wait for the tick. 0 or unset means 5 seconds. On timeout, `Step` fails with "Timeout waiting for physics step. Is the simulation running?"
`camera_requests`	Camera frames to capture after the tick. See Camera frames.
`action_groups`	Named groups of actions, each with the action-vector indices it writes, for setups where several controllers drive different joints. Groups apply on top of `actions`, or on top of the default joint positions when `actions` is empty. `SetActionGroup` preloads a group before the next `Step`.

StepResponse fields:

Field	Description
`success`, `message`	`false` with a reason when the step failed. Also `false` when the tick ran but a requested camera frame is missing; the observation is still valid then.
`observation`	An `AgentFrame`: `observations`, `actions`, `frame_number` and `timestamp_ms`.
`physics_step_duration_us`	How long the server waited for the tick, in microseconds of wall-clock time.
`camera_frames`	The requested camera frames.
`reward_signals`, `terminated`, `truncated`, `termination_flags`	Filled once a task contract is active (see Defining a task).
`info`	Defined in the schema but not filled in this release.

Let the engine keep time¶

One Step advances the simulation by exactly one tick of the gate's time runner, a fixed time step the engine owns. SceneService.GetSceneInfo returns that runner's rate as control_hz: one gated Step is exactly 1 / control_hz seconds of simulation time, and 0 means the rate is unavailable. Use it to convert step counts to simulation seconds instead of hard-coding a rate.

The wall-clock duration of a Step does not matter. In the deterministic simulation modes, a Step advances exactly one tick no matter how long it takes to return, whether a microsecond or an hour. An inverse-kinematics solve, the number of calls per tick, and host jitter make no difference. The engine simulates, observes and records in simulation time. A slow client only makes the run take longer on the wall clock; the gate never lets the simulation run ahead of the client's Steps.

So an artefact in a recording (jitter, stutter, uneven motion) comes from what is commanded in simulation time, not how fast it is sent. Per-tick command cadence, the recorder's frame rate relative to the control rate, and the time per Step are all simulation-time quantities.

Never pace the loop with a wall clock

Do not drive the control loop with time.sleep, time.monotonic or any other real-world clock, and do not hard-code a control rate (1 / hz, steps = seconds * hz). Under the action gate the simulation only advances on a Step, so a wall-clock wait just stalls the client while the simulation sits frozen. In SIMULATION_MODE_DETERMINISTIC_HIGH_PERF, simulation time and real time are unrelated entirely. Pacing off the wall clock makes the number of Steps per phase depend on host speed and jitter, which silently breaks reproducibility.

Express every duration in the engine's own time instead:

Count steps. One Step is one tick. Advance a phase by stepping a fixed number of times, converting seconds with control_hz where needed.
Read the simulation clock. MujocoSceneService.GetFullState().state.time is the MuJoCo simulation clock. To run a phase for N seconds of simulation time, Step until that clock has advanced by N.

# Advance ~2 s of SIMULATION time using the engine clock only (no wall clock).
def sim_time():
    return scene_mj.GetFullState(mj.GetFullStateRequest()).state.time

end = sim_time() + 2.0
while sim_time() < end:
    agent.Step(pb.StepRequest(actions=action))   # one tick

Not Python-specific

The loop above calls the generated stubs directly. It is the raw gRPC API, not the luckyrobots package, which is optional (see Python helper). The call sequence and the StepRequest / StepResponse fields are the same in every language. Generate stubs for Go, C++ or Rust as shown in Connecting a client, then call GetAgentSchema, SetSimulationMode, ResetAgent and Step the same way. A single step can be sent from the shell to confirm the wiring:

grpcurl -plaintext -d '{"actions": [0, 0, 0]}' \
    127.0.0.1:50051 hazel.rpc.AgentService/Step

Reporting progress¶

AgentService.ReportProgress lets a client show how a run is going in the gRPC Server panel. Each ProgressReport replaces the previous one. While finished is false, the panel shows Evaluation Progress with the task, phase, episode and step progress bars, elapsed time and status. When finished is true, it shows Evaluation Results until you click Dismiss.

Field	Shown as
`task_name`, `policy_name`	Task, with the policy name in brackets
`phase`	Phase
`current_episode`, `total_episodes`	The Episode progress bar (Episodes in the results)
`current_step`, `max_steps`	The Step progress bar
`elapsed_s`	Elapsed (Time in the results)
`status_text`	Status (Results in the results)
`finished`	Switches to the results view and releases the action gate

run_id is stored but not shown. A report with finished=true also releases the action gate, as described under How it behaves.

agent.ReportProgress(pb.ProgressReport(
    task_name="pick_place", policy_name="ACT", phase="running",
    current_episode=3, total_episodes=10, current_step=250, max_steps=600,
    status_text="Episode 3/10"))

Camera frames¶

A StepRequest can ask for camera frames, which are rendered after the tick and returned in the same StepResponse, in lock-step with physics.

import numpy as np

resp = agent.Step(pb.StepRequest(
    actions=action,
    camera_requests=[
        pb.GetCameraFrameRequest(name="TripodCamera", width=96, height=96),
        pb.GetCameraFrameRequest(name="TripodCamera", width=96, height=96,
                                 kind=pb.CAMERA_STREAM_DEPTH),
    ]))

for cf in resp.camera_frames:
    f = cf.frame
    if f.format == "raw":     # colour: RGBA, 4 bytes per pixel
        rgba = np.frombuffer(f.data, np.uint8).reshape(f.height, f.width, 4)
    else:                     # "gray16": depth, 16-bit little-endian codes
        depth_m = np.frombuffer(f.data, "<u2").reshape(f.height, f.width) * f.depth_scale

A request names a camera by name or by entity id. Names match the entity's tag, so give each camera a unique name. The entity must have a Camera component.
width and height set the image size; 0 uses the scene's current viewport size.
kind is CAMERA_STREAM_COLOR (the default) or CAMERA_STREAM_DEPTH. One camera that should produce both needs two requests.
Each returned frame carries the requested name (for a request by id, the ID as a decimal string), so match frames by name and format.
Colour frames are RGBA with format "raw" and channels 4. Depth frames are described under Sensors over gRPC.
A request with neither a name nor a non-zero id is rejected before the tick runs. If a camera cannot be captured, for example because no entity with that name has a Camera component, the tick still runs and the observation is returned, but that frame is missing, success is false, and message is "Step observation is available, but one or more requested camera captures are missing or invalid." The message does not name the camera. Check success on every Step.

CameraService.StreamCamera streams frames from one camera continuously, and ViewportService.StreamViewport streams the scene's primary camera (or the first camera, if none is primary) as a single viewport named Main. Both return RGBA frames with format "raw", whatever format the request asks for.

Path-traced camera frames¶

Camera frames can be path traced. Set it per camera in the scene's Recorder settings: in the camera's binding, under Camera Configuration on the Video tab of the Scene → Recorder settings page, set Render Mode to Path Traced. From then on, colour frames from that camera are path traced in Step camera requests, StreamCamera and, for the primary camera, StreamViewport. See Recording for camera bindings.

Each frame is rendered at the requested size with the binding's Samples and Bounces, which the binding's Quality preset also sets. The binding's Denoiser, DLSS Mode and Sub-frames do not apply, and no history is carried from one frame to the next.
More samples give a cleaner image and a proportionally slower Step.
The binding's recording Enabled checkbox does not matter here.
Depth frames, and colour frames from a camera whose Depth Mode is DepthOnly, are always rasterized. Cameras without a binding render rasterized.
Path tracing needs a GPU with hardware ray tracing and is not available on macOS. Where it is unavailable, the frame is not captured: Step returns success=false and the frame is missing, rather than a rasterized image. See Rendering.

The Samples field is shown when the binding's Denoiser is None (reference) or OIDN. With another denoiser, the Quality preset sets the sample count that camera frames use.

Recording vs. observation frames¶

Two different things produce camera images, and mixing them up is a common mistake.

Step camera frames are observations. A StepRequest.camera_requests entry renders a fresh frame from a named scene camera at that exact tick and returns it in StepResponse.camera_frames, in lock-step with physics. It is the image a policy sees on that step, suitable as a network input. It is not a recording mechanism; do not stitch these per-step frames into a video.
The Observer is the recorder. The engine-side Observer writes a structured dataset (state, actions and camera video) to disk. It runs on its own time runners at its own rate, set on the General tab of the scene's Recorder settings, fully decoupled from the control and Step rate. Record at, say, 30 Hz while stepping the control loop at any rate, so any run can be recorded without changing how it is stepped. Drive it from a C# script with Observer.StartRecording(), Observer.EndCurrentEpisode(success) and Observer.StopRecording(). See Recording for the editor setup.

Sensors over gRPC¶

Beyond colour frames, a client can read simulated lidar and metric depth. The two reach the client by different routes, and their settings live in bindings in the scene's Recorder settings, which a client can change but not create. Lidar needs a lidar binding. A depth frame uses the sensor settings of its camera's binding, and is clean depth when the camera has no binding or the recorder has not armed yet (see Depth). The Sensors section covers that model, the editor setup, and every field.

	Lidar	RGBD depth
Route	Polled with `LidarService.GetLidarScan`	Per `Step`, in `StepResponse.camera_frames`
Configured with	`LidarService`	`CameraService.GetCameraConfig` / `SetCameraConfig`
Addressed by	0-based `sensor` index	Camera entity `id` or `name`
Payload	`LidarScanResponse.ranges[]`, metres	`ImageFrame` with `format` `"gray16"`
No return is	`-1`	`0`, under a sensor model only
Streaming	Not streamed	Not streamed; `StreamCamera` is colour only

Depth¶

Depth rides on the step loop. Request it with a camera_requests entry whose kind is CAMERA_STREAM_DEPTH. A depth frame has one channel of 16-bit little-endian codes; convert with metres = code * ImageFrame.depth_scale. depth_scale is 0 on colour frames. On Step depth frames it is always 0.001 (1 mm per code, up to about 65.5 m), whatever Depth Scale (m/code) the camera's binding uses for recording. The binding's depth sensor model and diagnostic channel do apply, but only once the recorder has armed in the current Play run (with Arm Recorder in the Observer panel, or a script calling Observer.StartRecording()), because that is when it loads the bindings. Until then, every depth frame is clean depth. See RGBD and Depth for sensor models, and Diagnostic channels for what a diagnostic channel puts in place of depth.

CameraService.GetCameraConfig and SetCameraConfig read and write a bound camera's recording and depth settings, including these depth fields:

Field	Values
`depth_mode`	`0` Off (colour only), `1` RGBD, `2` Separate, `3` DepthOnly. The same choices as the binding's Depth Mode.
`debug_channel`	`0` off; `1`–`9` replace depth with a diagnostic term, in the order of the binding's Diagnostic channel list. Such a stream is not a depth map.
`record_depth`	Legacy. Read as `depth_mode != 0`. On write it is used only when `depth_mode` is not sent: `false` turns depth off, `true` keeps the camera's current depth mode (RGBD if it was off).

SetCameraConfig replaces every setting it carries, so read the configuration with GetCameraConfig, change what you need, and write it back. debug_channel and depth_mode are applied only when sent. The camera must already have a binding; otherwise the call fails with "camera has no binding in the scene's Recorder settings". The same happens, and GetCameraConfig returns found=false, for a bound camera until the recorder has armed once in the current Play run. An out-of-range depth_mode or sensor_model is rejected. Changes are ignored while a recording is in progress (the call still reports ok). Otherwise they are written to the running scene's Recorder settings, so later recording sessions use them, and they are discarded when Play stops.

Lidar¶

Lidar is polled, not stepped. Scans do not ride along on StepResponse. A client calls LidarService.GetLidarScan when it wants one, after SetLidarLive(true) so scans are produced outside recording. The scene also needs lidar recording set up; see Lidar.

lidar = rpc.LidarServiceStub(channel)
lidar.SetLidarLive(pb.SetLidarLiveRequest(live=True))

scan = lidar.GetLidarScan(pb.GetLidarScanRequest(sensor=0))
ranges = scan.ranges          # metres per beam; -1.0 means no return

RPC	Purpose
`GetLidarScan`	A scan: `ranges[]`, plus `secondary_ranges[]` with `want_secondary`, and `intensity[]` and `reflectivity_calibrated[]` with `want_intensity` (meaningful with `material=true`; `-1` where not resolvable)
`GetLidarBeamCount`, `GetLidarBeamAngles`	The beam count, and per-beam azimuth and elevation in radians
`SetLidarLive`	Fire the lidar every step, even when not recording
`SetLidarConfig`, `GetLidarConfig`	Write and read the scan configuration. `GetLidarConfig` also reports `secondary_capture` and `secondary_scope`.
`SetLidarReturnParams`, `GetLidarReturnParams`	Write and read the return model. Optional fields left out of `SetLidarReturnParams` keep their current values; the nine base fields are always written, so read first and change only what you need.
`SetLidarPreset`	Apply a return model preset by index
`SetLidarSecondaryCapture`	Choose which extra rays a recording captures. Ignored while recording.
`BakeLidarMaterial`, `GetLidarBakeStatus`	Bake the scene's materials for material-aware scans, and check the loaded bake, including whether it is stale
`SaveLidarBake`, `LoadLidarBake`	Save a bake to disk and load it again

For scan patterns, presets, the return model, material bakes and secondary returns, see Lidar.

Reading and writing MuJoCo state¶

MujocoSceneService exposes the full MuJoCo model and state. GetModelInfo returns the model dimensions and the joint and actuator descriptors. GetFullState returns qpos, qvel and ctrl (with optional filtering), and StreamFullState streams them. SetControl writes actuator targets, SetQpos writes positions, and ResetScene snaps the model back to its initial state (its first keyframe, if the model has one).

import mujoco_scene_pb2 as mj, mujoco_scene_pb2_grpc as mjrpc

scene_mj = mjrpc.MujocoSceneServiceStub(channel)
info = scene_mj.GetModelInfo(mj.GetModelInfoRequest())     # nq, nv, nu, joints, actuators
scene_mj.SetControl(mj.SetControlRequest(bulk=[0.0] * info.nu))

SetControl applies engine-wide but rejects actuators owned by an active policy slot or external agent, so direct writes do not fight a running policy.

Scene and simulation mode¶

SceneService controls play mode and the simulation rate. EnterPlayMode and ExitPlayMode start and stop Play from a client. Both return as soon as the request is sent; the change itself happens shortly after. SetSimulationMode selects how time advances. It works whether or not an agent is registered, and changes the scene's Scene Time Mode setting (Scene → Time Runners in Settings).

`SimulationMode`	Value	Scene Time Mode	Behaviour
`SIMULATION_MODE_NONDETERMINISTIC`	0	Game Realtime (Non-Deterministic)	Tracks the wall clock, dropping or repeating frames to keep pace. Not deterministic; not for training.
`SIMULATION_MODE_DETERMINISTIC_REALTIME`	1	Sim Realtime (Deterministic Capped)	Fixed-step and reproducible, capped at real-time speed. The default for new scenes; for watchable runs.
`SIMULATION_MODE_DETERMINISTIC_HIGH_PERF`	2	Sim High Perf. (Deterministic Uncapped)	Fixed-step and reproducible, as fast as the hardware allows. Best for training.

Set the simulation mode for training

New scenes use SIMULATION_MODE_DETERMINISTIC_REALTIME, which is reproducible but capped at real-time speed. For fast training, call SetSimulationMode with SIMULATION_MODE_DETERMINISTIC_HIGH_PERF before stepping. Never select SIMULATION_MODE_NONDETERMINISTIC: it ties stepping to the wall clock and produces irreproducible episodes even with the action gate enabled.

Entering Play rebuilds the MuJoCo model, so a brief gap before the scene is ready is normal. Poll GetAgentSchema or GetModelInfo until it succeeds before driving the agent.

Driving policies¶

The policy and motion-graph surface of RobotControllerComponent is exposed on AgentService: SetPolicyActive, SetPolicyCommandFloat and SetPolicyCommandBool, SetPolicyDrivenJoints, SetPolicyPriority, and the motion-graph calls SetMotionGraphActive, SetMotionGraphInput and FireMotionGraphTrigger. ListRobotControllers and ListPolicyDescriptors discover what a scene exposes. This is the same model documented in Controlling robots, reached over gRPC.

Defining a task¶

For reinforcement learning, the engine computes observations, rewards and terminations from a negotiated task contract. Three calls drive it:

GetCapabilityManifest lists the observation, reward, termination and randomization components the engine knows about for a given robot and scene.
ValidateTaskContract checks a contract without applying it.
NegotiateTask validates and activates a contract. Once active, every Step returns the reward_signals, terminated, truncated and termination_flags for that task.

Custom reward, observation and termination components are defined in C# with the [MdpReward], [MdpObservation] and [MdpTermination] attributes and discovered automatically. The component reference is the learn API.

Python helper¶

For Python, the luckyrobots package wraps the generated stubs in a higher-level client, so stubs do not need to be generated by hand. It is a convenience layer over the same gRPC API. The Open LuckyRobots SDK on GitHub button in the gRPC Server panel opens its repository, which is the reference for the package's API.

pip install luckyrobots                # core client
pip install "luckyrobots[rl]"          # adds Gymnasium for LuckyEnv / PolicyEnv

from luckyrobots import Session

with Session() as sess:
    sess.connect(timeout_s=30.0, robot="unitreego2")
    obs = sess.reset()
    for _ in range(1000):
        obs = sess.step(actions=[0.0] * len(obs.actions))

Session.connect() needs the robot name. LuckyEnv wraps a robot and scene in the Gymnasium API (reset, step, observation_space, action_space), and RobotController mirrors the policy calls from Controlling robots. The package ships pre-generated stubs, so everything it does is also available over the raw API above from any language.

Where to go next¶

Sensors covers simulated lidar and RGBD depth, including the LidarService and depth-frame details.
Controlling robots: policies and IK is the policy and motion-graph model the policy calls mirror.
Recording covers engine-side dataset recording and camera bindings.
Command Line lists the launch options, including --server.
AI Assistants (MCP) covers the editor's other automation interface, for AI coding assistants.