Lucky Robots Blog Open Roles

2.13 · Robot / MotionGraph

Two composing subsystems for robot behaviour: motion graphs for designed behaviour (IK, blending, state machines) and per-entity ONNX policies for learned behaviour. The graph always runs; policies override it on the joints they claim.

Module: Hazel/src/Hazel/Robots/ MotionGraph.h PolicyApplier Issue #587
For each joint, every physics substep: Walk active slots priority-ascending DrivenJoints claims it? first slot wins Policy-claimed PD → qfrc_applied Motion graph drives see routing table below yes no Routing once a driver is chosen Policy PD → qfrc_applied (torque control) Graph + actuator ctrl[] (position/general) Graph, no actuator PD → qfrc_applied Unclaimed gravity compensation
Per-substep merge: priority walk picks a driver per joint, then the case table chooses the actuation channel.

Overview

LuckyEngine pairs two control philosophies in the same robot. Motion graphs are visual, authored, deterministic — good for IK reach, blending, state machines, and anything you want to step through and inspect. Per-entity ONNX policies are learned controllers loaded from disk and ticked every physics substep. The two compose by joint: the motion graph always runs, and any active policy slot that whitelists a joint takes that joint over for the substep.

There is no global override switch and no "policy-mode vs graph-mode" flag. Authority is per-joint, decided from PolicySlot::DrivenJoints, every substep.

Motion graphs

MotionGraph::Graph (MotionGraph.h) is the base class — it owns nodes, value and event connections, input/output streams, and local variables. The concrete MotionGraph::MotionGraph specialises it for robots: it holds a non-owning mjModel* and mjData* pair, and Process(timestep) evaluates the graph to produce JointStates.

TypeHeaderRole
MotionGraph::GraphMotionGraph.hBase. Owns nodes, value/event connections, I/O streams, local variables.
MotionGraph::MotionGraphMotionGraph.hRobot-specialised graph. Non-owning mjModel* / mjData*. Process(timestep) produces JointStates.
NodeProcessorNodeProcessor.hAbstract node base. All concrete nodes derive from this.
JointStatesMotion.hMotionDuration, MotionTimePos, Nq positions, Nv velocities. Data allocated contiguously after the struct.
MotionGraphFactoryMotionGraphFactory.hRegisters node types; exposes the editor-diagnostics hook for multi-error/warning reporting.

Concrete nodes live in Hazel/src/Hazel/Robots/Nodes/:

IKNodes

Limb IK using Mujoco::InitIKChain on the read-only compiled model.

MathNodes

Scalar / vector arithmetic, clamps, lerps.

BlendNodes

Pose blending and weighted mixers.

TriggerNodes

Event-driven gates.

LogicNodes

Boolean / conditional flow.

MotionNodes

Source poses, motion clips.

StateMachineNodes

State graphs with transitions.

ArrayNodes

Array slicing, indexing, fan-out.

Editor-time validation

Nodes can return multiple errors and warnings via MotionGraphFactory's editor-diagnostics hook — the editor surfaces all of them, not just the first. LimbIK reuses Mujoco::InitIKChain on MujocoSceneAsset's read-only compiled model to flag overconstrained chains before the user enters play mode.

Multi-agent / RL training

For RL workflows, RobotManager (RobotManager.h) owns a RobotEnv and one or more RobotAgents. The split lets a single environment drive N agents through batched ONNX inference.

TypeHeaderRole
RobotManagerRobotManager.hOwns RobotEnv + RobotAgents.
RobotAgentRobotAgent.hObservation / action / control buffers, joint mappings (m_ActuatorIds, m_JointQposAdrs, m_JointQvelAdrs, m_DefaultJointPositions), policy reference.
RobotEnvRobotEnv.hMulti-agent Reset, ResetAgent, ApplyControls.
PolicyNetworkPolicyNetwork.hONNX Runtime integration. Static cache (Load, Get, ClearCache) and ForwardBatch(obs, actions, batchSize) for zero-copy batch inference.

Per-entity ONNX policies

Runtime policy inference is not a motion-graph node

Issue #587. Policies live as PolicySlot entries on RobotControllerComponent; they tick from MujocoSceneInstance, not from inside the graph. Anything that depends on policy output (PD gains, joint targets, action history) belongs on the PolicyRuntime, not on a node.

A per-MujocoSceneInstance PolicyApplier owns one PolicyRuntime per active slot and runs inference every physics substep, then merges the result with the motion graph's JointStates by per-joint priority.

Types under Hazel/src/Hazel/Robots/Policy/

TypeFields / APIRole
PolicySlotId, Name, DescriptorPath, Active, Priority, DrivenJoints, ClampObservationForUnclaimedJointsDeclarative entry on RobotControllerComponent. Id is the 1-based script id. Empty DrivenJoints = all actuated joints in the descriptor.
PolicyRuntimeONNX session, PD gains, joint mappings, CommandStore, target joint positions, last actionPer-slot live state. Allocated by PolicyApplier when a slot goes active.
PolicyApplier(Entity UUID, slotId) → SlotRuntimePer-scene-instance dispatcher. Called from MujocoSceneInstance::SimulateControl and SimulateWithSubsteps.
RobotController (C#)SetPolicyActive, SetFloat/SetBool, GetFloat/GetBool, SetMotionGraphActive, IsMotionGraphActiveEntity-scoped script wrapper. Accepts raw uint ids or unmanaged, Enum via Unsafe.As<TEnum, uint> — no boxing.

Per-substep merge rule (per joint)

  1. Walk active slots in priority-ascending order; the first slot whose DrivenJoints claims the joint wins.
  2. If no policy claims the joint, the motion graph's JointStates value drives it.

Routing after a claim

CaseChannel
Policy-claimedPD → qfrc_applied
Graph-claimed, joint has a position / general actuatorctrl[]
Graph-claimed, joint has no actuatorPD → qfrc_applied
Unclaimed actuated joint (no graph, no policy)Gravity compensation
MotionGraph Process(timestep) PolicyRuntime[*] ONNX inference PolicyApplier per-joint merge priority walk qfrc_applied PD torques ctrl[] position actuators gravity comp unclaimed joints
Motion graph + policy slots feed PolicyApplier; per-joint claim picks which MuJoCo channel each joint uses this substep.

Motion-graph gating

Graph::SetEnabled(bool) / IsEnabled control whether a graph runs. Scene::UpdateRobotControllers short-circuits to continue when a graph is disabled — it skips SetSceneTransform, Process, and SetPendingJointStates, so no stale graph output leaks into the next substep. Exposed in C# via RobotControllerComponent.SetMotionGraphActive(bool) and IsMotionGraphActive.

Descriptor generation

scripts/GeneratePolicyDescriptor.py produces one policy_descriptor.<key>.json per ONNX model. The editor's Add Policy to Entity flow (PolicyDescriptorGenerator::AddPolicyToEntity) appends a PolicySlot that references the descriptor and stamps a C# scaffold next to the ONNX file so script code can wire up SetFloat / SetBool inputs immediately.

Removed — do not re-introduce

These APIs are gone on purpose

The following symbols were deleted with issue #587 and the per-joint authority model. Re-introducing any of them re-creates the bypass paths that the merge rule was designed to eliminate:

  • PolicyInferenceNode — inference is no longer a graph node.
  • Command.* script API — replaced by RobotController.
  • ShouldBypassMotionGraphForPolicy — gating is now per-joint, not per-graph.
  • Global IK-overlay path: SetGlobalIKOverlayEnabled, AddGlobalIKActuator, etc.
  • IsJointIKOwned.
  • IsTorquePolicyAuthoritative.

Pitfalls

Priority is ascending

Priority is walked low-to-high; the first active slot whose DrivenJoints claims a joint wins. A second slot with higher Priority on the same joint will never run for that joint — this is intentional but easy to mis-set.

Empty DrivenJoints means everything

An empty DrivenJoints list whitelists all actuated joints in the descriptor. Leaving it empty on a high-priority slot will lock the motion graph out of every joint.

Don't add diagnostics to MotionGraphNodeEditorModel

Per-node editor diagnostics belong on MotionGraphFactory's hook (it supports multiple errors and warnings per node). Pushing knowledge into the editor model couples the UI to specific node types.

Extending

  • New motion-graph node — derive NodeProcessor in Robots/Nodes/, register in MotionGraphFactory.
  • New editor-time node diagnostics — extend MotionGraphFactory's diagnostics hook; return multiple errors/warnings if you have them.
  • New policy type — extend PolicyNetwork or add a new inference backend alongside the ONNX path. Wire it into PolicyRuntime; do not add a graph node.