Skip to main content
xarm-perception-sim-agent runs the xArm perception, planning, MuJoCo simulation, MCP server, and built-in agent together. It is simulation-only; it does not control a physical arm. See the manipulation capability overview for the underlying planning and perception stack.

​
Prerequisites

Install the manipulation dependencies:
The built-in agent requires an OPENAI_API_KEY for OpenAI Platform API access. Your OpenAI account must have billing enabled or available API credits. Do not put the key in this document or in shell history shared with others. The local LCM UDP multicast/loopback path must be available between the DimOS processes. On a restricted environment, allow local multicast traffic on the loopback interface; otherwise the agent input and module communication may not arrive.

​
Start and stop

Run in the foreground:
Or run it as a daemon:
Inspect and control the run from another terminal:
Use dimos log -f to follow the log while the run is active.

​
Daily interaction

For normal interactive use, start the human-friendly terminal client:
It connects to the running agent so you can send prompts and read responses in one session.

​
Try these prompts

Start with a non-motion state check:
Scan the scene for objects. This moves the arm to its observation pose:
Try basic motion commands:

​
Debugging and testing interfaces

Use agent-send for one-shot LCM input when testing or diagnosing the agent:
The blueprint also includes an MCP server. Use these commands for direct server inspection and tool-level testing:
For example:
look observes the current camera view without moving the arm. scan_objects is a single initialization-viewpoint perception snapshot: it first moves the simulated arm to its init position and then refreshes detections. It is not a guaranteed object inventory.