Skip to main content
The Substrate AI Assistant is not a generic code copilot. It is a simulation-aware agent that can see your 3D scene, read live telemetry, query the scene graph, and execute commands against a running simulation. Open the chat panel from the right sidebar to start a conversation.

​
How It Works

The assistant connects to your active simulation through the same bridge protocol that powers the rest of the IDE. It has access to specialized tools that let it observe and interact with the simulation environment directly.
  • Scene Graph — query the full entity hierarchy, including models, links, joints, and sensors
  • Telemetry Sampling — read recent values from any data channel with time-series history
  • Readiness Details — check which subsystems are online and which are still initializing
  • Topic Browser — discover all available data channels and their update rates
  • TF Tree — inspect the transform hierarchy between coordinate frames
  • Simulation State — combined readiness, telemetry, and background task status in one call
For the full tool inventory with parameters and examples, see the AI Tools Reference.

​
Vision and Recording

The assistant can capture screenshots and record video from the 3D viewer, then analyze them using Gemini vision. Screenshots are useful for quick visual checks — whether a model loaded correctly, an entity is in the right position, or a collision geometry looks correct. The assistant receives the image directly and can describe what it sees. Video recording captures the simulation window for up to 30 seconds. The assistant can then analyze the recording to observe behavior over time — stability during maneuvers, oscillations, drift, or unexpected collisions. Small recordings are analyzed frame-by-frame; larger files are uploaded to Gemini’s File API.
Try asking: “Record the simulation for 10 seconds while I run this script, then tell me if anything looks wrong.”
You can also paste or drag images directly into the chat input. The assistant will analyze them alongside any active simulation context.

​
Model Tiers

Standard

Claude Sonnet. Best for routine questions, parameter lookups, and straightforward debugging. Fast response times.

Max

Claude Opus. Use for complex multi-step reasoning, architecture decisions, and deep analysis of simulation behavior.

​
Context Awareness

The assistant automatically detects the running simulation, robot type, controller state, and system health. You do not need to explain your setup — ask questions directly.
The assistant reads simulation state on demand through tool calls. It does not continuously stream all telemetry. This keeps conversations focused and avoids unnecessary overhead.