How Movesia Reads Your Unity Scene Hierarchy

The Challenge

When an AI agent needs to understand a Unity scene with 1,800 GameObjects, you can't just dump the entire hierarchy into a prompt. Context windows have limits, token costs add up, and most of that data isn't relevant to the current task. The agent needs a way to explore the scene incrementally — drilling into branches that matter and ignoring the rest.

This is the core design problem we solved in Movesia's hierarchy browsing system, and the solution borrows heavily from a familiar concept: filesystem navigation.

Filesystem-Style Hierarchy Browsing

Instead of loading the entire scene tree at once, Movesia's Hierarchy Tool works like ls and cd in a terminal. The agent starts at the root and can list the direct children of any node, then navigate deeper into subtrees that are relevant to the current task.

// Unity-side C# handler for hierarchy requests
public static class HierarchyHandler
{
    public static JObject ListChildren(string path)
    {
        var parent = FindByPath(path);
        var children = new JArray();
 
        foreach (Transform child in parent.transform)
        {
            children.Add(new JObject
            {
                ["name"] = child.name,
                ["childCount"] = child.childCount,
                ["activeSelf"] = child.gameObject.activeSelf,
                ["components"] = GetComponentNames(child)
            });
        }
 
        return new JObject
        {
            ["path"] = path,
            ["children"] = children
        };
    }
}

Each response includes just the immediate children with minimal metadata: name, child count (so the agent knows if a node has subtrees worth exploring), active state, and a list of attached component type names.

The WebSocket Protocol

Communication between Movesia and Unity happens over a local WebSocket connection on port 8765. The protocol uses JSON-RPC 2.0 with a simple request-response pattern:

A typical interaction looks like this:

// Request: List root-level objects
{"method": "hierarchy.list", "params": {"path": "/"}, "id": 1}
 
// Response: 4 root objects
{
  "result": {
    "path": "/",
    "children": [
      {"name": "Main Camera", "childCount": 0, "components": ["Camera", "AudioListener"]},
      {"name": "Directional Light", "childCount": 0, "components": ["Light"]},
      {"name": "Player", "childCount": 3, "components": ["PlayerController", "Rigidbody"]},
      {"name": "Environment", "childCount": 47, "components": []}
    ]
  },
  "id": 1
}

The agent sees that "Player" has 3 children and "Environment" has 47. If the user asks about the player's setup, the agent drills into /Player without ever loading the 47 environment objects.

Why Not Load Everything?

We considered sending the full hierarchy on connection, but testing with real projects showed why this doesn't scale:

• A mid-size Unity project has 500-2,000 GameObjects
• Each object has 3-8 components with serialized properties
• The full data for a 1,000 object scene is roughly 2-4 MB of JSON
• Sending this on every query would consume ~8,000 tokens, wasting context on irrelevant objects

The incremental approach typically uses 200-500 tokens per hierarchy query, and the agent rarely needs more than 3-4 queries to find what it needs. For a scene with 1,800 objects, the agent might inspect 20-30 nodes total — less than 2% of the scene.

Asset Management Through the Hierarchy

The hierarchy system becomes especially important when working with assets from the Movesia Asset Store. When the agent imports an asset into a project — say a prefab for a stone bridge — it needs to understand where in the scene to place it. The Hierarchy Tool lets the agent inspect the existing scene structure, find the right parent object (like an "Environment" or "Level" container), and place the imported asset as a child at the correct position.

This also applies when editing scenes that already contain placed assets. If a user says "move the castle gate to the north wall," the agent uses the hierarchy to locate the gate object, reads its transform component, calculates the new position relative to the wall object, and applies the change. The incremental browsing design means this works efficiently even in large, asset-heavy scenes with hundreds of placed objects.

Component Data on Demand

Once the agent finds the right GameObject, it can request detailed component data separately through the Component Tool. This keeps hierarchy browsing lightweight and component inspection detailed:

// Request component details for the Player
{"method": "component.get", "params": {"path": "/Player", "type": "PlayerController"}, "id": 2}
 
// Response with serialized field values
{
  "result": {
    "type": "PlayerController",
    "fields": {
      "moveSpeed": 5.0,
      "jumpForce": 12.0,
      "groundLayer": "Ground",
      "isGrounded": true
    }
  },
  "id": 2
}

This separation of concerns — hierarchy for navigation, components for data — mirrors how developers actually use the Unity Editor. You browse the Hierarchy panel to find an object, then switch to the Inspector to see its details.

Handling Scene Changes

Unity scenes are dynamic. Objects get created, destroyed, reparented, and modified during Editor sessions. Movesia handles this with a lightweight change detection system:

The Unity package tracks modifications through Editor callbacks (ObjectChangeEvents in Unity 6) and maintains a version counter. When the agent queries the hierarchy, the response includes the current version. If the version has changed since the last query, the agent knows its cached understanding may be stale and can re-query the relevant subtree.

This avoids both polling overhead and stale data — the agent works with fresh information when it matters and cached data when the scene hasn't changed.

Error Detection in the Loop

The hierarchy and component systems don't just serve read-and-write purposes — they're also how the agent detects and resolves errors. When the agent modifies a scene (placing an asset, rewiring a component, creating new objects), it can immediately re-query the affected area of the hierarchy to verify the result. If something went wrong — a missing reference, a misplaced object, a component that didn't attach correctly — the agent catches it in the same turn and fixes it.

This is how Movesia achieves its auto-fix behavior: the agent operates in a tight loop of action, verification, and correction. The incremental hierarchy design makes this fast, because the agent only needs to re-check the specific subtree it just modified, not the entire scene.

Lessons Learned

Building this system taught us that the best AI tool interfaces are the ones that match existing mental models. Filesystem navigation is universally understood, so giving the agent ls-style browsing feels natural both for the agent's reasoning and for developers reading the tool calls in the chat log.

The other key lesson is that less data is usually better. Sending the agent only what it needs — when it needs it — produces better results than flooding it with context and hoping it finds the relevant parts. This principle extends beyond the hierarchy tool — it shapes how the agent interacts with the Movesia Asset Store, how it reads error logs, and how it manages assets across an entire project. Every interaction is scoped to exactly what's relevant to the current task.