// Execution Management (XEM)

Troubleshooting

First stop when an operator hits an unfamiliar error code. The companion Error Codes page lists every code with its HTTP status, envelope type, and JSON shape; this page adds the operator action and the xeroctl command that diagnoses it further.

Overview

Only codes that the router emits as error.code string values are listed. The public envelope shape is {"error": {"message", "type", "code", "param"}}; code is optional and many controller errors omit it, so always cross-check the envelope type when code is missing.

How to Read This Page

  • Code, the value in the error.code field of the router's JSON response.
  • Operator action, the single next step a reasonably experienced operator takes.
  • Diagnostic, the xeroctl command to run if the action fails or you need more context.

Where no dedicated diagnostic command exists yet, the row notes "(no dedicated diagnostic)" rather than inventing one. Capture the x-request-id response header on every failed call, it is the only correlation handle the router exposes to clients.

Scope & Auth

Note: scope_insufficient is emitted by three distinct middlewares (ManagementScopeMiddleware, InferenceScopeMiddleware, and ExecScopeValidator) with envelope type=authentication_error. The same string code therefore covers management-API, inference-API, and exec-tool scope gaps; the envelope message field disambiguates which surface rejected the call.

CodeOperator ActionDiagnostic
scope_insufficient Re-issue the key with the required scope, or switch to a key that already carries it. Read the envelope message to see whether management, inference, or exec scope was missing. xeroctl keys
cross_project_access The key belongs to a different project than the resource. Confirm the project_id path segment matches the key's owning project's external ID (ws_-prefixed for workspaces). xeroctl keys <key-id>
project_not_found The project external ID in the path does not exist or is not visible to the caller. Verify the ID with the operator who issued the key. xeroctl project members list
signing_public_key_required The endpoint enforces signed requests; attach the registered Ed25519 public key to the API key before retrying. xeroctl keys <key-id>
insufficient_quota Project credit budget is exhausted. Top up credits or wait for the next billing window. xeroctl settings get
billing_delinquent Billing is past due. Resolve invoices before further inference calls are accepted. xeroctl settings get

Agents

Note: enrollment failures (duplicate name, manifest validation, candidate rejection) surface as HTTP-status signals on /v1/enroll/refresh rather than as error.code string values. There is no machine-readable agent_* code for offline agents; absence shows up as missing rows in xeroctl agents or as exec_tool_not_found / tool_executor_unavailable at execution time.

CodeOperator ActionDiagnostic
agent_not_found Confirm the agent ID. If the agent was decommissioned, re-enroll or pick another agent in the workspace. xeroctl agents
candidate_not_found The discovery candidate has already been approved, rejected, or expired. List current candidates and decide on a fresh one. xeroctl agents candidates list <agent-id>

Workspaces & Models

CodeOperator ActionDiagnostic
workspace_not_found Verify the workspace external ID (ws_-prefixed); confirm the workspace was not deleted. xeroctl workspaces list
model_not_found The model id is unknown to the router. Re-target the request at a model that exists in the catalog. xeroctl models
endpoint_inactive The endpoint is provisioned but not currently serving. Re-provision it or pick an active endpoint. xeroctl endpoints show <endpoint-id>
endpoint_restricted The endpoint is gated by an allowlist or quota the caller does not satisfy. Update the endpoint's restrictions or use a different endpoint. xeroctl endpoints show <endpoint-id>
endpoint_task_mode_mismatch The endpoint's task_mode (chat, embedding, reranking, etc.) does not match the request's surface. Re-issue the request against an endpoint whose task mode matches. xeroctl endpoints show <endpoint-id>
model_not_scoring No worker for this model is currently scoring as ready. Wait for provisioning or scale the fleet. xeroctl endpoints list
invalid_tier The requested service tier is not configured for this endpoint. Pick a tier the endpoint advertises. xeroctl endpoints show <endpoint-id>

Execution

CodeOperator ActionDiagnostic
exec_tool_not_found The tool name does not resolve in the workspace's effective manifest. Verify the spelling and that an XEM publishing the tool is bound to the workspace. xeroctl exec tools
tool_not_mcp_visible The tool exists but is not exposed to MCP clients in this workspace. Adjust the workspace tool visibility or invoke through the agentic-loop surface instead. xeroctl exec tool-show <tool>
tool_executor_unavailable No online XEM can serve this tool. Verify XEM health, then enroll an additional XEM or scale the fleet. xeroctl agents
capacity_exceeded The fleet is saturated. Wait, retry with backoff, or add capacity. xeroctl exec list
rate_limit_exceeded Back off; honor the Retry-After response header and the retry_after field in the error envelope. xeroctl settings get
context_length_exceeded Request exceeds the model's context window. Compact the conversation, drop earlier messages, or pick a model with a larger context. xeroctl models

Approvals

CodeOperator ActionDiagnostic
approval_not_found The approval id is unknown or has been purged. List current approvals and retry against a live id. xeroctl approvals list
approval_not_pending The approval has already been approved, rejected, or timed out. Inspect its terminal state; re-issue the originating invocation if a new decision is needed. xeroctl approvals show <id>

Lifecycle

CodeOperator ActionDiagnostic
invocation_not_found The invocation id is unknown or has been pruned by retention. Confirm the id and the retention window. xeroctl exec list
invocation_terminal The invocation is already in a terminal state; the requested transition (for example cancel) is a no-op. xeroctl exec show <id>
internal_error Capture the x-request-id response header (not a body field) and the timestamp, then report to the platform team. The router does not surface a body-level request id; the header is the only correlation handle. (no dedicated diagnostic; provide the x-request-id header value)