// XEM Guides

Credential Rotation

Routine and emergency rotation for the four credentials a XEM touches.

Workspace credentials, the Curve25519 data-plane keypair, API keys used by clients calling /v1/exec/*, and the short-lived join keys used at enrollment. Each has its own discipline.

Summary

Four credentials, four postures. Skim the strip before drilling in.

  • // workspace Long-lived or STS, on disk under /var/lib/xerotier/credentials/. Replace file, reload service.
  • // curve Curve25519 keypair, data-plane identity of the XEM. xeroctl agents curve-rotate.
  • // api Client-side keys for /v1/exec/*. Lives on the caller, not the XEM. xeroctl keys create then revoke.
  • // join Single-use, TTL-bounded JWTs. Not rotated; bounded. Cap TTL. Revoke unconsumed on suspicion.

Press Alt+C while a code block is focused (Tab through to one) to copy it. No Meta/Cmd binding, so the OS clipboard chord is untouched.

Workspace Credentials

Workspace credential files live under /var/lib/xerotier/credentials/ and are referenced by credentials_ref in the manifest. Rotation is a two-step: replace the file on disk, then reload.

bashinstall + reload
# On the XEM host: sudo install -m 0600 -o root /tmp/new-kubeconfig \ /var/lib/xerotier/credentials/kubernetes-prod.yaml sudo systemctl reload xerotier-xem

The reload re-opens the credential file without interrupting in-flight invocations. The invocation that was already authenticated with the old credential completes; the next one picks up the new file.

For short lived tokens (e.g. cloud STS), set a filesystem watcher or timer to refresh the file before expiry. The XEM does not refresh tokens itself; it consumes what the operator provides.

CURVE Keys

The Curve25519 keypair is the data-plane identity of the XEM. Rotate on compliance schedule, on suspected compromise, or after a host rebuild.

bashrotate-curve
# From an operator workstation: xeroctl agents curve-rotate agt_abc123 --reason "scheduled"

The rotation is on-demand, not scheduled. The router instructs the agent to generate a fresh keypair and waits for a signed acknowledgement over the existing authenticated channel before promoting the new key. The agent ID is the router identifier (e.g. agt_abc123); the --reason string is recorded in the audit log and defaults to scheduled.

Expected transcript:

textrequest accepted
request_id: 7f0e... agent_id: agt_abc123 status: pending monitor: /v1/management/agents/agt_abc123/events

Monitor the events stream at the printed URL to observe the acknowledgement and promotion. The rotation is audit-logged with operator, reason, and request id.

If status resolves to failed, the recorded reason will be one of: acknowledgement timeout, signature mismatch, or revoked agent. Treat any of these as a containment trigger and pivot to Incident Response before retrying.

API Keys

API keys live on the caller side, not on the XEM host. Rotation uses the management scope:

bashrotate-api-key
xeroctl keys create --scope execution --name fleet-ops-2026q2 xeroctl keys revoke --name fleet-ops-2026q1

Create the new key before revoking the old one so callers have a clean cutover. The revoke is immediate, plan the client rollout before calling it.

Join Keys

Join keys are not rotated; they are bounded. Each is a single-use, TTL-bounded JWT, consumed on first enrollment and never reusable. The operational handle is the TTL: cap it to the shortest value that still lets operations complete the enrollment.

bashissue-join-key
xeroctl agents join-keys --create \ --name xem-fleet-02 \ --region us-east \ --router-addr tcp://router:5555 \ --ttl-seconds 600 # cap as low as operations allows

The --agent-role option accepts only inference or execution; XEM execution agents use execution. Omit it to accept the router default.

A leaked join key is contained by its TTL, but the window is not zero. Revoke unconsumed keys immediately on suspicion -- the TTL is a backstop, not a substitute for revocation.

Unconsumed and terminal (revoked, used, expired) keys can be listed, and individual keys revoked by id:

bashlist + revoke
xeroctl agents join-keys # active keys xeroctl agents join-keys --include-terminal # include revoked/used/expired xeroctl agents join-keys jk_abc123 --revoke --force

Recommended Cadence

CredentialRoutine cadenceEmergency
Workspace credentials (long-lived) 90 days Within 1 hour of suspected leak
Workspace credentials (STS / short-lived) Per token TTL Per token TTL (no faster)
CURVE keys 180 days Within 15 minutes of suspected leak
API keys 180 days Within 15 minutes
Join keys N/A (single-use) Revoke unconsumed immediately

These are defaults. Compliance regimes (PCI, HIPAA, FedRAMP) override with tighter requirements; follow the stricter of the two.

Adjacent Runbooks