Cluster operations#

Use the CLI or TUI to inspect applications, manage nodes, run jobs, and maintain the cluster.

Select a target before running commands:

orch target list
orch target use production
orch target test

Pass --target production explicitly in automation.

Cluster status#

orch status
orch nodes
orch tui
orch web

The TUI provides a keyboard-driven view of applications, instances, workers, deployments, events, and live logs.

orch web opens a browser dashboard for the selected target: cluster health, applications with replicas, routes, cron runs, deployment history, diffs and rollback, runtimes, nodes with cordon, drain and ingress controls, one-off jobs, events, and live logs. Press ⌘K or / to jump anywhere.

The dashboard is served from the orch binary on a loopback address. It prints a one-time URL; the page only works on this machine and the administrator credentials stay in the orch process. Pass --no-open to only print the URL, and --port N to choose the port.

orch status also shows when the administrator and controller certificates expire. Renew them before then; see certificate renewal.

Applications#

orch app status hello
orch app history hello
orch events --limit 50
orch logs hello web --replica 1 --tail 100

When a service is blocked, its status includes a reason such as missing capacity, a label mismatch, a missing secret, or a lost local-volume worker. Resolve that condition and orch will reconcile the desired state.

Node labels#

Labels let applications request specific workers:

orch node label worker-1 region=eu-west
orch node label worker-1 tier=compute
orch nodes

Use the same keys in a service's constraints. The orch.* label prefix is reserved.

Node maintenance#

Cordon a worker to stop new placement:

orch node cordon worker-1

Drain movable workloads before maintenance:

orch node drain worker-1

Services with local volumes remain pinned and are reported instead of being moved. After maintenance:

orch node uncordon worker-1

One-off jobs#

Run a command from a service definition:

orch job run hello web -- ./bin/migrate up
orch job show JOB_ID
orch job logs JOB_ID

Use a pre-deploy hook in orch.pkl when a migration must finish before a new revision rolls out:

deploy {
  preDeploy = new PreDeployHook {
    service = "web"
    command { "./bin/migrate" }
    args { "up" }
    timeout = "5m"
  }
}

Migration commands should be safe to retry.

Scheduled jobs#

orch cron runs hello cleanup
orch cron logs hello cleanup RUN_ID

Overlapping occurrences are skipped. Failed and missed occurrences are not automatically replayed.

Removing applications#

Inspect the application and its local data before removal:

orch app status hello
orch remove hello

Removing desired state stops its workloads. Treat local-volume data separately according to your retention policy.

Backups#

Create backups regularly and copy them off the controller:

sudo /opt/orch-controller/bin/orch cluster backup \
  --data-dir /var/lib/orch-controller \
  --output /secure/orch-backups/production.db \
  --secret-key-output /secure/orch-backups/production.key

Protect backups like administrator credentials: they contain controller state, PKI, and the secret-encryption key. See security and recovery before restoring a controller.

Upgrades#

Upgrade the controller first, then workers:

sudo ./new-orch cluster upgrade --binary ./new-orch --dry-run --json
sudo ./new-orch cluster upgrade --binary ./new-orch

sudo ./new-orch node upgrade --binary ./new-orch --dry-run --json
sudo ./new-orch node upgrade --binary ./new-orch

Keep the recovery path reported by each operation until normal traffic and a test deployment have succeeded.

Controller outages#

Workers continue running their last assigned workloads. New deployments, scheduling, node changes, and secret changes wait. Restore the controller at the same canonical HTTPS address so existing workers and operator contexts can reconnect.