Production checklist
orch runs a production cluster with a few one-time steps. Most of them take a single command. Do them before the cluster carries traffic you care about.
Before you start
- Install from a signed release.
install.shandinstall.ps1check the release signature and checksums; see release signatures. Workers upgraded byorch cluster upgradeare checked the same way. - Pin your manifests to the v1 schema. Start each
app.pklwithamends "orch:/v1/Application.pkl". Compatibility lists what stays stable across 1.x. - Check each host.
orch node doctorreports anything the installer could not fix by itself.
Backups
Send backups off the controller. A backup on the controller does not survive losing that host:
orch backup destination set s3://acme-backups/productionSee off-host backups.
Store the recovery key. The first
setprints it once. Put it in a password manager; restoring the controller from the bucket needs it.Limit who can write to the bucket. Restoring a backup restores the cluster CA and every secret, so anyone who can plant a backup there can take over the cluster when you restore it. Give the access key orch uses its own bucket or prefix, and do not share write access with other systems.
Back up stateful volumes. Give each volume that holds data a
backupblock; see volume backups.Practise a restore. Restore the newest backup into a scratch directory once, and again after major changes, so the first restore is not during an outage:
sudo -E orch cluster restore --from s3://acme-backups/production \ --data-dir /tmp/orch-restore-drillThe command checks the backup, the recovery key and the PKI before it writes anything. Delete the drill directory afterwards. After a real restore, follow the checks in backups.
Monitoring and alerts
Send cluster alerts somewhere you read. One command, and it sends a test message:
orch notify add slack https://hooks.slack.com/services/...Lost nodes, full disks, expiring certificates and stale backups are then reported; see cluster notifications.
Scrape
/metricswith Prometheus if you run it; see monitoring.
Keeping it current
- Patch the hosts. Run
orch node update --allon a schedule, monthly or when anode.reboot_requiredalert arrives. It drains, upgrades and reboots one node at a time; see operating-system updates. - Upgrade orch.
orch cluster upgradeupgrades the controller and then every worker; see upgrades. It also refreshes the bundled containerd. - Certificates renew themselves. Workers, the controller and administrator
contexts in regular use renew ahead of expiry. A
certificate.expiringalert means renewal is failing; see certificate renewal.
Capacity
- Run at least two workers for anything that must stay up while a node is
drained or updated, and give such services
replicas = 2or more. - Know the limits. Limitations lists what one controller supports and what orch leaves to you.