Host reference
This page lists what orch expects from a host and everything it changes on one: supported operating systems, network ports, the files, accounts, and systemd units it owns, the pinned dependencies it installs, and the capacity it keeps back for the host itself. Use it to prepare firewalls and images, to review a host before installation, or to find a file while debugging.
For the step-by-step setup, see installation. For symptoms and fixes, see troubleshooting.
Supported hosts
| Requirement | Supported |
|---|---|
| Operating system | Ubuntu 24.04 LTS or newer, or Fedora 42 or newer (any edition except ostree-based ones such as Silverblue or CoreOS, whose /usr is read-only) |
| Architecture | amd64 or arm64; nodes of both can share a cluster |
| Init system | systemd running as PID 1 |
| Control groups | Unified cgroups v2 with the cpu, memory, and pids controllers |
| Kernel modules | overlay and wireguard (installation loads them) |
| Address | A unicast IPv4 address assigned to a local interface, private or public |
Free disk under /var/lib |
512 MiB for a controller, 2 GiB for a worker |
| Clock | Synchronized; the controller refuses a year before 2024 or after 2100 when it creates certificates |
| Privileges | Root for orch init, orch join, orch cluster ..., and orch node ... host commands |
The same binary and the same checks are used for both roles. A controller and a worker can share one host: their files, accounts, and services do not overlap.
The advertise address must be assigned to an interface on the host itself.
orch node doctor compares it with the host's interface addresses. On clouds
that map a public address to the host through 1:1 NAT, so that the public
address never appears on an interface, advertise the private address instead.
Start from a fresh host. Installation refuses to adopt existing state: a
worker install fails when containerd.service, docker.service, or
caddy.service is already loaded, when an orch-ingress account already
exists, or when any path orch would own already exists. A controller install
fails on an existing orch-controller account or controller path.
Check a host before installing anything:
sudo orch node doctor --advertise-address 10.0.0.11
sudo orch cluster doctor --server https://10.0.0.10:9443 --listen 0.0.0.0:9443Both commands only inspect. Each check prints a state (pass, warn, fail
or unknown) and, when there is one, the remedy. See
orch node doctor findings for each check.
Network ports
orch never changes UFW, firewalld, nftables rules outside its own tables, or cloud firewalls. Open these ports yourself.
Inbound
| Port | Protocol | On | From | Purpose |
|---|---|---|---|---|
9443 (orch init --port, or the --listen port of orch cluster install) |
TCP | Controller | Workers, operators that use certificate targets, and GitHub Actions runners when you use orch github trust |
Controller API: agent sessions, enrollment, administration over mTLS |
| 22 | TCP | Controller | Operators | SSH targets (orch target add NAME ssh://root@HOST) reach the administrator socket over SSH |
| 51820 | UDP | Every worker | Every other worker | WireGuard mesh between nodes. With public advertise addresses, allow it only from the other workers' addresses |
| 80 | TCP | Ingress-enabled workers | Clients, and the ACME certificate authority | HTTP routes, the public = true catch-all listener, and HTTP certificate challenges |
| 443 | TCP | Ingress-enabled workers | Clients, and the ACME certificate authority | HTTPS routes |
Each tcp.publicPort |
TCP | Ingress-enabled workers | Clients | TCP ingress, forwarded to ready replicas |
Each tcp.nodePort / udp.nodePort |
TCP or UDP | Workers that run the service | Clients | Node-local ports published straight to the replica on that node |
The controller listen port must be 1024 or higher. orch init listens on
0.0.0.0 at that port and advertises https://ADDRESS:PORT to workers.
Some ports on workers are reserved: TCP 22, 53, 80, 443, and 9443 (the default
controller port) and UDP 53 and 51820 cannot be used as publicPort or
nodePort values.
Local listeners
These are used by orch but should not be opened in a firewall.
| Listener | Where | Purpose |
|---|---|---|
| Port 53, TCP and UDP, on the node's workload gateway address | Workers | Cluster DNS for SERVICE.APPLICATION.svc.internal names, served to containers on that node |
/var/lib/orch-controller/admin.sock |
Controller | Local administrator socket; only root and the controller account can open it |
/var/lib/orch-ingress/admin.sock |
Workers | Caddy's admin API, used only by the agent |
/run/orch-containerd/containerd.sock |
Workers | orch's dedicated containerd |
127.0.0.1, random or --port |
Workstation | orch web dashboard |
Because the cluster DNS server runs on every worker, orch node doctor fails
when another process listens on port 53 on all addresses (0.0.0.0, *, or
::). A resolver bound only to a loopback address, such as systemd-resolved
on 127.0.0.53, does not conflict. It also fails when anything other than
orch listens on UDP 51820, or on TCP 80 or 443 on all addresses or the
advertise address.
Outbound
| From | To | Purpose |
|---|---|---|
| Workers | Distribution package mirrors | apt-get or dnf installs the OS utilities listed below |
| Workers | github.com over HTTPS |
Download of the pinned containerd, runc, CNI plugins, and Caddy releases during orch join or orch node install |
| Workers | The controller port | Enrollment and the agent's session |
| Controller | Container registries over HTTPS | orch deploy resolves image tags to digests on the controller |
| Controller | token.actions.githubusercontent.com over HTTPS |
Only with orch github trust: GitHub's signing keys for workflow tokens |
| Workers | Container registries over HTTPS | Image pulls |
| Ingress-enabled workers | The ACME certificate authority | Automatic HTTPS certificates through Caddy |
Forwarding
Workers forward packets between the host interface, the WireGuard interface,
and the workload network (10.88.0.0/16 unless you chose another range with
orch network configure). Installation enables IPv4 forwarding, but a host or
provider firewall can still drop forwarded traffic. Allow it. On Ubuntu with
UFW enabled, the default routed policy is deny; on Fedora, configure
firewalld zones and policies. See
networking requirements.
Controller host
orch init and orch cluster install install the controller. orch init
fills in the options: it detects the advertise address, listens on
0.0.0.0:9443, names the cluster orch, and writes the break-glass
administrator context to /etc/orch/admin. orch cluster install takes each
of these explicitly.
Files and directories
| Path | Contents |
|---|---|
/opt/orch-controller/bin/orch |
Copy of the orch binary that ran the install; root-owned |
/usr/local/bin/orch |
Link to the controller's binary, so sudo orch works; replaces a node's link |
/var/lib/orch-controller/ |
Controller state, owned by orch-controller |
/var/lib/orch-controller/control.db |
SQLite database: desired state, revisions, encrypted secrets, history |
/var/lib/orch-controller/secret-master.key |
Secret-encryption key |
/var/lib/orch-controller/pki/ |
Cluster CA and controller certificate and keys (ca.pem, ca-key.pem, controller.pem, controller-key.pem) |
/var/lib/orch-controller/admin.sock |
Local administrator socket |
/var/lib/orch-controller-install/manifest.json |
Installation journal: options, owned files and their hashes, completed phases |
/var/lib/orch-controller-install/upgrades/ |
Recovery sets created by orch cluster upgrade |
/etc/systemd/system/orch-controller.service |
systemd unit |
/etc/orch/admin/ |
Break-glass mTLS administrator context from orch init (--admin-dir with orch cluster install) |
--recovery-dir |
Optional day-zero recovery set: database backup, secret key, and PKI |
/run/lock/orch-controller-install.lock |
Lock held while an install runs |
Account
orch-controller is a system account and group with home directory
/var/lib/orch-controller and a nologin shell. The service runs as this
account. The binary and unit stay root-owned.
systemd unit
orch-controller.service runs:
/opt/orch-controller/bin/orch controller serve --data-dir /var/lib/orch-controller --listen 0.0.0.0:9443It restarts on failure and is hardened with NoNewPrivileges,
ProtectSystem=strict, ProtectHome, PrivateTmp, PrivateDevices, and
write access only to /var/lib/orch-controller.
What a controller install changes
- Creates the
orch-controlleraccount. - Copies the running orch binary to
/opt/orch-controller/bin/orchand links it as/usr/local/bin/orch, replacing a copy of the same binary but no other file that orch did not create. - Creates the cluster identity: CA, controller certificate, database, and secret key, plus the administrator context.
- Writes and verifies the systemd unit (
systemd-analyze verify). On Fedora with SELinux enabled, restores file labels on the owned paths. - Enables and starts
orch-controller.serviceand waits until it answers with the new cluster identity. - Writes a verified recovery set when you pass
--recovery-dir.
It leaves firewall rules, DNS, routing, sysctl settings, and unrelated services unchanged. It installs no packages.
Each phase is recorded in the installation journal. Rerunning the same command
with the same binary and options resumes from the first unfinished phase. A
different binary, address, or option is refused with "existing controller
installation intent differs"; use orch cluster upgrade to change the binary.
Worker host
orch join and orch node install install a worker. orch join resolves the
join token, contacts the controller to verify its CA, then performs the same
installation as orch node install --join-file.
Preview the plan, including every package, artifact, and file, without changing the host or reading join credentials:
sudo orch node install --advertise-address 10.0.0.11 --dry-runInstalled dependencies
orch downloads these from the upstream GitHub releases, checks each SHA-256 against the digest embedded in the orch binary, and installs only the listed executables. No upstream installer script runs.
| Component | Version | Installed files |
|---|---|---|
| containerd (static build) | 2.3.7 | /opt/orch/bin/containerd, /opt/orch/bin/containerd-shim-runc-v2 |
| runc | 1.4.3 | /opt/orch/bin/runc |
| CNI plugins | 1.9.1 | /opt/orch/cni/bridge, /opt/orch/cni/host-local, /opt/orch/cni/loopback |
| Caddy | 2.11.3 | /opt/orch/bin/caddy |
The versions are fixed for a given orch release. The controller only places
workloads on a node that reports containerd 2.3.4 or a newer 2.3 patch release
and CNI plugins v1.9.1, and only routes ingress through a node that reports
Caddy v2.11.3. To change them, upgrade orch: orch node upgrade (and the
controller-driven worker upgrade) also replaces an older containerd with the
bundled one and restarts it, which leaves running containers in place.
It also installs these packages from the distribution's signed repositories, without upgrading the rest of the host:
| Distribution | Packages |
|---|---|
| Ubuntu 24.04+ | ca-certificates, nftables, iproute2, libcap2-bin, kmod, passwd, udev |
| Fedora 42+ | ca-certificates, nftables, iproute, libcap, kmod, shadow-utils, container-selinux, policycoreutils-python-utils, systemd-udev |
Files and directories
| Path | Contents |
|---|---|
/opt/orch/bin/orch |
Copy of the orch binary that ran the install |
/usr/local/bin/orch |
Link to /opt/orch/bin/orch unless the controller's link is already there |
/opt/orch/bin/, /opt/orch/cni/ |
Pinned dependencies from the table above |
/opt/orch/containerd.toml |
Configuration for orch's dedicated containerd |
/etc/systemd/system/orch-containerd.service |
systemd unit for containerd |
/etc/systemd/system/orch-agent.service |
systemd unit for the agent |
/etc/sysctl.d/90-orch.conf |
net.ipv4.ip_forward=1 |
/etc/modules-load.d/orch.conf |
Loads overlay and wireguard at boot |
/etc/systemd/network/10-orch-veth.link |
Keeps the MAC address of new veth* devices so CNI can recognize them later; it does not match physical NICs or change existing interfaces |
/var/lib/orch-install/manifest.json |
Installation journal: address, distribution, owned files and their hashes, completed phases, node ID |
/var/lib/orch-install/upgrades/ |
Agent state backups created by orch node upgrade |
/var/lib/orch-agent/ |
Agent state (mode 0700) |
/var/lib/orch-agent/agent.json, pki/ |
Node identity: configuration, cluster CA, agent certificate and key |
/var/lib/orch-agent/agent.db |
Local cache of the assignment, so workloads survive a controller outage |
/var/lib/orch-agent/logs/ |
Workload logs |
/var/lib/orch-agent/volumes/APPLICATION/VOLUME/ |
Local volume data; kept when the application is removed |
/var/lib/orch-agent/volume-backups/APPLICATION/VOLUME/ |
Volume backups taken on this node |
/var/lib/orch-agent/runtime-secrets/ |
Secret files projected into containers |
/var/lib/orch-agent/network/ |
CNI and WireGuard state and the containers' resolv.conf |
/var/lib/orch-containerd/ |
containerd content store and snapshots (images) |
/run/orch-containerd/ |
containerd runtime state and socket |
/var/lib/orch-ingress/ |
Caddy data (certificates), admin socket, caddy.pid, and caddy.log; owned by orch-ingress |
/run/lock/orch-install.lock |
Lock held while an install runs |
Local volume data, volume backups, and workload logs live under
/var/lib/orch-agent. Size the file system that holds /var/lib for images,
logs, volumes, and their backups, not only for the 2 GiB the installer checks.
Each volume backup is a compressed copy of the whole volume, and keep of
them are retained.
Account
orch-ingress is a system account and group with home directory
/var/lib/orch-ingress and a nologin shell. The agent runs Caddy as this
account. Installation gives /opt/orch/bin/caddy exactly one file capability,
cap_net_bind_service=ep, so Caddy can bind ports 80 and 443 without running
as root. The binary is owned by root:orch-ingress with mode 0750, so
other local users cannot run it to bind low ports. There is no separate Caddy
systemd service.
systemd units
| Unit | Runs | Notes |
|---|---|---|
orch-containerd.service |
/opt/orch/bin/containerd --config /opt/orch/containerd.toml |
Dedicated containerd with its own socket and state paths; the CRI plugins are disabled. Restarts always |
orch-agent.service |
/opt/orch/bin/orch agent serve ... |
Requires orch-containerd.service. Restarts on failure. KillMode=process, so restarting the agent does not stop running containers |
The agent unit passes these settings, all fixed at install time:
| Flag | Value |
|---|---|
--advertise-address |
The address given to orch join or orch node install |
--wireguard-port |
51820 |
--system-reserved-cpu |
500m |
--system-reserved-memory |
512Mi |
--containerd-socket |
/run/orch-containerd/containerd.sock |
--snapshotter |
overlayfs |
--cni-bin-dir |
/opt/orch/cni |
--caddy-binary |
/opt/orch/bin/caddy |
--ingress-user |
orch-ingress |
--ingress-data-dir |
/var/lib/orch-ingress |
Resource reservations
Each worker keeps 500 millicpu and 512 MiB of memory back for the host and
orch's own processes. The rest is allocatable to workloads. orch nodes shows
each node's total capacity, allocatable capacity, and what is already
assigned:
orch nodesThe scheduler places a replica only on a node whose remaining allocatable CPU
and memory cover the service's resources.cpu and resources.memory. When no
node has room, the service is blocked with insufficient_cpu or
insufficient_memory.
Network objects
The agent creates and owns these, named after the cluster and node:
| Object | Name |
|---|---|
| WireGuard interface | orchwg followed by 9 hexadecimal characters |
| Workload bridge | orch followed by 11 hexadecimal characters |
| nftables table | orch_ followed by a digest, for forwarding, NAT, and node-local ports |
Each node receives one /24 from the workload network. Containers on that
node get addresses from it, and the first usable address is the node's
gateway and DNS server.
What a worker install changes
- Installs the OS utilities listed above with
apt-getordnf. - Downloads and verifies the pinned artifacts and copies the orch binary to
/opt/orch/bin/orch, linked as/usr/local/bin/orchon PATH. - Creates
/var/lib/orch-agent,/var/lib/orch-containerd, and/var/lib/orch-ingress; creates theorch-ingressaccount; sets Caddy's low-port capability; writes the containerd configuration and both units; enables IPv4 forwarding; loadsoverlayandwireguardand makes them persistent; installs the veth link policy and reloads udev rules (without reapplying them to existing interfaces). On Fedora with SELinux enabled, it adds file-context mappings for the runtime paths using the distribution'scontainer-selinuxtypes and restores labels. It never changes the SELinux mode. - Enrolls with the controller using the join token or file.
- Enables and starts
orch-containerd.serviceandorch-agent.service. - Waits until the agent reports an authenticated assignment and a ready
network (five minutes by default;
--wait-timeoutchanges it).
It leaves host and provider firewall rules unchanged.
If installation stops, rerun the same command with the same binary, address, and join token or file. It resumes from the first unfinished phase. A readiness timeout leaves everything installed and running. An install with a different address or binary is refused with "existing installation intent differs; upgrades and reconfiguration are unsupported".
The first node that joins a cluster becomes ingress-enabled automatically.
Later nodes accept public traffic only after
orch node ingress NODE --enable.
Upgrades
orch cluster upgrade and orch node upgrade replace the managed orch binary
in place. Before the switch, a controller upgrade writes a recovery set below
/var/lib/orch-controller-install/upgrades/ and needs 512 MiB free; a worker
upgrade copies the agent state below /var/lib/orch-install/upgrades/ and
needs 256 MiB free. A worker upgrade restarts the agent while containers keep
running under containerd. After a controller upgrade, the controller upgrades
each worker this way, staging the downloaded binary below
/var/lib/orch-install/upgrades/staged/ and running orch node upgrade in the
transient orch-agent-upgrade systemd unit. See
upgrades.
Removing orch
orch has no uninstall command. To retire a worker, drain it and remove it from
the cluster with orch node drain and orch node remove, then reinstall or
discard the host. Local volume data under /var/lib/orch-agent/volumes is not
moved for you; copy it off first if you need it.