Tutorial: deploy a web app with a database
This tutorial picks up where the quickstart ends. You will deploy a small but complete application: a web service with two replicas behind HTTPS, a PostgreSQL database that keeps its data on one worker, secrets for the database credentials, a nightly cron job, and a migration that runs before each rollout. Then you will plan, deploy, inspect, update, and roll back the application, the same way you would in production.
Each step shows the exact commands. It takes about 30 minutes, most of it waiting for images and certificates.
What you will build
The application is called notes and has these parts:
| Part | Kind | Details |
|---|---|---|
web |
Service | Two replicas of your web image on port 8080, published at https://notes.example.com |
database |
Service | One PostgreSQL 17 replica with a local volume, pinned to a node labeled role=database |
database |
Volume | Local directory that holds the PostgreSQL data |
database-password, database-url |
Secrets | Stored encrypted on the controller and injected as environment variables |
cleanup |
Cron job | Runs ./bin/notes cleanup every night at 02:00 UTC |
| Pre-deploy hook | Hook | Runs ./bin/notes migrate before each new revision rolls out |
The web service reaches the database through cluster DNS at
database.notes.svc.internal. Only the web service is public.
Before you begin
You need:
- A cluster with a controller and at least one worker, set up as in installation. Two workers make the tutorial more interesting: one runs public traffic and the other holds the database.
- A target for the cluster on your workstation. This tutorial assumes it is
named
production. - Pkl 0.32.1 on your workstation's
PATH.orch validate,orch plan, andorch deployevaluateorch.pklwith it.mise installin the orch repository provides it. - A domain you control. This tutorial uses
notes.example.com. - An OCI image for the web application. This tutorial uses
ghcr.io/acme/noteswith tags1.0.0and1.1.0; replace it with your own. The image must:- listen for HTTP on port 8080;
- return a 2xx or 3xx status from
/healthzwithout depending on the database being reachable; - read the database connection string from
DATABASE_URL; - provide
./bin/notes migrate, which applies schema migrations and is safe to run more than once, and./bin/notes cleanup, the nightly task.
If your image is in a private registry, store its credentials first:
printf '%s' "$REGISTRY_TOKEN" | orch registry login ghcr.io \
--username robot --password-stdinSelect the target so you do not have to repeat --target production on every
command. In scripts, pass --target explicitly instead.
orch target use production
orch target test
orch nodesorch nodes lists each worker with its state, capacity, labels, and whether
it accepts ingress (ingress-role=true).
Step 1: Prepare the nodes
Choose the database node
A local volume lives on one worker. Choose which one by labeling it, and
constrain the database service to that label. Here worker-2 holds the
database:
orch node label worker-2 role=database
orch nodesThe labels= field for worker-2 now shows role=database. With a single
worker, label that worker instead.
Without a constraint, orch places the database wherever capacity allows the first time and then keeps it there because of its volume. The label makes the choice explicit, so you know which host to back up.
Check ingress and DNS
The first worker that joined the cluster accepts public traffic automatically. To use another one, enable ingress on it:
orch node ingress worker-1 --enableCreate a DNS A record for notes.example.com that points at the address of
each ingress-enabled worker, and allow TCP 80 and 443 to those workers in your
host and provider firewalls. Caddy on the ingress node requests the
certificate automatically, and the certificate authority must be able to reach
the node on those ports. Check the record from your workstation:
dig +short notes.example.comSee HTTPS for details.
Step 2: Store the secrets
The database needs a password, and the web service needs a connection string that includes it. Store both as application secrets. Values go through stdin, so they never appear in your shell history or the process list:
DB_PASSWORD="$(openssl rand -hex 24)"
printf '%s' "$DB_PASSWORD" | orch secret set notes database-password --from-stdin
printf '%s' "postgres://notes:[email protected]:5432/notes" \
| orch secret set notes database-url --from-stdin
orch secret list notesSecrets belong to an application by name, so you can store them before the
application exists. The controller encrypts them and delivers them only to
the workers that run a workload referencing them. orch.pkl refers to them by
name and can stay in source control.
The host name database.notes.svc.internal follows the pattern
SERVICE.APPLICATION.svc.internal. Every node runs a DNS server that answers
these names with the addresses of ready replicas. See
service discovery.
Step 3: Write the application
Create a directory for the application and an orch.pkl file in it. The
sections below build the file piece by piece; the complete file follows.
The web service
amends "https://orch.torsten.work/pkl/v1/Application.pkl"
name = "notes"
services {
["web"] {
image = "ghcr.io/acme/notes:1.0.0"
replicas = 2
resources {
cpu = "250m"
memory = "256Mi"
}
env {
["APP_ENV"] = "production"
}
secretEnv {
["DATABASE_URL"] = "database-url"
}
dependsOn { "database" }
http = new Http {
port = 8080
domains { "notes.example.com" }
healthPath = "/healthz"
}
}
}replicas = 2runs two instances. orch spreads them across eligible workers when capacity permits, and sends traffic only to ready instances.resourcesis what the scheduler reserves on a node.memoryis also the container's memory limit.secretEnvmaps the environment variableDATABASE_URLto the stored secretdatabase-url.dependsOnmakes the web replicas wait until the database is available before they start.httppublishes the service. Withdomains, orch routesnotes.example.comthrough the ingress nodes.httpsdefaults totrue, so the route is served over HTTPS with an automatic certificate.healthPathis required when a service has domains. orch sends an HTTP GET to it every two seconds; see health checks.
The database service and its volume
Add the database to services and declare its volume at the top level:
["database"] {
image = "docker.io/library/postgres:17-alpine"
resources {
cpu = "500m"
memory = "1Gi"
}
env {
["POSTGRES_DB"] = "notes"
["POSTGRES_USER"] = "notes"
}
secretEnv {
["POSTGRES_PASSWORD"] = "database-password"
}
constraints {
["role"] = "database"
}
mounts {
new Mount {
source = "database"
target = "/var/lib/postgresql/data"
}
}
tcp = new Tcp {
port = 5432
}
}volumes {
["database"] {}
}constraintsis an exact match on node labels. The database only runs on the node you labeledrole=database. If no node matches, the service stays blocked instead of running somewhere else.mountsattaches the local volumedatabaseat PostgreSQL's data directory. A service with a local volume must have exactly one replica, which is the default.tcpdeclares the port other services connect to and gives the database a TCP readiness check, so the web service waits until PostgreSQL accepts connections. It has nopublicPort, so the database is not reachable from outside the cluster.
If the worker that holds the volume is lost, orch reports volume_node_lost
and does not start an empty database elsewhere. Back up the data with the
database's own tools; see local volumes.
The nightly cron job
cronJobs {
["cleanup"] {
image = "ghcr.io/acme/notes:1.0.0"
schedule = "0 2 * * *"
command { "./bin/notes" }
args { "cleanup" }
secretEnv {
["DATABASE_URL"] = "database-url"
}
dependsOn { "database" }
timeout = "10m"
}
}The schedule uses five cron fields in UTC, so this runs every day at 02:00
UTC. A run that is still going when the next one is due causes that next one
to be skipped, and runs missed while a node was down are not replayed.
timeout stops a run that takes longer than ten minutes.
Rollout settings
deploy {
maxSurge = 1
maxUnavailable = 0
healthDeadline = "2m"
autoRollback = true
}During an update, orch may start one extra web replica (maxSurge = 1) and
never takes a ready one away before its replacement is ready
(maxUnavailable = 0). Each new replica has two minutes to become ready. If
the update fails, autoRollback returns to the last healthy revision.
The database restarts differently. A replica with a local volume is always stopped before its replacement starts, so the same data directory is never used by two processes. Expect a short database interruption whenever its definition changes.
The complete file
amends "https://orch.torsten.work/pkl/v1/Application.pkl"
name = "notes"
services {
["web"] {
image = "ghcr.io/acme/notes:1.0.0"
replicas = 2
resources {
cpu = "250m"
memory = "256Mi"
}
env {
["APP_ENV"] = "production"
}
secretEnv {
["DATABASE_URL"] = "database-url"
}
dependsOn { "database" }
http = new Http {
port = 8080
domains { "notes.example.com" }
healthPath = "/healthz"
}
}
["database"] {
image = "docker.io/library/postgres:17-alpine"
resources {
cpu = "500m"
memory = "1Gi"
}
env {
["POSTGRES_DB"] = "notes"
["POSTGRES_USER"] = "notes"
}
secretEnv {
["POSTGRES_PASSWORD"] = "database-password"
}
constraints {
["role"] = "database"
}
mounts {
new Mount {
source = "database"
target = "/var/lib/postgresql/data"
}
}
tcp = new Tcp {
port = 5432
}
}
}
volumes {
["database"] {}
}
cronJobs {
["cleanup"] {
image = "ghcr.io/acme/notes:1.0.0"
schedule = "0 2 * * *"
command { "./bin/notes" }
args { "cleanup" }
secretEnv {
["DATABASE_URL"] = "database-url"
}
dependsOn { "database" }
timeout = "10m"
}
}
deploy {
maxSurge = 1
maxUnavailable = 0
healthDeadline = "2m"
autoRollback = true
}Step 4: Validate and plan
Check the file:
orch validatevalid notes (2 services, 1 cron jobs) sha256:...orch validate evaluates the Pkl file and checks every value, such as names,
ports, durations, and references to volumes and services, without contacting
the cluster. It reports each problem with its path, for example
services.database.replicas: must be 1 when the service uses local volumes.
Then look at the plan:
orch planPlan for notes (sha256:...)
+ application: "notes"
+ deploy: {"max_surge":1,"max_unavailable":0,"health_deadline":"2m","auto_rollback":true}
+ services.database: {...}
+ services.web: {...}
+ cron_jobs.cleanup: {...}
+ volumes.database: {...}
Plan: 6 to add, 0 to change, 0 to remove.orch plan works locally. Without --against, everything is new. You will
compare two versions of the file in step 8. Use orch render to see the full
resolved JSON that would be sent to the controller.
Step 5: Deploy
orch deploy --waitThe controller resolves ghcr.io/acme/notes:1.0.0 and postgres:17-alpine to
immutable digests, stores the result as revision 1, and starts the rollout.
With --wait, the command prints progress until the revision is ready or the
wait times out (ten minutes by default; change it with --timeout). Each
progress line shows the application status, then one line per replica with
its node, phase, and health.
In order, you will see:
- The database is placed on
worker-2, its image is pulled, and PostgreSQL initializes the empty volume with the password fromdatabase-password. - Once port 5432 accepts connections, the database is ready, and the web replicas, which depend on it, start.
- Each web replica becomes ready after two successful
/healthzchecks. The route fornotes.example.comis published once the service is healthy.
The command ends with Ready: notes generation 1. Interrupting it does not
cancel the deployment; resume watching with orch app watch notes.
POSTGRES_PASSWORD only takes effect when PostgreSQL initializes an empty
data directory. Changing the secret later does not change the password of an
existing database.
Step 6: Run the first migration
The web service is running but the database is empty. Run the migration once as a one-off job, using the web service's image and environment:
orch job run notes web --wait -- ./bin/notes migrateorch prints the job ID, runs the command on a worker, and waits for it to finish. Read its output and check the result:
orch job logs JOB_ID
orch job show JOB_IDorch job list notes lists recent jobs for the application. In step 8 you
will make this migration run automatically before every rollout.
Step 7: Check status and logs
Application status
orch app status notesThe output starts with a summary:
Desiredis the newest generation and revision you submitted, andObservedis the generation the controller has acted on.Statusis the overall state, such asEXECUTION_STATUS_HEALTHYorEXECUTION_STATUS_BLOCKED, followed by aReasonandMessagewhen there is something to explain.Activeis the generation that is serving traffic.
Then it lists:
- one line per replica (
web/1,web/2,database/1) with its phase,health=, private address, restart count, andreason=; - one line per route, such as
route https://notes.example.com, with its state,certificate=, and number of ready upstreams; - one line per cron job with
next=, the time of the next run.
Open the site:
curl -I https://notes.example.comCaddy requests the certificate when the route is published, so the first requests can fail with a TLS error for a short while. If it keeps failing, see troubleshooting.
Logs
Read the last lines of each replica's output:
orch logs notes web --replica 1 --tail 100
orch logs notes web --replica 2 --tail 100
orch logs notes database --replica 1 --tail 50Replicas are numbered from 1, as in orch app status. Standard error is
written to your terminal's standard error.
For a live view of applications, replicas, events, and streaming logs, run
orch tui or open the browser dashboard with orch web.
Events
orch events --limit 20Events record changes made to the cluster: applied deployments and rollbacks, secret changes, node enrollment and updates, join tokens, and one-off jobs. Use them to see what changed, and when, before a problem started.
Run a command in a replica
orch exec runs a non-interactive command inside a running replica. Check the
database from inside its container:
orch exec notes database --replica 1 -- psql -U notes -d notes -c 'select 1'Try the cron job
The cleanup job will first run at 02:00 UTC. To try the same command now, run it as a one-off job:
orch job run notes web --wait -- ./bin/notes cleanupAfter the schedule has fired, list its runs and read the logs of one:
orch cron runs notes cleanup
orch cron logs notes cleanup RUN_IDStep 8: Ship an update
Version 1.1.0 of the image includes a schema change. This time, make the
migration part of the rollout so it cannot be forgotten.
Keep a copy of the deployed file so you can compare against it:
cp orch.pkl orch-deployed.pklEdit orch.pkl:
- Change the web image and the cron job image to
ghcr.io/acme/notes:1.1.0. - Add a pre-deploy hook to the
deployblock:
deploy {
preDeploy = new PreDeployHook {
service = "web"
command { "./bin/notes" }
args { "migrate" }
timeout = "5m"
}
maxSurge = 1
maxUnavailable = 0
healthDeadline = "2m"
autoRollback = true
}The hook runs ./bin/notes migrate as a one-off job with the new revision of
the web service: its new image, environment, and secrets. Replicas of the
new revision are only rolled out after the hook succeeds. While it runs, the
current replicas, including the database, keep running.
Because the hook holds back the whole application, it would not work on the very first deploy of an application whose database is part of the same file: the migration would run before the database exists. That is why step 6 ran the first migration by hand.
Validate and compare the new file with the deployed one:
orch validate
orch plan --against orch-deployed.pklPlan for notes (sha256:...)
~ cron_jobs.cleanup.image: "ghcr.io/acme/notes:1.0.0" -> "ghcr.io/acme/notes:1.1.0"
+ deploy.pre_deploy: {"args":["migrate"],"command":["./bin/notes"],"service":"web","timeout":"5m"}
~ services.web.image: "ghcr.io/acme/notes:1.0.0" -> "ghcr.io/acme/notes:1.1.0"
Plan: 1 to add, 2 to change, 0 to remove.The database is not in the plan. Deploy:
orch deploy --waitWhile the hook runs, orch app status notes shows the reason
pre_deploy_running and the job ID in its message. Follow the migration's
output with:
orch job list notes
orch job logs JOB_IDWhen the hook succeeds, orch replaces the web replicas one at a time: it
starts a 1.1.0 replica, waits for it to become ready, then stops a 1.0.0
replica. Traffic moves only to ready replicas, so the site stays up.
If a migration fails, the deployment stops with the reason
pre_deploy_failed and nothing from the new revision is rolled out. Read the
job's logs, fix the cause, then run the hook again with
orch job retry JOB_ID, or deploy a corrected file.
If the new replicas never become ready within healthDeadline, orch rolls
back automatically because autoRollback = true, and orch app history
records the outcome.
Step 9: Roll back
Suppose 1.1.0 passes its health checks but has a bug that users notice. List
the application's deployments:
orch app history notesEach line shows a generation, its revision ID, when it was created, its
outcome, and markers for the current and active generations. The images
of each revision follow, with tags resolved to digests.
Compare the revision you want to go back to with what is deployed now, then roll back to it:
orch app diff notes REVISION
orch rollback notes REVISION --waitA rollback creates a new generation from the stored revision and reuses its resolved image digests, so you get exactly the images that ran before, even if the tag has since moved. It goes through the same rolling update and health checks as a deploy.
Two things a rollback does not do:
- It does not undo database migrations. Write migrations so that the previous version of the application still works after they run, for example by adding columns before code uses them and removing them a release later.
- It does not change
orch.pkl. Your file still says1.1.0, and the nextorch deploywould roll forward again. Revert the change in your file or in source control before you deploy next.
Step 10: Back up the database
The database volume lives only on worker-2. Back it up with PostgreSQL's
own tools. For example, a cron job can mount the volume read-only and be
placed on the same node, or a cron job can run pg_dump against
database.notes.svc.internal and send the dump off the host. See
reading a local volume for the
read-only mount.
Also back up the controller, which holds the application's revisions and secrets; see backups.
Clean up
To remove the application and stop its workloads:
orch app status notes
orch remove notesRemoving an application does not delete the volume's data. It stays on
worker-2 in /var/lib/orch-agent/volumes/notes/database; keep or delete it
according to your retention policy. Remove the secrets and the node
label if you no longer need them:
orch secret delete notes database-url
orch secret delete notes database-password
orch node unlabel worker-2 roleNext steps
- Read the application manifest reference for every field.
- Learn more about secrets, jobs and schedules, and deployments.
- Keep troubleshooting at hand for when a rollout does not go as planned.