Jobs and schedules
Not every workload runs forever. orch has three ways to run a process that starts, does its work, and exits:
| Kind | Started by | Defined in | Typical use |
|---|---|---|---|
| One-off job | orch job run |
The command line, using a service's settings | Data fixes, admin scripts, manual migrations |
| Pre-deploy hook | Every new revision | deploy.preDeploy in orch.pkl |
Database migrations before a rollout |
| Cron job | A five-field UTC schedule | cronJobs in orch.pkl |
Reports, cleanup, backups |
All three share one runtime path on the worker: an isolated container with its own resources, a timeout, recorded state and exit code, and logs you can read afterwards. None of them is retried automatically, because the first attempt may already have changed external state. When a run fails, you decide whether to run it again.
Run states
Jobs and cron runs report the same states. The CLI prints them with a
JOB_RUN_STATE_ prefix, for example JOB_RUN_STATE_SUCCEEDED.
| State | Meaning |
|---|---|
WAITING |
Accepted, waiting for its node, its dependencies, or its start |
RUNNING |
The process is running |
SUCCEEDED |
The process exited with code 0 |
FAILED |
Nonzero exit, cancellation, or an outcome that could not be confirmed |
TIMED_OUT |
The run exceeded its timeout, or a one-off job missed its start deadline |
SKIPPED |
A cron occurrence that did not start; see overlap and missed runs |
Each run also carries a reason:
| Reason | Meaning |
|---|---|
assignment_pending |
The controller has placed the job and is waiting for the worker |
dependency_wait |
Waiting for the services in dependsOn to become available |
process_running |
Running |
exit_zero |
Exited successfully |
exit_nonzero |
Exited with a nonzero code; the exit code is recorded |
timeout_exceeded |
Stopped after its timeout |
start_deadline_exceeded |
Could not start before its start deadline |
overlap_forbidden |
Cron only: the previous occurrence was still running |
cancelled |
Cancelled with orch job cancel |
start_outcome_unknown |
The worker restarted while starting the process and could not confirm whether it ran |
runtime_object_missing |
The container disappeared before its result was recorded |
One-off jobs
A one-off job runs a command with the settings of an existing service in the current revision of an application: its image, plain and secret environment, secret files, resources, placement constraints and dependencies. It does not receive the service's ports or volume mounts.
Run a job
Put the command after --:
orch job run shop web --target production -- ./bin/rake orders:reindexIdempotency key: 6f1c0b9e5a7d4c2f9e8b1a3d5c7e9f01
job_3e8a... shop/web JOB_RUN_STATE_WAITING assignment_pending
revision=... image=ghcr.io/acme/shop@sha256:... node=... kind=manualThe first word after -- replaces the image's entry point; the rest become
its arguments. Use a shell explicitly when you need shell features:
orch job run shop web --target production -- /bin/sh -c 'bin/export > /tmp/out && wc -l /tmp/out'Useful flags:
| Flag | Default | Purpose |
|---|---|---|
--wait |
off | Wait for the run to finish and exit non-zero if it did not succeed |
--wait-timeout |
20m |
How long --wait waits |
--timeout |
15m |
Process timeout, from 10s to 24h |
--revision |
current | Refuse to start unless this is still the application's current revision |
--idempotency-key |
generated | Make the request safe to retry |
--json |
off | Machine-readable output |
--wait is convenient in scripts and CI:
orch job run shop web --wait --timeout 30m --wait-timeout 35m \
--target production -- ./bin/rake orders:reindexRetrying the request safely
orch job run prints its idempotency key to stderr before it contacts the
controller. If the command is interrupted or the network fails, run it again
with the same key. The controller returns the existing run instead of starting
a second one:
orch job run shop web --idempotency-key 6f1c0b9e5a7d4c2f9e8b1a3d5c7e9f01 \
--target production -- ./bin/rake orders:reindexInspect, wait, and read logs
orch job list --target production
orch job list shop --limit 50 --target production
orch job show JOB_ID --target production
orch job wait JOB_ID --timeout 1h --target production
orch job logs JOB_ID --tail 500 --target productionorch job listshows the newest jobs first, 20 by default and at most 100. When more exist, it printsNext page: --before ID.orch job showprints the state, reason, revision, image, node, kind (manualorpre_deploy) and, once finished, the exit code.orch job waitwaits up to--timeoutand exits non-zero unless the job succeeded.orch job logsprints the last--taillines (default 100) of stdout to stdout and of stderr to stderr.
The TUI (press o on an application) and the orch web dashboard also list
one-off jobs.
Cancel and retry
orch job cancel JOB_ID --target production
orch job retry JOB_ID --target productionCancelling stops the process and marks the run FAILED with reason
cancelled.
orch job retry creates a new run with a new ID that refers to the original.
It reuses the original image, command, arguments and secret versions, so a
retry repeats exactly what was attempted, even if the secrets were rotated
since. A retry is refused when:
- the original run succeeded (start a new run instead);
- the original run has not finished (cancel it and wait first);
- the application has a newer revision than the original run.
Placement and limits
The controller places a job when you submit it, on the first connected, ready
worker that matches the service's constraints and has enough free CPU and
memory for the service's resources. If no worker qualifies, the request
fails instead of waiting. At most 32 one-off jobs can be active in the cluster
at once.
The job waits up to one minute for the services in dependsOn to become
available; after that it ends as TIMED_OUT with start_deadline_exceeded.
A job stays on its worker. If that worker disconnects, orch does not move the
job or start it elsewhere, because the process may already have run. When the
worker returns, it reports the real outcome. If it restarted while starting
the process and cannot tell whether it ran, the run fails with
start_outcome_unknown and you decide whether to retry.
Job records are kept on the controller. Logs and containers of finished jobs stay on the worker, which keeps the newest 100 completed one-off runs.
When to use orch exec instead
orch exec runs a command inside a running replica, which suits interactive
inspection:
orch exec shop web --replica 1 --target production -- /bin/shUse orch job run when the work should run in its own container with its own
resources, be recorded, and keep its logs and exit code.
Pre-deploy hooks
A pre-deploy hook is a one-off job that orch starts automatically for every new revision of an application. The new revision rolls out only after the hook exits with code 0. It is the standard place for database migrations.
deploy {
preDeploy = new PreDeployHook {
service = "web"
command { "./bin/migrate" }
args { "up" }
timeout = "5m"
}
}| Field | Default | Rules |
|---|---|---|
service |
required | A service in this manifest |
command |
required | 1 to 256 entries; replaces the image's entry point |
args |
empty | At most 256 entries |
timeout |
"15m" |
10s to 24h |
What happens during a deployment
orch deploystores the new revision.- The controller starts the hook with the named service's settings from the new revision, so it runs the new image with the new configuration.
- While the hook runs, the application is
BLOCKEDwith reasonpre_deploy_running. All services and cron jobs of the previous revision keep running unchanged. - When the hook succeeds, the rollout proceeds as usual.
Because the old revision keeps serving while the hook runs, and keeps serving if the rollout is rolled back, migrations must be compatible with the code that is already running. Add columns and tables before code uses them, and remove them only in a later release.
The hook runs once per revision, including revisions created by
orch rollback when the restored configuration has a hook. Rotating a secret
does not create a revision and does not run the hook.
If no worker has capacity for the hook, or a referenced secret is missing, the
application is BLOCKED with reason pre_deploy_waiting until the problem is
resolved.
When the hook fails
If the hook exits non-zero, times out, or misses its start deadline, the
deployment is FAILED with reason pre_deploy_failed, and the previous
revision keeps running. orch deploy --wait reports the failure.
orch app status shop --target production
orch job list shop --target production
orch job logs JOB_ID --target productionFix the cause and then either:
retry the same hook, which resumes the same deployment if it succeeds:
orch job retry JOB_ID --target productionor deploy a corrected revision, which runs its own hook.
Deploying a newer revision while a hook runs requests cancellation of the old hook. The new hook starts after the old one has stopped.
Things to plan for
- The hook needs free CPU and memory for the service's
resourcesin addition to the running replicas. - The hook's
dependsOnservices must already be available within one minute, because the new revision's services do not start until the hook succeeds. If an application's database is one of its own services, deploy the database before you add a hook that depends on it. - The hook does not receive the service's volume mounts.
orch deploy --waitwaits 10 minutes by default. For long migrations, raise it with--timeout.- orch cannot guarantee that a migration ran exactly once. Use a migration tool that records what it applied and is safe to run again.
Cron jobs
A cron job is a template for a scheduled one-shot run. It is defined under
cronJobs and deployed with the rest of the application.
cronJobs {
["cleanup"] {
image = "ghcr.io/acme/shop:2.3.0"
schedule = "0 2 * * *"
command { "./bin/rake" }
args { "sessions:cleanup" }
secretEnv {
["DATABASE_URL"] = "database-url"
}
dependsOn { "database" }
timeout = "10m"
}
}Unlike a one-off job, a cron job has its own image, environment, secrets,
resources, constraints and dependencies. Every field is listed in the
manifest reference.
Schedules
Schedules use exactly five fields, always in UTC:
┌───────── minute (0-59)
│ ┌─────── hour (0-23)
│ │ ┌───── day of month (1-31)
│ │ │ ┌─── month (1-12)
│ │ │ │ ┌─ day of week (0-6, Sunday = 0)
│ │ │ │ │
0 2 * * *| Schedule | Runs |
|---|---|
*/15 * * * * |
Every 15 minutes |
0 * * * * |
At the start of every hour |
30 2 * * * |
Every day at 02:30 UTC |
0 6 * * 1-5 |
Weekdays at 06:00 UTC |
0 3 1 * * |
The first day of every month at 03:00 UTC |
Descriptors such as @daily, a sixth field for seconds, and CRON_TZ= or
TZ= prefixes are rejected by orch validate. Convert local times to UTC
yourself, and remember that daylight saving time does not move a UTC schedule.
Where cron jobs run
Each cron job has one owner worker, chosen by the scheduler like a service
replica: it must match the job's constraints and have room for its
resources, which stay reserved on that worker. The owner keeps the schedule
and starts runs from its own copy of the assignment, so schedules continue
while the controller is unavailable.
Ownership is sticky. If the job must move, for example because its constraints changed or the worker is drained, the old owner finishes any active run and releases the schedule before the new owner takes over. A worker that cannot be reached keeps the job until it is confirmed gone.
Timeouts and deadlines
| Field | Default | Range | Meaning |
|---|---|---|---|
startDeadline |
"1m" |
1s to 1h | How long after the scheduled time a run may still start, for example while waiting for dependsOn |
timeout |
"15m" |
10s to 24h | How long a started run may take before it is stopped |
Overlap and missed runs
- No overlap. If the previous run is still active when the next occurrence is
due, the new occurrence is recorded as
SKIPPEDwithoverlap_forbidden. - No catch-up. Occurrences missed while the owner was down or busy are not
replayed. One that cannot start within
startDeadlineisSKIPPEDwithstart_deadline_exceeded. - No retry. A failed or timed-out run waits for the next occurrence.
- Exactly-once execution is not guaranteed. Make jobs safe to run twice, and safe to skip once.
If a job must run at least once a day, schedule it more often than that and let it exit early when its work is already done.
Inspect runs and logs
orch cron runs shop cleanup --target productionjob_9c1f... 2026-10-14T02:00:00Z JOB_RUN_STATE_SUCCEEDED reason=exit_zero
job_47ab... 2026-10-13T02:00:00Z JOB_RUN_STATE_FAILED reason=exit_nonzeroRead the output of one run by its ID:
orch cron logs shop cleanup RUN_ID --tail 500 --target productionorch cron runs shows 20 runs by default; --limit accepts 1 to 100. Each
worker keeps the latest 20 runs of each cron job, including their logs, and
removes older ones. Copy anything you need to keep longer into your own log
storage. The TUI and orch web also show recent cron runs.
There is no command to trigger a cron job outside its schedule. To run the
same work now, use orch job run with a service that has the same image, or
temporarily deploy a more frequent schedule.
Reading local volumes from cron jobs
A cron job can mount a service's local volume read-only. This is the way to back up data that lives on one worker, such as a database's data directory or a SQLite file, next to the data itself.
services {
["database"] {
image = "postgres:17"
secretEnv {
["POSTGRES_PASSWORD"] = "postgres-password"
}
tcp = new Tcp { port = 5432 }
mounts {
new Mount {
source = "pgdata"
target = "/var/lib/postgresql/data"
}
}
}
}
volumes {
["pgdata"] {}
}
cronJobs {
["pgdata-archive"] {
image = "ghcr.io/acme/backup:1.0"
schedule = "0 3 * * *"
mounts {
new Mount {
source = "pgdata"
target = "/backup/source"
readOnly = true
}
}
}
}The rules:
- Every mount must set
readOnly = true. The worker also refuses a writable cron mount and binds the volume read-only. - Each mounted volume must be declared under
volumesand mounted by a service in the same application. - All volumes mounted by one cron job must belong to the same service, so they are on the same worker.
- Each run is placed on the worker that holds the volume. The job's own
constraintsandresourcesstill apply; if they exclude that worker, the job is blocked withlocal_volume_pinned. - Until the owning service has been placed and its volume exists, the job is
not scheduled and the application reports
cron_volume_pending.
The service keeps writing while the job reads, so copying files can capture them mid-write. Use a method that is consistent for your data, such as the database's own backup tool or a SQLite online backup, rather than a plain file copy of a busy database.
Examples
Database migrations
Run migrations from the application image before each rollout:
amends "https://orch.torsten.work/pkl/v1/Application.pkl"
name = "shop"
services {
["web"] {
image = "ghcr.io/acme/shop:2.4.0"
replicas = 2
secretEnv {
["DATABASE_URL"] = "database-url"
}
http = new Http {
port = 8080
domains { "shop.example.com" }
healthPath = "/healthz"
}
}
}
deploy {
preDeploy = new PreDeployHook {
service = "web"
command { "./bin/migrate" }
args { "up" }
timeout = "10m"
}
}orch plan
orch deploy --wait --timeout 20m --target productionTo run a one-time data migration outside a release, use a one-off job and wait for it:
orch job run shop web --wait --timeout 2h --wait-timeout 2h \
--target production -- ./bin/migrate data 2026_10_backfill_totalsNightly logical backup
When the database is reachable over the network, a cron job can stream a dump
to off-site storage. Here pg_dump connects to the database service through
its cluster DNS name:
cronJobs {
["nightly-backup"] {
image = "ghcr.io/acme/pg-backup:17"
schedule = "15 1 * * *"
command { "/bin/sh" }
args {
"-c"
"pg_dump --format=custom \"$DATABASE_URL\" | aws s3 cp - \"s3://acme-backups/shop/$(date -u +%F).dump\""
}
env {
["AWS_REGION"] = "eu-west-1"
}
secretEnv {
["DATABASE_URL"] = "backup-database-url"
["AWS_ACCESS_KEY_ID"] = "backup-aws-key-id"
["AWS_SECRET_ACCESS_KEY"] = "backup-aws-secret"
}
dependsOn { "database" }
resources {
cpu = "500m"
memory = "512Mi"
}
timeout = "2h"
}
}Store the secrets first:
printf '%s' 'postgres://backup:[email protected]:5432/shop' |
orch secret set shop backup-database-url --from-stdin --target production
printf '%s' "$AWS_ACCESS_KEY_ID" |
orch secret set shop backup-aws-key-id --from-stdin --target production
printf '%s' "$AWS_SECRET_ACCESS_KEY" |
orch secret set shop backup-aws-secret --from-stdin --target productionRun it once right away to check it, then watch the scheduled runs:
orch cron run shop nightly-backup --wait --target production
orch cron runs shop nightly-backup --target production
orch cron logs shop nightly-backup RUN_ID --target productionA backup is only useful if it can be restored. Test restores regularly, and alert when a run fails, for example by having the job report success to an external monitor as its last step.
Archiving a SQLite file
For data that only exists in a local volume, mount the volume read-only and use the database's own safe copy method:
cronJobs {
["sqlite-backup"] {
image = "ghcr.io/acme/sqlite-backup:1.0"
schedule = "0 */6 * * *"
command { "/bin/sh" }
args {
"-c"
"sqlite3 'file:/data/app.db?mode=ro' \".backup /tmp/app.db\" && upload /tmp/app.db"
}
mounts {
new Mount {
source = "app-data"
target = "/data"
readOnly = true
}
}
timeout = "30m"
}
}Here upload stands for whatever tool your image uses to ship the copy
off the worker. The app-data volume must be mounted by a service in the same
application.
See cluster operations for a short overview and the CLI reference for every flag.