Run lifecycle

A run is one execution of a workflow. It has an id (<org>:<project>:run-<random>), a durable event log, a status, and zero or more child runs. This page covers the states a run moves through, what cancel does, and how the substrate handles agent and server failures.

Status

A run is in one of four statuses:

Status	Meaning
running	Still executing, or waiting on something.
completed	Finished successfully.
failed	An activity or the workflow itself errored terminally.
cancelled	Cancelled by an operator.

completed, failed, and cancelled are terminal. A terminal run never goes back to running.

A run’s page opens on that status, then a timeline with one bar per step - where the time went, and which step was still going when the run ended:

Substate

While a run is running, the server derives a finer substate from its event log so the UI can show what the run is doing right now. At most one substate is open at a time.

Substate	The run is…
queued	Nothing scheduled yet - the run exists but has no activity in the queue.
executing	Running one or more activities on agents.
waiting_for_agent	An activity is scheduled but no agent has claimed it, and nothing else of the run is claimed either - the fleet is busy.
unroutable	Same, except no agent in the fleet advertises the capabilities the activity requires:, so waiting will not resolve it. Carries the required capabilities. Provision a runner that advertises them, or fix the job’s requires:.
sleeping	Parked on a cryo sleep timer; carries the resume time.
awaiting_signal	Blocked on cryo wait-signal or a YAML approval gate.
awaiting_child	Waiting on a child workflow to finish.
awaiting_call	Waiting on a run started with cryo call.
awaiting_lease	Waiting to acquire a concurrency lease (a job’s concurrency: group or cryo lock) held by another run; carries the group.

Approval gates surface as awaiting_signal - an approval is a signal whose name is the approval job’s name. Substate is only computed for running runs; terminal runs don’t have one.

Lifecycle events

When a root run reaches a terminal status it publishes an event on the project bus, so a handler can react to it:

• run.started - the run began
• run.succeeded - adds output
• run.failed - adds error
• run.cancelled - adds reason

All four carry run_id, status, workflow, the run’s tags / tag, trigger (what started it) and run_url. See lifecycle events.

Any handler can subscribe. Point one at run.failed for a failure notifier, at run.succeeded to fan out on completion, or at run.* for all four. These are platform-emitted, so cryo emit can’t forge them. See events and handlers for the handler side.

Cancelling

cryo cancel <run-id> --reason "superseded by a newer build"

Cancel does two things. It appends a terminal cancelled event to the run’s log, and it purges the run’s own pending and leased activities from the dispatch queue. Purging the queue matters: without it, an activity whose agent died before reporting - including one that killed its own agent - would re-dispatch forever on lease expiry.

What happens to the children

A run that reaches a terminal state cancels every child it was still waiting on. That is the workflow: job, the workflow node, and cryo call graph / cryo call node - everything the parent suspends for. Their result has nowhere to go once the parent has stopped, and until they notice they keep an agent claimed and hold whatever lease their jobs took.

It applies to any terminal outcome, not only to cryo cancel: a run that fails on one branch while another waits on a child leaves the same orphan. Each cancelled child does the same for its own children, so a cancel_on: event or one cryo cancel reaches the whole tree. A child stopped this way records parent <run-id> is no longer running as its reason.

The child’s own page says why it stopped, so a run nobody submitted and nobody cancelled by hand still explains itself:

Polyglot fan-out children go too. A cancelled run’s cryo activity-submit children are purged from the dispatch queue with its own activities, so a cancelled batch stops occupying agents instead of working through the rest of its list. One consequence to expect: a purged child never reports, so cryo activities <run-id> shows it as running from then on - the run is over, and nothing will come back to say what it was doing.

cryo spawn children are not cancelled. Spawning is fire-and-forget - a dispatcher that starts a hundred pipelines and exits is the point of the verb - so a spawned run outlives its spawner. Cancel one directly if you want it stopped.

Children that already finished are left alone, so cancelling a mostly-done run doesn’t rewrite what its completed steps recorded.

When an agent dies

An in-flight activity holds a 30-second lease. If the agent stops renewing it (crash, network partition, kill), the lease expires, the activity returns to the queue, and another capable agent claims it. The activity re-runs from the start. Completed cryo step boundaries return their cached output on replay, so the re-run skips work already done.

An agent run with cryo agent run also caps any single activity at one hour. A job that wedges past that ceiling is abandoned so it can’t hold the agent’s sequential claim slot forever.

When the server dies

The event log is durable. On restart the server picks up every run that had not reached a terminal status and continues it from its recorded history, with no operator action. A run that was mid-activity resumes waiting on that activity; a parked cryo sleep or cryo wait-signal resumes parked.

Terminal runs are never resurrected. Resume scans the whole log to classify a run, so a completed, failed, or cancelled run stays terminal across restarts even if stray queue rows remain.

Retries and timeouts

Two kinds of failure recovery, and they don’t overlap.

Infrastructure failure recovers automatically - no config. If an agent or the server dies mid-run (the two sections above), the run is re-dispatched and replays from the log. Completed cryo step / stepFn boundaries return their recorded results instead of re-running, so recovery is cheap and only the unfinished tail re-executes. You don’t ask for this and can’t turn it off; it’s what “durable” means here.

A step that fails is your result, and it isn’t retried unless you say so. When a command exits non-zero, that’s a workflow-level outcome - the run fails at that job. Cryosleep doesn’t silently re-run it. To retry on failure, opt in:

• YAML steps: and script: jobs take a retry: block - attempts (total, including the first) and backoff (delay before attempt 2, doubling after). See the YAML reference. A run out of attempts fails.
• Fan-out activities take --retries and --retry-backoff on cryo activity / cryo activity-submit, and --retry-on-exit to narrow which exit codes are worth another attempt. See the CLI reference. Unlike a job, a fan-out child out of attempts does not fail the run by itself: its last attempt’s output and exit code are what cryo activity-collect returns, and the calling script decides what that means.

A timeout caps how long one job’s activity may run. A job’s timeout: (humantime) overrides the agent’s default ceiling (one hour on an agent started with cryo agent run; the in-process agents behind cryo run / cryo dev don’t apply it). On expiry the agent kills the job’s process group and the activity fails - which then feeds retry: if set.

What durability does not do: it won’t turn a failing command into a success by replaying it, and it won’t retry your logic on a non-zero exit. Replay re-runs the program and serves completed steps from the log; retries (when you ask for them) re-run the failed unit; timeouts bound a stuck one. A determinism violation between steps fails loudly rather than being retried, because replaying it would fail the same way.

Reconcile sweep

Appending a run’s terminal event and purging its queue rows are two separate steps. A crash between them could leave queue rows behind for a run that’s already terminal. A reconcile sweep closes that window: it runs once at startup and then on a slow cadence, purging queue rows whose owning run has reached a terminal status. Combined with cancel’s own purge, an already-finished run can’t keep re-dispatching activities.

Retention

Finished runs don’t live forever. An hourly sweep deletes terminal runs - with their full history, logs metadata, and annotations - once they’re older than the retention window, and bus events past their own window. Running runs are never touched, however old; a six-month soak is safe. Age counts from when the run finished, not from when it was submitted.

How long runs are kept is set per org, and comes from the first of these that applies: a window pinned on the org, the org’s plan, then the server-wide default (CRYOSLEEP_RETENTION_RUNS, 90 days, or off to keep everything). Bus events follow the server-wide CRYOSLEEP_RETENTION_EVENTS alone, 30 days by default. Both take humantime values.