The authoring model
How durable workflows execute, and the rules that follow from it. Read this once before writing anything longer than a demo - the model is small, and every guideline below falls out of it.
Checkpoint and re-run
Section titled “Checkpoint and re-run”A workflow script is a normal program that the engine can run more
than once. Each durable call - step, sleep, wait-signal,
wait-event, activity-collect - is a checkpoint: its result is
recorded in the run’s event log the first time it completes. When
the run resumes (after a crash, a sleep, a signal, an agent
restart), the script executes again from the top. Durable calls
whose results are already in the log return them instantly, without
re-running anything; execution races through the recorded prefix and
picks up where it left off.
That’s the whole model. There is no bytecode capture, no special runtime, no requirement that your code be deterministic in any formal sense. The engine replays recorded results; it never re-executes recorded code.
Two consequences shape everything else:
- Between checkpoints, your code runs again on every pass. A log line between two steps prints on every resume. A counter incremented between steps counts passes, not events.
- The suspension points are real process exits. When a
sleepor a wait has to park, the script process is terminated and the agent slot is freed - a run sleeping for a month costs nothing. The call you wrote never returns on that pass; it returns on a later pass, from the log.
The rules
Section titled “The rules”Wrap side effects and nondeterministic reads in a step. Anything
that touches the outside world (deploys, API calls, file writes) and
anything a later decision depends on that could change between
passes - time, randomness, environment lookups, network reads -
belongs inside a step. The recorded result is then the same on
every pass, so decisions made from it are stable. Passes are not tied to
a machine: a run that suspends is dispatched again to whichever agent is
free, and the recorded result travels with the run, so a step’s value is
the same one no matter where the resume lands or how long it took.
Make steps idempotent where you can. A step that dies after acting but before its result is recorded will run again. “Create or update”, “push if absent”, version-stamped writes - the usual idempotency toolkit applies.
When the remote system dedupes for you - most payment, messaging and
ticketing APIs take an idempotency key or a transaction id - hand it a
token that stays the same across every way one call can happen twice.
On a graph node that’s node.idempotency_key:
{ "id": "charge", "type": "http", "config": { "method": "POST", "url": "https://api.example.com/charges", "headers": { "Idempotency-Key": { "$expr": "node.idempotency_key" } } } }It’s stable across a retry:, across a replay after a suspend, and
across a job re-dispatched because its agent died - all of those are the
same call site of the same run. Inside a fan_out it also carries the
item’s index, so ten notifications send ten keys rather than one. A
rerun of the run gets a new token, which is right: a rerun is a
deliberate new attempt.
In a script, the same token reaches a cryo step body as
$CRYO_IDEMPOTENCY_KEY:
cryo step charge -- ./charge.sh "$account"curl -fsS https://api.example.com/charges \ -H "Idempotency-Key: $CRYO_IDEMPOTENCY_KEY" \ -d "account=$1"It is scoped to that step of that job of that run, so it holds across a
retry:, a replay after a suspend and a re-dispatch to another agent,
and two steps get two keys. A fan-out item is its own job, so ten items
running the same charge step send ten different keys.
A step body deliberately sees none of the other CRYO_* variables - it
has no agent socket of its own, and a nested cryo step should fail
loudly rather than half-work. This one is put back afterwards because
it is a value to send onwards rather than a handle to call back with,
and it can’t be shadowed by a project secret of the same name.
Don’t reach for $CRYO_ACTIVITY_ID instead. It is visible in the script
body and it is one constant for the whole job, so three retries of a
dunning ladder would send one key, the provider would dedupe all three
into the first charge, and nothing would tell you the customer was never
charged again.
A YAML steps: job has no token of its own - its steps are one atomic
script with no durable boundaries between them, so the whole job re-runs
together. Put a call that must not double up in a script: job behind a
cryo step, or on a graph node.
For “has anyone already handled this record?” - the same webhook delivered twice, two schedules picking up one row - the answer isn’t a key you send anywhere, it’s a claim you take. See Durable state.
Keep between-step code cheap and repeatable. It re-runs on every
pass. Pure formatting, branching on recorded results, and building
argument lists are all fine; sending an email is not. The engine
enforces this: if a replay takes a different path and reaches a
different durable call than the first pass recorded at that point, it
fails with a determinism error instead of silently re-running. Branch on
values you captured in a step, not on the wall clock or fresh
randomness. A deployment that has to allow a divergence can set
CRYO_REPLAY_MODE=lenient on the agent, which removes the check for
everything that agent runs. When replay goes
wrong walks through a real determinism error,
how to find the branch that caused it, and what lenient mode actually
does to a run.
Names are identity where something refers back to them. These are unique per run, and a second use is rejected rather than silently returning the first call’s result:
activity <name>, becauseactivity-collect <name>retrieves that child’s result. Two submits under one name would make the second uncollectable.lock <group>with no command, because the lease is held to the end of the run - there is one of it, not one per call site.state cas/state incr, keyed by--as <name>(defaulting to the cell), because each attempt in a read-modify-write loop is its own question.
Everything else repeats. A step or stepFn name is its program
point, so a loop reaches one name as many times as it iterates and each
iteration gets its own memo; pass --label if you want the run view to
say which host or which attempt an iteration was for. A wait-signal
or wait-event name is a queue, so waiting on one twice takes the
first signal and then the second. And the primitives that carry no name
at all - sleep, emit, spawn - key off call order, so two sleeps
are two waits.
What all of them need is stability across passes: the same call reached in the same order every time. That is what replay is built on, and it is why a branch on wall-clock or randomness is a determinism error rather than a slightly different run.
A handler run started by an event is an ordinary workflow: it replays under the same rules as any run you submit yourself, so nothing here changes when a run is triggered off the bus rather than by hand.
Mind result sizes. Step results are stored in the event log. Record a path, an id, or a summary - not a build artifact.
What this buys you
Section titled “What this buys you”- Code changes don’t strand in-flight runs. Each run replays against the definition it started with, so deploying new workflow code leaves in-flight runs untouched - they finish on their pinned version, and new runs pick up the change. No migration step, no version-pinning ceremony.
- Any language works today. The durable calls are an exec away
(
cryo step …) or a socket write away (the SDKs); there’s no runtime to port. - Nothing to unlearn: a workflow reads exactly like the script you would have written anyway, with checkpoints at the moments that matter.
The trade: the engine cannot catch a rule violation for you. Code
between steps that misbehaves - an unwrapped random() steering a
branch, a side effect outside a step - produces a run that is
plausible but wrong rather than a loud error. The rules above are
short enough to actually follow.
Where code runs, per language
Section titled “Where code runs, per language”A polyglot script executes on an agent advertising the polyglot
capability, with the interpreter chosen by file extension or
--interpreter (see polyglot scripts). The
agent needs that runtime installed:
| Language | Runs as | Agent needs |
|---|---|---|
| bash | inline script (default) | nothing extra |
| Python | inline script, python3 |
python3 |
| TypeScript | inline script, bun run |
bun |
| JavaScript | inline script, node |
node |
| Go | go run (stdlib-only single file), or from a checkout / prebuilt binary when importing the SDK |
go toolchain |
| Rust | compiled binary, or cargo run from a checkout |
rust toolchain |
The Python, TypeScript, Go, and Rust SDKs speak the agent’s socket
directly; the bash prelude and plain cryo verbs do the same thing one
subprocess at a time. See workflows in your language
for the same workflow in several languages, and note that a
non-bash workflow needs its runtime (and, for the import path, its SDK)
present wherever the agent runs it.