When replay goes wrong

Every transcript on this page came from running the script beside it. Nothing here is illustrative.

A workflow that sleeps, waits for a signal, or survives a deploy is a program the engine runs more than once. Most of the time you never notice. This page is about the times you do: what the engine refuses, what the error is telling you, and how to find the line that caused it.

Read Authoring model first if you have not. The rules are short, and this page assumes them.

What a replay actually looks like

Start with the case where nothing is wrong, because the shape surprises people.

#!/usr/bin/env bash
set -euo pipefail
echo pass >> /tmp/pass-count
cryo step a -- echo one
cryo sleep 3s
cryo step b -- echo two

That file wrote two lines to /tmp/pass-count. The sleep evicted the script, and the resume ran it again from the top. cryo step a did not re-execute the second time - it returned its recorded result - but the echo outside any step did, because nothing recorded it.

That is the whole model in one file. Code between durable calls runs on every pass. Durable calls run once and are replayed from the log after that.

So the cost of getting this wrong is not an abstract correctness argument: it is an echo that happened twice, an email sent twice, a counter incremented twice.

The determinism error

Here is a script with a bug that looks reasonable:

#!/usr/bin/env bash
set -euo pipefail

if [ -f /tmp/cryo-marker-demo ]; then
  cryo step reuse-cache -- echo "cache hit"
else
  touch /tmp/cryo-marker-demo
  cryo step build -- echo "built it"
fi

cryo sleep 2s
echo "past the sleep"

It branches on a file, and the branch it takes creates that file. First pass: the marker is absent, so it builds. After the sleep the script replays, the marker now exists, and it takes the other branch:

$ cryo run diverge.sh
  run: dev:default:run-fqBeywDeCOqFqYKm
built it
cryo: script suspended for 2000ms; agent re-dispatch on wakeup

  - replaying from checkpoint (completed steps are cached, not re-run)
cryo: determinism violation at durable op #0: this run recorded "build" here,
but on replay the script reached "reuse-cache". A durable workflow must take
the same path between durable calls on every pass — a branch on something that
changes (wall-clock, randomness, an unmemoized read) diverges like this. Wrap
the nondeterministic value in a step, or run lenient
(CRYO_REPLAY_MODE=lenient) if the divergence is intentional.
  dev:default:run-fqBeywDeCOqFqYKm failed: script exited 1

Reading it

durable op #0 is a position in a sequence, not a line number. These calls take the next number, counting from zero, in one shared sequence:

step · sleep · wait-signal · wait-event · lock · activity · spawn · call · emit · state cas · state incr

So op #0 is the first of those the script makes, whichever it is. cryo artifact put / get, cryo activity-collect and cryo state add are durable - they record memos and replay like anything else - but they are not in this sequence and do not advance the count. add is out because two claims of one cell should share an answer; cas and incr are in because each attempt asks its own question.

recorded "build" is what stood at that position on the first pass. reached "reuse-cache" is what the replay found there instead.

To locate the bug, count calls from that list down your script until you reach the reported position, then look at what decides which call happens there. It is almost always the nearest enclosing if, loop bound, or early return. Here, op #0 is inside the if, and the condition reads a file that the first pass created - so the condition is the bug, not either branch.

The check is a guard rail rather than a proof, and it is worth knowing where it stops:

• It compares positions. Two different branches that both reach cryo step deploy at op #4 look identical to it, so a divergence that rejoins is not reported.
• It fires on the first mismatch only. Everything after that is unexamined.
• If the memo read backing the check fails, the check is skipped rather than failing your run.

Fixing it

Move the nondeterministic read into a durable call, so both passes branch on the same recorded answer:

#!/usr/bin/env bash
set -euo pipefail

cached=$(cryo step check-cache -- bash -c \
  'test -f /tmp/cryo-marker-demo && echo hit || echo miss')

if [ "$cached" = hit ]; then
  cryo step reuse-cache -- echo "cache hit"
else
  touch /tmp/cryo-marker-demo
  cryo step build -- echo "built it"
fi

cryo sleep 2s
echo "past the sleep"

$ cryo run fixed.sh
  run: dev:default:run-oQF3DQcMZDcUEIWL
built it
cryo: script suspended for 2000ms; agent re-dispatch on wakeup

  - replaying from checkpoint (completed steps are cached, not re-run)
built it
past the sleep
  dev:default:run-oQF3DQcMZDcUEIWL completed

check-cache recorded miss on the first pass and replayed miss on the second, so both passes took the build branch. The built it line appears twice because replaying a step prints its recorded output; the command behind it ran once.

This holds with any number of agents. The second pass can be claimed by a different machine, where /tmp/cryo-marker-demo does not exist at all, and the branch is still the build branch - the answer comes out of the log, not off a disk. That is the whole point of moving the read into a step, and it is why the broken version gets worse rather than better as you add agents.

The marker itself is still a local file, so don’t read this as a way to cache across runs. touch runs on whichever agent that pass landed on; a later run scheduled elsewhere sees nothing. Anything that has to be shared belongs in an entity-scoped state cell or an artifact (cryo artifact put/get), both of which live in the project rather than on a machine.

The general form: anything the run itself changes, or that changes on its own, has to be read inside a durable call. Wall-clock, $RANDOM, a directory listing, an HTTP GET, the contents of a file another job writes.

cryo step is the usual one. Any call that records a memo will do, so a read that has to happen on another machine can be a cryo activity, and cryo artifact get is memoized for exactly this reason - it pins which artifact you resolved, so a later pass branching on what it restored takes the same path.

CRYO_REPLAY_MODE=lenient

The error message offers lenient mode, so it is worth being precise about what it does. It does not make the divergence safe. It removes the check.

Same broken script, same divergence, lenient:

$ CRYO_REPLAY_MODE=lenient cryo run diverge.sh
  run: dev:default:run-Uzbt4OFTNUbQ8GeF
built it
cryo: script suspended for 2000ms; agent re-dispatch on wakeup

  - replaying from checkpoint (completed steps are cached, not re-run)
cache hit
past the sleep
  dev:default:run-Uzbt4OFTNUbQ8GeF completed

The run completes, and look at what it did: cache hit executed. reuse-cache was a name this run had never seen, so it was not a replay of anything - it ran for real, on the resume, doing work the first pass never did. The build step’s recorded result is still in the log, orphaned.

That is the trade. Strict mode fails a run that has gone off its recorded path. Lenient mode lets it keep going down a different one. off is a synonym for lenient; strict is the default and anything else you set means strict.

Reach for it when the divergence is deliberate and you understand what will re-execute - a debugging session, or a workflow you are mid-way through rewriting. It is a per-deployment environment variable on the agent, not a per-workflow setting, so turning it on turns it on for everything that agent runs.

Debugging a run that suspended many times

A run that sleeps between every item can suspend hundreds of times. Some of what you might expect to find is not there, so start with what is.

cryo steps <run-id> lists the run’s completed step checkpoints:

$ cryo steps dev:default:run-CH1vWEjVrMjsejIO
✓ fetch-batch     ./fetch.sh --since 2h  (exit 0)
✓ process     #0  ./process.sh web-1  (exit 0)
✓ process     #1  ./process.sh web-2  (exit 0)
✓ process     #2  ./process.sh web-3  (exit 0)
✓ summarise       ./summarise.sh  (exit 0)

This is the first thing to look at: a run stuck mid-loop shows exactly which iteration it reached. process is one step in the script reached three times, so it lists once per iteration with the iteration number; a step reached once has no number to show.

The third column says what each iteration was working on. It defaults to the command, which is usually enough. When it isn’t - a stepFn has no command, and a loop over a list may run the same command with the difference somewhere else - pass --label:

for host in web-1 web-2 web-3; do
  cryo step drain --label "$host" -- ./drain.sh "$host"
done

A label is display only. It never reaches the memo key, so two passes may label one step differently without the replay losing it.

Two limits to know. It lists only what finished, so the call a run is currently blocked on is the one absent from the list rather than a pending entry in it. And it shows step names only - sleeps, waits, locks, activities and artifacts have no line here.

What you will not find: a suspension count. Suspending is not a workflow event. The run above suspended three times, and its durable log records one activity from start to finish:

$ cryo events dev:default:run-CH1vWEjVrMjsejIO
 0 WorkflowStarted
 1 ActivityScheduled
 2 ActivityCompleted
 3 WorkflowCompleted

cryo status agrees - activities_completed: 1. That is deliberate: a suspension parks the same dispatch rather than completing it, so the run stays on one activity across every eviction. It also means you cannot ask the system how many times a run has replayed. If you need that number - and it is a fair thing to want when a script is misbehaving - record it yourself, the way the first example on this page did.

Every pass is stored, though - including the script suspended line, which the agent writes into the run’s log like any other output. Plain cryo logs hides them, showing only the final pass, so a run that replayed forty times reads like a script that ran once:

$ cryo logs dev:default:run-NN1xDlpSAOTkL9Tq
one
two
three

--follow keeps every pass and marks where each begins. On a run that has already finished it replays the whole history rather than tailing, which is the closest thing to a pass count the system offers:

$ cryo logs dev:default:run-NN1xDlpSAOTkL9Tq --follow
one
cryo:
  - replaying from checkpoint (completed steps are cached, not re-run)
one
two
cryo: script suspended for 3000ms; agent re-dispatch on wakeup

  - replaying from checkpoint (completed steps are cached, not re-run)
one
two
three

Three passes for a script with two sleeps. Read it as blocks: each replaying from checkpoint marker starts a pass, and the lines above it are what that pass produced. one appearing three times is the memoized step replaying its recorded output, not the command running three times.

So the debugging loop for a long-lived run is: cryo steps for how far it got, cryo logs --follow for what each pass did, cryo events for the run-level story, and your own instrumentation for anything you need counted.

What the engine does not catch

Worth stating plainly, because the checks above can read as more of a safety net than they are.

• Side effects between durable calls. The engine cannot see your echo, your curl, or your rm. Code outside a durable call runs on every pass and nothing warns you.
• Nondeterminism inside a call. A step whose command is itself nondeterministic records whatever it produced the first time. That is usually the point, and occasionally a surprise.
• Divergence in the calls it does not sequence. cryo artifact and cryo activity-collect replay from their own memos but take no position, so a branch that changes how many of those happen is invisible to the check.

The rule that survives all three: if it matters that something happened exactly once, it belongs inside a durable call.