Architecture

Cryosleep is a durable-execution substrate with product surfaces built on it: YAML pipelines, polyglot scripts, node graphs authored on the canvas, schedules, webhook-driven CI, and a web UI. This document explains how the system behaves - the execution model, the process topology, and the durability and security properties you can rely on. For the authoring-facing version of the execution model, see the authoring model.

The execution model

A run is an append-only journal; nothing else survives. Every run’s history is a sequence of history events: started, activity scheduled/completed, timer scheduled/fired, signal received/consumed, child scheduled/completed, terminal. There is no state snapshot. Status, the timeline UI, and substates like “sleeping until X” are all derived by folding over the log.

Recovery re-executes the program and skips what already happened. The executor rebuilds the workflow from scratch, polls it once, performs the effects it requests, appends the result events, and loops. On resume the body re-runs from the top; every durable call whose result is already in the log returns the recorded value without executing. Locals come back because the code re-runs; the journal exists to memoize completed effects and to pin values that must not change across replays - timer deadlines, signal payloads, child outputs.

Durable calls are the only durability boundary. activity, step, stepFn, sleep, wait-signal, wait-event, spawn, emit, and activity-submit/activity-collect journal or memoize their outcomes. Code between two durable calls re-runs freely on every replay and must therefore be deterministic in which durable call it reaches next. annotate is the one deliberate exception - a side channel with no durable boundary.

A durable call is identified by what it is, not by when it ran. Every one carries an Identity: the scope it was reached in, its family (activity, wait, sleep, call, signal, …), the author’s name for it, and an ordinal counting how many times this run has reached that site. A loop reuses one name and every iteration gets its own ordinal.

Both tiers key the same way; they only spell it differently. The orchestration tier holds an Identity struct, the polyglot tier an escaped journal path. Position used to be the key on the orchestration side and it was wrong: the counter restarts each replay, so two branches running concurrently drew from one counter and a call could be dispatched twice with the run still reporting success (TDD 0095). Naming a call by what it is removes that whole class.

A per-site guard still fails loudly by default if a replay reaches a different durable op where it expected one (opt out with CRYO_REPLAY_MODE=lenient). Name-keyed identity tolerates inserting or reordering steps between deploys.

A parked run costs one database row. It holds no process. A polyglot script that sleeps or waits is suspended and its process killed; the queue parks the entry until the wake time or the signal. In-driver workflows park as journaled timers or signal waits. Waking is: record the memo, flip the row to pending, let replay run the script back to the waiting point. Approvals that wait weeks cost the same as sleeps that wait seconds.

Workers never touch the journal. Agents are outbound-only: they long-poll for claims, hold a lease with a single-use claim token, and expose durable primitives to scripts over a local unix socket. All durability semantics live in the server and agent; the SDKs are thin skins over that socket protocol.

Push, with a poll backstop. Every hot path is event-driven: a lease release or a new claim wakes a waiting long-poll immediately, and broadcast channels fan out logs and run events. Behind each push is an interval loop - lease expiry, wake-parked, terminal-outcome drain, concurrency reconcile - so a lost notification degrades to bounded latency instead of a hang.

Event sourcing only where replay pays. The journal is event-sourced. The dispatch queue, concurrency tables, tenancy, secrets, and logs are ordinary current-state tables or byte streams. Replayability is the product for run history and a liability everywhere else.

Definitions are pinned per run. The YAML or script source travels in the run’s start event, so an in-flight run replays its submit-time definition no matter what has since been deployed. Runtime skew (the cryo binary itself changing under a suspended run) is handled operationally: agents advertise a build id and self-update by draining, never mid-lease.

Idempotency by deterministic identity. Workflow ids are caller-supplied idempotency keys - starting an existing id attaches instead of duplicating. Child ids, activity ids, and fan-out child ids are all derived deterministically from the parent and a sequence, and emits/spawns memoize by counter. Retries and replays converge on the same rows instead of duplicating work.

One disambiguation, because three things share the word “event”: a history event is a journal entry, internal to one run. The event bus is project-scoped pub/sub - cryo emit, handler workflows, and wait_for_event/cancel_on waiters. The run-detail stream sends “event frames”, which are a projection of the journal for the browser.

Graphs and scripts

There are two execution models, not three: YAML compiles to the graph IR and the graph interpreter runs it, so a pipeline job and a canvas node are the same thing to the engine. What differs is graphs against polyglot scripts, and the difference is where a run keeps its position.

A graph run’s position is its journal. The interpreter is the durable program. It selects a node, schedules an activity, waits for the result, selects the next - and writes each of those down, because writing them down is how it remembers across a restart. A wait node is a decision it makes itself: nothing is dispatched and no agent is involved.

A script run’s position is a process’s program counter, which cannot be persisted. So the whole script is dispatched as one activity, and the position is reconstructed instead of stored. On a cryo wait-signal with nothing waiting, your script is killed, the queue entry parks, and the agent slot is freed. When the signal lands the script runs again from line one, and each durable call it already made returns its recorded answer immediately until execution arrives back at the call that suspended.

Two things follow, and they are the whole trade:

The engine sees a graph’s every decision and only a script’s checkpoints. A graph’s shape is known before it runs. A script’s shape is whatever it recorded, which is why the canvas draws a script run from its checkpoints rather than from a document. A checkpoint is an answer, written when the call it belongs to returns, so on its own it would draw the steps around a long build and nothing for the build. What closes that is the positional marker the engine pins before every durable call: it names the call it is about, so a marker whose call has recorded nothing is a step running right now.
Anything a script does outside a durable call happens again on every pass. A curl -X POST at the top of a script that waits four times fires four times; inside cryo step it fires once and every later pass gets the recorded result. This is the discipline a graph doesn’t ask of you, and it is the price of writing control flow in your own language.

Reach for a graph when the orchestration is the interesting part - fan-out, approvals, per-node retries, anything you want resumable and visible node by node. Reach for a script when the logic is the interesting part and you would rather write a loop than express one in a document. They nest either way: a graph node can run a script, and a script can cryo submit children.

Where a run’s facts live

Two stores, split by the question being asked rather than by which surface is asking.

workflow_history answers “what did this run do, in order?”. It is the replay tape, walked forward, and the graph interpreter’s position is this log. Rows are keyed by the call’s identity, in columns (key_scope, key_family, key_name, key_ordinal) so a row stays queryable while its payload is opaque bytes.
journal_entries answers “did this named step already record an answer?”. Path-keyed and hierarchical (node/3/step/build/0), written by the agent over HTTP, read as a point lookup mid-flight.

The second store exists because a script cannot be replayed to rebuild its state - it can only be asked. Serving that from a replay tape would mean shipping the whole log to the agent and having it scan.

The split is not per tier. Every surface writes orchestration to the history log; YAML and the canvas compile to the same graph IR. The journal takes checkpoints recorded inside an activity, so a YAML job with steps: writes there too, via the agent. A graph with no code nodes never touches it.

One write surface over both. Metering, sealing and identity attach at a FactStore seam above the two stores rather than to each store’s own decorator chain. They used to attach per-chain, which is how the history log went unmetered and unsealed for months while the journal had both and looked done (TDD 0094).

A note on the word, since it is overloaded here: the prose above calls a run’s history “the journal” in the general sense of an append-only record. The Journal trait and journal_entries are specifically the path-keyed memo store. Where it matters, this document names the table.

Topology

                  ┌──────────────────────┐
                  │       web ui         │
                  │ (single-page app,    │
                  │  embedded in server) │
                  └──────────┬───────────┘
                             │ HTTPS
                             ▼
┌────────────────────────────────────────────────────────┐
│             cryosleep server (one binary)              │
│                                                        │
│  ┌──────────────────────┐  ┌────────────────────────┐  │
│  │ /api/v1/orgs/…       │  │ /api/v1/agent/…        │  │
│  │ user-facing routes   │  │ agent-facing routes    │  │
│  └───────────┬──────────┘  └───────────┬────────────┘  │
│              └──────────┬──────────────┘               │
│                         ▼                              │
│  ┌──────────────────────────────────────────────────┐  │
│  │ workflow driver: poll-once executor · resume ·   │  │
│  │ terminal hooks                                   │  │
│  └─────────────────────┬────────────────────────────┘  │
│                        ▼                               │
│  ┌──────────────────────────────────────────────────┐  │
│  │ journal · dispatch queue · log store · events    │  │
│  │ concurrency · secrets · … (pg / sqlite)          │  │
│  └──────────────────────────────────────────────────┘  │
│                                                        │
│     (`cryo dev` embeds an in-process agent here)       │
└────────────────┬────────────────────────┬──────────────┘
                 │ outbound HTTP long-poll claim loop
                 ▼                        ▼
      ┌───────────────────┐    ┌───────────────────┐
      │  cryosleep-agent  │  … │  cryosleep-agent  │
      │  shell + polyglot │    │  connector        │
      └───────────────────┘    └───────────────────┘

Logically three tiers - server, agents, durable store - though cryo dev collapses server and agent into one process via an embedded agent.

Multiple replicas, on postgres. The server tier scales horizontally: run replicas: 2 or more behind one load balancer. One replica drives a given run at a time, held by a durable per-run ownership lease; when a replica dies its lease lapses and another adopts the run on a 15s sweep. Signal and cancel wakes, and a browser’s live-log tail, cross replicas over postgres LISTEN/NOTIFY, so a tail attached to any replica sees bytes an agent streamed to another.

Scaling out needs postgres for both the database and the dispatch queue, a distinct CRYOSLEEP_REPLICA_ID per replica (the hostname is used when it is unset), and object-store logs so any replica can serve a finished run’s log. Sqlite deployments are single-process: the lease and both cross- replica buses are postgres-only. Ownership is a lease rather than a hard write fence, so a replica stalled longer than the lease TTL can briefly double-drive a run.

Outbound-only agents. Agents dial out; nothing dials in. A claim long-poll is held server-side for ~25s; work arriving mid-poll is granted immediately. Agents run one claim at a time - concurrency comes from running more agents, which keeps replay, timeouts, and lease semantics simple. Capabilities are advertised per claim and intersected with the token’s ceiling.

A third agent role: connector. http and connector nodes dispatch an http_request activity that requires the connector capability, and the resolved credential reaches the agent at claim time. connector is deliberately outside an agent’s default advertised set (shell,polyglot), so an operator opts a runner in with --cap connector or $CRYOSLEEP_AGENT_CAPABILITIES. A fleet with no connector-capable agent leaves that work queued indefinitely, with no error to read. cryo dev’s embedded agent advertises it, so local runs work without any of this.

Storage backends. Postgres is the production shape: journal, dispatch queue (with LISTEN/NOTIFY), concurrency store, and all control-plane stores are durable. Sqlite backs cryo dev and small self-hosts: the journal, concurrency store, and run logs are durable, so runs resume across a restart; the dispatch queue is in-memory and rebuilt from the journal on resume.

Components

Server - one binary. Hosts the user-facing HTTP API, the agent-facing claim/report API, the workflow driver (the poll-once executor and its terminal hooks), and the embedded web UI. It owns all durability logic: the journal, the dispatch queue, live-log fan-out, the scheduler, the retention sweeper, concurrency admission, and the multi-tenant control plane (orgs, projects, members, secrets, webhooks).
Agents - the workers. Each dials out, claims one activity at a time, runs it under a PTY, streams logs, and exposes the durable primitives to a script over a local socket. An agent carries a capability ceiling and a build id, and self-updates by draining.
Storage - Postgres or sqlite, behind the same interface.
SDKs - bash, Go, Python, Rust, TypeScript. Each is a client half (wraps the cryo CLI) and an in-script half (speaks the agent socket directly). They contain no durability logic.
Web UI - a single-page app embedded in the server binary; it streams logs and renders the timeline, pipeline strip, annotations, and the inline approval form.

Four workflow types are registered. The type names the kind of definition that was submitted; every run executes as a node graph over one journal.

cryosleep/pipeline/v1 - a YAML pipeline. It is the entry point: the pipeline compiles to the node-graph IR and the graph interpreter runs it.
cryosleep-graph/v1 - a node-graph document, authored on the canvas or written as JSON.
cryosleep-node/v1 - a single node kind: what cryo call node runs, and what a workflow node’s graph-ref children run.
cryosleep/script/v1 - a polyglot script, one opaque activity.

Life of a run

A YAML run. POST …/runs → role gate → org/project run caps → concurrency admission (below) → the driver journals the start event (the YAML is carried in the input) and spawns a runner → the graph interpreter walks the node graph; each ready node becomes an activity → the dispatch queue holds a pending row and a NOTIFY wakes a claim long-poll → at claim time project secrets are injected into the job’s env (never earlier - plaintext never sits in the queue) and a prefetched step cache comes along → the agent executes under a PTY, streaming log chunks (each renews the lease) → the outcome is journaled as completed → next DAG layer → terminal event → terminal hooks run: live-buffer eviction, outbound commit status, concurrency promotion.

A polyglot suspend and resume. The script calls cryo sleep 2h → the agent reports the suspension → the queue parks the row and the script process dies. Later the wake loop finds the due row, records the sleep memo first, then flips the row to pending. Any capable agent claims it; the script re-runs from the top; completed steps hit their memos (mostly from the prefetched cache); the sleep memo hits, so the sleep returns instantly and execution continues.

A signal. POST …/runs/{id}/signal/{name} → for a parked script, the signal memo is recorded and the row re-pended; for an in-driver wait, a signal-received event is journaled and the runtime re-polls, pairing it with a consumed event so replay consumes the same signal exactly once. Approvals are signal waits with a server-side validated field schema.

A webhook. Unauthenticated ingest, HMAC is the auth → dedup keyed on the signed body hash → trust gate (fork PRs run without secrets) → either delivered to the project handler or submitted directly (one, never both) → optional pipeline fetch from the repo at the pushed SHA → the pipeline’s top-level if: is evaluated server-side against the trigger scope → submit. On terminal, a stored outbound target posts the commit status.

Concurrency admission. A submit with a concurrency group tries an atomic acquire. Held → the policy decides: queue stores the serialized submit as a queue row and returns a position (the run has no journal row yet; its status is synthesized from the queue); cancel-running steals the hold and cancels the incumbent; cancel-queued drains the waiters and takes the queue slot; skip creates no run at all. On the holder’s terminal event, the oldest waiter is popped and re-submitted through the normal path. A periodic reconcile sweep promotes past dead holders and orphaned queues; both paths are idempotent, so hook and sweep can race harmlessly. Cancelling a queued run is deleting its row.

Log bytes. PTY → agent buffer → flush every ~150ms → posted to the server → appended to a capped in-memory snapshot (4 MiB) on that replica, published on the run’s broadcast channel, and forwarded uncapped to the durable log store. On postgres each chunk also lands in a shared short-window hot tier of recent chunks, which fans bytes out to tailers on every replica and is pruned when the run reaches terminal - so a browser hitting replica B sees bytes an agent streamed to replica A. That tier is a high-write UNLOGGED table, worth recognising when you watch the database. A browser tail gets the snapshot then live chunks (handed out atomically), with a keepalive every 15s of silence. After terminal eviction, reads come from the log store. Every byte that reaches object storage is written, but bytes that cannot be written are bounded: while the store is unavailable a run keeps at most 4 MiB of unflushed backlog (64 MiB across the replica), dropping the oldest and marking the gap in the stream, so an outage costs log fidelity instead of the control plane’s heap.

Wakeups: push and poll

Something happens	How the system notices	Worst case
Activity enqueued	NOTIFY → in-process notifier → held claim long-poll	25s poll cycle if NOTIFY lost
Signal arrives	memo + wake-parked + NOTIFY (scripts); journal + notifier (in-driver)	one loop interval
In-driver timer due	timer armed to the journaled deadline	exact; re-armed on resume
Parked script due	wake-parked interval loop	one loop interval
Agent dies mid-claim	lease-expiry interval loop re-pends the row	one loop interval
Cron tick due	scheduler poll + atomic tick claim	one tick
Group holder finishes	terminal hook promotes	~0; reconcile sweep backstop
New log/journal bytes	broadcast channels	immediate

The loops are cheap state-indexed queries, not table scans.

Durability and limits

State	postgres	sqlite (`cryo dev`)
Journal (runs, history)	durable, sealed at rest	durable, sealed at rest
Step / StepFn memos	durable, sealed at rest	durable, sealed at rest
Dispatch queue	durable + NOTIFY	in-memory, rebuilt from the journal on resume
Concurrency holders/queue	durable	durable
Run logs	durable in object storage when `CRYOSLEEP_LOGS_S3_*` is set, else in-memory	durable
Artifacts	blobs in object storage, index in the database	blobs in a directory, index in sqlite
Event waiters (`wait_for_event:`/`cancel_on:`)	durable	durable
Bus event history	durable	in-memory
Live log/event buffers	per-replica in memory, plus a shared cross-replica hot tier, both evicted on terminal	per-replica in memory, single-process

Limits: a run’s canonical input is uncapped — it reaches a pipeline as its input expression scope and any job as $CRYO_INPUT_FILE, neither of which is an environment variable; the live-log snapshot caps at 4 MiB per run (the durable log store is uncapped); concurrency queues default to a max of 100 and reject beyond it; event handler chains are capped at depth 5. The retention sweeper deletes terminal runs after 90d and bus events after 30d by default (configurable, postgres only).

Artifact storage

Blobs go to an S3-compatible endpoint when one is configured, and to a directory otherwise, so the verbs work on a laptop with nothing set up. The index lives in the same database as everything else.

Variable	Meaning
`CRYOSLEEP_ARTIFACTS_S3_BUCKET`	Bucket for artifact blobs. On its own, the endpoint and credentials are taken from `CRYOSLEEP_LOGS_S3_*`.
`CRYOSLEEP_ARTIFACTS_S3_ENDPOINT`	A dedicated backend. Setting this stops anything being inherited, so give it credentials of its own.
`CRYOSLEEP_ARTIFACTS_S3_ACCESS_KEY_ID`, `…_SECRET_ACCESS_KEY`, `…_REGION`, `…_ALLOW_HTTP`	The rest of a dedicated backend.
`CRYOSLEEP_ARTIFACTS_DIR`	Where blobs go with no S3 configured. Defaults to `artifacts/` beside the sqlite database.
`CRYOSLEEP_ARTIFACTS_PROXY`	Send transfers through the control plane even when the backend could sign. For a deployment whose agents cannot reach the object endpoint.
`CRYOSLEEP_ARTIFACTS_MAX_BYTES`	Per-project storage limit. Unset means no limit. Counted per distinct object, so two names for the same content count once.
`CRYOSLEEP_RETENTION_ARTIFACTS`	How long cached (`--key`) artifacts live — `30d`, `12h`, `off`. Unset means forever. Run artifacts ignore this: they are collected when their run is.

…_ALLOW_HTTP defaults to true, which suits an in-cluster endpoint. Agents fetch artifacts directly from this endpoint, so set it to false for anything an agent reaches over a network you do not control. Step results live in the journal and should stay small - record pointers, not artifacts. Replay cost grows with journal length, so a single run that loops on signals for a very long time accumulates cost; recurring work belongs in a schedule, where each tick is a fresh log.

Security model

Tenancy is org → project → member with roles (Owner/Admin/Member/Viewer); every user route extracts and authorizes the org/project from the path, and run-id prefixes are scope proof. Identity is OIDC or a dev-mode bootstrap; API tokens and agent tokens are separate, and an agent token carries a capability ceiling that claims cannot exceed. Claim tokens are per-claim, single-use, and echoed on every agent request.

A slug is permanent; name is the field that renames (ADR 0107). A run id is {org_slug}:{project_slug}:run-{random}, and that string addresses everything the run owns across four substrates: workflows (a LIKE on an id text_pattern_ops index), the journal (a path prefix), the object store’s log and artifact keys (a prefix delete), and durable state cells (org:proj:run-x, org:proj:entity:<name>). Only the first is a table a foreign key could repair, which is why deleting a project is a sequence of prefix operations rather than a cascade, and why a slug rename is a programme rather than a feature. Nothing derives a slug from a name, and no route accepts a slug update.

Outbound requests the control plane makes on a tenant’s behalf are bounded by where they may go. Registering a forge webhook sends a stored token to a caller-supplied API base, so that address must resolve publicly over https, its resolved addresses are pinned for the life of the request, and redirects are refused. CRYOSLEEP_FORGE_API_ALLOWLIST narrows it further to named hosts, which is also how a self-hosted deployment permits a forge on its own network. Connector egress is bounded a second time at the network layer, by a policy that excludes cluster-internal space.

Secrets are sealed with a per-org key, itself wrapped by the deployment master key (CRYOSLEEP_MASTER_KEY, fail-hard in prod; generated and persisted next to the db in dev). Plaintext exists only inside a claim grant: secrets are injected at claim time, so the dispatch queue never stores them, and the runner redacts their values from both live and captured logs. Runs triggered by untrusted events (fork PRs) get no secrets at all, and a step that asks for one fails rather than running without it, so the withholding is visible instead of silently changing what the step does. Script environments are sealed: a fixed allowlist plus explicitly opted-in CRYO_ENV_PASSTHROUGH globs; agent credentials are scrubbed before any job sees the environment.

Everything a run stores is sealed under the same per-org key: step and StepFn memos, durable state cells, and the history log itself. Two encodings, for one reason. Memos and state cells carry a Sealed envelope, which is base64 inside JSON and costs about a third; that is nothing on a secret. History rows can be megabytes and there are far more of them, so those are sealed by a codec at the backend and written as raw nonce || ciphertext bytes, compressed first because ciphertext does not compress (ADR 0103, ADR 0106).

A payload that will not open fails its run rather than reading as absent (ADR 0104). The alternative is worse than it sounds: an ActivityCompleted that cannot be decrypted looks exactly like an activity that never completed, so the executor would dispatch it again - and it may already have charged a card or deployed. Such a run ends as unreadable, a status of its own, because a key problem across the org and a bug in one workflow want opposite responses. The status lives on the run row rather than in the history, since the history is the part that will not open.