Skip to content
cryosleep

Reshaping data between steps

One step returns a wide record; the next one wants three fields under different names. A transform node is where that reshaping goes, so the steps either side of it stay about the work they do.

It runs inside the engine - no agent, no dispatch, no container - which is why it is cheap enough to put wherever the shape is wrong.

A transform node selected on the canvas: its inspector shows a field table with one row per output field — id, customer and total, each computed from the incoming record — and below it the live result of running that table against what the step above produced.

Two ways to write one

Section titled “Two ways to write one”

A table, one row per output field. Each row is a name and either a fixed value or a small expression. This is the default, and it is what most reshaping needs.

Code, a single JavaScript expression that returns the whole output. Reach for it when the output has fewer rows than the input, or a shape a per-field table cannot describe - grouping, joining, filtering.

A node is one or the other, never both. Switch to code on the table hands you the exact JavaScript your table compiles to, so the first thing you read is your own table rather than an approximation of it. There is no way back: going from code to a table would mean guessing an intent you may have edited past.

each and all

Section titled “each and all”

Above the editor is the question this step answers: does it run for each item, or once, over the whole list?

each (the default) runs your expression once per item. i is the item and index is its position. A value that isn’t a list is treated as a list of one, so a single record works without a special case.

all runs it once over the whole value, which is bound to input. Use it when the answer depends on more than one item, or when the number of rows changes.

It sits above the rows rather than among the node’s other settings because it decides what every row means — i in i.order_id is the item in one mode and nothing at all in the other.

Worked examples

Section titled “Worked examples”

Every example below is run by the test suite against its own input, and the result shown is the one it produced.

Keep a few fields, renamed

Section titled “Keep a few fields, renamed”

A fetch returns more than the next step needs. Name what comes out.

{
  "mode": "each",
  "fields": [
    { "name": "id", "expr": "i.order_id" },
    { "name": "customer", "expr": "i.buyer.email" },
    { "name": "total", "expr": "i.amount_cents / 100" }
  ]
}

Given:

[
  {
    "order_id": "A-1",
    "buyer": { "email": "kim@example.com" },
    "amount_cents": 2500,
    "internal_note": "ignore me"
  }
]

Result:

[{ "id": "A-1", "customer": "kim@example.com", "total": 25 }]

Add a field, keep the rest

Section titled “Add a field, keep the rest”

include_input carries the incoming record through, with your fields layered over it. It is off by default: a node that silently forwarded everything would make the table a lie about what comes out.

{
  "mode": "each",
  "include_input": true,
  "fields": [{ "name": "size", "expr": "i.amount > 100 ? 'large' : 'small'" }]
}

Given:

[
  { "id": "A-1", "amount": 250 },
  { "id": "A-2", "amount": 40 }
]

Result:

[
  { "id": "A-1", "amount": 250, "size": "large" },
  { "id": "A-2", "amount": 40, "size": "small" }
]

A named field wins over an incoming key of the same name, so this is also how you overwrite one field and pass the rest along.

A fixed value

Section titled “A fixed value”

Not every field is computed. A row can carry a literal, which is stored as JSON - 42 is a number, true is a boolean, and anything that isn’t JSON stays the text you typed.

{
  "mode": "each",
  "include_input": true,
  "fields": [
    { "name": "source", "value": "nightly-import" },
    { "name": "priority", "value": 3 }
  ]
}

Given:

[{ "id": "A-1" }]

Result:

[{ "id": "A-1", "source": "nightly-import", "priority": 3 }]

To write the string "42" rather than the number, switch the row to an expression and quote it there.

Keep only some rows

Section titled “Keep only some rows”

Filtering changes the number of rows, so it needs all and code.

{ "mode": "all", "expr": "input.filter(r => r.status === 'open')" }

Given:

[
  { "id": "A-1", "status": "open" },
  { "id": "A-2", "status": "closed" },
  { "id": "A-3", "status": "open" }
]

Result:

[
  { "id": "A-1", "status": "open" },
  { "id": "A-3", "status": "open" }
]

Group rows and total them

Section titled “Group rows and total them”
{
  "mode": "all",
  "expr": "Object.entries(Object.groupBy(input, r => r.team)).map(([team, rows]) => ({ team, count: rows.length, total: rows.reduce((s, r) => s + r.n, 0) }))"
}

Given:

[
  { "team": "x", "n": 2 },
  { "team": "y", "n": 3 },
  { "team": "x", "n": 1 }
]

Result:

[
  { "team": "x", "count": 2, "total": 3 },
  { "team": "y", "count": 1, "total": 3 }
]

Look a value up in another list

Section titled “Look a value up in another list”

One step fetched the rows, another fetched the thing they refer to. An expression is an expression, so statements go in an arrow function you call immediately - the same escape hatch JavaScript itself offers.

{
  "mode": "all",
  "expr": "(() => { const by = Object.fromEntries(input.teams.map(t => [t.team, t.lead])); return input.items.map(i => ({ ...i, lead: by[i.team] })); })()"
}

Given:

{
  "items": [
    { "name": "a", "team": "x" },
    { "name": "b", "team": "y" }
  ],
  "teams": [
    { "team": "x", "lead": "kim" },
    { "team": "y", "lead": "sam" }
  ]
}

Result:

[
  { "name": "a", "team": "x", "lead": "kim" },
  { "name": "b", "team": "y", "lead": "sam" }
]

Split text into unique values

Section titled “Split text into unique values”
{
  "mode": "all",
  "expr": "[...new Set(input.flatMap(r => r.tags.split(',').map(t => t.trim())))].sort()"
}

Given:

[{ "tags": "urgent, billing" }, { "tags": "billing" }]

Result:

["billing", "urgent"]

Number the rows

Section titled “Number the rows”

index is the item’s position, and it is only bound in each mode.

{
  "mode": "each",
  "include_input": true,
  "fields": [{ "name": "position", "expr": "index + 1" }]
}

Given:

[{ "id": "A-1" }, { "id": "A-2" }]

Result:

[
  { "id": "A-1", "position": 1 },
  { "id": "A-2", "position": 2 }
]

Building one on the canvas

Section titled “Building one on the canvas”

The awkward part of reshaping data is that you cannot see what you are reshaping. Four things address that:

Start from an example. A transform with an empty table offers a few working starting points - keep and rename, add a field, group and total. Each carries the data it was written against, so picking one shows you a result before anything upstream has run.

Run the step above. On a graph that has never run, this is where the first real data comes from. Select the node, press Run this node, and its output becomes the sample every node below it is edited against. It really runs: an http node here calls the service.

Pick a field instead of spelling it. Click into a row’s expression and the fields the incoming data actually has are offered underneath, each with the value it holds:

order_id  "A-1"    buyer.email  "kim@example.com"    amount_cents  2500

Picking one inserts i.buyer.email at the caret, so you can pick the field and then type / 100 after it. The list comes from the data rather than a declared schema, because the step above is usually an http node whose output shape nothing declares.

Watch the result as you type. Every row shows what it produced:

id        =fx  i.order_id             → "A-1"
total     =fx  i.amount_cents / 100   → 25

Each row is evaluated on its own, so a mistake in one is reported against that row and the others keep showing their values. That evaluation runs nothing - the endpoint behind it refuses any node kind that would have an effect - so it can fire on every keystroke.

Together those are the loop: add a node, run the one above it, pick an example, edit, watch the shape change. That is the whole recording - running the step above, picking a starting point, renaming a field, and switching the finished table to the code it compiles to:

What an expression can use

Section titled “What an expression can use”

The JavaScript here is deliberately small. Object, Array, String, Number, Math, JSON, Map, Set, RegExp and the typed arrays are all present. Modern methods are too - Object.groupBy, Array.prototype.flatMap, spread, optional chaining.

What is absent is anything that reaches outside: no network, no filesystem, no Date, no Math.random, no timers, no imports.

That is not a sandbox afterthought, it is the point. A transform is re-derived rather than recorded, so it runs again on every replay of the run - and a value that came from a clock or a coin flip would be a different value the second time, which is a run that diverges from its own history. If you need the current time or a random id, take it in a code node, where the result is recorded once and replayed. See the authoring model.

Two limits bound a runaway expression: a tick budget that stops an infinite loop, and a 64MB ceiling on what it can allocate. Both fail the node with a message rather than hanging the run.

What gets checked, and when

Section titled “What gets checked, and when”

Save refuses what it can see is wrong, so it doesn’t become a run that fails minutes later:

  • an expression that isn’t valid JavaScript, naming the row it is in
  • a table row with no name, or with both a fixed value and an expression
  • two rows with the same name - in JavaScript the later one silently wins, which makes it the mistake nobody finds by reading the output
  • mode that isn’t each or all
  • a config with both expr and fields, or neither

A row you have added but not filled in is not an error. It saves, and it compiles to nothing, so a table can be built one click at a time. Nor is a name typed before its expression - that row holds null until you finish it, which is visible in the result rather than silently dropped.

When something is wrong

Section titled “When something is wrong”

Messages are about what you wrote, not about what the engine noticed:

you wrote you get
i.order_id + this isn’t valid JavaScript yet - a ( is not closed
i.name.toUpperCse() i.name.toUpperCse is not a function - check the spelling
i.customer.email i.customer has no value here, so .email cannot be read - this item has order_id, buyer and amount_cents

That last one lists the fields the data actually has, which is the same list the field picker offers - the way out of the error is the way you would have avoided it.

Each row is evaluated on its own, so a mistake in one is reported against that row and every other row keeps showing its value. A typo does not blank the panel.

In a graph document

Section titled “In a graph document”

The canvas writes JSON; this is what it writes. Nothing stops you writing it by hand, or generating it.

{
  "version": "cryosleep-graph/v1",
  "nodes": [
    {
      "id": "fetch",
      "type": "http",
      "config": { "url": "https://api.example.com/orders" }
    },
    {
      "id": "shape",
      "type": "transform",
      "config": {
        "mode": "each",
        "fields": [
          { "name": "id", "expr": "i.order_id" },
          { "name": "total", "expr": "i.amount_cents / 100" }
        ]
      }
    },
    {
      "id": "notify",
      "type": "shell",
      "config": { "run": "./post-summary.sh" }
    }
  ],
  "edges": [
    { "from": "fetch", "to": "shape" },
    { "from": "shape", "to": "notify" }
  ]
}

A transform reads the value the edge into it carried, and hands its result to whatever is wired after it. Downstream nodes can also name it directly - nodes.shape.output - like any other node.

Any part of the config can itself be computed, with the usual markers:

{
  "id": "shape",
  "type": "transform",
  "config": { "mode": "each", "expr": { "$expr": "vars.shaping_expression" } }
}
Section titled “Related”