Skip to main content

​
Overview

A pipeline is how a derived value in ProDex shows its work. It’s a structured, navigable record of a data derivation: the sources that were read, the transformations that ran (with the code), and the outputs those transformations produced, bound to the artifacts they landed in. When a number on a dashboard, a parameter on a simulation model, or a quantity on a BOM was derived rather than stated, the pipeline is the audit trail behind it. Pipelines are what turn raw data into reusable, auditable inputs to the rest of ProDex. For how they fit into the wider data lifecycle, see Data Overview.
Formerly “Templates.” Older versions of these docs called pipelines “templates.” That word now belongs to configuration templates in the BOM configurator, which are unrelated. If you’re looking for reusable BOM structures, that’s the page you want.The concept changed too, not just the name. A pipeline is no longer something you execute — there’s no Run Pipeline button, no gates, and no run states. It’s now an auditable record of transformations that already happened.

​
What a Pipeline Is (and Isn’t)

A pipeline is a document of work that already happened, not an executable program. Dexter authors it as an analysis lands, keyed by slug. When the same derivation is re-done — a fresh upload, a corrected interpretation, additional outputs — the pipeline is saved again under the same slug and updates in place. A genuinely different derivation gets a new slug.
A pipeline is provenance, not verification. It shows how a value was derived, not whether it matches reality. Whether a simulation model’s parameters match the floor belongs to the model’s validation, not to its pipeline.

​
Where Pipelines Live

On the Data page, in the Pipelines tab. Selecting a pipeline opens it as a canvas: sources on one side, transformations in the middle, outputs on the other, with node detail (name, description, bindings, values) revealed when you select a node. Every source and output node is clickable through to the artifact it binds, so the canvas doubles as a navigation surface: from a chart on a dashboard, click into the pipeline; from a node on the pipeline, jump to the file it read or the model parameter it landed.

​
The Shape of a Pipeline

A pipeline is a DAG of nodes and edges, plus a small amount of framing. Framing:
  • Name: human-readable title
  • Goal: one to two sentences on the question this derivation answers — the framing a reader lands on before touching a node
  • Description: the context a reader needs to make sense of the graph
Three node types:
  • Source nodes: entry points. Each source binds to a specific input: an uploaded file, a connected integration, or an upstream ProDex artifact (a model, a BOM, a lookup table).
  • Transformation nodes: the work that ran. Each transformation carries a markdown description with a narrative and a fenced code block (SQL or Python) showing exactly what executed. Granularity follows the explanation: mechanical plumbing folds into the step it serves; any step that changes what the data means stays its own node.
  • Output nodes: where the derivation landed. Each output binds to a specific downstream artifact and carries the materialized values it produced.
Edges connect nodes into the DAG, showing which source feeds which transformation and which transformation produces which output.

​
Node Fields

Source node
  • Name: human-readable label (e.g., “MES Q3 export”)
  • Description: markdown covering what the source is, its structure, and any interpretation notes (unit conventions, code meanings, quirks) that future work on this source should start from
  • Binding: the endpoint type and slug — upload (a file under Uploads), data_integration (an integration slug), or an artifact kind and slug
Transformation node
  • Name: an action phrase (“Filter to completed cycles at station 710”), not a noun (“Filter step”)
  • Description: markdown with a sentence or two of prose plus the fenced code that ran, in one field
Output node
  • Name: short label (“Op 4411 median cycle time”, “Line A model parameters”)
  • Description: markdown explaining what this output produced and how to read it
  • Binding: the artifact this output landed in (see below)
  • Values: the materialized results, zero or more. Each carries a name, a one-line value (as it would appear in a table cell — “42.5 s”, “4.2 parts/hr ± 0.3”), a confidence level (high / medium / low) signalling how firmly the number should be relied on downstream, and an unbounded markdown detail for the full story.
An output’s values list may be empty. Two valid cases: the derivation hasn’t produced anything for this output yet, or the bound artifact itself is the value — a BOM has no headline number; the BOM is the result.

​
Where Outputs Can Land

Twelve artifact types can bind as output endpoints today: Source nodes can bind to upload, data_integration, or any of those artifact types except report — a pipeline can read from any existing artifact, so one pipeline can chain off the outputs of another.
An output can point at an artifact that doesn’t exist yet. If a derivation lands before its destination is created, the output binds by slug anyway: the pipeline saves with a warning and resolves the reference the next time it’s saved, once the target artifact exists. You don’t have to create the destination first.

​
Authoring

Pipelines are authored by Dexter as part of the analysis that produced them, and land once that analysis has settled — typically at the end of an Exploratory Analysis. While a derivation is still being explored, Dexter keeps the runnable scripts and working notes with the dashboard directly; the pipeline is recorded afterward, so the graph reflects the work actually done rather than a reconstruction from memory. The pipeline surface is the record of a finished derivation, not the place exploration happens. Your role on the canvas is to read, audit, and click through. To change a pipeline, ask Dexter to revise it; Dexter re-saves it under the same slug. Naming discipline matters and shows up on the canvas:
  • Transformation nodes name the action, not the topic: “Join operator codes to name tags on operator_id” reads better than “Operator mapping.”
  • One output per derived result: the fan-out on the canvas shows which transformation produced which value.
  • A goal and a description: a reader should grasp the point of the pipeline from the framing before touching a node.

​
Reading a Value’s Provenance

Any materialized value on a dashboard, a model parameter, or a BOM quantity that was derived through a pipeline is clickable back through it. From the value, open the output node that produced it, read the transformations that shaped it, and open the source it came from. Every non-trivial number in a well-documented ProDex workspace should be traceable this way — that traceability is the whole point of the pipeline surface.

​
Re-Doing a Derivation

When the same derivation is re-done — a fresh upload for the same source, a corrected interpretation of a code, additional outputs added to the same analysis — Dexter saves the pipeline against the same slug. The canvas updates in place, and downstream values on the bound artifacts pick up the new results. A genuinely different derivation (different question, different sources, different destination) gets a new slug and its own pipeline.
Overwrites don’t surface prior revisions. There is no user-facing version history on a pipeline slug and no run-history concept, since pipelines don’t execute. If you need to preserve an old derivation alongside a new one, ask Dexter to save the new work under a new slug instead.

​
Not to Be Confused With

  • Configuration templates: reusable BOM structures with configurable decision points in the BOM configurator. Unrelated to pipelines.
  • Runs: a run is a specific execution of a simulation model. Pipelines don’t have runs — they are documents, not programs.
  • Gates: a gate is a user sign-off checkpoint on a project’s phase. Pipelines have no gates; there is nothing in a pipeline to approve or reject.