Pipelines

Beta

​

Overview

A pipeline is repeatable data preparation that Dexter builds for you. It reads defined inputs, such as uploaded files or a connected integration. It applies the same transformations and checks every time, and it produces the same kind of output, such as a dataset or a report. You describe what should come in and what should come out. Dexter asks about the rest, writes the steps as scripts, and tests them on your data. Pipelines live on the Pipelines tab of the Data page, where you run them and inspect every stage. Use a pipeline when the same data arrives again and again, like a weekly MES export or a monthly stock report, and should be prepared the same way each time. For a one-off question, just ask Dexter. For a recurring procedure that needs judgment or a conversation on every run, use a custom workflow instead.

​

Creating a Pipeline

Once the Pipelines tab has at least one pipeline, a Create pipeline button appears at the top of the tab. It opens the ETL pipeline form:

• Name
• Sources: what files or data come in, their naming patterns and how often they arrive
• Transformation: what should happen to the data, such as joins, business rules and clean-up
• Output: the dataset or report every run must produce
• Validation checks (optional): conditions your data should meet. A failed check is flagged on the run as a warning; if a condition should stop the run outright, say so and Dexter builds it that way. If you leave this empty, Dexter proposes checks and confirms them with you.
• Source files (optional) and Integrations (optional): existing uploads and integrations to build from

Create workflow sends the form to Dexter in chat. Dexter asks clarifying questions before it builds anything. While the tab is still empty, ask Dexter in chat to build a pipeline. You can also open the same form from the + on the Workflows section of the Dexter sidebar, then choose ETL pipeline. Either way, expect Dexter to confirm the details with you before it writes the pipeline. That means where each input comes from and how often, what one row represents, what the columns and codes mean, the business rules, and which checks should only warn and which should stop a run.

​

What Dexter Builds

A pipeline is a sequence of steps, each backed by a script. The first step checks that the inputs are present and in the expected shape, and stops the run with a specific error if they aren’t. Each business stage the pipeline produces is saved as a versioned table in Datasets, so you can inspect every stage, not only the final result. The final output is a table or a report file. Before finishing, Dexter runs the pipeline against your data, checks the outputs, and walks you through the result. To change a pipeline, such as a new column, a different rule or an extra check, ask Dexter to revise it.

​

The Pipeline Page

Open a pipeline from the Pipelines tab. Its page puts everything in one place:

• Run. Starts a run. Its menu offers Run all steps and Run from a selected step. Each run shows its state (Running, Succeeded, Failed, Waiting for a decision) and how it started (Started manually, Started by Dexter, Started by schedule).
• The steps. View them as a Pipeline graph, Steps as nodes, or a Step list. Select a step to see its Script, its Events (what it read and wrote), and a Summary. Generate summary has Dexter describe what the script does.
• Versioned table outputs. Every run that changes a table publishes a new version (v1, v2, and so on). A run that writes the same rows keeps the existing version. For each version you can see the Changes since the previous version and its change note. Open a table in Datasets to work with it. To ask Dexter about a value, annotate its cell.
• Checks. A check flags rows in a published table that are wrong, such as negative stock where that’s impossible, or a code missing from the master list. A failing check doesn’t stop the run. The run records which rows failed so you can fix the source data or the rule and run again.
• Fix with Dexter. When a run fails, Fix with Dexter opens a conversation where Dexter works on the failure. When a run succeeds but some checks fail, Investigate does the same for the failing checks.
• Decisions. A step can pause for a judgment call from Dexter or you. The run shows Waiting for a decision, and Open Dexter takes you to the conversation where it’s being settled. The run then continues through its remaining steps.
• New input files. A file input can be tied to a folder in Uploaded files, and each run then reads the newest file in that folder. Upload new version on the input adds a file there, and Run pipeline on the confirmation starts a run with it. Older files stay as history.
• Schedule. If your organization also has Scheduled Jobs, Schedule lists the pipeline’s schedules. Use New schedule to describe when it should run, and Dexter confirms the schedule with you before saving it. Each schedule has Run now and Pause or Resume.
• Edit details. In the page menu. Changes the pipeline’s name and description.

​

Older Pipelines

The Pipelines tab can also list older pipelines. Their cards carry a Legacy badge, and when you open one, its title shows Legacy Read only. They have no Run button:

• Records of earlier analyses. A graph of the sources that were read, the transformations that ran, and the outputs they produced. Nothing runs them; Dexter can read them and build on what they found.
• Pipelines built as workflows, before pipelines were set up for your organization. These still run from the Workflows section of the Dexter sidebar, like any custom workflow. Ask Dexter to port one if you want it to run from the Pipelines tab. The ported pipeline writes new tables, and Dexter tells you which apps or other workflows it couldn’t move over to them.

​

Not to Be Confused With

• Custom workflows: recurring procedures Dexter follows in conversation, with room for judgment and questions on every run. A pipeline is for preparation that should run the same way every time.
• Runs: simulation runs of a model. Pipeline runs are separate and appear only on the pipeline’s own page.
• Configuration templates: reusable BOM structures in the BOM configurator. Unrelated to pipelines.