A workflow is a scheduled group of loads: its name, its schedule and the tables or transformations it runs. Use this page to learn how workflows, runs and nodes fit together, where each workflow comes from and which runs count.
Every workflow has one type. The type decides what the workflow runs and which permissions apply to it.
| Type | What it runs | Belongs to |
|---|---|---|
| Lake | The Lake tables assigned to it | One Lake data source |
| Warehouse | Its transformations, in dependency order, together with their history steps | The Warehouse |
| WriteBack | The Writeback tables assigned to it | One Writeback destination |
Each Lake or Writeback table is assigned to one workflow. A Warehouse workflow always shows the Warehouse icon; a Lake or WriteBack workflow shows its data source's icon.
A run is made of nodes. A node starts when the nodes it depends on are done. A run ends Finished when none of its nodes fails, and Failed when one of them ends in an error. In a Warehouse run, a gate node waits for a table that another workflow fills.
DataMind OS creates some workflows for you, and you can add your own.
You can add your own workflows to a Lake data source, a Writeback destination or the Warehouse. The Workflows card on the Dashboard has an Add workflow action that opens the create dialog directly. For the full steps, see Create and edit a workflow.
System-made workflows cannot be removed. A system-made workflow appears on the Workflows screen only once it has at least one table, or, for a Warehouse workflow, at least one transformation whose schedule is on.
The left panel lists your workflows. The right panel shows the status timeline, a row of workflow tabs and the runs table. For a tour of each part, see The Workflows screen. The workflow list and the runs table update by themselves while a screen is open, so you do not need to reload the page. To reload the runs table on demand, click the refresh button, which shows the tooltip Refresh jobs.
Every run has a Run Scope and an Execution Type.
| Column | Values | What it means |
|---|---|---|
| Run Scope | Workflow, Single Node, Test | What the run covers: a full workflow run, one table sync or transformation run started from the Lake or the Warehouse, or a transformation test. |
| Execution Type | Automatic, Manual | Who started the run: the schedule, or a person. |
Single Node and Test runs are hidden by default. Only Workflow runs show until you use the Run Scope column filter. Only a full Workflow run counts for a schedule slot or for an automatic retry. See Job runs and Schedules and when a workflow starts.
To start a run yourself, outside its schedule, see Run a workflow by hand.
The Workflows item in the sidebar appears when you hold at least one of these permissions:
You see only the workflow types you hold a view permission for. Workflows and runs of any other type stay hidden from you. For every permission, see Workflow permissions.
The Workflows screen works on desktop only. On a phone-sized screen, DataMind OS opens Minds instead.
DataMind OS creates them together with the source or destination: Standard, Snapshot and Replication for a Lake data source, and Standard for a Writeback destination. They appear in the list once they have at least one table.
The workflow is the definition and its schedule. A run is one execution of the workflow. A node is one step of that run, such as one table load or one transformation.
They are the workflow's type, shown after its name. DataMind OS creates all three for a Lake data source, and Standard for a Writeback destination. When you register a workflow yourself, you can choose Standard or Snapshot; Replication is not offered in the form. The type cannot be changed after creation.
System-made workflows without tables are hidden. Assign tables to one of them from the Lake, and it appears. See Workflows in the Lake.
No. A workflow belongs to one data source. You choose the data source when you create the workflow, and you cannot change it afterwards.
No. Only a full Workflow run counts for a schedule slot or for an automatic retry. A table sync, a transformation run or a test is a Single Node or Test run: it never uses the schedule's slot and never holds the schedule back. These runs stay hidden in the runs table until you filter the Run Scope column. See Schedules and when a workflow starts and Job runs.
No. On a phone-sized screen, DataMind OS opens Minds instead.
No. Statuses, the workflow list and the runs table update by themselves. Click Refresh jobs only if you want to reload the runs table on demand.
When at least one of its nodes ends in an error. A run without such a node ends Finished. To find the failing node and act on it, see Monitor and fix a run.