Create and edit a workflow

A workflow holds the schedule and settings for the work it runs: loading a data source's tables in the Lake, running Warehouse transformations, or pushing Writeback tables to a destination. Use this page to add a workflow, change its settings later, or delete one that no longer has tables or transformations assigned.

Add a workflow

On the Workflows screen, click +. Its tooltip reads Add Workflow. You can also click Add workflow on the Workflows card on the Dashboard, or Add Workflow in the Workflows view of the Lake or Writeback.
The dialog's header reads Add a Workflow; its main panel is titled Register Workflow. If the Lake / Warehouse switch appears at the top, choose the kind you need. The switch appears only when you create a workflow, you can add Warehouse workflows, and the Warehouse is set up. Choosing Warehouse hides Datasource and Workflow Type, because the workflow belongs to the Warehouse.
Unless you chose Warehouse, open Datasource and choose a Lake data source or a Writeback destination. The list shows only what you may add, and it reads Select a datasource until you pick one. Type in Search datasources… to filter the list; when nothing matches, it reads No datasources found. Then choose a Workflow Type: Standard or Snapshot. When you open the dialog from a source's or destination's Workflows view, Datasource is already filled in for it and locked.
Enter the Workflow Name, Batch Size and Restart Count.
Under Schedule Settings, choose a Unit. Fill in Frequency, Schedule Day and Start Time. For Minute or Hour, also fill in End Time. Check the sentence under Timezone: it starts with your browser's time zone, and the times you enter are read in it. To use another one, click Change and search by city or country.
Under Workflow Settings, set Schedule Status and Ignore Error.
Click Create. A success notification reads Metadata created successfully.
Put work on the workflow: tables from the Lake or Writeback, or Warehouse transformations. Then run it once and follow it in Job runs.

Settings

Setting	What it does	Default
Datasource	The source or Writeback destination the workflow belongs to. It cannot be changed after creation.	None. Choose one.
Workflow Type	Standard or Snapshot. It cannot be changed after creation.	Standard
Workflow Name	The name of the workflow. It must be at least 2 characters long and unique. It cannot contain the words standard, snapshot or replication.	Empty
Batch Size	How many nodes of one run may work at the same time. Enter 1 to 100.	10
Restart Count	How many times DataMind OS restarts a failed run by itself, when a step failed with a temporary database error. Enter 0 to 10. See Monitor and fix a run.	0
Unit	How the schedule repeats: Minute, Hour, Day or Month.	Minute
Frequency	The gap between runs, from 1 to 100.	1
Schedule Day	The day of the month for a Month schedule, from 1 to 31. Used only with Month.	1
Start Time	The first time the schedule may start a run.	00:00
End Time	The last time the schedule may start a run. Shown only for Minute and Hour, and it must be after Start Time.	23:59
Timezone	The time zone that Start Time, End Time and Schedule Day are read in. Editing the workflow keeps it unless you choose another.	Your browser's time zone
Schedule Status	On lets the schedule start runs. Off stops it, the same as Off on the card.	On
Ignore Error	When on, the schedule keeps starting runs after a failed or aborted run.	Off

For how these fields decide when a run starts, see Schedules and when a workflow starts.

Note

For an ANY API data source, a hint under Batch Size shows the cap on how many ANY API tables may load at once across all workflows. When your number is higher than the cap, the hint adds that only the capped number of these run in parallel. An administrator sets the cap under Admin > System Configuration > Health; see Scheduler, health, maps and connector safety.

Permissions

Your permissions depend on the type of workflow.

To	You need
Add a workflow	Add Lake Workflows, Add Warehouse Workflows or Add WriteBack Workflows
Edit a workflow, or switch its schedule on or off	Modify Lake Workflows, Modify Warehouse Workflows or Modify WriteBack Workflows
Delete a workflow	Remove Lake Workflows, Remove Warehouse Workflows or Remove WriteBack Workflows

Without an add permission, the + button is hidden, and Datasource lists only what you may add. Without a modify permission, the card shows View instead of Edit: the dialog opens read-only with locked fields, and it has no buttons at the bottom. For the full picture, see Workflow permissions.

Edit a workflow

Hover the workflow's card and click Edit. You can also right-click its tab and choose Edit.
Change the fields you need. Datasource and Workflow Type stay locked.
Click Save Changes. The button stays disabled until you change a field. A success notification reads Metadata updated successfully.

Delete a workflow

Move its work to another workflow: tables in Workflows in the Lake or Run, schedule and monitor Writeback, and transformations in Rearrange transformations.
Hover the card and click Delete, or right-click its tab and choose Delete.
In Delete Metadata, click Confirm.

Limits and edge cases

A Batch Size above 100 is refused. The field goes up to 1000, but the save accepts at most 100, so enter 100 or less. The refusal arrives as a toast titled Failed to create workflow, or Failed to update workflow when you edit.
A name that another workflow already uses cannot be saved. The save fails with the same kind of toast.
A name that contains standard, snapshot or replication, in any capitalisation, is refused even inside another word. A Validation Error toast names the words: "Workflow name cannot contain reserved words: standard, snapshot, replication". The check is skipped when you keep a workflow's existing name, so a system-made name such as "Orders (Standard)" can stay.
For Minute and Hour, End Time must be set and after Start Time. Otherwise the save is refused with "End time is required when unit is Minute or Hour" or "End time must be after start time".
The data source is fixed once the workflow exists. A workflow cannot be moved to another data source; the attempt is refused with "Workflow cannot be changed. Please create a new sub-workflow instead." To load from another data source, create a new workflow.
A new workflow does not start on its schedule until it has run once. A Lake or Writeback workflow also cannot run until at least one of its tables is deployed. See Schedules and when a workflow starts.
Delete is refused while tables or transformations are assigned. The message gives the count, for example "Orders still has 3 table(s) assigned and cannot be removed. Move them to another workflow first."
DataMind OS does not delete system-generated workflows. Their cards still show Delete, but the server refuses with "Orders (Standard) is a system generated workflow and cannot be removed."

Questions, answered

Why does saving say the name cannot contain reserved words?

The name contains standard, snapshot or replication, in any capitalisation, even inside another word such as my-replication. Remove the word. When you edit a workflow, keeping its existing name is allowed.

What happens if I enter 200 for Batch Size?

Saving fails. DataMind OS accepts 1 to 100, so enter 100 or less. The field goes higher than that, but the save does not accept more than 100. See Troubleshooting workflows.

What does Batch Size change?

It sets how many nodes of one run work at the same time. Higher values let large workflows finish sooner, but they put more load on the source and the database. Nodes above the number wait and start as others finish.

Why is Datasource greyed out when I edit?

The data source and the workflow type are fixed once the workflow exists. To load from another data source, create a new workflow.

Why don't I see the Lake or Warehouse switch?

The switch appears only when you create a workflow, you can add Warehouse workflows, and the Warehouse is set up. It never appears when you edit a workflow.

Why does Delete say the workflow still has tables assigned?

The workflow still holds tables or transformations. Move them to another workflow, then delete again. The message gives the count, and the Delete a workflow steps on this page list where to move each kind.

Can I delete the (Standard) workflow of my source?

No. DataMind OS creates these workflows for each source and Writeback destination, and it refuses to delete them.

Why is Create or Save Changes greyed out?

The button turns on only after you change a field. When editing, if the dialog is locked entirely and shows no buttons at the bottom, you lack the modify permission for this workflow type.

What is the difference between Add a Workflow and Register Workflow?

They are two labels on the same dialog: Add a Workflow in the header at the top, and Register Workflow on the main panel below it. When you edit a workflow, both read Edit Workflow.

Can I rename a workflow after I create it?

Yes. Edit the workflow and change Workflow Name. The rules stay the same: at least 2 characters, no standard, snapshot or replication, and not a name another workflow already uses. The reserved-word check is skipped only while the name stays exactly as it is.