Loading the documentation…
Every run of a workflow is listed in the runs table with its status. Use this page to open a run, see its nodes, find out why it failed, and then restart, stop or delete it.
Each run has a list of nodes, the steps it loads or transforms. To open it, do one of these:
The window title shows the workflow name, and a number next to it shows how many nodes are visible. The footer shows Current Progress: N%, an estimate from how long recent runs of the workflow took, with a progress bar while the run is Running. Next to it are the buttons you can use: Restart, Stop and Delete. For each node type and status, see Nodes in a run.
The window updates on its own while it is open. Close it with the button in the top corner, whose tooltip reads Close. If no run is selected, the window reads Job Nodes and shows No job selected.
A Pending node that waits for a table another run or a schema change uses shows an hourglass Waiting chip next to its status, in the node window and on the workflow diagram. Hover the chip to read why it waits, for example Waiting for "Customers [CRM]" held by job #812 (Customers). The node starts by itself when the table is free. A Meltano load node waits the same way for the previous load of its data source to end in the Meltano service; its note names landing area of data source <code>. To see the holder and the line of waiting work, open ETL locks.
Which buttons appear depends on the run's status:
| Run status | Restart | Stop | Delete |
|---|---|---|---|
| Running | No | Yes | No |
| Running Error | No | Yes | No |
| Failed | Yes | No | Yes |
| Aborted | Yes | No | Yes |
| Finished | No | No | No |
A status opens the error only when the node has error text. Skipped nodes and system nodes cannot be opened from their name.
Restart reruns the same run. Use it after you have fixed the cause of the failure.
When you restart a run:
A Finished run cannot be restarted; start a new run with Run instead. A run with the run scope Schema change has no Restart: a queued schema change runs once, so make the change again instead. You cannot restart a run while another run of the same workflow is live. DataMind OS shows The job is running, please abort it before restarting. On success, you see Job restarted successfully.
Stop ends a run that is still going.
When you stop a run:
Stopping a run that has already ended as Failed or Aborted is refused with The Job is already in error state. There is nothing left to stop.
A stopped run is not restarted automatically. It also pauses its workflow's schedule, as described in Schedules and when a workflow starts.
Delete removes a run from the runs table. Use it for failed or stopped runs you no longer need.
Deleting a run cannot be undone. A running run cannot be deleted, so stop it first.
On success, you see Job deleted successfully. If the run has already been deleted, Delete is refused with a message that the job does not exist. Refresh the runs table.
A run that has ended but whose locks are still being released cannot be deleted yet: Job #<id> is still releasing its ClickHouse work (<k> lock(s)); try again when the Locks tab no longer lists it. Deleting the workflow is refused the same way, with Workflow "<name>" still has job #<id> releasing its ClickHouse work; try again in a minute. Wait a moment, then delete again.
Set Restart Count in the workflow's form, so that DataMind OS restarts a failed run by itself. For where this field sits, see Create and edit a workflow.
| Setting | What it does | Default |
|---|---|---|
| Restart Count | How many times DataMind OS restarts the latest failed run of the workflow by itself. Accepts 0 to 10. | 0 |
| Ignore Error | Lets the schedule keep starting runs after a failed or stopped run. | Off |
How automatic retries behave:
| Action | Permission |
|---|---|
| Restart, Stop | Execute/Stop Lake Workflow Objects, Execute/Stop Warehouse Workflow Objects or Execute/Stop WriteBack Workflow Objects, for the workflow type of the run |
| Delete | Delete Lake Workflows, Delete Warehouse Workflows or Delete WriteBack Workflows, for the workflow type of the run |
Without the permission, the buttons are hidden in the node window and greyed out in the run's menu. The server also refuses the action. See Workflow permissions.
No. Restart the failed run. Only the nodes that did not finish run again.
A restart runs the nodes planned when the run started. Start a new run with Run to include tables added since then.
The run is still live, so it can only be stopped. Stop it first, then restart it. A Finished run cannot be restarted; start a new run with Run. Restart also needs the Execute permission.
Only runs with status Failed or Aborted can be deleted, and a finished run cannot be. Stop a running run first, then delete it. Deleting also needs the Delete permission for the workflow's type.
No. Only temporary database problems are retried, such as running out of memory, too many simultaneous queries, no free connection, a socket or query timeout or a network error. A wrong SQL query or a missing table stays failed until you fix it.
An Aborted run pauses the schedule, like a Failed run does. Run the workflow again with Run, or turn on Ignore Error in the workflow's settings to keep the schedule running after a failed or stopped run. See Schedules and when a workflow starts.
Yes, if the run before it finished. The schedule checks the latest run that is left. If that run finished, the schedule continues.
Someone stopped the run with Stop.
The table was deleted or renamed after the run started, so DataMind OS cannot find it by the node's name. The message reads Table "<node name>" not found. In a Warehouse workflow, it reads Transformation "<node name>" not found instead.
Another run is loading the same data source, and a data source is loaded by one run at a time. The message names that run: Data source "<code>" is being loaded by job <job ID> (<workflow name>).
No. While the Workflow Scheduler is paused, no scheduled workflows run and no automatic retries happen. Restart the failed run by hand once the scheduler is back on.
The run resumes by itself. Steps that did not finish run again from their start, and each one gets a note that starts with Interrupted:. When the server running a step stopped responding, the note also counts the interruptions, for example (1/, and after three of them the step fails with Interrupted 3 times: the executor died while running this node. You do not need to restart the run; watch its progress in the node window.