Monitor and fix a run

Every run of a workflow is listed in the runs table with its status. Use this page to open a run, see its nodes, find out why it failed, and then restart, stop or delete it.

Open the nodes of a run

Each run has a list of nodes, the steps it loads or transforms. To open it, do one of these:

  1. Open the menu at the end of the run's row and choose View.
  2. Double-click the run's row.
  3. Click the workflow name in the row.

The window title shows the workflow name, and a number next to it shows how many nodes are visible. The footer shows Current Progress: N%, an estimate from how long recent runs of the workflow took, with a progress bar while the run is Running. Next to it are the buttons you can use: Restart, Stop and Delete. For each node type and status, see Nodes in a run.

The window updates on its own while it is open. Close it with the button in the top corner, whose tooltip reads Close. If no run is selected, the window reads Job Nodes and shows No job selected.

Which buttons appear depends on the run's status:

Run statusRestartStopDelete
RunningNoYesNo
Running ErrorNoYesNo
FailedYesNoYes
AbortedYesNoYes
FinishedYes, but nothing rerunsNoNo

Find why a run failed

  1. Double-click the run with status Failed in the runs table.
  2. Look for nodes with status Error. Click the node's status to open Error Stack Trace, which shows the full error. Click Copy to copy it.
  3. Click the node's name to open the table it loaded, or the transformation it ran in a Warehouse workflow. Fix the problem there. For the meaning of a failed step's message, see Troubleshooting workflows.
Note

A status opens the error only when the node has error text. Skipped nodes and system nodes cannot be opened from their name.

Restart a run

Restart reruns the same run. Use it after you have fixed the cause of the failure.

  1. Open the run's menu and choose Restart, or click Restart in the node window.
  2. Confirm in the Restart Job dialog, which asks "Are you sure you want to restart this job?".

When you restart a run:

You cannot restart a run while another run of the same workflow is live. DataMind OS shows The job is running, please abort it before restarting. On success, you see Job restarted successfully.

Stop a run

Stop ends a run that is still going.

  1. Open the run's menu and choose Stop, or click Stop in the node window.
  2. Confirm in the Stop Job dialog, which asks "Are you sure you want to stop this job?".

When you stop a run:

Stopping a run that has already ended as Failed or Aborted is refused with The Job is already in error state. There is nothing left to stop.

A stopped run is not restarted automatically. It also pauses its workflow's schedule, as described in Schedules and when a workflow starts.

Delete a run

Delete removes a run from the runs table. Use it for failed or stopped runs you no longer need.

  1. Open the run's menu and choose Delete, or click Delete in the node window.
  2. Confirm in the Delete Job dialog, which asks "Are you sure you want to delete this job?".
Warning

Deleting a run cannot be undone. A running run cannot be deleted, so stop it first.

On success, you see Job deleted successfully. If the run has already been deleted, Delete is refused with a message that the job does not exist. Refresh the runs table.

Retry temporary failures automatically

Set Restart Count in the workflow's form, so that DataMind OS restarts a failed run by itself. For where this field sits, see Create and edit a workflow.

SettingWhat it doesDefault
Restart CountHow many times DataMind OS restarts the latest failed run of the workflow by itself. Accepts 0 to 10.0
Ignore ErrorLets the schedule keep starting runs after a failed or stopped run.Off

How automatic retries behave:

Permissions you need

ActionPermission
Restart, StopExecute Lake Workflow Objects, Execute Warehouse Workflow Objects or Execute WriteBack Workflow Objects, for the workflow type of the run
DeleteRemove Lake Workflows, Remove Warehouse Workflows or Remove WriteBack Workflows, for the workflow type of the run

Without the permission, the buttons are hidden in the node window and greyed out in the run's menu. The server also refuses the action. See Workflow permissions.

Questions, answered

Do I need to start the whole workflow again after I fix a table?

No. Restart the failed run. Only the nodes that did not finish run again.

Why did tables I added not load after a restart?

A restart runs the nodes planned when the run started. Start a new run with Run to include tables added since then.

Why is Restart greyed out on my run?

The run is still live, so it can only be stopped. Stop it first, then restart it. Restart also needs the Execute permission.

Why is Delete greyed out on my run?

Only runs with status Failed or Aborted can be deleted, and a finished run cannot be. Stop a running run first, then delete it. Deleting also needs the Remove permission for the workflow's type.

Will Restart Count retry a run that failed on a SQL error?

No. Only temporary database problems are retried, such as running out of memory, too many simultaneous queries, no free connection, a socket or query timeout or a network error. A wrong SQL query or a missing table stays failed until you fix it.

Why did my schedule stop after I stopped a run?

An Aborted run pauses the schedule, like a Failed run does. Run the workflow again with Run, or turn on Ignore Error in the workflow's settings to keep the schedule running after a failed or stopped run. See Schedules and when a workflow starts.

Does deleting a failed run let the schedule start again?

Yes, if the run before it finished. The schedule checks the latest run that is left. If that run finished, the schedule continues.

What does Aborted by User mean?

Someone stopped the run with Stop.

Why does Table "orders" not found appear when I click a node?

The table was deleted or renamed after the run started, so DataMind OS cannot find it by the node's name. The message reads Table "<node name>" not found. In a Warehouse workflow, it reads Transformation "<node name>" not found instead.

Why was my restart refused because a data source is being loaded?

Another run is loading the same data source, and a data source is loaded by one run at a time. The message names that run: Data source "<code>" is being loaded by job <job ID> (<workflow name>). Restart once that job finishes.

Does automatic retry work when the Workflow Scheduler is paused?

No. While the Workflow Scheduler is paused, no scheduled workflows run and no automatic retries happen. Restart the failed run by hand once the scheduler is back on.

What happens to a running run when the platform restarts?

The run resumes by itself. Steps that did not finish run again from their start, and the interrupted work is closed with a note in the run's history. You do not need to restart the run; watch its progress in the node window.