Workflows in the Lake

A workflow is the scheduled job that loads a datasource's Lake tables. Use the Workflows view in the Lake to add workflows, change their schedules, run them, read their runs and move tables between them, without leaving the Lake. For how a schedule decides when a run starts, see Schedules and when a workflow starts.

Open the Workflows view

  1. Open Lake in the sidebar and click Workflows in the Lake header. While the Lake is open, press Alt+4.
  2. To open the view on one datasource, right-click it in the Lake tree and choose Show Workflows (Alt+W).

The left panel holds two lists: Datasource over Workflows. Select a datasource, then select one of its workflows. The view starts on the first datasource in the list, and it restores the datasource and workflow you used last when you come back. Type in Filter datasources... to narrow the list. Drag the divider between the panel and the right side to resize it; it stays between about 300 and 500 pixels wide (320 by default).

At the top sits the Runs / Transfer switch. Runs is the default; its tooltip reads View job run history for the selected workflow. Transfer moves tables between workflows; its tooltip reads Move deployed tables between workflows of the same type.

Each datasource row carries small counts, named by their tooltips: Tables, Workflows, Deployed, Pending Deployment and Missing PK, plus a branch icon whose tooltip reads Workflows with no tables. The right side shows the selected datasource and workflow with the workflow's type chip; until you pick a workflow, it reads · pick a workflow.

The view lists only datasources with at least one table. If none qualifies, it reads No datasources ready and asks you to deploy tables first. A filter that matches nothing reads No matching datasources.

Add a workflow

  1. In the Workflows view, select the datasource in the left panel.
  2. Click Add Workflow. It is hidden without Add Lake Workflows, and greyed out until a datasource is selected (hover title: Select a datasource first).
  3. The Add a Workflow form opens with the datasource already chosen. Name the workflow, set its type, schedule and settings, then click Create.

For what each field does, see Create and edit a workflow. A new workflow starts with its schedule switched on.

Run a workflow and switch its schedule

Point at a workflow card in the left panel to show its buttons:

When a run cannot start, Run now sits in a hover panel headed Can’t run yet. The panel explains that none of the workflow's tables are deployed, so it has nothing to load into, and offers Show datasource to open the datasource in the Lake.

Read a workflow's runs

  1. Select the workflow's card in the left panel. Keep the mode on Runs, which is the default.
  2. The right side lists the workflow's runs. Double-click a run to open its nodes and see each step.

Until you pick a workflow, the right side reads Select a workflow and tells you to choose one from the left panel. For the run list and its columns, see Job runs; to find out why a run failed, see Monitor and fix a run. For the runs of a single table, see Run, stop and troubleshoot loads.

Move deployed tables between workflows

  1. Select the workflow that holds the tables in the left panel, then switch the mode to Transfer.
  2. Under From, tick the tables to move in Source tables. Type in Search tables... to narrow the list. Select All ticks every table the list shows that can move; it reads Clear when everything is ticked.
  3. Under To, open Select destination workflow. The view selects the first valid destination for you, and each choice shows how many tables it holds.
  4. Click the round arrow between the two panels. Its tooltip reads Transfer N tables.
  5. The move starts at once, with no confirmation. When it finishes, Success and "Transferred N table(s)." appear, and the tables show under the target workflow, marked New for a moment.

The steps track along the top: Select tables, Choose destination, Confirm transfer. While the workflows reload, the destination side reads Updating workflows.... The button under the arrow, Swap source and destination workflows, trades the two sides so you can move tables the other way; it clears your ticked tables. If a move fails, an Error toast appears and your selection stays.

What can move between workflows

Permissions

ToYou need
See the view, its workflows and runsView Lake Workflow Objects
Add a workflowAdd Lake Workflows
Edit a workflow or switch its scheduleModify Lake Workflows
Move tables into a workflowModify Lake Workflows on the destination workflow
Use Run nowExecute Lake Workflow Objects
Delete a workflowRemove Lake Workflows

Without a permission, its button is hidden or greyed out. For the full list, see Workflow permissions and Who can do what in the Lake.

Limits and edge cases

Questions, answered

Why is my datasource missing from the Workflows view?

The view lists only datasources that have at least one table. Import a table into the datasource first. See Import tables. If no datasource qualifies at all, the view reads No datasources ready; an active filter with no match reads No matching datasources.

Why is the Run now button greyed out?

Hover it to read the panel headed Can’t run yet: none of the workflow's tables are deployed, so it has nothing to load into. Deploy at least one of its tables, then run the workflow. See Deploy tables. A workflow with no tables at all shows no Run now button.

Why did my new workflow start running on its own?

New workflows are created with Schedule Status on and a starting frequency of every minute. Open Edit to change the frequency, or turn the schedule off with the power toggle on the card. See Schedules and when a workflow starts.

Why can't I tick a table to transfer?

Only deployed tables with a primary key can move. A table without a key shows No PK and cannot be ticked, even when it is deployed; add a key on the table's Columns step (see Configure a table). A table that is not deployed does not appear in the list at all.

Why is the transfer arrow greyed out?

It switches on when you have ticked at least one table and a destination workflow is chosen. It is also off while a transfer is running. When you cannot modify any destination workflow, its tooltip reads You do not have permission to modify any destination workflow here.

Why doesn't the destination list show the workflow I want?

The view offers only workflows of the same type on the same datasource, it never offers replication workflows, and it leaves out workflows you are not allowed to modify. If nothing qualifies while tables are ticked, it warns No other workflows with the same type on this datasource.

What does the swap button do?

Swap source and destination workflows trades the two sides of the view, so you can move tables in the other direction. A swap needs a chosen destination workflow, and permission to modify the workflow that will receive the tables after the swap. It clears your ticked tables, so tick them again for the new direction.

How do I see what a run did?

Select the workflow, keep the mode on Runs and double-click the run. Its nodes open, so you can see each step. For the full steps, see Monitor and fix a run and Job runs.

How do I pause loads for one datasource?

Turn off the schedule on each of its workflow cards with the power toggle. Turn the schedules back on to resume the loads.

Why can't I delete a workflow from here?

Delete does not appear on system-generated workflows, and it is hidden without Remove Lake Workflows. A workflow that still has tables refuses the delete: move its tables to another workflow with Transfer first, then delete the empty workflow.