Loading the documentation…

Lake

The Lake is where DataMind OS keeps raw copies of your source data, exactly as it arrived. You connect your external systems, bring their tables in and load them. Start here when you add a new source, or when you want to know how its data reaches DataMind OS.

How a source becomes a loaded table

Every table moves through the same four steps:

  1. Connect a datasource, which is one connection to an external system.
  2. Import the tables you want. Importing registers them in the Lake. It does not load any data.
  3. Configure each table, then Deploy it. Deploying creates the table in the Lake with its current column settings.
  4. Load the table, either by hand or through its workflow.

Open the Lake screen

  1. Click Lake in the sidebar.

If you open the Lake without a view, it returns to the last Dashboard, Relationship Diagram or Workflows view you used, provided that was within the last 24 hours. Otherwise it opens Dashboard.

The right side of the screen has four header buttons:

ButtonShortcutWhat it opens
DashboardAlt+2Your datasources and their tables.
Relationship DiagramAlt+3Your Lake tables on a diagram.
WorkflowsAlt+4Lake workflows. Shown only if you can view Lake workflows.
Add datasourceAlt+NThe steps to connect a new source.

Alt+1 shows or hides the left panel, which holds the Lake tree. The tree is organised as folder, datasource, database, schema, table and column. A datasource that is not in a folder sits at the top level.

Connect a datasource

  1. Click Add datasource, or press Alt+N, and choose the source type.
  2. Fill in the connection steps. Use Back and Next to move between them, and Revert to reset the form.
  3. On the last step, click Add.

When the datasource is created, the Lake switches to Dashboard and opens the new datasource in a tab. For most source types, it then opens the source panel so you can import its tables. If the datasource is still processing or installing, the source panel opens when it is ready. You can also open the source panel at any time from a datasource's menu with View Source.

Import tables

  1. In the source panel, tick the tables you want.
  2. Click Import. The button stays disabled until you have selected something.

Importing does not load data. The imported tables wait until you configure, deploy and load them.

Configure and deploy a table

  1. Open the table from the Lake tree.
  2. On the Table tab, open General. Set Sync Type and Workflow Name.
  3. Open Columns and make sure at least one column is a primary key.
  4. Click Deploy in the banner at the top of the table.

Until the table is deployed, its Settings tab shows Table Not Deployed, or Primary Key Required when no column is a primary key, in place of its settings. For the load patterns you can choose in Sync Type, see Choose a sync type.

Load a table

You can load a table in one of two ways.

By hand: open the table, go to the Settings tab and click Synchronize.

On a schedule: on Table > General, keep Synchronize switched on in the Synchronization section. The table is then included in its workflow, and the workflow loads it on schedule. To set up and run workflows, see Workflows in the Lake.

To see what a load did, open the table's Sync Log. For stopping, aborting and troubleshooting loads, see Run, stop and troubleshoot loads.

Who can do what

Four permissions control the Lake. Your administrator assigns them through your role.

PermissionWhat it lets you do
View Lake ObjectsSee the Lake, its datasources and its tables.
Create/Edit Lake ObjectsConnect datasources, import tables and change their settings. It also lets you view the Lake.
Execute/Stop Lake ObjectsLoad tables and abort a running load.
Delete Lake ObjectsDelete datasources and tables.

The Workflows button needs its own Lake workflow permissions. For the full list, see Who can do what in the Lake.

Limits and edge cases

Where to go next

Questions, answered

Why don't I see Lake in the sidebar?

Your role needs View Lake Objects or Create/Edit Lake Objects. Without either, Lake does not appear in the sidebar. Ask your administrator to check your role.

What is the difference between importing and deploying a table?

Importing registers a source table in the Lake, so you can configure it. Deploying creates the table in the Lake with its current column settings. Only a deployed table can be loaded.

Does importing a table load its data?

No. Data lands only when the table is loaded, either with Synchronize on its Settings tab or through its workflow.

Why does the Settings tab show Table Not Deployed?

The Settings tab shows its actions and the Sync Log only after the table is deployed. Click Deploy in the banner at the top of the table. If the tab shows Primary Key Required, define a primary key first. See Configure a table.

Can I get back to the view I used last?

Yes, if you used it within the last 24 hours. Opening Lake brings back the last Dashboard, Relationship Diagram or Workflows view. After 24 hours, it opens Dashboard.

Which keyboard shortcuts work on the Lake screen?

Alt+1 shows or hides the left panel. Alt+2 opens Dashboard, Alt+3 opens Relationship Diagram, Alt+4 opens Workflows and Alt+N opens Add datasource. Alt+4 does nothing if you cannot view Lake workflows.