How DataMind OS fits together

DataMind OS moves data in one direction. Sources feed the Lake, the Lake feeds the Warehouse, and the Warehouse feeds queries, Minds, reports, presentations and Writeback. Read this page to see which area does which job and where each one lives in the sidebar.

Follow data from a source to a result

Under Data pipeline on the Dashboard, four numbered cards show the order of work:

  1. Lake lands raw data from your sources. Start with Connect a source and load your first table.
  2. Warehouse shapes the Lake data into clean, modelled tables. Start with Warehouse.
  3. Writeback sends curated data back to your source systems. Start with Writeback.
  4. Workflows schedule the loads for the Lake, the Warehouse and Writeback, and show how each run goes. Start with Workflows.

Once the data is loaded and shaped, use it in one of these areas:

AreaWhat you do thereStart here
EditorQuery and explore your data with SQL.Editor
MindsAsk anything across your data.Minds
DataPrismBuild reports and explore semantic models.DataPrism
SlidesBuild and present decks from your data.Slides

Know the Warehouse building blocks

In Warehouse, the Add New screen offers these objects:

ObjectWhat it isRead more
DimensionDescriptive data, such as customers, products or dates, that facts join to.Create a dimension or fact table
FactMeasurable events, such as sales or orders.Create a dimension or fact table
TransformationThe query that fills a dimension or fact table from Lake data.Build, test and finalize a transformation
DictionaryLookup values, filled by SQL queries and refreshed on a schedule.Dictionaries
StaticReference data you type or upload yourself, with no live source.Static tables
ViewAn object defined by a SQL query.Views
Semantic source and Logical modelGovernance objects. Every dimension and fact table names a logical model and a source system, and each source system belongs to a semantic source.Semantic sources, source systems and logical models

See what watches the data

Observability shows platform health and performance. Lineage traces each table from source to finish, and it covers tables, Minds, DataPrism models and Slides decks. Use it to see what an object depends on (upstream) and what depends on it (downstream). Start with Observability and Lineage.

Take a query into the Editor

Several areas hand their SQL to the Editor, which opens it in a new console:

FromButton
A query in a Mind's answerOpen in Query Editor
The SQL of a DataPrism report chartOpen in Query Editor
The query of a Warehouse transformation or viewOpen in Editor
The query of a Writeback tableOpen in Editor
A query in Query Management, in ObservabilityOpen in Editor

These buttons appear only if you have Execute Query or Add/Remove Consoles.

Choose the right kind of Mind

A Mind is an AI agent you chat with about your data. You pick its kind when you create it.

KindWhat it does
Analytical MindAnswers questions over the tables you link in Sources.
Warehouse BuilderBuilds facts, dimensions, views and transformations for you. It sees every Lake and Warehouse table and is owned by the admin role.
Lake BuilderImports source tables into the Lake and manages connectors, syncs and load settings for you. It sees every Lake and Warehouse table and is owned by the admin role.
Note

The Warehouse Builder and Lake Builder options appear in the Mind form only if your role lets you create them.

Find each area in the sidebar

The sidebar on the left lists every area your role can open. It has four groups, separated by lines:

Find your way around explains how to use the sidebar and the account menu, including the small BETA and SOON badges on some items and More, where items move when your window is too short to show them all.

The sidebar spells the area WriteBack and the area Admin. The Dashboard cards call them Writeback and Administration.

Your administrator gives you one or more roles, and each role carries permissions. The permissions decide which sidebar items you see:

Sidebar itemShown when you have
WorkflowsView Lake Workflow Objects, View Warehouse Workflow Objects or View WriteBack Workflow Objects
MindsView Minds
EditorExecute Query or Add/Remove Consoles
DataPrismView DataPrism Reports or View DataPrism Semantic Models
SlidesView Decks or View Others Decks
LakeView Lake Objects
WarehouseView Warehouse Objects
WriteBackView WriteBack Objects
LineageAny one of View Lake Objects, View Warehouse Objects or View WriteBack Objects
AdminAny Administration permission, for example View Users or View Roles
ObservabilityAny Observability permission

Dashboard, Universe, Documentation and BI are shown to every signed-in user. An area you cannot open is not shown at all. A group with no visible items also loses its separator line. If you type the address of an area you cannot use, you see an Access Denied page.

See the flow on the Dashboard

The Flow view of the Dashboard draws the same picture with your own figures. Your connected sources are on the left, the platform is in the middle with figures for Data Lake, Data Warehouse, Data Sources, AI Minds and User Activity, and the outputs Minds, Editor, Slides, DataPrism and Writeback are on the right.

It is one of the Dashboard's three views; The Dashboard covers the view switch. To open each box and read what the figures count, see The Flow view.

Questions, answered

What is the difference between the Lake and the Warehouse?

The Lake holds raw data as it arrives from your sources. The Warehouse holds clean, modelled tables that are built from Lake data. Load the Lake first, then shape the Warehouse.

Do I need the Warehouse to query my data?

No. The Database Explorer in the Editor lists Lake tables as well as Warehouse tables, for the role you have selected, so you can query Lake tables directly. A Writeback table can also read straight from a Lake table. See Run your first query.

Which Mind should I use?

Use an Analytical Mind to ask questions about the tables you link to it. Use the Warehouse Builder or the Lake Builder when you want the AI to build Warehouse objects or import sources for you. See Create a Mind.

Where do schedules live?

In Workflows. Each workflow has a type of Lake, Warehouse or WriteBack, and each one has its own schedule. See Schedules and when a workflow starts.

How do I get data out of DataMind OS into another system?

Use Writeback. Add a destination, which can be a database, Google BigQuery or RabbitMQ, and then add the tables that send data to it from a Lake or Warehouse table. You can also connect a tool you already use, such as a SQL client, Excel or Power BI, from Universe. See Writeback and Integrations.

What is the difference between DataPrism and Slides?

DataPrism builds interactive reports on semantic models, which readers can filter and explore. Slides builds presentations, and a chart, table or text can be filled from a live query. Use DataPrism for reports people explore, and Slides for decks you present or email.