The Lake

The lake is the storage layer between your sources and your queries. Connectors write into it; everything else reads from it. A query never touches the source database, so a slow report cannot slow down the system your product runs on.

Organisation

Tables are addressed as lake.<table>, and the table name is the source table's name. There is one copy per source table, not one copy per connector: two connectors reading the same table into the lake land in the same place, and that collision is reported rather than silently merged.

Schema	Holds	Written by
`lake`	Synced source tables	Connectors
`lake_history`	Superseded versions of changed rows	Connectors, on every incremental sync
`lake_meta`	Sync runs, watermarks, row counts	The platform

History

An incremental sync that updates a row does not overwrite it. The previous version moves to lake_history with the interval it was valid for, and the live table keeps only the current state.

sql

select
    order_id,
    status,
    valid_from,
    valid_to
from lake_history.orders
where order_id = 'A-10241'
order by valid_from

That is what makes a question like "what did this order look like last Tuesday" answerable at all - and it is also why storage grows faster than the sources do. A table with heavy updates keeps every version of every changed row.

Tip

Point-in-time queries are cheap only if you bound them. Always filter valid_from in a history query; without it the engine reads every version of every row.

Sync behaviour

:::timeline

First sync

A full load. Every row the reader can see is written, and the sync is recorded as full.

Later syncs

Incremental, driven by the cursor column. Only rows newer than the stored watermark are read, and an update to an existing row becomes a new version in history.

Retries

A sync that fails is retried with backoff. A sync that fails three times is marked failed and the next scheduled run starts from the last good watermark, so nothing is read twice. :::

Checking what is in the lake

The table tree in the editor is built from the lake's own metadata, not from the connectors, so a table that no longer has a connector still appears until it is dropped. Two queries answer most questions about the state of a sync:

sql

select table_name, max(synced_at) as last_sync, sum(rows) as rows
from lake_meta.sync_runs
group by table_name
order by last_sync desc

sql

select count() from lake.orders where synced_at >= today()

If the second number is zero and the first says the sync ran, the source really is empty - the connector read it and found nothing, which is a different problem from a connector that did not run.