Loading the documentation…

ETL locks

While a load, a transformation run, a history run, a schema change or an OPTIMIZE uses a table, it holds a lock on that table, and other work that needs the same table waits until the lock is released. The Locks view of the ClickHouse tab lists every holder and, under each one, what waits for it. Use it when a run stays Pending, a schema change waits for its table, or a table seems stuck.

How work shares a table

DataMind takes and releases the locks for you; you never set them by hand. Each kind of work locks only what it needs:

Work	What it locks
A Dimension run, or a Fact run with a periodic pattern	Its own source system's part of the table, and it shares the rest. Runs for different source systems of one table load side by side; two runs for the same source system take turns.
A Fact run with any other pattern	The whole table.
A history run	Its source system's part of the table's history, and it shares the table.
A Lake table sync or a Writeback load	The whole table.
A transformation test run	Only the transformation's own work tables, never the table it writes to.
A schema change	The whole table. It starts only when no load uses the table.
An OPTIMIZE	It shares the table, so Dimension and periodic Fact loads run beside it.

The periodic patterns are listed on Transformation patterns and load settings. Waiting work starts in the order it arrived. A schema change that is queued for a table goes ahead of the loads of that table that arrive after it. What a person sees when they change a busy table is described on When a table is busy.

Open the Locks view

Open Observability from the sidebar and click ClickHouse.
Click Locks in the view toggle, right after Queries.

The view needs View Active Queries or Manage ETL Locks; without one of them, Locks is not in the toggle. A Lake, Warehouse or WriteBack edit permission alone does not show it.

The list refreshes itself every 5 seconds. The refresh button (tooltip Refresh data) reloads it at once.
Type in Filter locks... to narrow the list.
With nothing held, the view reads No ETL locks are held right now. If the list cannot load, it reads The ETL locks could not be loaded., with a Retry button.

Read a lock

Each row is one holder: a run step, a schema change or an OPTIMIZE that holds at least one lock.

Column	What it shows
Resource	The table, with the source system in brackets when the lock covers one source system, for example `Customers [CRM]`. A History chip marks a Dimension's history table, and a Scratch chip marks a transformation's work tables, shown by the transformation's name. Hover the cell to list every resource of the holder with its mode.
Mode	X (tooltip Exclusive: nothing else may use it meanwhile) or S (tooltip Shared: other shared holders may use it too).
Holder	A `job #<id>` link that opens the run, then the step and the transformation. A holder without a run, such as a schema change that is applied straight away, shows its own label.
Started by	The person who started the run or the change. It shows a dash for a scheduled run.
Since	How long ago the lock was taken. Hover it for the exact time.
Heartbeat	How long ago the server process that holds the lock last reported in, for example `12 s ago`. It turns red after 60 seconds, and reads `none` when no report has arrived.
Process	The server process that holds the lock, with its role and version.
State	Held, Releasing or Stale, plus Long-held when it applies. Hover a badge for the explanation.
Statements	How many ClickHouse statements of the holder run now, and the age of the oldest one.

The states mean:

Held: the holder runs and keeps its lock until it ends.
Releasing: the holder has ended or was aborted. The lock is released only once ClickHouse shows none of its statements and none of the row changes (mutations) it started, and only then does the next waiting work start. Hover the badge to see what it still waits for; by default it reads Its ClickHouse statements are finishing.
Stale: the server process that holds the lock stopped responding. DataMind takes such a lock over by itself, usually within a minute or two, and the step runs again.
Long-held: the lock has been held for more than 30 minutes, and more than three times as long as that step usually runs, judged by its runs over the last 30 days. The tooltip reads, for example, Held for 2 h 10 min, far longer than this step usually runs.

See what waits for a table

Expand a holder row with the arrow in its Resource cell.
Read the waiting rows under it: Waiting #1, Waiting #2 and so on, in the order they will run.
Hover the hourglass chip of a waiting row to read why it waits.

Waiting work whose holder is not in the list is grouped under Waiting, no listed holder. A schema change that has waited more than 10 minutes gets a Long wait badge (tooltip This schema change has been waiting for the table for more than 10 minutes.).

The waiting notes name the resource as <table>, <table> [<source system>], <table> History [<source system>] or scratch tables of "<transformation>":

Note	What it means
`Waiting for "<resource>" held by job #<id> (<step>)`	A step of that run uses the resource.
`Waiting for "<resource>": schema change by <email>`	A person's schema change uses it.
`Waiting for "<resource>": OPTIMIZE by <email>`	An OPTIMIZE uses it. When it was started from the maintenance actions of the ClickHouse tab, the note ends `OPTIMIZE by the DBA page`.
`Waiting for "<resource>": job #<id> is releasing its ClickHouse work`	The holder has ended, and its statements are still finishing.
`Waiting for "<resource>": a queued schema change (job #<id>) runs first`	A queued schema change is ahead in the line.
`Waiting for "<resource>": job #<id> (<step>) is queued first`	Another run is ahead in the line.

The same notes appear on a waiting step in the run's steps and diagram; see Monitor and fix a run.

Abort a holder

Use this when a holder is stuck or blocks urgent work. It needs Manage ETL Locks, and it is not offered for a lock that is already Releasing.

Click Abort holder & release at the end of the holder's row.
Read the dialog, for example: Abort job #<id> (<step>) and release its locks? Its ClickHouse statements are killed and the job ends Aborted. For a holder without a run, it reads Abort <holder> and release its locks? Its ClickHouse statements are killed.
Optionally, type why in Reason (optional, kept in the audit log), up to 500 characters.
Click Abort holder & release. Pressing Enter does not confirm.

The run ends Aborted, whatever its workflow type, and its ClickHouse statements are stopped. The lock is released as soon as ClickHouse shows nothing of it; the dialog waits up to 20 seconds for that. The step's note reads Aborted from the Locks tab by <email>: <reason>.

Holder aborted, locks released: done, and the next waiting work can start.
Holder aborted with Still releasing: ...: ClickHouse still shows some of its work. The lock stays Releasing until that is gone.
Abort failed: the toast gives the reason.

Aborting from this view does not need the Lake, Warehouse or WriteBack permission that Stop in the runs table needs. It also stops an OPTIMIZE, which DataMind never stops on its own.

Release a stale lock

Release now appears only on a Stale lock whose holder has nothing running in ClickHouse, and it needs Manage ETL Locks.

Click Release now in the row.
Read Release stale lock: Release the stale lock of job #<id>? The holder process <process> stopped heartbeating and nothing of it runs in ClickHouse. The node is re-queued.
Click Release now.

The step is queued again and runs once its tables are free. This counts as an interruption: after three interruptions, the step fails with Interrupted 3 times: the executor died while running this node. A queued schema change is not run again: it ends with an error, and you make the change again.

You then see Stale lock released, or Stale lock releasing with Still releasing: .... When the release is refused, Release failed gives the reason:

The holder process <process> (<role>) is alive (last heartbeat <n> s ago). Use "Abort holder & release".
<n> ClickHouse statement(s) of this holder are still running (oldest <m> s). Use "Abort holder & release" to kill them first.
ETL lock <id> not found; it may have been released already.

Every abort and release is recorded with the person, the time, the reason and the outcome.

Permissions

Permission	What it lets you do
View Active Queries	See the Locks view: every holder, the person who started it, its server process, and what waits.
Manage ETL Locks	See the view, and use Abort holder & release and Release now. A builder Mind can also cancel a schema change that someone else queued.

The ETL Locks Manager preset bundles both. It is part of Observability Administrator, not of Observability Viewer, so give it to other roles in the role editor. See the Permission reference.

Limits and edge cases

The view lists the holders of every workflow type, with their run names and the people who started them, also to viewers who cannot open those workflows.
A Lake, Warehouse or WriteBack permission does not open the view, even for the runs you started.
DataMind never stops a running OPTIMIZE by itself. Its lock is released when the statement ends, or when you abort it here.
The Warehouse Builder and Lake Builder Minds can read this list and act on it with your permissions; see Warehouse tool cards in chat.

Questions, answered

A run stays Pending with a Waiting chip. What does it wait for?

For a table that other work uses. Hover the chip to read the note, which names the table and the run or person holding it. Open Locks to see the holder and its place in the line.

What is the difference between Abort holder & release and Release now?

Abort holder & release stops a holder that is still working: its run ends Aborted and its statements are stopped. Release now is for a Stale lock whose process stopped responding and has nothing running in ClickHouse: the step is queued again instead of ending.

Why is there no Release now button on a lock?

Release now shows only for a Stale lock with nothing running in ClickHouse. For any other lock, use Abort holder & release, which needs Manage ETL Locks as well.

Why does a lock stay Releasing after its run ended?

ClickHouse still shows statements or row changes (mutations) the holder started. The lock is released when they are gone, and the next waiting work starts then. Hover the Releasing badge to see what it waits for, and check Mutations for a change that does not finish.

Can two source systems of one table load at the same time?

Yes, for a Dimension and for a Fact with a periodic pattern: each run locks only its own source system's part. A Fact with any other pattern, a Lake table and a Writeback table load one run at a time.

Does a test run block changes to the table?

No. A test run locks only the transformation's own work tables, so loads and schema changes of the table go on. A second test of the same transformation is refused while one runs, because both would use the same work tables.

Why can't I see the Locks view?

It needs View Active Queries or Manage ETL Locks. A Lake, Warehouse or WriteBack permission does not open it. Ask an administrator for the ETL Locks Manager preset, or for View Active Queries to only look.

What happens to a step whose stale lock I release?

It is queued again and runs when its tables are free. A step that is interrupted three times fails with Interrupted 3 times: the executor died while running this node.

Do I need a Workflows permission to abort a holder?

No. Abort holder & release needs only Manage ETL Locks, whatever the workflow type of the run.