Loading the documentation…

Run logs and step messages

While a run works, it writes a message for each thing it does, such as moving rows or clearing a day, and a reason when a step is skipped. This page explains the messages you meet and where to read them. For the steps a run is made of, see Nodes in a run.

Where a run's messages appear

In the workflow diagram. Open a run's diagram, then open a step's details. The panel shows the step's Notes and Error when the step has them. See Workflow diagram.
On the step. The nodes of a run carry a Note and an Error Stack. The Note explains a skip or records an outcome, and the Error Stack holds the full text of a failure. See Nodes in a run and Monitor and fix a run.
In a table's Sync Log. Every load of a Lake table and every run of a Writeback table is listed in its Sync Log, with Status, Run Scope, Note, Processed Rows and Error Stack. See Run, stop and troubleshoot loads and Synchronize a table and read the Sync Log.

Note

Each kind of step writes its own messages. A load of one table shows that table's messages, while a Warehouse run shows the messages of the transformations it ran. The tables below cover the messages you meet most often.

Messages of a Lake load

Message	What it means
Column List Generated	The columns this load will fill were resolved.
Temp Table Created	A temporary copy of the table was created for this run. Rows land there first.
Data Inserted In Temp Table	The rows read from the source landed in the temporary copy.
Data Successfully transferred from TMP table to the Main one	The temporary copy was moved into the table. A looped load adds how many iterations it took.
Data Successfully transferred to the Main Table	The same outcome on a snapshot load.
Explicit Day `<date>`	A reload of chosen days loaded that day.
Day Replaced	A reload of chosen days replaced that day's data in the table.
Day Cleared	The chosen day produced no rows, so its data was cleared.
Rows Outside Requested Days	The extract returned rows for other days. They are discarded, and the step lists them.
Undated Files Skipped	Files whose names carry no date are not loaded. The step lists a sample.
Loop Chunk `<n>`	One step per piece of a looped load.
Full Reload From Date	The date a first run starts from, next to the date the run loaded for.
Sync Days For First Run	How many days the first run covered. It can say the span was clamped to the earliest source date, or that the source has no dated rows.
Snapshot DDL Generated and Snapshot DDL Executed	The snapshot column of a Snapshot table was prepared for this run.
Tokenization Script Generated and Tokenization Script Executed	Values in protected columns were tokenized while loading.
Processed Rows	The count this load measured.

Some steps keep the exact statement the run sent, so an entry can look like SQL; that text is what the platform executed for that table or day.

Messages of a publish

A Fact run and a Lake load collect their rows on the side and publish them when they end; see How Dimension and Fact runs write data and Reloads and partitions. These steps record the publish:

Message	What it means
Staging Created	A production Fact run created the temporary table it collects its rows in.
Staging Fenced	A Lake load set its extract aside as this run's own copy, ready to publish.
Publish Gates Passed	The checks before the publish passed. The text counts the days and rows to publish and the days to clear.
Partitions Replaced	Each day the run loaded replaced the same day in the table, as a whole.
Partitions Appended	The run added its days: a Diff Insert (Periodic) run, or a first load into an empty Lake table.
Partitions Cleared	Days of the window that the source no longer has were cleared, at the end of the publish.
Table Exchanged	A Truncate Insert (Full) run swapped the whole Fact table for the new copy.
Table Replaced	A Lake Full Reload replaced the whole table at once. With `The extract and the table are both empty; nothing to publish.` there was nothing to replace.
Partition Replaced	A snapshot load replaced the partition of its snapshot.
Upsert Staging Built	An upsert run merged the stored rows and the extract into one new copy, which then replaced the table at once.
Duplicate Keys Collapsed	A warning: the extract repeated keys, and the last loaded row of each key was kept.
Empty Extract	An upsert run's extract was empty, so the table was left unchanged.
Insert Timestamp Materialized	Older data of an upsert table was completed with its insert timestamp before the run merged.
Safety Guards Skipped	The run was started without its data safety guards. The text says what would have failed.
Publish Verified	The table was checked after the publish and holds what the run published.
Publish Re-issued	The database did not confirm a publish statement, so it was sent again; the text gives the attempt, up to 3.
Staging Dropped	The run's temporary tables were removed after the publish.
Staging Drop Failed	A warning: the temporary tables could not be removed, and the text says the run itself succeeded. They are removed automatically later.

A failed publish leaves the table as it was. The message of a refused publish, and what to do, is on Warehouse troubleshooting and Lake troubleshooting.

Messages of a Warehouse transformation

The steps follow the run: the transformation generates its statements, works in staging copies, then writes the table. A production Fact run then publishes with the steps in Messages of a publish.

Message	What it means
ETL Statements Generated	The SQL for the transformation was generated.
Drop Script Executed	The staging copies were cleared and recreated. A production Fact run writes Staging Created instead.
Source Insert Script Executed	The transformation's query result landed in the staging copy.
Duplicate Check Script Executed	The staging copy was checked for duplicate keys before anything reaches the table.
Target Insert Script Executed and Rest Script Executed	The final write steps for the table.
Loop Chunk Executed	One step per piece of a looped transformation, numbered like 2/5, with the posting date and the days it covered.
Full Reload From Date and Sync Days For First Run	The window a first run used.
Earliest Source Date Not Found	The search for the earliest date in the source did not answer, so the run fell back to the set start date.
Full Reload From Date Not Reached	The start date is after the date the run loaded for, so there was nothing to load yet.
Processed Rows	The rows the run produced. A Dimension run counts them on its target output; a production Fact run counts the rows it staged for publishing, and its test runs count the target output.
Emptied Staging Tables	A Dimension run emptied its staging copies after the run.
Empty Staging Tables Failed	The cleanup failed. This is a warning, and its text says the run itself succeeded.
Tokenization Script Generated and Tokenization Script Executed	Values in protected columns were tokenized.
Eligibility Warning Ignored	A run started by hand went ahead past a warning-level check.
TransformationRefused	The run stopped before sending any statement, because the query contains something the platform refuses. The text says why.
Error	The run failed. The text is the failure reason; a database error keeps its code.

For what a Warehouse run writes, see How Dimension and Fact runs write data.

Messages of a Writeback and a History run

A Writeback run pushes rows to your destination:

JDBC Statements Generated and Dump Data To Temp Table: the statements were built and the rows were collected in a temporary copy.
Data successfully transferred to remote table: the rows reached the destination.
Processed Rows: the rows the push collected.

A History node writes versions of changed records:

History Statements Generated: the full SQL for this history run is recorded before it executes, and the step's text shows the statements it generated. Read the entry to inspect the generated statements, including when execution later fails.
Processed Rows: the count of the last insert, or unknown when it could not be read.
A skipped history step records why, or reads History is not eligible to run.

A connector source adds a Meltano Pipeline Finished step before the steps of its tables: the source is pulled once per run, and the step carries no row count. For the node behind it, see Nodes in a run.

Row counts a run reports

The count comes from the step that measured it. A Warehouse Dimension run counts its target output, a production Fact run counts the rows it staged for publishing, a History run takes the count of its last insert, and a Lake load prefers the count the loader reported.
When a count cannot be measured, the Sync Log leaves Processed Rows blank and a step can read unknown. A missing count never means zero, and it is never a guess.
A skipped run reports nothing, because it never started.
The nodes table shows this column as 0 when no count was reported, so a 0 there does not prove that nothing was handled. The note on that column in Nodes in a run explains it.

First runs and full reloads

A first run that fills history starts from the Full Reload Start Date when one is set. Without it, the run uses the earliest date it can find in the source.
Full Reload From Date records the window's start next to the date the run loaded for. Sync Days For First Run records how many days the run covered; when the source is younger than the start date, the span is clamped to the source's earliest date and the step says so. When the source has no dated rows, the step says that too, and the run covers no days of history.
A start date after the date the run loaded for makes the load skip; the run records Full Reload From Date Not Reached, and nothing is loaded yet.
A looped load stays one step in the diagram and repeats Loop Chunk lines as it walks the window, newest piece first. On a connector load the newest piece can be a partial period. A looped Lake load keeps no progress between chunks, so a run that fails part way starts again from its first chunk next time.
If the platform stops while a step is running, the entry is closed with a note that it was interrupted, and the step runs again after the platform is back. See Monitor and fix a run.

Notes you may meet on a step

A step's Note can read as:

Skipped due to previous node error: a step it depends on failed, so this one did not run.
Gate not satisfied: followed by the reason: a cross-workflow gate's wait ended without the table.
Preparation cancelled: followed by the reason: the fetch of a source's files was stopped before it finished.
Aborted by User: the run was stopped by hand.

Each of these is explained, with what to do next, in Nodes in a run and Monitor and fix a run.

Questions, answered

Why does one step repeat with "Loop Chunk" in its name?

A looped load or transformation covers a long window in pieces. Each piece writes one line: Loop Chunk <n> on a Lake load and Loop Chunk Executed on a transformation. The node stays one step; read the lines to see how far the run got.

Why is the row count missing for a run?

The count could not be measured. The Sync Log leaves Processed Rows blank, and a step can read unknown instead of a number, because the platform records nothing rather than a guess. A skipped run shows nothing: it processed nothing. See Row counts a run reports.

Why did my first run load so many days?

A first run that fills history goes back to the Full Reload Start Date or, without one, to the earliest date in the source. Full Reload From Date and Sync Days For First Run in the messages show the window it used. For how a Lake table sizes its first load, see Choose a sync type; for a transformation, see How Dimension and Fact runs write data.

Why was my run skipped with "Full Reload From Date Not Reached"?

The start date is after the date the run loaded for, so there was nothing to load yet. Set an earlier Full Reload Start Date so the load can start.

What do "Day Replaced" and "Day Cleared" mean?

They come from a reload of chosen days. A day the extract returned rows for is Day Replaced: the table's data for that day is replaced with the fresh rows. A day that produced no rows is Day Cleared: its data is cleared. Read the two steps to see which days changed.

Why does my run show "Rows Outside Requested Days"?

The extract returned rows for days you did not ask for, so they are not loaded; the step lists the days it discarded. Check the days you chose and the table's incremental column.

My run shows "Staging Drop Failed". Did it fail?

No. The step is a warning: the run published its data, and only the removal of its temporary tables failed. They are removed automatically later; the Temporary ETL tables panel in ClickHouse storage and maintenance lists them meanwhile.

My Warehouse run shows "Empty Staging Tables Failed". Did it fail?

No. The step is a warning: the run's own work finished, and only the cleanup of the staging copies failed. The step text says the run itself succeeded. Watch the next run.

Why did a run write "Eligibility Warning Ignored"?

You started it by hand. A manual run goes ahead when the only problem is a warning-level check, and records this step. An error-level check still stops the run.

Why don't all runs show the same steps?

Every kind of step writes its own messages. A snapshot load writes Snapshot DDL steps, a reload of chosen days writes Explicit Day steps, and a write-back writes JDBC steps. A run shows only the messages of the steps it ran.