Stored files

The Files section of a file datasource's Source step lists every file the datasource holds. Use it to check what a load reads, fetch new files from a server or bucket, download or delete stored copies, and reload the days whose files changed.

To reach it, open a saved datasource: double-click it in the Lake tree, or right-click it and choose Edit (see The Lake tree), then open the Source step. For where the files come from, see Load files.

For an upload source, the drop area sits at the top of the section, so you can add more files to a saved datasource. Added files are stored right away and loaded by the next run of each table. Days that fall outside a table's sync window appear below for a reload.

Read the file list

The list has six columns:

Column	What it shows
File	The file's name. Point at it for the full stored path and extra lines such as New version of {path}, Stored as {path}, Remote {path} or Source encoding {enc}.
Size	The stored size, in units from B to TB; - when it is not known yet.
Status	Where the file is in processing; see the badges below.
Date	The date found in the file's name or path when a table group extracts one; otherwise -. A ready file whose group expects a date but has none shows no date.
Groups	The table groups that match the file, by label. An Excel workbook shows Workbook instead, because its sheets are the tables listed below it. A ready file that no group matches shows No group.
Rows	Rows loaded from the latest load; Not loaded until a run reads the file. Point at the figure for the per-table breakdown.

The Status badges, in one line each:

Badge	Meaning
Uploading	The upload has not finished yet.
Processing	Converting to UTF-8 and checking the content. The file is loaded once it is ready.
Ready	The file is stored and checked; loads and downloads can use it.
Failed	Processing failed. Open the message under the file name for the error.
Replacing…	A new version of this file is on its way. Loads keep using the current file until the new one is ready.
Replace failed	The new version could not be prepared. The current file is still loaded.
Duplicate	Loaded, but the content is identical to {dup}. Both files are read - delete one to avoid double rows.

Two more states to know:

While the list loads it reads Loading files.... With nothing to show, it says No files yet. Drop files above to add them. for uploads, No files fetched yet. Fetch now copies the files from the server. (or from the bucket) for server and bucket sources, and No files match the search or status filter. while a search or filter is on.
A group that no longer exists shows as Removed group on the files that used it.

Search, filter and select files

Search files narrows the list; it applies about 300 ms after you stop typing.
The status filter offers All statuses, Ready, Processing, Uploading and Failed.
Tick the checkboxes to select files. The header checkbox selects everything selectable on the page, and a file that is still being prepared cannot be selected.
With a selection, Delete {n} appears in the toolbar.
The refresh icon's tooltip reads Refresh the file list. The list also refreshes on its own about every 4 seconds while a fetch runs or a file is still being prepared.
The pager reads {first}-{last} of {total} files and moves with Previous page and Next page; one page holds 50 files. Searching, changing the filter or turning the page clears your selection.

Download stored files

Each ready row has a download icon, tooltip Download the stored copy. It is disabled while that file is downloading.
Download all packs every stored file into one zip. While it works, the button reads Downloading {pct}%, Downloading {size} or Preparing zip.... The zip is named after the datasource.
When the stored files add up to more than 256 MB, the download is handed to the browser and opens in a new tab. A single file above 256 MB downloads in a new tab too.
If the server refuses to build the zip, the toast reads Too large to download at once with The stored files are too large for one zip. Download them one by one from the list.

Delete stored files

Select the files, or use a row's delete icon (tooltip Delete file). A file that is still being prepared cannot be deleted: the icon is disabled with Wait until processing finishes.
In the confirmation, check the listed files (the first eight names, then and {n} more) and the description, then click Delete. The dialog title counts the files, for example Delete 3 files?.
The toast confirms {n} files deleted.

What the description covers:

For a server or bucket source: The stored copies are removed., and files still on the server or in the bucket are fetched again by the next sync.
For an upload source: The stored files are removed.
For Excel: Deleting a workbook also deletes its sheets.
Always: Rows already loaded stay in the Lake until a run reloads their days. If some of those days fall outside a table's sync window, the toast adds that you can reload them from the banner above the list.

Note

Deleting a stored copy never changes loaded data by itself. A day is replaced only when a run reloads it.

Fetch new files from the server

For SFTP, FTP and S3 sources, the section opens with a sync card.

The card says	What it means
Fetching files from the server	A fetch is running. New and changed files are copied in the background, and the list refreshes as they arrive.
Last sync succeeded	The last fetch finished. Scheduled runs fetch new files before loading; Fetch now copies them right away.
Last sync failed	The fetch failed. Fix the cause and fetch again; files fetched before the failure stay available.
Last sync finished with warnings	The fetch finished, but some files or post-processing steps reported problems.
No sync status yet	Nothing has been fetched yet. Fetch now copies new and changed files from the server.
Sync status not refreshed	The sync ran longer than the status check. Click Refresh to see where it stands.

For bucket sources the same lines say from the bucket instead of from the server.
Fetch now starts a fetch at once. It is disabled while a fetch is running or starting. The toast confirms Fetching files with New and changed files are copied from the server in the background., or reports Fetch could not start.
If the last fetch left a message, its first line shows under the card. Click it to open Last sync messages and read the detail.

Reload days changed outside the sync window

When stored files changed days that fall outside a table's load window, a banner appears above the list.

The banner is headed Files changed outside the sync window: Scheduled runs only reload the days inside each table's sync window. Reload these days so the Lake tables match the files.
Each row names a table, summarises its days as {n} day(s) ({first} to {last}), and offers Reload affected days.
Clicking it opens the confirmation Reload affected days?: {summary} of {label} are reloaded from the current files. Each day is replaced as a whole; days with no files left are emptied. Click Reload days to start; closing the dialog cancels the reload before it starts.
After it starts, the toast reads Reload started with {n} days of {label} are being reloaded., the row shows - reload started, and the button becomes Reload again. The banner keeps checking for about ten minutes.
When more than 366 days changed for one table, its button is disabled: More than 366 days changed - too many for one reload. Run a full reload of this table instead. (see Choose a sync type and Run, stop and troubleshoot loads). The dialog lists up to 60 days, then and {n} more.

Limits and permissions

One page holds 50 files, and search applies about 300 ms after you stop typing.
A day reload covers up to 366 days per table, and its dialog lists up to 60 days.
A download larger than 256 MB switches to a new-tab download; a zip the server cannot build is refused with Too large to download at once.
Reading the list and downloading need View Lake Objects; adding or removing files and Fetch now need Add/Modify Lake Objects; reloading days needs Execute/Abort Lake Objects. The full rules are on Who can do what in the Lake.

For the messages you may see in this section, read Lake troubleshooting.

Questions, answered

Why is my file still Processing?

The file is being prepared: Converting to UTF-8 and checking the content. The file is loaded once it is ready. Wait a moment; the list refreshes on its own, and the file loads only after it turns Ready.

What does Duplicate mean?

Two stored files hold identical content, so both are read and their rows appear twice. Delete one of the two copies. The badge's tooltip names the file this one matches.

Why can't I delete a file?

A file that is still being prepared cannot be deleted: its delete icon is disabled with Wait until processing finishes. Wait for the row to settle, then delete it.

Do deleted files remove rows from the Lake?

No. Rows already loaded stay in the Lake until a run reloads their days. If the deleted files covered days outside a table's sync window, reload them from the banner so the table matches the files.

Why did Download all open a new tab?

Because the stored files add up to more than 256 MB, so the zip downloads through the browser in a new tab instead of through the page. A single file above 256 MB downloads in a new tab too.

More than 366 days changed. What now?

One reload covers up to 366 days per table, so the button is disabled and the banner asks for a full reload instead; see Run, stop and troubleshoot loads.

What does no date mean next to a file?

A table group extracts a date from each of its files and found none in this file's name or path. Date-incremental loads skip this file, so its rows do not load. Give the file a name that matches the group's date format, or turn the extraction off if the group does not need it.

The sync card says Last sync failed. What should I do?

Fix the cause and fetch again; files fetched before the failure stay available. Click the message line under the card to open Last sync messages for the detail, then click Fetch now.

Why is Fetch now greyed out?

A fetch is already running or starting, so there is nothing to trigger. Wait for it to finish; the list refreshes as files arrive.

What do the Rows and Groups columns show?

Rows counts the rows loaded from the file in the latest load, with a per-table breakdown in the tooltip; Not loaded means no run has read the file yet. Groups lists the table groups that match the file, by label. Workbook marks a workbook whose sheets are the tables below it, and No group means no table loads this file.