Load files

Load CSV, TSV, JSON, Parquet and Excel files into the Lake from an upload, an SFTP or FTP server, or an S3 bucket. Use a file datasource when your data arrives as files and new files must load on a schedule.

Create a file datasource

Open Lake in the sidebar and click Add datasource. Choose CSV, TSV, JSON, Apache Parquet or Excel Workbook.
On General, enter a Name of at least 3 characters and a Code of letters and numbers, at least 2 characters. Choose a Folder if you want one. Set Timezone and Datetime values when the files hold datetimes; see Timezones and datetime values. Click Next.
On Source, choose where the files come from, as described below. Click Next.
On Format, check the preview and adjust the options. The options are described on File formats and preview. Click Next.
On Tables, choose how files become tables. The modes and their settings are described on Group files into tables. Click Next.
On Review, set how long stored files are kept, then click Add.
Open the View Source panel and import the tables you need. See Import tables.

Choose where the files come from

On Source, answer Where do the files come from? with one of three options:

Option	Use it when
Upload files	The files are on this computer. You drop them into the form.
SFTP / FTP	A file server holds the files. Each run copies new and changed files.
S3 bucket	An Amazon S3 or S3-compatible store holds the files. Each run copies new and changed objects.

The choice is fixed after creation: The source is fixed once the data source is created. While uploads are still in the form, remove them first to switch: Remove the uploaded files to switch to SFTP / FTP or an S3 bucket.

Note

Amazon S3 is also a datasource type of its own, with its own form. This page covers the S3 bucket option. For the other one, see Connect Amazon S3.

Upload files

Enter the Code on the General step first. The upload area stays disabled until the code is set.
Drop files or folders onto the upload area, or click Select folder. Folder paths are kept.
If a file already sits at the same path, the Replace file? prompt names it and asks Replace or Keep current. Until you confirm, a second upload of that path is refused; delete the stored file first to replace it. While a replacement is being prepared, its row shows beside the original and neither row can be deleted: Some files are being replaced and cannot be deleted until the new version is ready. Loads keep using the current file until the new one is ready.

A few mechanics and limits:

Accepted extensions are .csv; .tsv, .tab and .txt; .json, .jsonl and .ndjson; .parquet; and .xlsx and .xlsm for Excel. CSV, TSV and JSON files can also be compressed as .gz or .zst, and uploads other than Excel accept .zip.
Each file can be up to 50 GB, and each Excel workbook up to 1 GB.
Large files upload in 16 MB parts and resume after network errors. A file can need up to 10,000 parts.
When an upload finishes, the file is prepared before it can load: Upload completed; the file is being prepared
An upload session lasts a day. If it expires, upload the files again. A session belongs to the person who started it: another user sees This upload session belongs to another user
A partly typed upload draft survives a page reload: the name and code come back while those fields are empty.
While uploads sit in the form, the code is locked to them: Code is locked to "{code}" by the uploaded files. Remove the files to change it.
Next stays disabled until at least one file is ready: Upload at least one file, Retry or remove the uploads that failed or Wait until every upload has finished processing.

SFTP / FTP

Setting	What it does	Default
Protocol	SFTP (SSH), FTPS (FTP over TLS) or FTP (unencrypted). Changing it moves the port to that protocol's default, unless you typed another port.	SFTP (SSH)
Port	The server port.	22 for SFTP, 21 for FTPS and FTP
Authentication	Password or Private key. Private key works with SFTP only. FTP and FTPS use a password.	Password
Remote folder	The folder to read, absolute or relative to the login folder.	Empty: the login folder
Subfolders	Include subfolders or This folder only.	Include subfolders
Stability wait (seconds)	Skips files modified more recently than this until the next run, so half-written files are not copied. Maximum 3600.	60
Mirror deletes	Deletes the stored copy when the file is removed from the server. Off keeps every copy. Needs Leave files on the server.	Off
After a file is copied	Leave files on the server, Move files to another folder or Delete files from the server.	Leave files on the server
Move to folder	Shown for Move files to another folder. Must be outside the remote folder.	Empty

Click Test connection before you save. For SFTP, check the server fingerprint, then click Trust this host; saving needs a trusted key, and Next stays blocked with Trust the server host key before continuing. For FTPS with a self-signed or private certificate, check the fingerprint with your server administrator, then click Pin this certificate; until it is pinned, Next stays blocked with The server certificate is not issued by a trusted authority. Pin it before continuing. A certificate from a public authority works without pinning. If you change the server or its sign-in details after a test, the form asks for a new test: The server changed - run the connection test again.

Warning

Plain FTP sends the password and every file unencrypted. Use SFTP or FTPS unless the server is on a trusted private network.

S3 bucket

Choose Amazon S3 and pick the bucket's region from the list. Or choose S3-compatible and enter the Endpoint URL. For an S3-compatible store, an empty Region (optional) means us-east-1.
Path-style addressing appears for S3-compatible stores and is on by default.
Bucket takes the bucket name only, without s3:// or a folder. Folder (prefix) limits the files to one prefix. Empty reads the whole bucket.
Under Access, choose Access key or Public bucket (anonymous). The public option works only when the bucket allows public listing and reads.
Objects stay in the bucket. Each run copies new and changed objects only.
Changing the provider, region or endpoint, bucket or access key ID asks for the secret access key again.

Fetch new and changed files

A server or bucket source copies only files that are new or changed. Each scheduled run checks the file size and modified time before copying.
Click Fetch now to copy them right away instead of waiting for the next run. It works for SFTP, FTP and S3 sources. If a fetch is already running, a new request is refused because the files are already being fetched.
When a changed file is copied, it becomes a new version of the stored file. Loads keep using the current version until the new one is ready.
Large folders are handled in bounded steps: a connection test or a fetch that runs too long stops and asks you to narrow the folder. A very large listing is truncated, and The remote listing was truncated; counts cover the listed files only
New files that fall outside a table's sync window wait for a reload; see Stored files.

Set the format

On Format, check the preview and set the options for the file type you load. CSV and TSV dialects, JSON shapes, encodings, Excel workbook options and the preview are described on File formats and preview.

The options apply when you save: stored files are re-read with the new settings.
Parquet files carry their own schema, so there is nothing to configure.
Some options can be overridden per group on the Tables step. If a workbook has no sheet with data, the preview says The workbook has no sheet with data.

Choose tables

Union by pattern makes each group of files one table, with the files of the group appended together. Use it for daily files that share one structure.
Each file separately is the default. Every file becomes its own table, named after the file. Every Excel sheet becomes its own table.
For Excel Workbook sources, the step asks How do sheets become tables? The layouts are Each sheet, Union by sheet name, Union inside workbook and Union all.

Suggested groups lists groups that match your files; click Add all to add them all. An imported table keeps its Table name and its group cannot be removed. Full grouping detail is on Group files into tables.

Columns added to every file table

Every table loaded from a file source carries six extra columns: _file, _path, _file_size, _file_arrived_at, _file_date and _rescued_data. They behave like ordinary table columns and stay hidden from the preview sample.

_file and _path identify the stored file a row came from; _file_size and _file_arrived_at carry its size and arrival time.
_file_date holds the date extracted from the file name or path. Turn the extraction on for a group to fill it; see Group files into tables.
_file_date can be the Date Column of an incremental load, and it is listed first for files. A date column inside the files makes every run read every file, and the form warns you. See Choose a sync type.
_rescued_data holds the raw text of a value that did not fit its column when Bad values is Rescue.

Review and save

The Summary lists the source, format and tables, and flags files that match no table or more than one. Under Stored files, turn on Delete stored files after a number of days to set Keep files for (days), which defaults to 90. With the switch off, files are kept until you delete them. The switch is hidden for S3 buckets, and disabled for servers that leave files in place: a removed copy would be fetched again on the next sync.

Manage the stored files

After saving, open the source and go to its Source step. The Files section lists every stored file, with search, statuses, download and delete; it is described on Stored files.

Deleting a stored copy does not remove rows already in the Lake. A run replaces a day when its files are reloaded.
When files change outside a table's sync window, the step offers Reload affected days. The table must use an incremental reload sync type and have a date column; one reload covers up to 366 days, and each day is replaced as a whole.

Who can do what

View Lake Objects lets you see a file datasource, read its file list, download stored files and run a connection test.
Add/Modify Lake Objects lets you create or change a file datasource, upload or delete files, and click Fetch now.
Execute/Abort Lake Objects lets you reload affected days.

The full permission rules are on Who can do what in the Lake.

Limits and edge cases

A file datasource cannot be duplicated.
For Excel workbooks read from a server or bucket, only .xlsx and .xlsm files up to 1 GB each are read. Other files are ignored.
When the stored files are too large for one zip, Download all fails with Too large to download at once. Download them one by one from the list.
A server address that points at the platform itself, a loopback or a link-local address is refused: it cannot be used for a file source.

For the messages you may see while you set up a file datasource, read Lake troubleshooting.

Questions, answered

Can I change the source later?

No. The source is fixed once the data source is created. While uploads are still in the form, remove them first to switch to SFTP, FTP or an S3 bucket. To load from another place, create a new datasource.

Why are the name and code locked while I upload files?

Files are stored under the code, so the General step locks them while uploads sit in the form. Remove the uploaded files on the Source step to change them, or save the datasource first.

How do I stop half-written files from loading?

Set Stability wait (seconds) on the Source step. A file modified more recently than this is skipped until the next run. The default is 60 seconds and the maximum is 3600.

What happens when I replace a stored file?

Loads keep using the current file until the new copy is ready, then later loads and downloads return the new content. The stored path and its date are kept. While the replacement is prepared, its row shows beside the original and neither row can be deleted.

What does Fetch now do?

It copies new and changed files from the server or bucket right away, instead of waiting for the next scheduled run. It works for SFTP, FTP and S3 sources. If a fetch is already running, a new one is refused because the files are already being fetched.

The server's host key or certificate changed. Is it safe to continue?

Treat it as a possible attack. If nobody re-installed or replaced the server, check with your server owner before you continue. A changed key shows The server host key changed. Review it and trust the new key before continuing. A changed certificate shows The server certificate changed. Review it and pin the new certificate before continuing.

Which columns are added to my tables?

Every table carries _file, _path, _file_size, _file_arrived_at, _file_date and _rescued_data. They record where each row came from, hold the date extracted from the file name or path, and keep the raw text of values that did not fit. They stay hidden from the preview sample.

Can I use the date in the file name for incremental loads?

Yes. Turn on Extract a date from each file (_file_date) for the group, then choose _file_date as the Date Column in the table's incremental settings. A date column inside the files, rather than in the name, makes every run read every file, so use _file_date for file-level loads. See Choose a sync type.

How large can a file be?

A file can be up to 50 GB. An Excel workbook can be up to 1 GB. Large files upload in 16 MB parts and resume after network errors.

Why does the app say the upload session belongs to another user?

An upload session belongs to the person who started it, and only that person can use its files. Start your own upload, or ask that person to finish or save the datasource.