Connect Amazon S3

An Amazon S3 source reads files from one folder in a bucket and combines everything that matches the chosen format into one table. Use it when a bucket folder holds files in a single format; for the S3 bucket option inside a file source, see Load files.

Create an Amazon S3 source

Open Lake and click Add datasource, then choose Amazon S3 in the picker.
On General, enter a Name of at least 3 characters and a Code of letters and numbers, at least 2 characters. Choose a Folder if you want one. These are the shared fields every source has; see Supported connectors. Click Next.
On Bucket, set the folder URL and the access keys, described below.
On Format, choose the file type and the format, described below.
Check the summary and click Add. On an existing source the button reads Save, and Duplicate creates a copy.

The summary lists Name, Folder URL, Access key ID and the chosen Format; for a CSV format it shows the delimiter in parentheses. It never shows the secret key.

Point the source at a folder

The Bucket step holds the Folder and AWS credentials sections.

Folder URL takes the full address of the bucket folder that holds the files, written in the form https://bucket.s3.region.amazonaws.com/prefix/, for example https://my-bucket.s3.eu-central-1.amazonaws.com/exports/.
AWS Access Key ID and AWS Secret Access Key take an IAM user key that can list the bucket and read the files in this folder. After you save, the form never shows the secret key again: it keeps a stored value, with Replace beside it.

The step names whatever is missing or malformed:

Folder URL is required
Enter a valid URL (e.g. https://bucket.s3.region.amazonaws.com/prefix/)
AWS Access Key ID is required
AWS Secret Access Key is required

Choose the file format

On Format, choose a File type first, then a Format under it. Changing the file type selects that family's first format. A new source starts on CSV with header.

The four file types are Delimited text (CSV or TSV files, one row per line), JSON (JSON lines or JSON documents), Columnar (Parquet, Arrow or ORC files with their own schema) and Other (XML or raw file contents).

File type	Format	Files read	Notes
Delimited text	CSV with header	.csv files	The default for a new source.
Delimited text	CSV without header	.csv files	Columns are named c1, c2, …
Delimited text	CSV with header and types	.csv files	Row 1 holds the names, row 2 the ClickHouse types.
Delimited text	TSV with header	.tsv files
Delimited text	TSV without header	.tsv files
Delimited text	TSV with header and types	.tsv files
Delimited text	TSV raw	.tsv files	Tab-separated without escape sequences, no header.
Delimited text	TSKV	.tskv files	Tab-separated key=value pairs.
JSON	JSON lines	.json files	One JSON object per line (NDJSON).
JSON	JSON document	.json files	One document whose "data" array holds the rows.
Columnar	Parquet	.parquet files
Columnar	Arrow	.arrow files	Apache Arrow IPC files.
Columnar	ORC	.orc files
Other	Raw file contents	every file	Every file under the folder, one row per file.
Other	XML text	.xml files
Other	XML as JSON	.xml files

Below the cards, a line states which files the chosen format reads: Files ending in .csv are read. for a CSV format, Every file under the folder is read. for Raw file contents, and Pick a supported format to see which files are read. while no supported format is selected.

The step also shows a banner: All matching files under this folder (subfolders included) are combined into one table.

CSV delimiter appears for the CSV formats and is ignored by the others. Choose a comma, a semicolon or a pipe to match the files; the default is a comma.

Test connection on this step reads the matching files with the chosen format, so it needs the format above. It is optional, and you can continue without it.

Edit a saved source

When you open a saved Amazon S3 source, the form opens on Bucket.
If you change File format or CSV delimiter, the form warns Changing the format changes the existing table structure. Run Sync Schema after you save. Run Sync Schema on the datasource after saving; see Run, stop and troubleshoot loads.
A format you saved earlier that is no longer offered for new sources stays in the picker with (legacy) after its name, so the edit form keeps working. The entry says Kept from the saved settings, no longer offered for new sources when the stored value still works, or No longer supported - choose another format when it cannot be used. Saving with a format that is no longer supported is refused with This format is no longer supported - choose another one.

Limits and edge cases

Every matching file under the folder is read, subfolders included, and all of it goes into one table.
Files are matched by their extension. Raw file contents is the exception: it reads every file, whatever its extension.
The Amazon S3 source does not offer Incremental Reload Loop (Truncate Insert). See Choose a sync type.

For the messages you may see while you connect an Amazon S3 source, read Lake troubleshooting.

Questions, answered

What should the Folder URL look like?

The full address of the bucket folder that holds the files, written like https://bucket.s3.region.amazonaws.com/prefix/, for example https://my-bucket.s3.eu-central-1.amazonaws.com/exports/. If it does not parse as a URL, the field shows Enter a valid URL (e.g. https://bucket.s3.region.amazonaws.com/prefix/).

Are subfolders read too?

Yes. Every matching file under the folder is read, subfolders included, and they all go into one table.

Which credentials do I need?

An IAM user key that can list the bucket and read the files in this folder. Enter its AWS Access Key ID and AWS Secret Access Key on the Bucket step.

Can I limit which files the source reads?

Two things decide it: Folder URL sets the folder, and Format sets which file extensions are read. Point the URL at a subfolder to narrow the scope, and pick a format whose extension matches only the files you want. Raw file contents reads every file.

Which JSON format should I pick?

JSON lines reads one JSON object per line (NDJSON). JSON document reads one document whose "data" array holds the rows. Both read .json files; pick the one that matches the shape of your files.

My CSV uses semicolons. What do I set?

On the Format step, set CSV delimiter to a semicolon. The setting appears for the CSV formats and is ignored by the others.

I changed the format after saving. What does Sync Schema do?

The format decides how the files are read, so changing it changes the structure of the existing table. The form warns Changing the format changes the existing table structure. Run Sync Schema after you save. Run Sync Schema on the datasource after saving; see Run, stop and troubleshoot loads.

Why does my saved format show as (legacy)?

The format was saved before it was retired from the list offered to new sources, so the picker keeps it visible with (legacy) after its name. If its description says Kept from the saved settings, no longer offered for new sources, the stored value still works. If it says No longer supported - choose another format, pick a supported format before you save.

Does Test connection read my files?

Yes. The test on the Format step reads the matching files with the format you chose, so it needs a format first. It is optional, and you can continue without it.

Is this the same as the S3 bucket option in Load files?

No. The standalone Amazon S3 source on this page reads one folder into one table. The S3 bucket option of a file source uses the file form instead, where you can preview files and load them as one or several tables; see Load files and Supported connectors.