Connect BigQuery

Use a BigQuery source to read the tables of a Google Cloud project into the Lake. The connection form is a wizard with the steps General, Service account and Datetime. This page covers the key you need, the fields the key fills in, and how BigQuery datetime values are read.

Add a BigQuery source

  1. Open Lake and click Add datasource in the header, or open the New menu in the toolbar above the tree. Choose Google BigQuery under Data Storage in the picker; Supported connectors shows how the picker is grouped.
  2. On General, enter a Name of at least 3 characters and a Code of letters and numbers. Choose a Folder if you want one, then click Next.
  3. On Service account, add the JSON key of a service account. Project ID and Service account email fill in from the key. Test connection sits on this step and is optional, so you can continue without it.
  4. On Datetime, choose the source Timezone and how Datetime values are read. The step also shows the summary of the new source.
  5. Click Add to save. On an existing source the button reads Save, and Duplicate makes a copy. Then import the tables you want.

Saving a source needs Add/Modify Lake Objects, and opening a saved one needs View Lake Objects. See Who can do what in the Lake.

Add the service account key

The Service account step holds the key the connection runs with. The card carries this hint: "Create a JSON key for the service account in Google Cloud console under IAM & Admin > Service accounts > Keys."

The key must be the JSON key of a service account. An empty key field shows "Service account key is required". Text that cannot be read as a JSON key shows "Invalid JSON format in service account key".

Check the project and email fields

When the key is accepted, two fields fill in from it. Their placeholder reads Filled from the key.

You can edit both fields. When a value no longer matches the key, the form adds a note beside it: Differs from the key's project or Differs from the key's client email, followed by the key's own value in brackets. The notes are warnings only, and the form saves either way.

Set the timezone

On the Datetime step, Timezone and Datetime values decide how the source's datetime values are read on load. The step shows this note for BigQuery:

"BigQuery TIMESTAMP columns are stored in UTC. DATETIME columns have no timezone and are read as written."

If the connection test detected the source timezone, the step suggests it and you can apply it. The settings, the suggestion and the rules that always apply to values are described on Timezones and datetime values.

Review and save

The Datetime step also carries the review summary of the new source, with the rows Name, Project ID, Service account and Key. The Key row summarises the key, for example Stored key or New key, rather than showing the raw JSON. Click Add to save; on an existing source the button reads Save.

When you edit a saved source, the form opens on the Service account step, and the stored key is loaded back into the field as formatted JSON so you can read or replace it. Name and Code are locked in edit mode.

Use BigQuery as a destination

BigQuery also works in the other direction. Open WriteBack, click Add destination and choose Google BigQuery. That form is a single page, not a wizard: it holds the same Service account fields and saves the connection as a destination. It has no Folder picker. See Add and manage destinations.

Questions, answered

Which key do I need for BigQuery?

A JSON key created for a service account: open Google Cloud console, go to IAM & Admin > Service accounts > Keys, and create a key for the account you want to use. The step takes the key file, or its pasted contents.

Can I paste the key instead of uploading the file?

Yes. Click Paste the JSON instead, paste the contents of the JSON key file, and click Use this key. Cancel closes the box without using the pasted key.

What does Filled from the key mean?

Project ID and Service account email fill in from the key you add. Project ID is the project the queries run and are billed in, usually the key's own project. Service account email is the email of the service account the key belongs to. You can edit both.

Why do I see Differs from the key's project?

Project ID no longer matches the key's own project, for example because you typed another project. It is a warning only and does not block the save. Change the value back if it was not intended.

What if I clear the Project ID or Service account email?

Both fields are required. When one is empty, the form shows "Project ID is required" or "Service account email is required". Add the key again, or type the value, before you save. An address that is not an email shows "Invalid email format".

Why was my key rejected?

An empty key field shows "Service account key is required". Text that cannot be read as a JSON key shows "Invalid JSON format in service account key". Drop or paste the file again; the key must be the JSON key of a service account.

How are BigQuery timestamps read?

"BigQuery TIMESTAMP columns are stored in UTC. DATETIME columns have no timezone and are read as written." The Timezone and Datetime values you choose on the Datetime step decide how they land in the Lake. See Timezones and datetime values.

What opens when I edit a saved BigQuery source?

The form opens on the Service account step, and the stored key is loaded back into the field as formatted JSON. Name and Code are locked in edit mode. Click Save once you have changed something.

Why is there no Folder picker on the BigQuery destination form?

Destination connections save without a folder, so the form leaves the picker out. Identify the destination with its Name and Code. See Add and manage destinations.