Configure a custom connector

The Connection step of a custom connector holds its configuration: a YAML file, shown as Configuration (meltano.yml), that tells the connector how to reach and read your system, plus the secrets the file refers to. You meet it while you create the connector and whenever you edit one. For the connector end to end, see Connect Anything (custom connector).

Changing the configuration or the vault needs Add/Modify Lake Objects; see Who can do what in the Lake.

Edit the configuration

1. Open the connector and go to Connection.
2. Write the settings in the editor under Configuration (meltano.yml).

The file arrives heavily commented. The comments start hidden, so you see the settings themselves; click Show comments to read the explanations, and Hide comments to hide them again. Edits keep the comments in the file.

• Example opens Example Configuration, a reference copy with every setting commented. Copy puts it on your clipboard.
• The commented copy opens with a header block: Datasource Name, Extractor, Variant and Label, plus a Documentation or Repository line when the connector has one.
• The connector's own documentation is included at the end of the file, under TAP DOCUMENTATION.
• Settings are grouped under standard headings: AUTHENTICATION, CONNECTION, DATA SELECTION, DATA STRUCTURE, BATCH PROCESSING, SCHEMA CONFIGURATION, STREAM TRANSFORMATIONS (Advanced), PERFORMANCE TUNING, TESTING & DEVELOPMENT and OTHER SETTINGS.

When comments are shown, each setting is annotated on its own line:

Placeholders read YOUR_VALUE_HERE, and YOUR_SECRET_HERE where a secret belongs. The file also ends with a VALIDATION block, listing what a valid configuration needs.

Reference a secret instead of writing its value

1. Store the value: under Secrets, click Manage Secrets. Add secret stores a name and a value, and Copy reference copies the {{secret:NAME}} text for it.
2. Paste the reference in place of the value in the file. A reference must be the whole value of its setting. A name is up to 200 characters: start with a letter, a digit or an underscore, then letters, digits, underscores or dots.

• A malformed reference is refused with "Malformed secret reference at: {paths}. Use the exact form {{secret:NAME}}."
• A reference without a stored value is reported in the Secrets section: "{path} references secret {secret}, which is not stored." with Add "…" and Edit config beside it. While you create the connector, Next stops with "Store a value for {names} in Secrets, or remove the reference before continuing."; a sync that finds one is refused with "Config references secret(s) that are not set: {list}. Set them in the Secrets panel."
• While a reference has no stored value, the Connection step carries the marker "1 secret reference without a stored value".
• A stored value is never shown again, to anyone. Keep your own copy; Replace types a new value for a stored secret and Clear removes one. The vault's full rules are on Connector secrets.

If the file already holds a value where a secret belongs, the Secrets section suggests moving it: "{path} holds an inline value. Move it to the Secrets panel and reference it as {{secret:NAME}}?". Click Move to Secrets; the value moves to the vault and the line is rewritten to the reference as an unsaved change, so save to apply it. Plaintext values are reported in the Secrets section, never blocked, and the suggestion can be dismissed.

Run Test connection

The Test connection card runs the connector's check with the configuration in the editor and the stored secrets. The card says what it does: "Runs the connector check with this configuration and the stored secrets. Optional - you can continue without it."

1. Click Test connection. The button reads Testing… while the check runs.
2. A pass shows Connection test passed, with the detected source timezone when the connector reports one.
3. A failure shows Connection test failed with the reason, and a copy button for the message.

• Failures come in two kinds, and the message says which: a problem with the file itself, for example "file must contain a 'config:' section at the root level", or a problem reaching the source. Fix the cause and test again.
• The button stays unavailable while the connector is not installed and up to date ("Available once the connector is installed and up to date."), while the file is empty, or while Fill with AI hid inline secret values ("Fill with AI hid 1 inline secret value. Type the real value or move it to Secrets before testing.").
• The result clears when you change the configuration or the vault, so a pass always refers to what is on screen.
• If the source does not answer, the test ends with a timeout; check the address and the network, then test again.

Apply your changes

• Save keeps the configuration and any secret values you staged in the vault, and confirms with "Connection saved".
• Save and sync saves and starts a data load, confirmed with "Connection saved - data sync started". It appears on the Connection and Review steps of an existing connector, and its tooltip reads "Available once the connector is installed and up to date".
• While the connector is not ready, the step shows a notice, for example "The connector is still installing - syncing waits for it.", with Go to Build.
• A failed save reports on the step, with "Failed to save connection" or "Failed to save secrets".
• Revert drops your unsaved changes, including staged secret values. Earlier saved versions of the file are under History; see Connect Anything (custom connector).
• While you create the connector, an empty file stops Next with "Add the configuration before continuing.".

Questions, answered

I only see a few settings. Where are the explanations?

The comments start hidden, so the settings stand out. Click Show comments in the editor to bring back the explanations, the header block and the connector's documentation. Example opens the full commented copy at any time.

Which settings are required?

The ones marked # ✓ REQUIRED in the commented file. Give each one a value, or a secret reference when it holds a secret; everything else can stay commented, and the connector uses each # Default: where it has one. The VALIDATION block at the end of the file lists what a valid configuration needs.

What is the exact form of a secret reference?

{{secret:NAME}}, standing alone as the whole value of its setting, with the name stored in the vault. A name runs up to 200 characters: start with a letter, a digit or an underscore, then letters, digits, underscores or dots. Anything else is refused: "Malformed secret reference at: {paths}. Use the exact form {{secret:NAME}}."

A warning says the configuration references a secret that is not stored. What do I do?

Store a value for that name or remove the reference. The Secrets section names both ("{path} references secret {secret}, which is not stored.") and offers Add "…" and Edit config. While you create the connector, Next stops with "Store a value for {names} in Secrets, or remove the reference before continuing."; a sync that finds an unset reference is refused with "Config references secret(s) that are not set: {list}. Set them in the Secrets panel."

What do the placeholder texts in the file mean?

Two placeholders can stand in for a value:

• <redacted - re-enter this value>: a sensitive value was not written into the file. Enter the real value, or store it in the vault and reference it, before you save or test.
• <inline secret - move to the vault>: a value is hidden here until you deal with it. Type the real value, or move it to Secrets, before testing; the message beside the test says the same.

Does Test connection save my configuration?

No. The settings are applied for the check only, and your saved configuration is restored afterwards. Only Save or Save and sync keeps a change.

Why is Test connection unavailable?

Three reasons: the connector is not installed and up to date yet ("Available once the connector is installed and up to date."), the configuration is empty, or Fill with AI hid inline secret values ("Fill with AI hid 1 inline secret value. Type the real value or move it to Secrets before testing."). Fix the named one and the button returns.

Do I have to run Test connection?

No. It is optional, and the card itself says "Optional - you can continue without it." Run it when you want to know the settings work before a load does.

I changed the configuration after a test passed. Is the result still valid?

No. The result clears as soon as you change the configuration or the vault, so a pass always refers to what is on screen. Run the test again to check the change.

When I save, are the secret values I staged in the vault saved too?

Yes. Staged vault values save with the step when you continue or save; if one fails, the save stops and reports it ("Failed to save secrets"). Use Revert before saving to drop them instead.