Connector builders in Lake
Every connector you build or describe yourself has an AI builder of its own, working in the connector's chat: a Python tap for a Connect Anything connector, a recipe for a Connect Any API one. This page covers the two builders, the rules they follow with secrets, how work passes between them, and what their checks and loads really cover.
The two connector builders
| Connector | What its builder builds | When to use it |
|---|
| Connect Anything (a custom connector) | A real Python tap, taken from the API's documentation through code and install to configuration, test and sync | A source a recipe cannot describe, such as unusual sign-in, or custom signing and cursor logic |
| Connect Any API | A recipe: a declarative manifest with no executable code | A REST, OData or GraphQL API you can describe as a recipe; a source that needs Python behaviour is a custom connector, not a recipe |
- One connector can carry both builds at once, a tap and a recipe. Say which one you mean when you ask for a change: the same symptom is fixed differently in each.
- The recipe side has its own walkthrough: Build an ANY API connector.
What the custom connector builder does
A custom connector's builder takes the connector from your API's documentation to a running sync. Its order is always the same: documentation, code, install, configuration, test, sync.
- It probes the real API before it writes the code. It reads your API's documentation, then checks the API's own answers with test requests you can see in the chat as they run.
- The first load is a full load. A custom connector carries a
full_load setting, on by default; the configuration's own comments tell you to switch it off after the first successful load.
- A load lands in the connector's landing database first. A background job then moves the data into your Lake tables.
- You can work on the same files. The builder's work lives in the connector's files; open the code editor to read them and to make your own changes: The connector code editor.
Configuration and secrets
- The settings section is what comes back. When the builder writes the configuration, it returns the settings part of the file: connection settings and their secret references. The connector's program code is not part of that answer.
- Secrets are references, never values. A credential in the configuration is written as a reference of the form
{{secret:NAME}}; the value itself is stored by you in the connector's Secrets. A value the file already masks stays exactly as shown: the builder never replaces it with a literal or a guess.
- A pasted credential leaves the chat. On a custom connector, paste a credential into the chat and the builder moves it into the connector's Secrets, takes it out of the stored conversation and leaves a
<redacted:NAME> marker in its place. The marker is not a value: the vault refuses it, and it cannot be used as one.
- A wrong name is renamed, not re-entered. A secret stored under the wrong name is renamed to the setting's own name, and its value is kept. The builder never repeats a credential back to you; it points to the connector's Secrets instead.
- A missing setting is asked for, not invented. When a non-credential setting is missing, a short message asks for it; the builder never writes a made-up placeholder.
The vault's own rules are on Connector secrets.
Changes go through the connector's builder
- A connector changes only through its own builder: every change it saves is kept as a version. A change made anywhere else is invisible to the next run.
- Until the builder has saved, confirmed and published a change, the connector still runs the old version, and Lake work that depends on it is refused.
- Hand-off. When a table-side build meets a connector problem, the Lake Builder hands the work to the connector's builder. The brief it writes stands alone: the connector, the symptom in your words, what was ruled out, the change to make and how to verify it. The brief is posted into the connector's chat as your request, and a secret value never goes into it.
- The wait. The builder that handed the work over waits for the result: ten minutes by default, thirty at most. If its wait runs out, the work carries on and the report still arrives. The wait can also end as finished, failed, stopped or waiting for you.
- Never from a scheduled run. A hand-off does not start in a scheduled run: a scheduled build that meets a connector problem ends with the brief, ready for a person to send.
- After the report. The table side re-imports only when the columns the connector produces changed, then loads and verifies. A fix counts as done only when the report says so and the reload and checks pass.
The cards, the wait and the report are described on Connector chats and hand-offs.
What a check or a load extracts
- Checks are samples. Every extraction a custom connector's builder starts directly while it works is capped: each stream stops after at most 100 records, so a non-zero count proves rows exist, never how many.
- A workflow run extracts in full. The only full extraction of a custom connector's source is a workflow run; see Workflows in the Lake.
- After a sample, load through the run. A full-reload table loaded straight from a sample extraction is left holding the sample; load such a table through its workflow run, which reads the source in full.
Questions, answered
Should I use a custom connector or an ANY API one?
An ANY API connector when your source is a REST, OData or GraphQL API you can describe as a recipe; a custom connector when the source needs real code, such as unusual sign-in or custom signing and cursor logic. A recipe cannot carry executable code.
Can one connector have both a tap and a recipe?
Yes. A connector can carry both builds at once. Say which one you mean when you ask for a change, because the same symptom is fixed differently in each.
Why does a changed connector still run the old version?
A change is real only when the connector's builder has saved, confirmed and published it. Until then the old version runs, and work that depends on the connector is refused. A change made outside the builder is invisible to the next run.
Why does my connector carry a full_load setting?
So the first load is a full one. It is on by default, and the configuration's comments tell you to switch it off after the first successful load.
Why did a builder's check show only some rows?
Because a check on a custom connector runs a sample: each stream stops after at most 100 records, so a non-zero count proves rows exist, not how many. A workflow run is the full extraction.
Why does my table hold only a sample of rows?
A custom connector's table was loaded from a sample extraction. A full-reload table loaded that way keeps the sample; load the table through its workflow run to read the source in full.
How long does the hand-off wait for the connector's builder?
Ten minutes by default, thirty at most. If the wait runs out, the work carries on in the connector's chat and the report still arrives. Waiting for you means a person is needed there.
What goes into a hand-off brief?
It stands alone: the connector, the symptom in your words, what was ruled out, the change to make and how to verify it. A secret value never goes into a brief.
Can a scheduled run hand work to a connector's builder?
No. A hand-off does not start in a scheduled run: a scheduled build that meets a connector problem ends with the brief, ready for a person to send.
When is a connector fix really done?
When the connector's builder reports the change and the table side's own reload and checks pass. The report alone is not the finish.