Connect any API

Use this page to connect a REST, OData or GraphQL API that no built-in source type covers. You describe the API as a recipe, DataMind OS discovers the data sets it can return (its streams), and you import the streams you need as Lake tables.

Create the connector

The wizard has five steps: General, Recipe, Connection, Streams and Review.

1. Click Add datasource in the Lake header, then choose Connect Any API. In the New menu of the tree toolbar, it is pinned at the top.
2. On General, enter a Name and a Code, and optionally choose a Folder. The Code uses only letters and numbers, at least 2 characters, and it prefixes the names of the tables you import.
3. Click Create & continue. The connector exists from this point, so the assistant can work on it, and it confirms with "Connector created - build its recipe next". Name and Code are locked.

Build the recipe

While the recipe is empty, the Recipe step offers three ways to start, plus Open recipe editor:

• Describe your API to the AI: the assistant reads your API docs or screenshots and writes the recipe with you.
• Paste manifest: opens the recipe editor with a starter manifest for you to replace.
• Import zip: opens a file picker for a recipe zip.

Then:

1. Build the recipe in the recipe editor, and press Ctrl+S to save a version.
2. Click Validate & confirm once your changes are saved. While you have unsaved changes, the button reads Validate.
3. Close the editor and click Next. A failing check lists its issues in the Recipe check card and keeps you on the step: "Fix the recipe issues listed below, then continue." While there is no recipe, Next asks you to "Build the recipe first: describe your API to the assistant, paste a manifest or import a zip." If the step cannot open the recipe at all, it reads "This connector is not available."

The check reads the recipe the way a load will. It refuses the save when the recipe is not valid YAML for the manifest schema, when it declares no version or check stream, when it does not declare exactly one of fixed or dynamic streams, or when it uses unknown top-level entries, custom code components or templates. A secret value written literally into the recipe is refused: store it in Secrets and reference it instead. A stream name uses letters, digits and underscores, up to 128 characters; a column name is non-empty, up to 255 characters, and cannot hold quotes, backslashes, backticks or control characters; the name __hb is reserved. Required settings must be present, and a host the recipe calls stays an issue until you approve it on the Connection step.

Every save is kept as a version you can restore from the versions list, and validating is what marks the recipe Confirmed. Restoring an earlier version leaves the connector Unconfirmed until the recipe is validated again. Reverting past the oldest version answers "Cannot revert past the initial recipe version", and reverting with nothing recorded answers "No changes since the last checkpoint".

To change the recipe later, click Open recipe editor. For the editor itself, see The recipe editor.

Set up the connection

On Connection:

1. Fill in the Connection settings that the recipe declares: text, secret, number, switch, picker, list, JSON, file or either/or fields, built from the recipe itself. A file setting accepts a text file up to 48 KB. Each field validates itself and names itself when something is missing, for example "API key is required".
2. Under Hosts, click Approve for each host the recipe calls, or Approve all when two or more are pending. To approve a host that is not listed, enter it under Add host, with an optional port. A pending host keeps the step open with "Approve the pending hosts below - the recipe can only call approved hosts." Approving a host on a connector that already has data access grants also pins those grants to the host, so the form asks you to confirm first; see Hosts, streams and data access.
3. Optionally, click Test connection. It runs the recipe's check stream with these settings. Without Execute/Abort Lake Objects, the test stays disabled with "You need execute permission on this connector to test it."
4. Open Advanced to set Verify TLS certificate and Rate limit (requests / min); see the table under Settings and defaults below.
5. Click Next. The step saves your settings. The settings and the test belong to the saved connector: before it is saved, the form answers "Save the connection first, then fill the Connection settings" or "Save the connection first, then test it".

Discover streams

On Streams, a new connector discovers its streams when you reach the step, unless they are already up to date. While discovery runs, the panel reads "Discovering streams from the API…" with Stop, and the step notes "Saving, validating and discovering streams - this can take a minute…". Until something is discovered, the table reads "No streams yet - Save and sync to validate the recipe and discover streams." Each row shows:

Then:

1. To change columns before import, click Schema on the stream. Edit the Name, Type, Nullable and PK values, then click Save schema. Reset to inferred returns the detected columns.
2. Click Import on each stream you want as a Lake table; it confirms with "{name} imported to Lake". A stream whose cursor is not a datetime cursor imports in full, and its row notes "Integer cursor - imports as Full (phase 2)".
3. After you change the recipe or its settings, click Discover streams. If you have unsaved changes or the recipe is not confirmed, it saves and validates first. When the streams no longer match the recipe, the step says "The streams are out of date with the recipe or its settings."

Review and finish

1. On Review, check Name, Code, Recipe, Secrets, Hosts, Streams and Source datetime. The datetime value reads Same as platform, convert to platform time or as written; for the options, see Timezones and datetime values.
2. Click Finish. "Finish opens the Lake picker so you can import streams as Lake tables."
3. Set up, deploy and load each table as you would any Lake table. See Import tables, Configure a table and Run, stop and troubleshoot loads.

In edit mode the footer also offers Save and sync, which saves, validates the recipe and discovers streams; it does not load data. It stays disabled while there is nothing to sync, with the hint "Build the recipe first - there is nothing to sync yet." or "Nothing to sync - the streams match the saved recipe."

Work on an existing connector

• Opening a saved connector lands on the step that needs attention: Recipe when there is no recipe or it is Unconfirmed, Connection when a sync failed, a secret is missing or hosts are pending, and Streams otherwise.
• Duplicate skips the Recipe step and runs through General, Connection and Review; the copy starts from the same recipe and no secrets are copied: "Secrets are not copied when duplicating. Enter them again." See Connector secrets.
• You can also work from a chat beside the connector, where the assistant reads the recipe, edits it as diffs, fills public settings (never secrets: "Secret values are never accepted in chat or tool arguments."), validates it, runs the connection check, samples streams and reads the load diagnostics. The cards are described in Connector operations and data access in chat.

Data access

A data access grant lets the recipe drive its requests from the rows of one Lake or Warehouse object, read as the stream grant_<name>. Only the admin role sees the Data access section; the assistant can raise a request, but an admin creates the grant. To add, preview or revoke one, see Data access grants and Hosts, streams and data access.

Settings and defaults

Permissions

A user with Add/Modify Lake Objects and no execute permission can still save the connector, including approving a host, but cannot test or sync it. See Who can do what in the Lake.

Limits and edge cases

• Only the last validated recipe loads. An edited recipe never loads until it is validated again: a load that finds no validation answers "Native API recipe of data source "…" has never been validated - run Save and sync first", and other actions are refused with "Data source "…" has unconfirmed recipe changes - run Save and sync before this action".
• The recipe can call only the hosts you have approved. A sync started before approval fails with "Approve the recipe hosts"; approve the pending hosts in the connector form and save first. Approving a host on a connector with live grants also needs the admin role.
• If the recipe or its settings change after discovery, the Streams step says the streams are out of date, a sync before discovery answers "has no discovered streams yet - run Discover streams first", and a table that asks for a stream outside the discovered set fails with "Stream "…" of "…" is not in the discovered catalog - run Sync Schema".
• Sizes: a recipe zip is at most 20 MB, a manifest at most 1 MB ("manifest exceeds 1 MB"), and a file setting at most 48 KB. Importing a zip replaces manifest.yaml and the public connection settings, never contains secrets ("Secrets are never inside a recipe zip - re-enter them in the form after importing."), and makes the connector Unconfirmed until Save and sync.
• Custom SQL is not supported for these connectors, so a table cannot be built from your own SQL query.
• Administrators can cap how many API tables load at the same time. See Scheduler, health, maps and connector safety.
• When a load fails, start with the run's Sync Log. A run with no error stack opens Runner Diagnostics instead and shows what the load did; parts you cannot access are hidden. See Run, stop and troubleshoot loads.

Questions, answered

Does Save and sync load data?

No. It saves, validates the recipe and discovers streams. Data lands only when a table is loaded, either by hand from the Lake or by a workflow. See Run, stop and troubleshoot loads.

Why is my connector marked Unconfirmed?

The saved recipe has changed since it was last validated. The chip's tooltip reads "The saved recipe has not been validated since it last changed. Save and sync (or Sync Schema in the tree) validates it and discovers streams." Click Save and sync, or run Sync Schema in the Lake tree, to validate it and discover streams. Restoring an earlier recipe version also leaves the connector Unconfirmed until it is validated again.

The recipe calls a host I did not approve. What now?

The Connection step stops until the pending host is approved: "Approve the pending hosts below - the recipe can only call approved hosts." Approve it under Hosts, or revoke it, then click Next. A sync started before approval fails with "Approve the recipe hosts". See Hosts, streams and data access.

Why is Test connection disabled?

Either the recipe is not saved yet, or you do not have Execute/Abort Lake Objects. The test runs the recipe's check stream, so save the recipe first and make sure your role includes that permission; the disabled text reads "You need execute permission on this connector to test it."

Why is Save and sync unavailable?

It needs Execute/Abort Lake Objects and something to sync. With no recipe it asks you to "Build the recipe first - there is nothing to sync yet."; when the streams already match the saved recipe, it reads "Nothing to sync - the streams match the saved recipe."

My API is slow or rate-limited. What should I change?

Set Rate limit (requests / min) under Advanced on Connection. Use Parallel pages only for offset or page-number APIs with a stable sort, under Parallel pages per stream on Streams.

Can I reuse the recipe in another environment?

Yes. In the recipe editor, use Export recipe as zip and, in the other environment, Import recipe zip. The zip holds the manifest and public settings and never a secret, and the imported connector stays Unconfirmed until you click Save and sync. See The recipe editor.

Why is Discover streams greyed out?

It needs a recipe, Execute/Abort Lake Objects and no discovery already running.

My API load failed. Where do I start?

Open the table's Sync Log and read the failed run. A Connect Any API run with no error stack opens Runner Diagnostics instead, showing what the load did; parts you cannot access are hidden. See Run, stop and troubleshoot loads and Lake troubleshooting.