Webhook and RabbitMQ

Use a webhook when another system pushes events to DataMind OS. Use RabbitMQ when your events already sit in RabbitMQ queues. In both cases, events collect in a buffer first, and the Lake tables you import from the connection load from it.

Connect a webhook

Open Lake in the sidebar and click Add datasource. Choose Data Storage > Webhook, or pick Webhook under Popular.
On General, enter a Name of at least 3 characters and a Code of letters and numbers, at least 2 characters. Choose a Folder if you use folders.
Click Next. On Sender & security, choose a Sender and fill in the security fields that appear.
Click Next. On Table & retention, set the Table name and Retention.
Click Add. Copy the Endpoint URL and the token or signing secret now. The secret is masked when you leave the screen and cannot be shown again.
Paste the endpoint URL into your sender's webhook settings, then click Done.

Tip

Click Request guide under the endpoint URL to see the exact request your sender must make. It shows the endpoint, the headers the request carries and an example, built for the sender you chose. The secret is never shown there.

Sender and security

Sender	What it does
GitHub, Shopify, Wolt	Checks the vendor's signature. The signature format is set for you, so you only enter the signing secret.
Static token in a header	Checks a shared token that the sender puts in a header. This is the default, with the header name X-API-Key.
Custom HMAC signature	Checks an HMAC signature. You set the signature format yourself.
No authentication	Checks nothing.

Each sender checks exactly what its row describes, and nothing else. Matching the Header name ignores case.

Warning

With No authentication, nothing is verified. Anyone who learns the URL can write events into this connection. Use it only when the sender cannot sign or send a token.

Stripe and Slack senders are not supported, because they pack several values into one header.

With Custom HMAC signature, the Signature format section shows these settings:

Setting	What it does	Default
Algorithm	The hash the sender signs with: SHA-256, SHA-1 or SHA-512.	SHA-256
Digest encoding	How the digest is written: Hex or Base64.	Hex
Signature prefix	Text the sender puts before the digest, if any.	Empty
Timestamp header	Set this only if the sender signs "timestamp.body". It also turns on replay rejection.	Empty
Replay window (seconds)	Appears when Timestamp header is set. How far the sender's timestamp may drift before the event is refused.	300

Table and retention

Setting	What it does	Default
Table name	The table that receives one row per event, with the body kept as raw JSON. It is fixed once the connection is saved.	table
Retention	Keep forever keeps every event. Delete after N days deletes older events.	Keep forever

After you connect

The sender makes a POST request to the Endpoint URL shown on the connection. A request the connection accepts gets HTTP 200 with {"received": true}, and the event is written to the buffer.

A request the connection refuses gets HTTP 401 with Webhook authentication failed: <reason>. The reason names the check that did not pass, and it never contains the stored secret:

Reason	What it means
`missing_auth_header`	The request has no credential header, or the header name does not match Header name.
`token_mismatch`	The token does not match the connection's current or previous token.
`signature_mismatch`	The signature does not match what DataMind OS computes from the request.
`timestamp_outside_tolerance`	The request's timestamp falls outside the Replay window (seconds).
`connection_auth_misconfigured`	The connection's own sender settings are incomplete, so no request can match.

Two more answers a sender can meet:

HTTP 404 Unknown webhook endpoint: the address matches no connection. The sender posts to the wrong place.
HTTP 400 Empty or unreadable request body: the request carries no readable body, so there is nothing to store.

Accepted events wait in the buffer until the table is synchronised. The buffer exists from the moment the connection is saved, so events are kept even before you import a table. The endpoint URL never changes when you edit the connection.

Rotate a webhook secret

Open the connection and go to Sender & security.
Enter the new token or signing secret. Keep Keep the old secret working during changeover on. It is on by default, so both secrets are accepted until you stop the old one. No event is refused while your sender catches up.
Click Save, then update your sender to use the new secret.
When the sender no longer uses the old secret, click Stop accepting in Sender & security, then click Save. If you change your mind before saving, click Undo.

Connect RabbitMQ

Click Add datasource > Data Storage > RabbitMQ, or pick RabbitMQ under Popular. Fill in General as you do for a webhook.
On Broker, enter Host, Port, Virtual host, Management API port, Username and Password. Turn on TLS to connect over AMQPS. The ports follow the switch while they still have their default values.
Click Test connection. This is optional. It logs in over AMQP and asks the management API which queues it can see.
Click Next. On Tables, choose one option under Queues:
- The first option lets DataMind OS create a queue for each table and bind it to your exchange with a routing key. Enter the Exchange and choose an Exchange type.
- Consume existing queues reads queues that already exist on the broker. DataMind OS declares nothing.
Under Discovered tables, click Refresh. Each routing key, or each queue in the second option, that the broker reports becomes a table you can import. If the management API cannot be reached, define the tables manually.
To define tables yourself, open Define tables manually, click Add table, and enter a Table name and a Routing key or Queue name. Once a table is listed, only listed tables are offered and discovery is skipped. Leave Columns empty to sample the columns from live messages at import, or add columns and tick Key on the one that identifies a row.
Click Next. On Load & review, set Keep consumed messages for (days) under Buffer. Open Advanced if you need more consumers or queues.
Click Add, then import the tables you need.

For every broker field in detail, including its default, see Connect RabbitMQ.

Imported RabbitMQ tables also get _message_id and _ingested_at columns. Without a key column, _message_id is the key.

Creating a webhook or RabbitMQ connection needs Add/Modify Lake Objects. See Who can do what in the Lake.

To send data out to RabbitMQ instead, see Publish to RabbitMQ.

Limits and edge cases

Webhook request bodies up to 1 MB are accepted. Empty bodies are refused.
The endpoint does not ask for a DataMind OS sign-in, so the sender security you choose is the only check on who can write events.
The event keeps the body and a normalised list of headers. Credential headers, such as Authorization and Cookie, and the header that carries your token or signature, are never stored with an event.
Expired events cannot be replayed, so a table with the Full Reload (Truncate Insert) sync type loses that history on its next full reload. See Connect Webhook for the retention setting and Choose a sync type.
One endpoint feeds one table. Each connection keeps its own buffer, so two connections stay apart even when their table names are similar.
Deleting the connection removes its buffer with it.
If you change a webhook sender to a different scheme, the stored secret cannot be reused. Enter the new token or signing secret.
While tables are imported from a RabbitMQ connection, its queue option, Exchange and Exchange type are locked until those tables are removed; see Connect RabbitMQ.

Questions, answered

I lost the webhook secret. What do I do?

A secret cannot be shown again. Enter a new token or signing secret in Sender & security, keep Keep the old secret working during changeover on, and follow the steps in Rotate a webhook secret above.

Can I rotate the secret without losing events?

Yes. Both the current and the previous token or signing secret are accepted during changeover, so events keep arriving while your sender catches up. Save the new secret, update the sender, then stop accepting the old one.

Does the endpoint URL change when I edit the connection?

No. The URL stays the same when you change the sender or the security settings.

What does the sender get back on success?

HTTP 200 with {"received": true}.

What happens to events that arrive before I import a table?

They wait in the buffer. The buffer exists from the moment you save the connection, and the table loads from it on its next sync.

Why is my table empty when events arrive?

Events land in the buffer first. The table fills the next time it is synchronised, by hand or by its workflow. See Run, stop and troubleshoot loads.

How do I stop replayed requests?

Set Timestamp header if the sender signs "timestamp.body". The endpoint then refuses any request whose timestamp falls outside the Replay window (seconds), which is 300 by default.

Are the sender's headers stored with the event?

The event keeps its headers as a normalised list, but never credential headers such as Authorization or Cookie, and never the header that carries your token or signature.

Does my sender need a DataMind OS account?

No. The endpoint answers any sender that can reach it, without a sign-in. The sender security you choose is the only check, so take care before choosing No authentication.

What happens to the buffer when I delete the connection?

The buffer is removed with the connection, and senders that keep posting to the old endpoint get HTTP 404 Unknown webhook endpoint.