Platform operations and security

This page describes how DataMind Platform is run and secured: how a request reaches it, how a release and its migration work, what is backed up, and the contracts the platform's services keep. It is written for the people who operate the platform and run its releases.

The request path

Every public request enters through the platform's edge, its single public entry point. The edge fronts the core application (accounts, keys, billing and catalogues) and the AI gateway (model calls, embeddings and web tools). Machine callers present the company API key at the edge (Use the API); console sessions are validated there too. The applications behind the edge accept no ingress of their own.

At the edge, a request can be stopped at four points, checked in this order:

1. Rate limit. Sign-in and password-recovery requests pass through a tighter per-minute bucket than session refreshes, password resets and invitation links; both buckets run in fixed one-minute windows. A refusal is a 429 reading Too many requests. Please wait a moment and try again., with a Retry-After hint to honour before retrying.
2. Private gate. One internal, service-to-service surface sits behind a gate that admits only the configured source addresses. Every other request is answered exactly like an address that does not exist, so the surface is never revealed.
3. Authentication. The edge validates the session or the API key and attaches the caller's identity itself. A failure is a 401; if the session store is unreachable, the answer is a fail-closed 503, never a bypass.
4. The service. Only a request that passes all three checks reaches an application.

Header trust. Identity headers that arrive from a client are stripped before anything else runs, on public routes as well, and identity is added only by the edge after its check. A caller cannot impersonate another user by sending an identity header.

Request size. One setting drives the size limit at every layer, so no layer rejects what another would accept. The default ceiling is 100 MiB. An oversized request is refused with a 413 that names the limit; the edge's copy reads Upload exceeds the maximum allowed size of {limit}.

Output ceiling. Kartlos-backed requests carry an output ceiling. When a provider refuses a request because the output would be too long, the AI gateway retries it downward through the ceiling ladder (64,000, then 48,000, then 32,000 tokens) with the same prompt; the retries count as one request, so the ledger holds one row for it. Pay-as-you-go requests take no such retry.

Health probes and start-up configuration

Each service exposes two probes:

• /api/v1/health is a trivial check that never touches a dependency; it is safe as both a liveness and a readiness probe.
• /api/v1/health/ready is the start-up probe: it proves the database and the cache answer, and the AI gateway's version also proves its model catalogue. It is wired as a start-up gate only, never as steady-state readiness, so a momentary dependency outage cannot pull every replica out of rotation at once.

Every service validates its whole configuration at start-up and refuses to boot on a missing or invalid required value, so a misconfiguration surfaces as a failed start, never as a first-request error. Optional integrations degrade instead of failing the start: a feature whose settings are absent is offered as unavailable, not broken.

The request ledger

Every model request that passes the gates writes one row to the usage ledger, and a request refused before forwarding writes none. Writes happen after the response, never on the request path: a database hiccup can cost a row, but it can never change what a caller receives. A failed write is retried a few times, then logged loudly, because a lost row cannot be reconstructed.

Rows settle in three cadences:

• A post-response check lands within seconds of a finished request and fills in the provider's final figures.
• An hourly sweep retries the remainder in a bounded batch, and only one replica sweeps per tick.
• A nightly sweep retries the leftovers within a seven-day horizon, during quiet hours; rows older than the horizon drop out on their own.

Two settlement rules stand out:

• Pay-as-you-go rows are re-rated to the provider's settled cost times the platform markup, so the charged amount converges on the provider's final number. Plan-backed requests never re-rate, and a settled zero stays zero: a request the provider waived is never billed.
• The ledger is append-only and independent of the records it mentions, so billing history survives a key rotation or a company closure.

Stored request and response snapshots are redacted before they are written: credential values (keys, tokens, passwords, cookies, secret entries and tool-call arguments) become markers like <redacted:name>, and secrets inside URL query strings are redacted in place. For a company that blocks personal-data transmission, bodies are not stored at all. Body snapshots are an operator setting, on while the platform is in its testing phase.

Environments and releases

The platform runs in three staged environments, development, staging and production, on the same set of services. Only production is internet-facing; the other environments are reachable only inside the private network.

A release runs the same steps whether a person or automation starts it, so the two cannot drift. The order is fixed:

1. Only the services whose files changed are rebuilt, each under its own image tag; the release resolves the correct tag for each service separately, because a release rebuilds only part of the set.
2. The database migration runs first and must finish before the new version starts rolling; a failed migration fails the release before any new replica serves traffic.
3. The services roll one by one, each new replica receiving traffic only after it passes its health check. In-flight responses, streaming answers included, are drained first; the edge allows up to ten minutes for a stream to finish.
4. A smoke check through the public path closes the release.

Database changes are never rolled back: old code runs against the new schema until the last old replica is gone, so every migration must tolerate the previous version of the code. A bad release is answered by shipping the next version.

Backups and routine maintenance

• Production keeps 35 days of point-in-time recovery, built on daily full backups plus continuous logs; the other environments keep 7 days.
• A database server deleted by mistake can be recovered for 5 days.
• Invoice PDFs stay protected for 30 days after deletion.
• The cache is deliberately not backed up: its counters heal from the ledger, and production keeps an hourly snapshot only to bound the loss. Cache contents are never a source of truth.
• Nodes and the platform runtime are patched automatically in a weekly window, Sunday 00:00-08:00 UTC, one node at a time.
• Certificates renew automatically about 30 days before expiry. The edge picks up a renewed certificate only on its next restart, so a renewal is followed by a restart and a check that the new certificate is the one served.

Monitoring and alerts

A shared monitoring stack keeps dashboards for the cluster, the application logs and the traffic each environment serves. Logs are kept 30 days and metrics 90 days.

Alerts cover a cluster that stops reporting, a rising 5xx rate, a deployment that falls below its desired replica count, crash-looping replicas, replicas stuck pending, an unreachable cache, nodes not ready, high disk usage and certificate expiry. Alerts route to the operations channel, and an idle environment never pages: a no-data condition is treated as healthy on every rule.

Shared service contracts

• Envelope. API responses share one shape: success is { success: true, message, data }, and a failure is { success: false, statusCode, error, message, path, timestamp }. Raw payloads such as files and streams skip the envelope. One global filter converts every error into this shape; an unmapped failure reads Internal Server Error with unknown error.
• Pagination. List endpoints page with limit (default 100, maximum 500) and offset.
• Audit stamping. Every write through the API is stamped with the acting user and the time; work the platform runs on its own carries no user stamp.
• Scheduling. Background work (billing rollovers, usage-alert passes, catalogue refreshes and reconciliation sweeps) runs on shared cadences, and each job runs on exactly one process: serving replicas run no jobs, so a job never runs twice. A missed billing boundary self-heals within the hour.
• Client seams. The console is built on a generated client for the platform API; session refresh and retry behaviour lives in the consumer, and streaming clients reconnect with backoff. Contract changes are additive: a shared shape gains fields, and existing ones are not renamed.
• Prompt publishing. Prompt content is assembled from reviewed source files by a build step that checks its structure and size, and a draft marker in the header blocks publishing entirely, so a draft can never be served. Published versions are immutable; a change is a new version. Which version a caller receives follows its build date (Agent prompt versions).

Questions, answered

In what order does the platform check an incoming request?

Rate limit, private gate, authentication, then the service. Each stage fails closed: a refusal ends the request there, and an application accepts nothing that did not pass all three checks at the edge.

Can a caller reach the applications without going through the edge?

No. The edge is the only public entry point and the only place a session or an API key is checked; the applications accept no other ingress. Identity headers sent by a client are stripped on every request and added only by the edge, so spoofing one has no effect.

What happens if a request is too large?

It is refused with a 413 that names the size limit. One setting drives the limit at every layer, so the edge, the AI gateway and the application agree; changing it is an operator action.

How do I check whether the platform is healthy?

Call /api/v1/health for a trivial check that never touches a dependency, or /api/v1/health/ready for the start-up probe that proves the database and the cache answer. The start-up probe is a start-up gate only, so a brief dependency outage does not take every replica out of rotation.

What happens to running requests during a release?

They finish. Replicas drain in-flight responses, streaming answers included, with up to ten minutes allowed, and a new replica only receives traffic after it passes its health check.

Are database changes ever rolled back?

No. The migration runs before the new version starts and is never reverted: old code runs against the new schema until the last old replica is gone. Every migration must tolerate the previous version; a bad release is fixed forward.

How long are backups kept?

Production keeps 35 days of point-in-time recovery and the other environments 7 days; a deleted database server can be recovered for 5 days, and invoice PDFs stay protected for 30 days after deletion. The cache is not backed up: it rebuilds from the ledger.

When does routine maintenance happen?

Node patches and runtime upgrades roll in a weekly window, Sunday 00:00-08:00 UTC, one node at a time. Certificates renew about 30 days before expiry, and the edge is restarted afterwards so it serves the new certificate.

Do stored request snapshots contain credentials?

No. Snapshots are redacted before they are stored; credentials become markers like <redacted:name>, and for a company that blocks personal-data transmission, bodies are not stored at all.

What stops a draft prompt from being served?

The publish gate. Prompt content is assembled and checked first, and a draft marker in the header blocks publishing, so a draft can never be served; published versions are immutable and follow the caller's build date (Agent prompt versions).