Update the DataMind Installer

An update replaces the DataMind Installer itself. The DataMind OS stack it manages is a separate compose project and is not pulled, recreated or restarted by an Installer update.

Understand what an update changes

A release can change two things, and the Installer checks both:

The image is built by .azure-pipelines/build-image.yml. The Installer's own compose file is published to blob storage by .azure-pipelines/publish-compose.yml on every push to main, so the published file is the canonical one.

Install channels and image tags

The Installer resolves its target from the channel, not from a hand-typed version.

• Production docker-compose.yml sets image: unistream.azurecr.io/delamain:${VERSION:-prod-latest} and DELAMAIN_CHANNEL: prod; the dev compose file sets the dev equivalents.
• The channel is taken from DELAMAIN_CHANNEL if set; otherwise it is the part of the running image tag before the first - (so prod-4821 resolves to channel prod).
• The check always requests "<channel>-latest" from the registry, whatever tag is running.

Check whether an update is available

In the interface, a pill in the header shows "Manager update available" when one exists; its tooltip shows the current digest and the remote digest. The check runs on load and every 30 minutes. Clicking the pill opens a confirmation dialog that states the Installer restarts, the DataMind OS stack is not touched, and to expect roughly one to three minutes without a connection.

Over the API, GET /api/system/self-update-check returns:

curl -s "http://localhost:${NEST_PORT:-8000}/api/system/self-update-check" \
  -H "Authorization: Bearer <token>"

Start the update

From the interface

Confirm the dialog. The interface calls POST /api/system/self-update and then waits for the backend to come back (see Observe the update).

From the API

curl -s -X POST "http://localhost:${NEST_PORT:-8000}/api/system/self-update" \
  -H "Authorization: Bearer <token>"

The response is { "started", "targetDigest", "message" }. The endpoint is authenticated but carries no additional role requirement, so any signed-in user or service caller can start it.

The Installer refuses to start when:

From the host

An operator can always update by hand over SSH:

docker compose pull && docker compose up -d

This is also the recovery path the helper prints when both the switch and its rollback fail.

What the helper container does

The Installer cannot replace itself from inside itself — recreating the backend would kill the process that asked for the update. So the running backend does the safe part first, then hands the switch to a detached helper container named delamain-self-updater, spawned from the new image.

Pre-flight, before anything changes

Still inside the running backend (SelfUpdateService.preflight):

1. Tags the image that is currently running as rollback — unistream.azurecr.io/delamain:rollback (the SELF_UPDATE_ROLLBACK_TAG). This both pins the rollback target and shields the old image from the Installer's own dangling-image prune.
2. Downloads the published compose file for the channel and writes it to /var/tmp/delamain/self-update/docker-compose.yml on the host (a path bind-mounted into both the backend and the helper at the identical location).
3. Pulls the target image.

If any of these fail, the update aborts with "nothing has changed".

Spawning the helper

The backend records a self-update action in the deployment audit trail with the target digest and the current schema migration, force-removes any leftover delamain-self-updater container, then runs:

docker run -d --name delamain-self-updater \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v <work-dir>:<work-dir> \
  -v <staging-root>:<staging-root> \
  unistream.azurecr.io/delamain:<new-tag> \
  sh /usr/src/app/scripts/self-update.sh \
    <project> <work-dir> <compose-file> /var/tmp/delamain/self-update/docker-compose.yml \
    rollback <new-tag> backend

The helper mounts the host Docker socket, the compose project's working directory, and the shared staging directory. The final argument is the Installer backend's compose service name, backend, so the wait judges only that service and its dependencies.

The switch, and the one rollback

Inside the helper (scripts/self-update.sh):

1. Backs up the live compose file to <compose-file>.pre-update.
2. Installs the staged compose file over the live one. Local edits to the compose file do not survive an update; the previous version stays at <compose-file>.pre-update.
3. Switches onto the new tag:
   bashCopy

   VERSION="<new-tag>" docker compose -p <project> -f <compose-file> \
     up -d --wait --wait-timeout 300 backend

   --wait blocks on the container's healthcheck, so a container that starts and immediately dies counts as a failure. The VERSION environment variable outranks the project's .env, so an operator-pinned VERSION cannot turn the update into a silent no-op.
4. If the switch succeeds, writes ok to <work-dir>/.self-update-status and exits 0.
5. If the switch fails, restores <compose-file>.pre-update and runs the same command pinned to the rollback tag. Success writes rolled-back and exits 0 — the update did not stick.
6. If the rollback also fails, writes failed and exits 1. The Installer is down and needs manual recovery over SSH: fetch the Installer compose from blob storage, then docker compose pull && docker compose up -d. The helper's output stays available with docker logs delamain-self-updater.

The escalation chain is fixed and short: one switch, one rollback, stop. There are no retry loops and no down-migrations — reverting a schema is more destructive and less tested than the forward path, so a genuinely breaking migration makes a release non-rollbackable instead.

The --wait-timeout for both attempts is 300 seconds.

The status file

The helper writes <work-dir>/.self-update-status in the compose project's working directory. Its value is exactly one of ok, rolled-back or failed.

That file is a breadcrumb for a human reading the host. It is not read back by the Installer — the project's working directory is not mounted into the backend container. So on its next boot the Installer reconciles by comparing image digests instead.

Observe the update

When the new build returns, the interface reports one of: updated to the new tag; rolled back and still on the previous tag (with the reason from lastUpdate.detail); or restarted on a tag that matched neither digest.

Confirm the outcome and read the reconciliation

On boot the backend looks for the last unfinished self-update action and reconciles it:

• If the running digest equals the recorded target, the action is marked completed.
• Otherwise it is marked failed, with a detail that says whether the database schema moved. If the schema migrated past the recorded preUpdateMigration before the rollback, the detail reads ROLLED BACK ONTO A CHANGED SCHEMA and the event is logged at error level — that build must tolerate the new schema and needs manual verification.

Confirm which build is live:

curl -s "http://localhost:${NEST_PORT:-8000}/api/version" \
  -H "Authorization: Bearer <token>"

The digest is exact; the tag is not, because a tag moves while a container keeps running.

Roll back

An update is rolled back automatically, at most once, as described above. To roll back deliberately on the host, pin the tag and recreate the backend:

VERSION="rollback" docker compose -p <project> -f <compose-file> \
  up -d --wait --wait-timeout 300 backend

rollback is the tag the pre-flight applied to the previously running image. It is protected from the Installer's dangling-image prune, so it is still on the host.

Pin a version

• The Installer's image tag comes from VERSION in the Installer's .env; the production compose file defaults it to prod-latest when unset.
• The built-in self-update overrides VERSION with the channel's -latest tag by design, so a pinned value does not block an in-app update.
• To hold a specific version, do not use the in-app self-update: set VERSION to the tag you want and deploy with docker compose on the host.
• DELAMAIN_CHANNEL selects the channel the check and the update use; the production compose file sets it to prod.

Trigger an update from the DataMind OS side

The Installer exposes the update over its own API, and the DataMind OS control plane reaches that API. The mechanism, as it appears in this repository:

• The Installer and the DataMind OS stack both attach to the shared, externally created Docker network unistream, where the Installer's backend is published under the alias delamain-backend.
• The control plane authenticates not with a user JWT but with a shared service token, stored as curato.token in the Installer's secrets volume and handed to the platform's .env. It is presented as a bearer token with x-actor-user-id and x-actor-email headers, purely so the audit trail names a person.
• The Installer's global JWT guard accepts that token and treats the caller as an admin, before Passport is consulted. It also forwards the acting user's id into the deployment audit trail.

So the code path from the DataMind OS control plane to the Installer's update and deployment endpoints exists and is authenticated end to end.

Verify the update