Troubleshooting

Run every command on the host, in /opt/delamain, unless the row says otherwise.

Did it refuse, or did it fail halfway?

What you seeWhat it meansWhat to do
A message appears and no job startsNothing changed. The DataMind Installer checked first and refusedFix the value or state the message names, then try again
A job started, showed progress, then failedPart of DataMind OS may already be recreatedRead the job in Job Logs. Then run Start or Update again to finish, or Stop to take the stack down cleanly

Run the diagnostics

The DataMind Installer

CommandA healthy answer
curl -sS http://localhost:8000/api/health{"status":"ok","db":"up"}
docker compose ps -adelamain-postgres and delamain-backend up and (healthy); delamain-backend-init exited with code 0
docker compose logs --tail=100 backendNo repeating error, and no start-up lines repeating. Every job's Compose output is here too
docker compose logs backend-initThe database migrations ran without an error
docker logs delamain-self-updater, only after an Installer updateThe last DataMind Installer update ends with [self-update] outcome: ok
docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}' <container>OOMKilled is false. true means the container was killed for memory

Check it worked. When every command gives its healthy answer, the DataMind Installer is sound.

DataMind OS

CommandA healthy answer
sudo docker compose ps -a, run in deploymentEvery DataMind OS service up

Continue in Status with the DataMind OS service that is not healthy.

Look up the error message

Each message is the exact text the DataMind Installer, its screen or Docker shows. … stands for the detail that follows.

Install and configuration

MessageCauseFix
Failed to download env template from Azure: …Shown at Save & continue in the install wizard, or when an Update job fails while downloading. The Azure client secret is wrong or expired, or the host cannot reach AzureRe-enter the client secret exactly. Check outbound access in System requirements
Failed to download docker-compose.yml from Azure: …Same as the row above, for the Compose fileSame as the row above
Azure client secret is invalid or expired — re-enter it.Shown in Configuration. Azure rejected the stored secretEnter the new secret in Configuration. Then repeat the host login with it: docker login -u <client_id> unistream.azurecr.io
Blob download failed: …The host did not reach DataMind's install-file storage after two attempts. The network error code follows in brackets, for example ETIMEDOUTAllow outbound HTTPS to unistream.blob.core.windows.net, see System requirements. Then repeat the action
Azure AD token request failed: …The host did not reach login.microsoftonline.com to sign in with the client secret. The network error code follows in bracketsAllow outbound HTTPS to login.microsoftonline.com. Then repeat the action
Missing required configurations: …A required setting is empty. The names follow, separated by commasFill each named setting in Configuration, then try again
Configuration with code "…" not foundThe setting does not exist in the published templateUse a setting the template contains
Missing deployment files: …A DataMind OS file was missing and could not be restored. The reason follows —Fix the reason: usually outbound access, the Azure secret, or a missing required setting
PLATFORM_URL cannot be emptyNo platform address was enteredEnter the address DataMind OS is reached at
PLATFORM_URL must be an http:// or https:// address, got "…"The address has another schemeUse http:// or https://. See TLS and certificates

Jobs, images and containers

MessageCauseFix
Start is greyed out with N images missing — run Update firstSome images were never pulled to this hostRun Update, or pull on the host as in Disaster recovery
Missing local images: …A job pulled, but some images did not arriveRead the pull lines in Job Logs. Check registry access, the Azure secret and free disk space
No output for <n> minutes — aborted as hungA job wrote nothing for that long. Usually an outbound call is blocked, or a container never startsRead Job Logs for the last line before the silence. Fix that step, then run the job again
DataMind Installer restarted while this job was runningThe DataMind Installer restarted mid-job, so the job was closedRead docker compose logs backend for the reason, then run the job again
Connection to DataMind Installer lost — reconnecting…The browser lost the live log stream. The job keeps running on the hostWait. If it persists, check the health endpoint, and turn response buffering off in your proxy, see TLS and certificates
Container failed to start: …Shown in a service's panel on Status. Docker could not start that containerRead the log under the message, and act on the detail
Service catalog is outdatedShown on Status. The service grouping comes from an older configuration templateRun Update
Error response from daemon: …Docker refused an action. The detail names the reasonAct on the detail. The two most common follow
network unistream declared as external, but could not be foundThe shared network does not existRun docker network create unistream, then start again
port is already allocatedAnother process holds the portFind it with ss -ltnp | grep ':8000', using the port in the message. Stop that process

Sign-in

MessageCauseFix
Invalid email or passwordThe sign-in name or password is wrongAn administrator resets the password in Users
Account is deactivatedAn administrator deactivated the accountAn administrator activates it again
Initial registration is not allowed once a user exists.The first administrator already existsSign in with that account
Internal server errorIn production the DataMind Installer shows this text for an unexpected errorRead docker compose logs --tail=100 backend for the full error

The DataMind Installer's own update

MessageCauseFix
A self-update is already in progress.An update is already runningWait for it to finish. If .self-update-status already shows a result and no update is running, restart the DataMind Installer with docker compose restart backend in /opt/delamain, then update again
A … job is running (started …) — updating delamain now would kill it. Wait for it to finish.A DataMind OS job is runningWait for the job to end, then update
Update preparation failed; nothing has changed: …The pull or staging failed before the switchFix the detail, usually registry access or disk space, then update again
Could not start the updater; nothing has changed: …The update helper container could not startFix the detail, then update again
This container carries no compose project labels, so its install directory cannot be found. Self-update requires DataMind Installer to have been started by docker compose.The DataMind Installer was started without Docker ComposeStart it with docker compose up -d in /opt/delamain
Manager update rolled back — still on …The new build did not turn healthy, and the previous build runs againRead the toast detail. Only when it starts with ROLLED BACK ONTO A CHANGED SCHEMA, check the schema
ROLLED BACK ONTO A CHANGED SCHEMA: …The rollback succeeded, but the newer build had already migrated the databaseCheck the schema

Spot a restart loop from a missing pg.pass

The backend container restarts in a loop when pg.pass is missing from the secrets volume.

How to spot it

bash
docker compose ps -a
docker compose logs --tail=20 backend
docker compose logs backend-init
SignWhat you see
Container statedelamain-backend shows Restarting
The backend logPostgreSQL password file not found: /usr/src/app/secrets/pg.pass, repeating
The backend log, with POSTGRES_PASS in the Installer's .envJWT secret loaded from file, Encryption key loaded from file and curato service token loaded from file repeat, with no error after them
The init container's logPostgreSQL password file not found: /usr/src/app/secrets/pg.pass
The backend log, after the database container restartedpassword authentication failed for user. The database container wrote a new pg.pass, which the existing database does not accept. Apply the same fix

Confirm that the file is missing. The database container mounts the same volume at /run/secrets:

bash
docker exec delamain-postgres ls -l /run/secrets

Fix it

Restore pg.pass alone from the secrets archive of your Backup and restore. The database keeps its original password, so only the backed-up file matches it. Run as root:

bash
BACKUP=/root/datamind-backup
DATE=2026-10-01   # replace with the date in your backup file names
docker run --rm \
  -v delamain_delamain_secrets:/data \
  -v "$BACKUP":/backup:ro \
  unistream.azurecr.io/docker-hub/library/postgres:16 \
  tar xzf "/backup/datamind-installer-secrets-$DATE.tgz" -C /data ./pg.pass

The other three files stay as they are. Remove POSTGRES_PASS from the Installer's .env if it is there, then start the stack again:

bash
cd /opt/delamain
docker compose up -d

Check it worked. curl -sS http://localhost:8000/api/health answers {"status":"ok","db":"up"}.

Recover from a failed Installer update

The update switches once. If the new build is not healthy within 300 seconds, it rolls back to the previous image. Your database dump restores the schema.

Read how it ended

bash
cat /opt/delamain/.self-update-status
docker logs delamain-self-updater

Read the helper's log before you try another update: the next attempt removes the helper container and its log.

.self-update-statusLast helper linesState and next step
ok[self-update] outcome: okThe new build runs
rolled-backrolled back to rollback — the update did NOT stickThe previous build runs. Check the schema
failedrollback failed too. delamain is down and needs manual recovery over SSH:The DataMind Installer is down. The helper already put the previous Compose file back. Bring the previous build back
failedcould not restore the backed-up compose fileThe DataMind Installer is down, and docker-compose.yml is still the new file. Run sudo cp docker-compose.yml.pre-update docker-compose.yml in /opt/delamain, then bring the previous build back
failedcould not back up the compose file — refusing to switch without a rollback path or could not install the new compose file — nothing has changed yetThe switch never ran, and the previous build still runs. Fix the cause, usually disk space in /opt/delamain. Then restart the DataMind Installer with docker compose restart backend in /opt/delamain, and update again

Bring the previous build back

Before the switch, the update tagged the previous image rollback. Start it:

bash
cd /opt/delamain
VERSION=rollback docker compose up -d backend

VERSION=rollback applies to this command only. A later docker compose up without it starts prod-latest again.

Check it worked. curl -sS http://localhost:8000/api/health answers {"status":"ok","db":"up"}, and docker compose ps shows delamain-backend (healthy).

If the previous build does not start

When the rollback image is gone or does not turn healthy, start the current published build instead:

bash
cd /opt/delamain
sudo curl -fsSL https://unistream.blob.core.windows.net/delamain/docker-compose.yml -o docker-compose.yml
docker compose pull
docker compose up -d

If the pull is refused, sign in to the registry again with docker login -u <client_id> unistream.azurecr.io.

If it fails again, read docker compose logs backend, then restore the dump you took before the update, as in Roll back only the DataMind Installer database.

Check the schema

List the last migrations the database has applied:

bash
docker exec delamain-postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT id, name FROM typeorm_migrations ORDER BY id DESC LIMIT 5;"'

The ROLLED BACK ONTO A CHANGED SCHEMA message reads the database migrated from "<old>" to "<new>". If the top row is <new>, the previous build runs on the newer schema. Restore the dump you took before the update, as in Roll back only the DataMind Installer database, then update again. If the update rolls back again, contact DataMind with the update helper's log.

Check it worked. After the restore, the top row is <old>, and the health check answers {"status":"ok","db":"up"}.

Contact DataMind

Send your report to https://datamind.ge/contact with the exact message, the job from Job Logs, the output of the health check and docker compose ps -a, and the update helper's log if a DataMind Installer update was involved. Remove every token and password from logs before you send them. Never send deployment/.env, the secrets volume or the Azure client secret.

Questions, answered

The DataMind Installer restarts over and over. What now?

See Spot a restart loop from a missing pg.pass.

The DataMind Installer's own update failed. Will it come back?

Yes. See Recover from a failed Installer update.