Logs and diagnostics

There are five distinct log sources. Knowing which one you need is most of the work:

How do I see the Installer's own logs?

Read the backend container's stdout. Everything the Installer logs — startup, secret generation, configuration writes, compose invocations and their output — goes to stdout.

docker logs -f delamain-backend

The default production log level is error, warn, log and fatal. Set NEST_NODE_ENV=dev to also get debug and verbose, and set NEST_TYPEORM_LOGGING=true to log every SQL statement the Installer runs.

How do I see the logs of one DataMind OS service?

Open the service in the interface, or read the container's logs on the host. The interface streams a per-service log feed and shows the container's state, health and exit code alongside it.

docker compose logs -f <service>

Or address the container directly by name:

docker logs -f $(docker ps --filter label=com.docker.compose.service=<service> --format '{{.Names}}') --tail 200

The Installer's own endpoint for the same thing is a server-sent event stream:

curl -N "http://localhost:8000/api/deployment/logs/<service>?tail=200" \
  -H "Authorization: Bearer <access-token>"

Omit the service to stream all compose services in one feed: GET /api/deployment/logs.

How do I see the logs of a deployment job that already finished?

Read the job's event stream — it replays from the database. Every event a job emitted is persisted with a sequence number, so you can re-read a finished run, including the compose output.

curl -N "http://localhost:8000/api/deployment/jobs/<jobId>/logs" \
  -H "Authorization: Bearer <access-token>"

Add ?after=<seq> (or send a Last-Event-ID header) to resume from a known point instead of replaying from the beginning. This is also how the interface reconnects after a dropped stream.

What can I read directly in PostgreSQL?

The job history and the audit trail, with psql inside the Installer's own database container.

docker exec -it delamain-postgres psql -U postgres -d delamain

Recent deployment actions and their outcomes:

SELECT action, status, exit_code, detail, created_at
FROM deployment_actions ORDER BY created_at DESC LIMIT 20;

Event counts for one job:

SELECT type, count(*) FROM deployment_job_events WHERE job_id = '<jobId>' GROUP BY type;

The database name defaults to delamain and the user to postgres. The password is in delamain_secrets/pg.pass — the psql client inside the container authenticates over the local socket, so you normally do not need it.

How do I download a log window for support?

Use the download endpoint; it returns a plain-text file covering a time range. The window is mandatory: you must give a start. An end defaults to now, and a container-name filter is optional.

curl -s "http://localhost:8000/api/deployment-log/download?since=2026-09-29T08:00:00Z&until=2026-09-29T09:00:00Z" \
  -H "Authorization: Bearer <access-token>" \
  -o logs.txt

The response is text/plain with a filename of the form logs-<timestamp>.txt. Every line is prefixed with the container it came from, so one file covers the whole deployment.

What should I collect for a support ticket?

Six things, and the first two are not optional — without them the build under discussion is unidentifiable.

The digest matters because a tag can be re-pointed by a pull while the old container keeps running; two systems can both claim to run prod-latest and be different builds. The digest cannot lie.

How do I check whether the Installer backend is alive?

Hit the health endpoint; it is public and it verifies the database too.

curl -s http://localhost:8000/api/health

The response is:

{"status":"ok","db":"up"}

A database failure returns a service-unavailable status rather than a silent success. This is the same endpoint the container healthcheck uses, so a healthy response also means the container is not being restarted.

How do I see the current state of every container?

Read the deployment status and look at each item's condition.

curl -s http://localhost:8000/api/deployment/status \
  -H "Authorization: Bearer <access-token>"

Interpret the conditions like this:

A clean exit is treated as DOWN, not FAILED: exit code 0, a SIGTERM exit, or a SIGKILL that was not an out-of-memory kill.

How do I know whether a job is still running or hung?

List the jobs and read their status and last event sequence. Only one job runs at a time, so the list is short.

curl -s http://localhost:8000/api/deployment/jobs \
  -H "Authorization: Bearer <access-token>"

The Installer protects against a silently stuck job itself: if a job produces no output for 15 minutes it is aborted as hung, with the message recording how long it was silent. Note that an Apply changes job deliberately gets a longer inactivity allowance, computed from the healthcheck timings of the services in the largest tier — ClickHouse's healthcheck alone can take minutes.

How do I cancel a running job?

Delete the job; the Installer terminates the child process and destroys any open pull streams. Cancellation is administrator-only.

curl -X DELETE http://localhost:8000/api/deployment/jobs/<jobId> \
  -H "Authorization: Bearer <access-token>"

Where do container logs live on disk?

In the Docker daemon's own log driver — by default the json-file driver inside the Docker data directory. The Installer's compose file sets no log options, so retention and size limits are governed by the host's Docker configuration, not by the Installer.

To cap growth, configure the daemon instead of the stack:

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "5" }
}

Where does compose output go?

Two places: the job's own event stream, and the Installer's stdout. The Installer buffers compose output line by line, emits each line as a job event, and also writes it to its own logger with the job id as a prefix:

compose[<jobId>] stdout: Container lurien  Started

The second copy exists because the in-memory job registry is not durable: the Installer's stdout is shipped by the log collector to the deployment's log store, so compose output survives a restart. If the Installer is down and you cannot reach the API, its own container log is still the fastest place to read what compose did.

Where do I find logs from an Installer self-update?

Three places, in order of usefulness:

The helper container is a separate container by design, since it kills the process that requested the update — so its logs are not in the Installer's own log.

Where does the host migration write its log?

To /var/log/unistream-migrate-host-<timestamp>.log on the host. The migration script echoes every command it runs and tees all output to that timestamped file, so the run reads like a manual session.

The same output is also streamed to your browser during the migration and is persisted as events against the migration job, so you can re-read it afterwards like any other job.

How far back do logs go?

It depends on the source, and the shortest limit is 30 days.

For long-term retention, export the audit trail rather than relying on container logs — see Data and backups.

For the complete diagnostic procedure, see Logs and Health and monitoring.