Health, monitoring and logs

Check the health of each layer

DataMind OS and the DataMind Installer are separate stacks. Check each one on its own.

CheckWhat it provesHealthy answer
Status, open a service, read its Health rowWhether that DataMind OS container passes its own Docker health checkhealthy. A service without a health check has no Health row: read the state beside its name
curl -sS http://localhost:8000/api/health on the hostWhether the DataMind Installer is up and reaches its own database{"status":"ok","db":"up"}
docker compose ps -a in /opt/delamainThe state of the DataMind Installer's three containersdelamain-postgres and delamain-backend running and (healthy), delamain-backend-init exited with code 0

What a failure looks like

What you seeWhat it means
HTTP 503 with "message":"Database connection failed"The DataMind Installer runs but cannot reach its database container. Its log shows Health check failed — DB unreachable: and the reason
curl cannot connectThe backend container is not running, or is restarting. Read docker compose logs --tail=100 backend
A service's Health row is not healthyThat DataMind OS container fails its health check. Read its log in Status

The full 503 body:

json
{"success":false,"statusCode":503,"message":"Database connection failed","error":"Service Unavailable","timestamp":"…","path":"/api/health"}

For causes and fixes, see Troubleshooting.

Where the logs go

LogWhere it goesHow long it lasts
The DataMind Installer's logdocker compose logs backend in /opt/delamain. In production it logs errors, warnings and normal eventsSee Rotate the container logs
Each DataMind OS service's logThat container's standard output, streamed into Status when you open the serviceAs long as Docker keeps that container's log
A job's outputStatus → Job Logs. Every Compose line of a job is also in the DataMind Installer's log, prefixed compose[<job id>]See How long job history is kept
The audit trailThe DataMind Installer's database: who ran which action. It is part of the database dumpAs long as the database

DataMind OS includes Fluent Bit, which ships the container logs on the host to VictoriaLogs while DataMind OS runs. That includes the DataMind Installer's log and every job's output.

How long job history is kept

Job Logs shows a job until one hour after it ends. After that, its Compose lines remain in the DataMind Installer's log (compose[<job id>]) and in VictoriaLogs. The DataMind Installer deletes stored job events older than 30 days each time it starts. A job that was running when it restarted is closed with DataMind Installer restarted while this job was running.

A job that writes no output for a set time stops with No output for <n> minutes — aborted as hung. The time is 15 minutes, and longer for Apply changes when a service's health check needs more time.

Rotate the container logs

The DataMind Installer's Compose file caps each of its containers at 5 log files of 20 MB, in the uncompressed json-file format that the DataMind OS log pipeline reads. A container takes the setting when it is created.

Check it worked.

bash
docker inspect delamain-backend --format '{{.HostConfig.LogConfig.Type}} {{json .HostConfig.LogConfig.Config}}'
text
json-file {"compress":"false","max-file":"5","max-size":"20m"}

If the output shows no max-size, the install directory still has an older Compose file. Update the DataMind Installer from the top-bar pill, as in Updating: it installs the current Compose file and recreates the containers.

Questions, answered

Where are the logs?

In each container's Docker log, and in VictoriaLogs while DataMind OS runs. See Where the logs go.

Does the DataMind Installer's health check prove DataMind OS is up?

No. It proves the DataMind Installer reaches its own database. Check each service's Health row in Status.

How long is job history kept?

See How long job history is kept.

Are container logs rotated?

Yes. See Rotate the container logs.