Move DataMind OS and the DataMind Installer, with all data and settings, from your current server to a new one in one maintenance window. Every step says where to run it and how to check that it worked, and the current server stays untouched, so you can always go back.
Go through the steps in order. Each one says where to run it, checks the result with you, and tells you when to send the output to DataMind.
Have these ready:
Two rules for the whole move:
Run these checks as soon as the new VM exists, and paste the output here. The page reads it for you and says what is fine and what to fix. Then send it to DataMind.
On both servers, as root
On the new VM, run the first block. On the current server, run the second one. On each, sign in as the OS user, run sudo -i on its own, then paste the block: Docker answers only to root. Paste each output into its own box below.
On the new VM:
grep PRETTY_NAME /etc/os-release; uname -m; nproc; free -h; df -h
docker --version; docker compose version; id "${SUDO_USER:-$USER}"
echo "user: ${SUDO_USER:-$USER}"
echo "public IP: $(curl -sS -m 10 https://api.ipify.org)"
docker info -f '{{.Driver}} {{.CgroupDriver}} cgroup-v{{.CgroupVersion}}'
uname -r; stat -fc %T /sys/fs/cgroup; grep -c memory_recursiveprot /proc/mounts
grep -o -w -E 'sse4_2|popcnt|avx2' /proc/cpuinfo | sort -u
curl -sS -o /dev/null -w 'registry: %{http_code}\n' https://<registry>/v2/
for h in login.microsoftonline.com platform.datamind.ai; do curl -sS -o /dev/null -w "$h: %{http_code}\n" https://$h; done
nc -zv -w 5 <data source host> <data source port> # once for every data source DataMind OS connects to
getent hosts <data source host> # only for a data source given by name, not by addressOn the current server:
nc -zv <new VM IP> 22
grep PRETTY_NAME /etc/os-release; docker --version; docker info -f '{{.Driver}}'
echo "user: ${SUDO_USER:-$USER}"Check it worked. Every line in the result list is green or explained. Send the output to DataMind and wait for our confirmation.
Send the output to DataMind.
One-time preparation, done before the maintenance window. Steps marked Send to DataMind need our confirmation, so nothing is decided during the window.
Work as root, inside tmux, so a dropped SSH session does not stop a running step. Start with tmux new -A -s migration, then run sudo -i as a separate command before you paste anything else. The same tmux new -A -s migration reconnects you later: it opens the session, or rejoins it when it already exists.
On the current server, as root
Install tmux and rsync if they are missing, then find the DataMind Installer: it is the container that publishes port 8000.
apt-get install -y tmux rsync
INSTALLER=$(docker ps -q --filter publish=8000)
docker inspect -f '{{index .Config.Labels "com.docker.compose.project.working_dir"}}' "$INSTALLER"
docker inspect -f '{{range $name, $net := .NetworkSettings.Networks}}network: {{$name}}{{println}}{{end}}' "$INSTALLER"Check it worked. One line shows the install directory, and the network: lines name the Installer's Docker networks. The shared one is the network whose name does not end in _default.
Send the output to DataMind.
On the current server, as root
These settings depend on the server's address. The last two show whether DataMind OS sends e-mails through your own mail server. The lines hold no passwords.
INSTALL=<install directory> # the directory printed in step S1 grep -E '^(PLATFORM_URL|SSL_CERT_PATH|SSL_KEY_PATH|NATIVE_API_DENY_HOSTS|NATIVE_API_DENY_CIDRS|REDASH_MAIL_SERVER|REDASH_MAIL_PORT)=' "$INSTALL/deployment/.env"
Check it worked. At least the PLATFORM_URL= line appears.
Send the output to DataMind.
On the current server, as root
DataMind tells you which folders go into the backup, whether the new VM's disk is large enough, and which scheduled jobs of this server must move too, such as the certificate renewal.
INSTALL=<install directory> # the directory printed in step S1
cd "$INSTALL/deployment"
PROJECT=$(docker compose config | awk '/^name:/{print $2; exit}')
docker ps -aq --filter "label=com.docker.compose.project=$PROJECT" \
| xargs docker inspect -f '{{range .Mounts}}{{if eq .Type "bind"}}{{println .Source}}{{end}}{{end}}' \
| grep -v '^$' | sort -u
docker system df -v
docker system df
df -h /root
crontab -l; ls /etc/cron.d; systemctl list-timers --all --no-pagerCheck it worked. The free space on / is larger than the size in the Local Volumes row: the backup is written there first. DataMind confirms the folder list and goes through the scheduled jobs with you.
Send the output to DataMind.
On the current server, as root
A key lets the copies run without stopping for a password.
[ -f /root/.ssh/id_ed25519 ] || ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519 ssh-copy-id <user>@<new VM IP> ssh <user>@<new VM IP> true && echo "ssh ok"
If ssh-copy-id fails because the new VM does not accept passwords, add the line from cat / to / on the new VM yourself.
Check it worked. The last line reads ssh ok.
On the new VM, as root
Install the tools, set Docker's image storage and create the shared Docker network.
Docker 29 and later store images differently on a fresh install. If the current server printed overlay2 in the check, switch the new VM to the same storage before any image arrives: it keeps both servers identical and uses less disk. If / already exists, send it to DataMind instead of running that block.
apt-get install -y tmux rsync
Only if the current server shows overlay2:
echo '{ "features": { "containerd-snapshotter": false } }' > /etc/docker/daemon.json
systemctl restart dockerdocker info -f '{{.Driver}}'
docker network inspect <network> >/dev/null 2>&1 || docker network create <network>Only if step S2 showed your own mail server:
nc -zv -w 5 <mail server> <mail port>
Check it worked. docker info prints the same storage driver as the current server. If you ran the mail line, it reports succeeded.
On the new VM, as root
The backup is copied into this folder, owned by the OS user that receives it.
mkdir -p /home/<user>/datamind-backup chown <user>: /home/<user>/datamind-backup chmod 700 /home/<user>/datamind-backup
Check it worked. No output means it worked.
On the new VM, as root
The new VM holds the programs, the backup and the restored data at the same time.
df -h /
Check it worked. DataMind confirms the free space is enough. If it is not, enlarge the disk before the window.
Send the output to DataMind.
Your IT team
The new VM has a public address. The Installer (port 8000) and several DataMind OS services accept connections on every network address of the VM, and Ubuntu's firewall (ufw) does not block ports that Docker opens. Set these rules in your provider's network firewall.
If your provider has no network firewall, tell DataMind: we send you rules for the VM itself. Step D3 checks the result.
On the current server, as root
The new VM gets an exact copy of the programs (Docker images) the current server runs, so nothing is downloaded during the window. Image tags in the registry move with every release, so a download could bring a newer version.
From here on, do not update DataMind OS or the Installer on the current server.
INSTALL=<install directory> # the install directory confirmed by DataMind
cd "$INSTALL"
PREFIX=$(docker compose config | awk '/^name:/{print $2; exit}')
cd "$INSTALL/deployment"
PROJECT=$(docker compose config | awk '/^name:/{print $2; exit}')
for p in "$PROJECT" "$PREFIX"; do docker ps -aq --filter "label=com.docker.compose.project=$p"; done \
| xargs docker inspect -f '{{.Config.Image}} {{.Image}}' | sort -u > /root/datamind-images.txt
OSIDS=$(docker ps -aq --filter "label=com.docker.compose.project=$PROJECT" | xargs docker inspect -f '{{.Image}}' | sort -u)
while read -r image id; do
cur=$(docker image inspect -f '{{.Id}}' "$image" 2>/dev/null)
[ "$cur" = "$id" ] && continue
if ! grep -q " $cur$" /root/datamind-images.txt; then
docker tag "$id" "$image" && echo "TAG RESET: $image"
elif grep -qx "$id" <<< "$OSIDS"; then
docker tag "$id" "$image" && echo "SHARED NAME: $image now on ${id:7:12}"
else
echo "SHARED NAME: $image stays on ${cur:7:12}"
fi
done < /root/datamind-images.txt
wc -l < /root/datamind-images.txtCheck it worked. The count is above 20. A TAG RESET line is fine: a newer download was waiting under that name, and the name now points back to the build that runs, so the copy in S10 is exactly what runs today. A SHARED NAME line is fine too: the Installer and DataMind OS run two builds under one name, and the name points to the build DataMind OS runs, because its database files are copied as they are. The Installer's database moves as a dump, so it runs on either build. Only names move: every program keeps running the build it runs.
On the current server, as root
This runs for a while: about 10 to 30 minutes, depending on the connection. It also measures the copy speed between the servers, which DataMind needs to plan the window.
START=$(date +%s)
docker save $(awk '{print $1}' /root/datamind-images.txt) | gzip -1 \
| ssh <user>@<new VM IP> 'cat > datamind-backup/datamind-images.tar.gz'
echo "exit codes: ${PIPESTATUS[*]}"
SIZE=$(ssh -n <user>@<new VM IP> 'stat -c %s datamind-backup/datamind-images.tar.gz')
awk -v s="$SIZE" -v t="$(( $(date +%s) - START ))" 'BEGIN { printf "speed: %.1f MB/s\n", s / 1048576 / (t > 0 ? t : 1) }'Check it worked. The line exit codes: reads 0 0 0: saving, packing and sending all finished without errors. The last line shows the copy speed.
Send the output to DataMind.
On the new VM, as root
gzip -t /home/<user>/datamind-backup/datamind-images.tar.gz && echo "archive ok" docker load -i /home/<user>/datamind-backup/datamind-images.tar.gz && rm /home/<user>/datamind-backup/datamind-images.tar.gz docker images | wc -l
Check it worked. The first line reads archive ok, docker load prints one Loaded image line per program and no Error line, and the last number is above 20. The copy is deleted only after a load without errors, so a failed load can be repeated without S10.
Part A starts the maintenance window: from step A4 on, users cannot work until Part E. Before A1, make sure DataMind is online in your usual chat. Part A then stops DataMind OS on the current server and writes a complete backup to /. Run every step in the same tmux session: later steps use the variables set in A1.
On the current server, as root
Run tmux new -A -s migration, then sudo -i, as two separate commands. Then paste:
echo "running as: $(id -un)"
INSTALL=<install directory> # the install directory confirmed before the window
BACKUP=/root/datamind-backup
DATE=$(date +%F)
umask 077
mkdir -p "$BACKUP"
cd "$INSTALL"
PREFIX=$(docker compose config | awk '/^name:/{print $2; exit}')
PGIMAGE=$(docker compose config --images | grep postgres)
SECRETS_VOL=$(docker inspect -f '{{range .Mounts}}{{if eq .Type "volume"}}{{println .Name}}{{end}}{{end}}' "$(docker compose ps -q postgres)" | grep secrets)
echo "$SECRETS_VOL" > "$BACKUP/installer-secrets-volume.txt"
cd "$INSTALL/deployment"
PROJECT=$(docker compose config | awk '/^name:/{print $2; exit}')
declare -p INSTALL PREFIX BACKUP DATE PGIMAGE PROJECT SECRETS_VOL > /root/migration-vars.sh
echo "$INSTALL | $PREFIX | $PROJECT | $SECRETS_VOL | $DATE"If the session is lost later, run sudo -i, then source / to get the variables back.
Check it worked. The first line reads running as: root, and the last line prints five values: the install directory, the Installer's project name, the DataMind OS project name, the Installer's secrets volume and today's date. None of them is empty, and no line says Permission denied. The page saves the date for Part B.
In the Installer (web screen)
Nothing may change the data while it is backed up.
On the current server, as root
The file lists every table in the data warehouse with its row count. Part D compares the new VM against it.
cd "$INSTALL/deployment"
CH_PASS=$(grep '^CLICKHOUSE_PASS=' .env | cut -d= -f2-)
docker exec "$(docker compose ps -q clickhouse)" clickhouse-client --password "$CH_PASS" -q "SELECT database, table, sum(rows) FROM system.parts WHERE active AND database NOT IN ('system','INFORMATION_SCHEMA','information_schema') GROUP BY database, table ORDER BY database, table FORMAT TSV" > "$BACKUP/counts-before.tsv"
wc -l "$BACKUP/counts-before.tsv"Check it worked. The count is above 0.
On the current server, as root
Click Stop on Status in the Installer, or run:
cd "$INSTALL/deployment" docker compose stop docker compose ps
Check it worked. docker compose ps lists no running services.
On the current server, as root
Its database, its secrets and its files hold every setting and password. They are backed up as a set.
cd "$INSTALL"
docker exec "$(docker compose ps -q postgres)" sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB"' \
> "$BACKUP/datamind-installer-db-$DATE.sql"
docker run --rm -v "$SECRETS_VOL":/data:ro -v "$BACKUP":/backup "$PGIMAGE" \
tar czf "/backup/datamind-installer-secrets-$DATE.tgz" -C /data .
FILES="docker-compose.yml deployment"
[ -f .env ] && FILES="$FILES .env"
tar czf "$BACKUP/datamind-installer-files-$DATE.tgz" $FILES
docker image inspect -f '{{.RepoTags}} {{index .RepoDigests 0}}' \
"$(docker inspect -f '{{.Image}}' "$(docker compose ps -q backend)")" > "$BACKUP/datamind-installer-version-$DATE.txt"
(cd "$BACKUP" && sha256sum "datamind-installer-db-$DATE.sql" \
"datamind-installer-secrets-$DATE.tgz" "datamind-installer-files-$DATE.tgz" \
"datamind-installer-version-$DATE.txt" installer-secrets-volume.txt > "datamind-installer-$DATE.sha256")
head -n 3 "$BACKUP/datamind-installer-db-$DATE.sql"
grep -c 'PostgreSQL database dump complete' "$BACKUP/datamind-installer-db-$DATE.sql"
tar tzf "$BACKUP/datamind-installer-secrets-$DATE.tgz"
gzip -t "$BACKUP/datamind-installer-files-$DATE.tgz" && echo "files archive ok"Check it worked. The dump starts with -- PostgreSQL database dump, the count after it is 1 (the dump is complete), the secrets archive lists pg., jwt., encr. and curato., and the last line reads files archive ok.
On the current server, as root
Every Docker volume and the host folders DataMind confirmed go into $BACKUP/, together with the list of image builds and a checksum file. The archives use the fastest compression: most of the data is compressed already, so stronger compression would only lengthen the window.
HOSTDIRS="<directories confirmed by DataMind>" # separated by spaces
OSBACKUP="$BACKUP/os-$DATE"
mkdir -p "$OSBACKUP"
VOLUMES=$(docker ps -aq --filter "label=com.docker.compose.project=$PROJECT" \
| xargs docker inspect -f '{{range .Mounts}}{{if eq .Type "volume"}}{{println .Name}}{{end}}{{end}}' \
| grep -v '^$' | sort -u)
[ -n "$VOLUMES" ] || echo "WARNING: no volumes found"
printf '%s\n' $VOLUMES > "$OSBACKUP/volumes.txt"
docker ps -aq --filter "label=com.docker.compose.project=$PROJECT" \
| xargs docker inspect -f '{{.Config.Image}} {{.Image}}' | sort -u \
| while read -r image id; do echo "$image $(date -d "$(docker image inspect -f '{{.Created}}' "$id")" +%s)"; done \
> "$OSBACKUP/images.txt"
for v in $VOLUMES; do
docker run --rm -v "$v":/data:ro -v "$OSBACKUP":/backup "$PGIMAGE" \
tar -I 'gzip -1' -cf "/backup/$v.tgz" -C /data .
done
printf '%s\n' $HOSTDIRS > "$OSBACKUP/hostdirs.txt"
for d in $HOSTDIRS; do
tar -I 'gzip -1' -cf "$OSBACKUP/host$(echo "$d" | tr / _).tgz" -C / "${d#/}"
done
for f in "$OSBACKUP"/*.tgz; do gzip -t "$f" 2>/dev/null || echo "DAMAGED: $f"; done
(cd "$OSBACKUP" && sha256sum *.tgz *.txt > SHA256SUMS)
wc -l < "$OSBACKUP/volumes.txt"
ls "$OSBACKUP"/*.tgz | grep -v '/host_' | wc -lCheck it worked. No WARNING or DAMAGED line appears, and the last two numbers are equal: one archive per volume. A DAMAGED line usually means the disk filled up: stop and send it to DataMind.
On the current server, as root
Stop the Installer and turn off automatic restart, so a reboot cannot bring the old DataMind OS back while the new VM runs. The command first saves each container's restart setting, so a rollback can put it back exactly; run again, it keeps the settings it saved the first time. Do not run docker compose down: the containers stay for a rollback.
cd "$INSTALL"
docker compose stop
if [ ! -s /root/restart-policies.txt ]; then
for p in "$PROJECT" "$PREFIX"; do docker ps -aq --filter "label=com.docker.compose.project=$p"; done \
| xargs docker inspect -f '{{.Name}} {{.HostConfig.RestartPolicy.Name}}' | sed 's#^/##' > /root/restart-policies.txt
fi
for p in "$PROJECT" "$PREFIX"; do docker ps -aq --filter "label=com.docker.compose.project=$p"; done \
| xargs docker update --restart=no > /dev/null
echo "saved: $(wc -l < /root/restart-policies.txt)"
echo "still running: $(docker ps --format '{{.Names}}' | grep -cxF -f <(cut -d' ' -f1 /root/restart-policies.txt))"Check it worked. saved: shows a number above 20, and still running: reads 0: no DataMind OS or Installer container runs.
Part B copies / over the VPN and proves on the new VM that every file arrived complete.
On the current server, as root
In the same session. If the copy stops, run the same command again: it continues where it stopped.
rsync -av --partial --progress /root/datamind-backup/ <user>@<new VM IP>:datamind-backup/
Check it worked. The command ends with a sent . summary and no error.
On the new VM, as the OS user
As the OS user that received the copy. Run tmux new -A -s migration on its own first, then:
DATE=<the date in the backup file names> # for example 2026-11-14 cd /home/<user>/datamind-backup && sha256sum -c "datamind-installer-$DATE.sha256" cd /home/<user>/datamind-backup/os-$DATE && sha256sum -c SHA256SUMS
Check it worked. Every line ends with OK. A line that ends with FAILED belongs to a file damaged on the way: run B1 again, then repeat B2.
On the new VM, as root
Only after B2 shows OK for every file. Run sudo -i on its own first, then:
mv /home/<user>/datamind-backup /root/datamind-backup chown -R root:root /root/datamind-backup chmod 700 /root/datamind-backup
Check it worked. No output means it worked.
Part C rebuilds the Installer and the DataMind OS data from the backup, in this order. The Installer's secrets go back before anything starts: a first start without them generates new keys, and the existing data could no longer be read.
On the new VM, as root
In the root session from B3:
INSTALL=<install directory> # the same install directory as on the current server BACKUP=/root/datamind-backup DATE=<the date in the backup file names> OSBACKUP="$BACKUP/os-$DATE" SECRETS_VOL=$(cat "$BACKUP/installer-secrets-volume.txt") declare -p INSTALL BACKUP DATE OSBACKUP SECRETS_VOL > /root/migration-vars.sh echo "$SECRETS_VOL"
If the session is lost later, run sudo -i, then source /.
Check it worked. The last line prints the Installer's secrets volume name, not an empty line.
On the new VM, as root
mkdir -p "$INSTALL" && cd "$INSTALL" tar xzf "$BACKUP/datamind-installer-files-$DATE.tgz" ls -a
Check it worked. The list shows docker-compose. and deployment.
On the new VM, as root
PGIMAGE=$(docker compose -f "$INSTALL/docker-compose.yml" config --images | grep postgres) echo "PGIMAGE=$PGIMAGE" >> /root/migration-vars.sh docker volume create "$SECRETS_VOL" docker run --rm -v "$SECRETS_VOL":/data -v "$BACKUP":/backup:ro "$PGIMAGE" \ tar xzf "/backup/datamind-installer-secrets-$DATE.tgz" -C /data docker run --rm -v "$SECRETS_VOL":/data:ro "$PGIMAGE" ls /data
Check it worked. The list shows curato., encr., jwt. and pg.. A later warning that this volume was not created by Docker Compose is harmless.
On the new VM, as root
PostgreSQL starts alone, the command waits up to three minutes until it is healthy, then loads the dump. Nothing is downloaded: the programs arrived before the window.
cd "$INSTALL"
docker compose up -d postgres
for i in $(seq 90); do [ "$(docker inspect -f '{{.State.Health.Status}}' "$(docker compose ps -q postgres)")" = healthy ] && break; sleep 2; done
docker inspect -f 'postgres {{.State.Health.Status}}' "$(docker compose ps -q postgres)"
docker exec -i "$(docker compose ps -q postgres)" sh -c \
'psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"' \
< "$BACKUP/datamind-installer-db-$DATE.sql"
echo "exit code: $?"Check it worked. One line reads postgres healthy, the last line reads exit code: 0, and no line starts with ERROR:. If one does, send the first ERROR: line to DataMind.
On the new VM, as root
cd "$INSTALL" docker compose up -d sleep 30 curl -sS http://localhost:8000/api/health; echo
Check it worked. The health check answers {"status":"ok","db":"up"}, and you can sign in at http:/ (over the VPN) with your existing Installer account. If the health check fails, wait a minute and run the curl line again.
On the new VM, as root
Every volume and host folder goes back before DataMind OS starts for the first time on this VM. A volume that already holds data is left alone and named, so a second attempt never lands on top of the first one.
for v in $(cat "$OSBACKUP/volumes.txt"); do
docker volume create "$v" >/dev/null
if docker run --rm -v "$v":/data "$PGIMAGE" sh -c '[ -z "$(ls -A /data)" ]'; then
docker run --rm -v "$v":/data -v "$OSBACKUP":/backup:ro "$PGIMAGE" \
tar xzf "/backup/$v.tgz" -C /data || echo "FAILED: $v"
else
echo "NOT EMPTY: $v"
fi
done
for d in $(cat "$OSBACKUP/hostdirs.txt"); do
tar xzf "$OSBACKUP/host$(echo "$d" | tr / _).tgz" --numeric-owner -C / || echo "FAILED: $d"
done
for v in $(cat "$OSBACKUP/volumes.txt"); do docker volume inspect "$v" >/dev/null 2>&1 || echo "MISSING: $v"; done; echo "volumes checked"Check it worked. The last line reads volumes checked, with no MISSING, FAILED or NOT EMPTY line before it. FAILED usually means the disk filled up. NOT EMPTY means an earlier attempt left data in that volume: stop and send the output to DataMind, and see R3 under If something goes wrong.
On the new VM, as root
The copied list holds the current server's addresses. DataMind OS rebuilds it with the new VM's addresses at the first start, but only when it is missing.
V=$(grep 'clickhouse_user_scripts$' "$OSBACKUP/volumes.txt") if [ -n "$V" ]; then docker run --rm -v "$V":/data "$PGIMAGE" rm -f /data/api_extract_deny.json docker run --rm -v "$V":/data:ro "$PGIMAGE" ls /data | grep -c api_extract_deny.json else echo "NO USER SCRIPTS VOLUME" fi
Check it worked. The last line prints 0. NO USER SCRIPTS VOLUME is expected only on older versions: send it to DataMind.
On the new VM, as root
The programs copied before the window must be the build the current server ran.
while read -r image created; do
c=$(docker image inspect -f '{{.Created}}' "$image" 2>/dev/null) && c=$(date -d "$c" +%s) || c=missing
echo "$image $c"
done < "$OSBACKUP/images.txt" | diff "$OSBACKUP/images.txt" - && echo "same build"Check it worked. The last line reads same build. Any other output: stop and send it to DataMind before you start DataMind OS.
In the Installer (web screen)
In the Installer, open Configuration, set these values and save. Saving writes them to deployment/ at once; the start in C10 puts them live.
If your Configuration has no NATIVE_API_DENY_HOSTS (older versions; step S2 printed no such line), skip that row: saving a setting your version does not have rejects the whole save.
| Setting | New value |
|---|---|
NATIVE_API_DENY_HOSTS | <new VM IP> <new VM public IP>, plus any other address hostname -I shows on the new VM. Remove the current server's addresses |
NATIVE_API_DENY_CIDRS | The new VM's network range, if the current server had its own range here |
PLATFORM_URL, SSL_CERT_PATH, SSL_KEY_PATH | Unchanged, unless DataMind told you otherwise |
Then check on the new VM:
grep -E '^(PLATFORM_URL|NATIVE_API_DENY_HOSTS|NATIVE_API_DENY_CIDRS)=' "$INSTALL/deployment/.env"
Check it worked. The output shows the new VM's addresses and none of the current server's.
In the Installer (web screen)
Click Start on Status in the Installer. Start uses the restored settings and sets the memory limits for this VM. Then check on the new VM:
cd "$INSTALL/deployment" && docker compose ps -a docker compose logs clickhouse-init 2>&1 | grep api_extract_deny
Check it worked. Every long-running service shows Up, with (healthy) where it has a health check, and the one-time init services show Exited (0). If the last line reads left untouched, send it to DataMind. Writing api_extract_deny. or no line at all (older versions) is correct.
The new VM is ready for users when the row counts match and every check below passes. Users are still on the old address, so a failed check costs nothing: see If something goes wrong.
On the new VM, as root
Run it right after the start, before any scheduled job adds rows.
cd "$INSTALL/deployment"
CH_PASS=$(grep '^CLICKHOUSE_PASS=' .env | cut -d= -f2-)
docker exec "$(docker compose ps -q clickhouse)" clickhouse-client --password "$CH_PASS" -q "SELECT database, table, sum(rows) FROM system.parts WHERE active AND database NOT IN ('system','INFORMATION_SCHEMA','information_schema') GROUP BY database, table ORDER BY database, table FORMAT TSV" > "$BACKUP/counts-after.tsv"
diff "$BACKUP/counts-before.tsv" "$BACKUP/counts-after.tsv" && echo "same counts"Check it worked. The last line reads same counts. Two log tables, log. and _system_parts_snapshot, grow by themselves after the start: more rows there is normal, fewer is not. Any other difference, or an error instead of the comparison: send the output to DataMind and wait for our answer before Part E.
On a tester's computer
To test with the real address before DNS changes, add the line <new VM IP> <your DataMind OS address> to the hosts file of the tester's computer, and remove it after the checks. If your users come over the internet, use <new VM public IP> in that line instead. Use a computer that reaches the new VM the same way your users will.
Check it worked. If one of them fails, send DataMind what you see.
On a tester's computer
From a computer outside your VPN, on the internet.
Linux:
for p in 22 4000 4222 5432 5433 6333 6334 6379 6901 7000 7070 8000 8001 8002 8081 8082 8123 8222 8443 8888 9000 9002 9003 9090 9428 9440; do nc -zv -w 3 <new VM public IP> $p; done
macOS:
for p in 22 4000 4222 5432 5433 6333 6334 6379 6901 7000 7070 8000 8001 8002 8081 8082 8123 8222 8443 8888 9000 9002 9003 9090 9428 9440; do nc -zv -G 3 -w 3 <new VM public IP> $p; done
Windows PowerShell:
foreach ($p in 22,4000,4222,5432,5433,6333,6334,6379,6901,7000,7070,8000,8001,8002,8081,8082,8123,8222,8443,8888,9000,9002,9003,9090,9428,9440) { $c = New-Object Net.Sockets.TcpClient; try { $open = $c.ConnectAsync('<new VM public IP>', $p).Wait(2000) } catch { $open = $false }; $c.Close(); "$p open: $open" }Check it worked. Every port is closed: timed out or refused on Linux and macOS, open: False on Windows. Port 22 must also be closed from the internet: SSH only through the VPN. An open port must be closed before Part E.
Once Part D passes, point users to the new VM and keep the current server as a fallback for a few weeks.
Your DNS team
This is the step that sends users to the new VM. Before your DNS team switches, tick the three points below.
Your DNS team points the DataMind OS address to the new VM. Check it from a user's computer:
nslookup <your DataMind OS address>
Check it worked. The answer shows the new VM's address: the address over the VPN if your users come through the VPN, or the public address if they come over the internet.
Your IT team
With the same address, the restored certificate keeps working until its expiry date. Open the address in a browser and look at the certificate before you tell users.
Check it worked. A user signs in at the usual address and sees their data. That is the end of tonight's work: Part F starts the next morning.
The first one to three days after the switch. These steps confirm what only shows with real use, and tidy up after the move.
In DataMind OS
The morning after the switch, check that the scheduled work ran on the new VM.
Check it worked. All three are true. If one is not, tell DataMind what you see.
On the new VM, as root
From now on the new VM needs its own backups. Set them up as Backup and restore describes, for the DataMind Installer and for DataMind OS data, and take the first one now.
Check it worked. A complete first backup sits on another machine.
On the new VM, as root
Once the backup from F2 is safe on another machine, delete the copy of the move's backup on the new VM: it takes as much room as all your data. The current server keeps its own copy until Part G. This cannot be undone.
rm -rf /root/datamind-backup && echo "backup copy removed" df -h /
Check it worked. The first line reads backup copy removed, and df shows the space back.
Your DNS team
Before the switch, your DNS team lowered the record's TTL to 5 minutes. Now it goes back to its usual value, often one hour or one day.
Check it worked. Your DNS team confirms the usual TTL, and the address still points to the new VM.
On the new VM, as root
Only if step S2 showed a certificate under /: Let's Encrypt certificates last 90 days, and the renewal that ran on the current server must now run on the new VM. Its settings came over with /; the renewal tool itself is installed here and tested once.
apt-get install -y certbot systemctl list-timers --all --no-pager | grep -i certbot; ls /etc/letsencrypt/renewal/ certbot renew --dry-run
Check it worked. A certbot timer is listed, the renewal folder holds a file for your address, and the test ends with Congratulations, all simulated renewals succeeded. Set up again any other scheduled job of your own that step S3 listed, as agreed with DataMind.
Send the output to DataMind.
The move is complete: welcome to your new server! Keep the current server switched off but untouched for two to four weeks after the switch, in case anything was missed. Then retire it with these steps.
Ask DataMind in your usual chat to confirm that nothing needs the current server any more: the certificate renewal and every scheduled job from step S3 run on the new VM. After the next steps there is no way back to it.
Check it worked. DataMind confirmed.
Your IT team
The current server still holds a key to the new VM (step S4) and network access to your data sources.
/ on the new VM: the line that ends with root@ and the current server's nameOn the new VM:
awk '{print $NF}' /home/<user>/.ssh/authorized_keysCheck it worked. The list no longer shows root@ with the current server's name.
Your IT team
Its disks still hold all your data and every password. Wipe them, or have your provider delete the server together with its disks. This cannot be undone.
Check it worked. Your IT team confirms the disks are gone.
Tell DataMind in your usual chat that the current server is gone. We update our records, and the move is finished.
Check it worked. DataMind confirmed.
Until DNS points to the new VM, going back to the current server loses nothing: its data is exactly as it was at the end of Part A.
After DNS has switched and users have worked on the new VM, going back would lose everything they changed since. Contact DataMind first: we plan that case with you.
To try the move again later, run R3 on the new VM first, then start again from step A1 with a new backup: users may have changed data in the meantime.
On the new VM, as root
source /root/migration-vars.sh cd "$INSTALL/deployment" && docker compose stop cd "$INSTALL" && docker compose stop
On the current server, as root
Put back the restart setting each container had before step A7, then start both again. The saved settings file is renamed afterwards, so a later attempt saves fresh ones:
source /root/migration-vars.sh while read -r name policy; do docker update --restart="$policy" "$name" > /dev/null; done < /root/restart-policies.txt mv /root/restart-policies.txt /root/restart-policies.restored.txt cd "$INSTALL" && docker compose start cd "$INSTALL/deployment" && docker compose start
Check it worked. Users sign in at the old address and see their data.
On the new VM, as root
Only after R1, and when DataMind agrees. It removes what the attempt restored on the new VM, so the next attempt does not land on top of it: the restored data, the Installer's database and the copy of the backup. The programs from S9 to S11 stay, the host folders from C6 are written over by the next attempt, and the current server is not touched.
source /root/migration-vars.sh cd "$INSTALL/deployment" && docker compose down -v for v in $(cat "$OSBACKUP/volumes.txt"); do docker volume rm "$v"; done cd "$INSTALL" && docker compose down -v docker volume rm "$SECRETS_VOL" rm -rf /root/datamind-backup && echo "new VM emptied"
Check it worked. The last line reads new VM emptied. A no such volume line is harmless. Then start again from step A1 on the current server.
It depends on the size of your data and the link between the two servers. Step S10 measures the copy speed, and with the data size from step S3, DataMind estimates the window with you before you agree on it. As a rough guide, every 100 GB of data takes about three hours over a 100 Mbit/s link, most of it copying. Copying the programs (steps S9 to S11) happens before the window, so it does not count. Part F takes a few minutes a day in the first days, and Part G comes two to four weeks later.
Yes, until DNS points users to the new VM. The current server is stopped but untouched, and its data stays exactly as it was at the end of Part A. Follow If something goes wrong. After users have worked on the new VM, going back would lose their changes: contact DataMind first.
A download brings the newest build, and the new VM must run the exact build the current server ran, so that the restored data matches it. Copying the images from the current server also means nothing is downloaded during the window.
A database copied while it is running can be inconsistent. Part A stops DataMind OS before the copy and keeps it stopped, so the backup is one consistent set.
Stop and send the output to DataMind before you continue. Each step checks what it did, so a problem is found where it happens and not hours later.
Your progress and your values are saved in this browser. Two things leave it:
Yes. The Installer and several DataMind OS services accept connections on every network address of the VM, and Ubuntu's ufw does not block ports that Docker opens. Close them in your provider's network firewall (step S8) and check the result in step D3.
Keep it switched off, but do not delete it, for two to four weeks: it is your way back. Then Part G retires it: DataMind confirms, its access is removed, and its disks are wiped, because they hold all your data and passwords.