Operations Runbook

This document is based on the Enterprise New offline application.

Daily Inspection

1. Run the following commands from the deployment package root directory:

   ./bin/verify.sh --site config/site.conf
   ./bin/smoke-business-emqx.sh --site config/site.conf

2. Confirm the following items:
   • The 3 Swarm nodes are Ready.
   • APP VIP /healthz and /readyz return 200.
   • DB VIP:5432 is accessible.
   • The PostgreSQL primary, standby, and witness topology is normal.
   • backend, redis, emqx, sourceflow, eventflow, marimo, and opcua-server are all running on the active application node.

Application Checks

1. Run the following commands on the active application node:

   cd /opt/tier0-enterprise/tier0-deploy
   bash bin/compose.sh ps
   bash bin/compose.sh logs --tail=200 backend
   curl -fsS http://127.0.0.1:8088/healthz
   curl -fsS http://127.0.0.1:8088/readyz

   Note

   Do not bypass bin/compose.sh and run raw docker compose commands directly. The wrapper script loads the runtime environment generated by the installer.

2. Health-check interpretation:

   • healthz failure: check the backend process, image, and container logs first.
   • readyz failure: also check DB VIP, Redis, and database migration logs.
   • Home page reachable but login unavailable: confirm ADMIN_INITIAL_PASSWORD in .env, then check backend IAM migration logs and user tables. Do not judge deployment success only from the HTTP port.

Protocol Entry Checks

nc -vz <APP_VIP> 1883
nc -vz <APP_VIP> 8883
nc -vz <APP_VIP> 8083
nc -vz <APP_VIP> 8084
nc -vz <APP_VIP> 4840

Note

EMQX and OPC UA Server are fixed on the active application node. APP VIP failover only guarantees the APP VIP:80 HTTP entry. Protocol ports are not routed through HAProxy TCP proxying, so this design does not promise high availability for protocol entries.

Database Checks

Run the following on database nodes:

systemctl status tier0-postgresql
systemctl status tier0-repmgrd
sudo -iu postgres psql -Atqc 'select pg_is_in_recovery();'
sudo -iu postgres /usr/local/bin/repmgr -f /etc/repmgr.conf cluster show

• pg_is_in_recovery=false means primary.
• pg_is_in_recovery=true means standby.
• DB VIP must only be located on the current primary.
• If standby replication is interrupted or the witness reports upstream errors, preserve the site first. Do not rerun the full deployment directly.

APP VIP Failover

1. Before testing, record the node that currently owns the VIP and verify:

   curl -fsS http://<APP_VIP>/healthz
   curl -fsS http://<APP_VIP>/readyz

2. After stopping keepalived on the current VIP node, confirm VIP takeover on another application node, then retest both URLs. The HAProxy backend points to the active application node on 8088/TCP, so all three application nodes must be able to reach port 8088 on the active node.

3. After restoring keepalived, run verify.sh and smoke checks again. Do not stop active application containers during the drill.

DB Primary Failure Switchover

Before a failure drill, create a backup and confirm replication is healthy:

./bin/backup-db.sh --site config/site.conf

After stopping tier0-postgresql on the current primary, observe repmgr promoting the standby and DB VIP takeover. The target RTO is within 2 minutes. Historical tests on the same architecture were about 59-64 seconds.

Caution

• The old primary must not be started directly back into the production topology. Stop PostgreSQL, repmgrd, and keepalived first; remove any remaining DB VIP; then run basebackup from the current new primary, register the node as a standby, and finally reconnect the witness.
• After failover, do not rerun the full installer directly. Static primary/standby definitions in the inventory may disturb the current topology.

Backup and Restore

• Backup:

  ./bin/backup-db.sh --site config/site.conf

• Restore:

  ./bin/restore-db.sh --site config/site.conf --restore-file /backup/tier0-db/postgres-YYYYmmddTHHMMSS.dump

Note

Before restoration, confirm the current primary. After restoration, run verify.sh, smoke checks, and actual platform login verification for the default-user path.

Cleanup and Retest

./full-cleanup.sh --env install.env --yes
./install.sh --auto --yes

Caution

Full cleanup deletes Tier0 runtime data, Docker/Swarm data, and database backups on the 6 nodes. It also performs final cleanup on the control node after the control container exits. Use this entry only for test environments that are confirmed disposable. If customer data or an incident site must be preserved, use normal --cleanup and complete a backup first.

Common Issues

Backend Startup Failure

cd /opt/tier0-enterprise/tier0-deploy
bash bin/compose.sh logs --tail=300 backend
bash bin/compose.sh ps

Note

Focus on DB URL, DB VIP, Redis, migration, and mounted directory permissions.

EMQX Startup Failure

cd /opt/tier0-enterprise/tier0-deploy
bash bin/compose.sh logs --tail=200 emqx

Note

Check certificates, configuration files, volume permissions, and whether ports 1883/8883/8083/8084 are occupied.

repmgrd Repeated Restarts

systemctl status tier0-repmgrd
tail -100 /var/log/repmgr/repmgr.log
sudo -iu postgres psql -Atqc 'show shared_preload_libraries;'

Note

Confirm that shared_preload_libraries includes repmgr, and that the systemd unit uses repmgrd --daemonize=false.

Material Verification Failure

./bin/verify-materials.sh

Caution

Do not skip SHA256 verification and continue deployment. Re-acquire the delivery package or rebuild the materials following the traceable process.