Audience: internal ops. Every entry is symptom → diagnose (exact commands) → fix,
and is code-grounded with file:line. When something is on fire, jump to the matching
section and run the commands verbatim — they're safe (read-only) unless the Fix step says
otherwise.
First move, always. SSH to the droplet and get the lay of the land before you touch anything:
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.State}}'A container in
Restarting (n) ... agois crash-looping — thatnis the exit code.
Stack name iskeystone-gobuild, so containers arekeystone-gobuild-<service>-1.
The whole prod stack is defined in docker-compose.prod.yml. Caddy terminates TLS for
every public subdomain and reverse-proxies to the internal containers (Caddyfile:16-110);
the databases, OpenProject, MinIO, and the hub itself are not published to the host —
users only ever reach them through Caddy.
| Symptom | Likely cause | Jump to |
|---|---|---|
SSL_ERROR / "secure connection failed" on a subdomain right after a DNS move |
Caddy backed off after failing ACME against the old IP | Cert won't issue |
| Schedule create/edit returns HTTP 502 | OpenProject container down/unreachable | OpenProject 502 |
chat.gobuild.ca 502, container Restarting |
Chatwoot stale Puma PID / OOM crash-loop | Chatwoot crash-loop |
| No emails / signing links going out | DigitalOcean blocks outbound SMTP | Email not sending |
Container Restarting or Exited |
boot failure, OOM kill, or unhealthy dep | Container won't start |
Random container killed, OOMKilled in inspect |
mem_limit too tight |
OOM |
🔴 Real incident — the one that bit us. After repointing a domain's DNS to a new
droplet, https://<domain> showed "secure connection failed" / SSL_ERROR even though
the container was healthy.
SSL_ERROR_*, or an untrusted/wrong cert.Caddy tried to issue the cert before DNS pointed here (or while it still resolved to the
old IP). The ACME HTTP-01 challenge hit the old box, failed, and Caddy did what it's designed
to do: backed off into a long retry / staging fallback. It will not retry promptly on its
own — you're now stuck in the ~20-minute backoff window even though DNS is now correct.
For partner custom domains, add one more suspect: on-demand TLS only issues if the hub's
ask oracle approves the host (Caddyfile:4-7, 98-109). An unregistered domain never gets
a cert at all — that's a hub/registration problem, not a backoff.
# 1) Does DNS actually point at THIS droplet now? Compare to the box's public IP.
dig +short <domain>
curl -s ifconfig.me # this droplet's public IP — the two must match
# 2) What is Caddy actually serving / complaining about?
docker logs --tail=100 keystone-gobuild-caddy-1 | grep -iE 'acme|challenge|obtain|staging|rate|error'
# 3) Inspect the live cert + its issuer (staging issuer == still broken)
echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null \
| openssl x509 -noout -issuer -dates
dig must return this droplet's IP. If it doesn't, stop — fix DNS first, the certLet's Encrypt (R10/R11/E5…) = good, real cert. Issuer containingSTAGING / (STAGING) = Caddy fell back; you're in backoff.Once dig confirms DNS points here, kick Caddy — restarting drops the backoff timer and
it re-attempts ACME immediately, issuing a prod cert within seconds:
docker restart keystone-gobuild-caddy-1
# then re-verify the issuer flipped to a real Let's Encrypt cert:
echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null \
| openssl x509 -noout -issuer -dates
Client still sees the failure after you've confirmed a good prod cert? That's
client-side DNS cache — their machine/router/ISP resolver is still holding the old IP
(or a cached negative/cert error). It resolves on its own as TTL expires. To confirm it's
not us, resolve through a public resolver and hit us directly:dig +short <domain> @1.1.1.1 curl -svo /dev/null https://<domain> 2>&1 | grep -iE 'issuer|subject|SSL certificate verify'If those are clean, the platform is fine — have them flush DNS / try another network.
Related monitoring: the fleet monitor tracks cert expiry (not issuance) for opt-in
domains and raises a cert_expiry incident — critical if expired, warning inside
monitor_cert_warn_days (controlplane/monitor.py:28-41, 171-184).
Creating or editing a schedule item in the app returns HTTP 502 with a body like
{"detail":"OpenProject error: ..."}. The rest of the app works.
Schedule writes flow hub → OpenProject through the adapter, and OpenProject is the
authoritative planner. When the OpenProject container is down, mid-restart, or unreachable on
the internal network, push_schedule_item raises OpenProjectError, which the router
translates straight to a 502 — on both create and update:
app/routers/schedule.py:71-74app/routers/schedule.py:98-101Note the hub row is already committed (schedule.py:68-69) before the push; the 502 means
"saved locally but not mirrored to OpenProject." OpenProject is internal-only — it's
intentionally not exposed via Caddy (docker-compose.prod.yml:27-36, Caddyfile:15).
# Is the container up and settled? (OpenProject:15 is a heavy boot)
docker ps --filter name=openproject
docker logs --tail=120 keystone-gobuild-openproject-1
# Reach it from INSIDE the hub, exactly the way the adapter does (http://openproject:80)
docker exec keystone-gobuild-hub-1 \
python -c "import urllib.request as u; print(u.urlopen('http://openproject:80/health_checks/default', timeout=10).status)"
docker ps shows it Restarting/Exited, this is really a container-won't-start problem.Up but the in-container fetch hangs/errors, it's still booting or wedged.docker restart keystone-gobuild-openproject-1
# watch it come back to a steady state (can take a minute+ on this image):
docker logs -f keystone-gobuild-openproject-1
Once it's healthy, re-run the failed schedule save from the app — the hub row is fine; you're
just re-triggering the mirror to OpenProject.
🔴 Real incident, now self-healing in compose. chat.gobuild.ca returned 502 and
chatwoot-rails sat in a restart loop.
chat.gobuild.ca → 502 Bad Gateway (Caddy proxies to chatwoot-rails:3000,Caddyfile:42-45).docker ps shows keystone-gobuild-chatwoot-rails-1 as Restarting.A server is already running. Check /app/tmp/pids/server.pid.Two failure modes that feed each other:
tmp/pids/server.pidmem_limit was too tight; the kernel OOM-killed Rails, which isBoth are addressed in docker-compose.prod.yml:
docker-compose.prod.yml:154-157.mem_limit raised 1200m → 2g for rails (:178) and 900m → 1200m for sidekiq:192) after the OOM that started it.docker ps --filter name=chatwoot
docker logs --tail=80 keystone-gobuild-chatwoot-rails-1 | grep -iE 'already running|pid|error|killed'
# Was it OOM-killed? (true == kernel reaped it for memory)
docker inspect keystone-gobuild-chatwoot-rails-1 \
--format '{{.State.OOMKilled}} exit={{.State.ExitCode}} restarts={{.RestartCount}}'
Normally just restart — the self-healing boot command clears the PID for you:
docker restart keystone-gobuild-chatwoot-rails-1
docker logs -f keystone-gobuild-chatwoot-rails-1 # wait for "Puma ... Listening on 0.0.0.0:3000"
If it still loops, the compose fix isn't deployed on this box (old image/config). Recreate so
the current command: and mem_limit: take effect:
docker compose -f docker-compose.prod.yml up -d chatwoot-rails chatwoot-sidekiq
If OOMKilled is true and it keeps happening, this is an OOM
problem — bump the limit.
🔴 Real incident / permanent constraint. No emails, no signing links, no fleet alerts.
Resend request failed / SMTP connection timeouts on ports 25/465/587.DigitalOcean blocks outbound SMTP (25/465/587) on droplets — smtp-auth just times out.
So the platform never uses SMTP: everything sends over Resend's HTTPS API (port 443).
https://api.resend.com/emails (app/integrations/email/resend_client.py:65).docker-compose.prod.yml:88-92.send_email path (controlplane/monitor.py:97-101).If email is dead, the usual culprit is a missing/rotated RESEND_API_KEY, not the
network — send_email raises EmailError("RESEND_API_KEY is not configured.") when it's
unset (resend_client.py:43). Note the resolver prefers a tenant's own key and only then
falls back to the platform key (resend_client.py:38-39).
# 1) Is the platform key present in the hub's env?
docker exec keystone-gobuild-hub-1 printenv RESEND_API_KEY | sed 's/^\(re_....\).*/\1…(set)/'
# 2) Can the droplet reach Resend over 443 at all?
docker exec keystone-gobuild-hub-1 \
curl -s -o /dev/null -w '%{http_code}\n' https://api.resend.com
# 3) Hub logs for the real Resend error (status + body are logged)
docker logs --tail=100 keystone-gobuild-hub-1 | grep -iE 'resend|email'
200/401/403 from Resend = network is fine; read the error bodyResend <status>: ... from resend_client.py:69) — usually a bad/expired key or anfrom domain.RESEND_API_KEY in .env, then recreate the affecteddocker compose -f docker-compose.prod.yml up -d hub ops documenso
from domain unverified in Resend → verify gobuild.ca (SPF/DKIM) in the Resend dashboard.Covers any service stuck Restarting/Exited — the generic path when the specific sections
above don't match.
docker ps shows a container Restarting (n) or Exited (n); the subdomain it backs returns
502 (Caddy can't reach an upstream that isn't listening).
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.State}}'
# Read the crash reason — last lines before it died
docker logs --tail=120 keystone-gobuild-<service>-1
# Structured post-mortem: OOM? exit code? how many restarts?
docker inspect keystone-gobuild-<service>-1 \
--format 'oom={{.State.OOMKilled}} exit={{.State.ExitCode}} err={{.State.Error}} restarts={{.RestartCount}}'
Then narrow by exit code / pattern:
A server is already running → Chatwoot stale PID.oom=true → OOM.docker-compose.prod.yml:16-20, 136-140), and the hub /depends_on: condition: service_healthy:67-71, :179-183, :255-258, :108-112). Check the DB first:docker exec keystone-gobuild-hub-db-1 pg_isready -U keystone
docker logs --tail=50 keystone-gobuild-hub-db-1
SECRET_KEY_BASE, unmounted key) → the.env / the mounted secret and recreate.# Simple wedge (stale lock, transient dep flap):
docker restart keystone-gobuild-<service>-1
# Picked up new .env / compose changes — recreate the service:
docker compose -f docker-compose.prod.yml up -d <service>
Control-plane changes (
ops) reuse the hub image (docker-compose.prod.yml:203).
Rebuilding onlyopsships stale code — you must rebuild the hub image too:GIT_SHA=$(git rev-parse --short HEAD) docker compose -f docker-compose.prod.yml up -d --build hub ops
Is it actually up once it's running? The hub exposes cheap probes:
GET /healthz — liveness, process is up, returns {"status":"ok"} and nothing thatapp/routers/health.py:11-15).GET /readyz — readiness, proves the DB is reachable via SELECT 1app/routers/health.py:18-22); a 500 here means the hub is up but its DB isn't.docker exec keystone-gobuild-hub-1 curl -s localhost:8000/healthz
docker exec keystone-gobuild-hub-1 curl -s localhost:8000/readyz
A container is killed and restarted seemingly at random — often under load or during a
background job. docker inspect shows OOMKilled=true. For Chatwoot specifically it also
manifests as the stale-PID crash-loop.
Each memory-hungry service is capped with a mem_limit; when it exceeds the cap the kernel
OOM-kills it, not the app. Current caps in docker-compose.prod.yml:
| Service | mem_limit |
Ref |
|---|---|---|
| chatwoot-rails | 2g (was 1200m) | :178 |
| chatwoot-sidekiq | 1200m (was 900m) | :192 |
| documenso | 1g | :107 |
| minio | 512m | :124 |
| chatwoot-db | 600m | :141 |
| chatwoot-redis | 256m | :149 |
| wiki | 1g | :254 |
The chatwoot rails/sidekiq limits were raised after an OOM left a stale Puma PID that
crash-looped the container (:178 comment) — the canonical example of an OOM cascading into
an outage.
# Confirm it was memory, not a crash
docker inspect keystone-gobuild-<service>-1 --format 'OOMKilled={{.State.OOMKilled}} exit={{.State.ExitCode}}'
# Live memory vs limit across the stack — who's riding their cap?
docker stats --no-stream --format 'table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'
# Droplet-wide memory pressure (are we globally out of RAM, not just one cap?)
free -h
docker restart keystone-gobuild-<service>-1 to recover service now.mem_limit in docker-compose.prod.yml, then recreate:docker compose -f docker-compose.prod.yml up -d <service>
# was … style at:178/:192).free -h shows the whole droplet is out of RAM (sum of limits > box RAM), raising oneThe control-plane monitor (controlplane/monitor.py) polls every active hub on a loop
(default 180s, 25s warmup: :220-224) and only alerts on state transitions — it opens an
Incident on the failing edge and resolves it when the condition clears, so you don't get
re-paged every cycle (:65-88). Alert delivery: email always; SMS only for critical
(:93-107).
| Incident kind | Fires when | Severity | Ref |
|---|---|---|---|
server_down |
hub unreachable after a confirm-retry | 🔴 critical | :44-55, :121-125 |
migration_drift |
DB behind code / multiple heads (up_to_date=false) |
🟠 warning | :130-140, admin_api.py:136-138 |
disk |
disk ≥ warn/crit pct from /vitals |
🟠/🔴 | :147-159, admin_api.py:296-301 |
email / sms sends failing |
≥5 fails and ≥50% of sends failed in last hour | 🟠 warning | :24-25, :160-169, admin_api.py:308-317 |
cert_expiry |
opt-in domain cert expiring/expired | 🟠/🔴 | :171-184 |
provision_failed |
a provisioning job errored in last 24h | 🟠 warning | :186-203 |
A failed alert send is swallowed so it can never crash the monitor (
:100-101,:106-107).
Corollary: if email is down, you won't be paged that email is down. Watch the
email sends failingincident and the email section together.
Open-incident count feeds the ops nav badge (monitor.py:208-215).
# What's running / crash-looping
docker ps --format 'table {{.Names}}\t{{.Status}}'
# Logs (last N, or follow)
docker logs --tail=120 keystone-gobuild-<service>-1
docker logs -f keystone-gobuild-<service>-1
# Post-mortem (OOM? exit code? restarts?)
docker inspect keystone-gobuild-<service>-1 \
--format 'oom={{.State.OOMKilled}} exit={{.State.ExitCode}} restarts={{.RestartCount}}'
# DNS points here?
dig +short <domain>; curl -s ifconfig.me
# Live cert + issuer (staging == Caddy backed off)
echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null \
| openssl x509 -noout -issuer -dates
# Recover
docker restart keystone-gobuild-<service>-1 # kick a wedged container
docker compose -f docker-compose.prod.yml up -d <service> # pick up .env / compose changes
GIT_SHA=$(git rev-parse --short HEAD) \
docker compose -f docker-compose.prod.yml up -d --build hub ops # control-plane deploy