Status: 🟢 Live — with load-bearing gaps. A background monitor in the
ops
container polls every hub, opens/resolves incidents, and alerts on the transitions.
That part works and is code-grounded below. But the monitor is not itself watched,
alerting depends on Resend being up, and there is no external dead-man's-switch. The
🟡 gaps at the bottom are the things that will actually bite you at 2am — read them.
Audience: internal GoBuild ops. Assumes SSH to the droplets and access to the
control plane atops.gobuild.ca. Nothing here is client-facing.
Related: Infrastructure & Ops Runbook · Troubleshooting & Common Failures
Ground-truth sources: controlplane/monitor.py, app/routers/health.py,
app/routers/admin_api.py, controlplane/hubclient.py, controlplane/main.py,
app/config.py, docker-compose.prod.yml.
The ops container runs a FastAPI app (the control plane). On startup its lifespan
launches an async loop, run_monitor_loop() (controlplane/monitor.py:220), which every
~180s calls run_checks() (controlplane/monitor.py:112). For each active
FleetInstance (controlplane/models.py:41) it makes signed Admin-API calls to the hub —
/_admin/health, /_admin/version, /_admin/vitals — and opens or resolves Incident
rows (controlplane/models.py:81) on state transitions only. Alerts fire on those
edges: email always, SMS only for critical (controlplane/monitor.py:93-107). The
/fleet and incident-feed pages are the human view of the same state.
ops container ──every ~180s──> hub /_admin/{health,version,vitals} (VPC-private, signed)
│
├─ transition detected ─> Incident row (open/resolved)
└─ alert on edge ───────> alert_email (always) + alert_sms_to (critical only)
All checks live in run_checks() (controlplane/monitor.py:112-205). Each cycle iterates
FleetInstance rows where status == "active" (controlplane/monitor.py:116).
| # | Check | Kind | Severity | Source |
|---|---|---|---|---|
| 1 | Hub reachable (/_admin/health, with one confirm-retry) |
server_down |
🔴 critical | monitor.py:120-127 |
| 2 | DB migration drift (up_to_date == false) |
migration_drift |
🟠 warning | monitor.py:130-140 |
| 3a | Disk ≥ crit % | disk |
🔴 critical | monitor.py:150-153 |
| 3b | Disk ≥ warn % | disk |
🟠 warning | monitor.py:154-157 |
| 3c | Email/SMS send-failure spike (last hour) | send_failed |
🟠 warning | monitor.py:160-169 |
| 4 | TLS cert expiry (opt-in, see gaps) | cert_expiry |
🔴/🟠 | monitor.py:171-184 |
| 5 | Recent provisioning failures (< 24h) | provision_failed |
🟠 warning | monitor.py:186-203 |
server_down)_reachable() (monitor.py:44-55) hits /_admin/health and, on a first failure, sleeps
_RETRY_SLEEP = 3.0s and retries once — so a transient blip doesn't page you. Only after
the second failure does it raise a critical server_down incident. Note the target is
the instance's internal base_url (e.g. http://hub:8000, hubclient.py:1-6) — this
is a VPC-private reachability check, not a public-edge check through Caddy/TLS.
On success it stamps inst.last_health_at (monitor.py:127) — that timestamp is what the
/fleet and dashboard pages show as "last seen" (controlplane/main.py:998).
migration_drift)Pulls /_admin/version and raises a warning when up_to_date is False
(monitor.py:134). up_to_date is computed hub-side in _migration_status()
(admin_api.py:115-139): the DB is on-head only when there is exactly one alembic head and
the applied revision equals it. Anything else — DB behind the code, or multiple heads — is
drift.
disk, send_failed)Pulls /_admin/vitals (admin_api.py:283-318), which reports:
disk_pct / disk_free_gb from shutil.disk_usage("/") — the droplet root disk.db_mb — Postgres database size (collected but not alerted on).sends_1h — a fleet-wide 1-hour window of out-bound email/SMS grouped into{sent, failed} (admin_api.py:308-317). The 1h window means a spike self-clears withinDisk thresholds are settings (config.py:53-54): warn ≥ 85%, crit ≥ 95%. Send-spike
logic (monitor.py:160-169): raise only when failed ≥ 5 and the failed fraction is
≥ 0.5 (_SEND_FAIL_MIN = 5, _SEND_FAIL_RATIO = 0.5, monitor.py:24-25) — so a couple
of bad addresses don't page you, a real outage does.
Cert checks (monitor.py:171-184) run against public hostnames listed in
monitor_cert_domains and warn at monitor_cert_warn_days (default 14), critical once
expired. This list is empty by default (config.py:55) → off unless configured
(see gaps). Provisioning failures (monitor.py:186-203) open a one-shot warning for jobs
that errored in the last 24h and auto-resolve when the job leaves the error state.
_alert() (controlplane/monitor.py:93-107) is the only egress:
alert_email is set (default gautam@gobuild.ca,config.py:50) it sends via Resend (send_email, monitor.py:97-99). A failed send isexcept … log.exception) so a broken alert never crashes the monitor — whichseverity == "critical" and alert_sms_to is setmonitor.py:102-105). alert_sms_to defaults to empty (config.py:51) → no SMS_clear, monitor.py:86-88) — you get an email✅ Resolved and nothing more.Alerts fire on edges only: _raise() no-ops if an open incident with that key already
exists (monitor.py:65-67), so you are not re-paged every 180s for the same fire.
Tunables (app/config.py:48-56):
| Setting | Default | Meaning |
|---|---|---|
monitor_enabled |
true |
master switch |
monitor_interval_seconds |
180 |
poll cadence (floored at 30, monitor.py:221) |
alert_email |
gautam@gobuild.ca |
always-on email recipient |
alert_sms_to |
"" (unset) |
E.164 SMS number for critical only |
monitor_disk_warn_pct / _crit_pct |
85 / 95 |
disk thresholds |
monitor_cert_domains |
"" (unset) |
public hostnames for cert checks |
monitor_cert_warn_days |
14 |
cert warn window |
Defined in app/routers/health.py — unauthenticated, deliberately low-signal:
GET /healthz (health.py:11-15) → {"status":"ok"}. Liveness only; intentionallyGET /readyz (health.py:18-22) → runs SELECT 1; confirms the DB is reachable.There is no /vitals on the public health router. Vitals live behind the signed
Admin API at /_admin/vitals (admin_api.py:283) and are only reachable by the control
plane. The public liveness/readiness pair is what an external uptime checker would poll —
but nothing external polls them today (see gaps).
Admin-API probes the monitor uses (all signed, VPC-private, controlplane/hubclient.py):
/_admin/health (hubclient.py:41), /_admin/version (hubclient.py:45),
/_admin/vitals (hubclient.py:55).
The /fleet page (controlplane/main.py:900-936) calls /_admin/version on every
instance and derives:
git_sha — baked into the image at build time (Dockerfile ARG GIT_SHA →APP_GIT_SHA, surfaced via /_admin/version, config.py:40-43).git_sha across reachable hubsmain.py:932-933) → the fleet is running mixed code.up_to_date is False → schema behind code.This is the human view; the monitor's migration_drift incident (check #2) is the paging
view of the same up_to_date signal.
From docker-compose.prod.yml: only the two Postgres services carry a real Docker
healthcheck — hub-db (pg_isready, compose :16-20) and chatwoot-db
(:136-140). The hub and ops containers have no healthcheck — they run
restart: unless-stopped, which restarts a crashed process but does nothing for a
process that is up-but-wedged. mem_limits set OOM ceilings (documenso 1g, chatwoot-rails
2g, sidekiq 1200m, etc.) but nothing watches memory approaching them.
These are the holes. Each is real and each is code-grounded.
🟡 The monitor watches everything except itself. run_monitor_loop() lives inside
the ops container (monitor.py:220). If ops is down, misconfigured, or the loop
dies, nobody polls and nobody tells you. The loop swallows per-cycle exceptions and
keeps going (monitor.py:227-233), which is good for resilience but means a silently
degraded monitor looks identical to a healthy one. There is no external heartbeat.
🟡 Email-down blind spot. Every alert emails via Resend, and a failed send is caught
and logged, not surfaced (monitor.py:100-101). If Resend is down, the monitor cannot
page you — and since alert_sms_to is empty by default (config.py:51), there is no
second channel. Ironically, "email sends failing" is itself one of the things the monitor
would try to email you about.
🟡 Cert-expiry is off by default. The code exists (_cert_days_left,
monitor.py:28-41; check #4) but monitor_cert_domains defaults to empty → skipped
(config.py:55, loop guard monitor.py:173). Until someone sets it, an expiring public
cert is invisible to the monitor. Note the reachability check (#1) hits the internal
URL, so it won't catch a public TLS failure either.
🟡 No OOM / memory trending. vitals reports disk and db_mb but not memory
(admin_api.py:295-306). mem_limits exist (docker-compose.prod.yml) — and the
chatwoot comments record real OOM kills that forced limit bumps (:178, :192) — yet
nothing watches a container creeping toward its ceiling. You find out when it's already
OOMKilled. (See Troubleshooting → OOM.)
🟡 Internal-only reachability. The server_down check probes http://hub:8000 over
the VPC. A failure at the public edge — Caddy down, TLS not issued, public DNS
wrong — does not trip it. Nothing synthetically exercises app.gobuild.ca from
outside.
🟡 No external uptime check. /healthz and /readyz exist and are perfect for an
external prober, but no UptimeRobot / healthchecks.io / Cronitor calls them. All
monitoring is inside-out.
🟡 Disk incident with no automated action. Disk warn/crit opens an incident and
emails, but there is no automated cleanup or runbook link fired with it — remediation is
entirely manual.
External dead-man's-switch (highest value). Have each run_checks() cycle ping an
external heartbeat service (healthchecks.io / Cronitor). If the pings stop — because
ops is down, the loop died, or Resend is down — the external service pages you.
This single addition closes gaps #1 and #2 at once. Wire it at the end of a successful
cycle in run_checks() (monitor.py:204) or the loop (monitor.py:229).
Set alert_sms_to. Give critical incidents a second channel independent of email
(config.py:51). Cheap; removes the single-channel risk in gap #2.
Populate monitor_cert_domains with the real public hostnames (app.gobuild.ca,
ops.gobuild.ca, etc.) so the already-written cert check (#4) actually runs
(config.py:55).
External uptime check on /healthz + /readyz through the public edge, to catch
Caddy/TLS/DNS failures the internal probe can't (gaps #5, #6).
Add memory to vitals and a warn/crit threshold pair, mirroring disk, so OOM is
caught before the kill (gap #4).
Attach a runbook link + a remediation hint to the disk incident so the page tells
the on-call what to do, not just that it's full (gap #7).
ops container. Restart it and monitoringdocker compose -f docker-compose.prod.yml up -d --build hub opshub and ops rebuilt — the ops image is the/fleet and the incident feedcontrolplane/main.py:975+); nav badge count via open_incident_count()monitor.py:208-215).alert_email (default gautam@gobuild.ca) on every incident;alert_sms_to (unset) on critical only. Confirm these are current — see openalert_email defaults to gautam@gobuild.caconfig.py:50) — is that a monitored, shared inbox or one person's mail? Should it be aalert_sms_to set in prod? It's empty by default (config.py:51), which meansops (and therefore themonitor_cert_domains populated? The cert check is written but off by default —server_down check is VPC-internalhttp://hub:8000). Is anyone verifying that app.gobuild.ca is actually reachable anddb_mb (already collected) and container memory be alerted on? Both are