Status: 🟢 Live. This is how the production fleet (
app.gobuild.ca+ the control
plane onops.gobuild.ca) actually gets updated today. Every command below is
code-grounded inscripts/update.sh,deploy.py,docker-compose.prod.yml, and
Dockerfile. The rollout model is deliberately boring: SSH to one box at a time and run
one script. The clever bits (deploy.pyorchestration) are still 🟡 WIP — see Reality
checks.
Audience: internal GoBuild ops only. Assumes SSH access to the droplets and push
access to the private repo. Nothing here is client-facing. (House terms: a Job is a
builder's project; a Client is a builder's homeowner customer. Neither is what we
deploy — we deploy the hub that serves them. The "customer boxes" below are tenant
org droplets, not Clients.)
Author code on the Dev box → push to origin/main → SSH to each droplet and run
bash scripts/update.sh, canary (the app box) first, one box at a time. For a
control-plane change you must additionally rebuild with hub ops together — see
the gotcha.
Code lives in the private repo git@github.com:gobuildca/gobuild.git, default branch
main (evidence: CUSTOMER_DROPLET_OPS.md §5). The golden rule:
Author on the Dev box, deploy everywhere else. A live box only ever pulls — never
edit code on a box that serves traffic.
update.sh enforces this two ways so you can't get it wrong:
update.sh:43)..env, certs/, and custom_pages/ are git-ignored and never block it.git pull --ff-only, update.sh:61) and aborts if theupdate.sh:57). A deploy targetTypical flow:
# On the Dev box (GoBuild-Dev, *.dev.gobuild.ca) — the source of truth for latest code:
git add -A && git commit -m "feat(...): ..." # commit BEFORE you deploy (see GIT_SHA below)
git push # publish to origin/main
There is no formal PR gate today — main is the single line and the Dev box is where you
prove a change before rolling it. Whether that should change is an open question.
Commit before you deploy. The running code's git SHA is baked into the image and
surfaced at/_admin/version(see §4). If you deploy uncommitted
or unpushed code, the fleet dashboard's version column will not match reality.
bash scripts/update.sh
Run this on any droplet. Every box runs the identical script; the only thing that
differs per box is its git-ignored .env. It does the whole safe lifecycle and stops
the instant anything looks wrong (set -euo pipefail, update.sh:22):
| Step | What | Code |
|---|---|---|
| 1. Safety checks | Refuse if the tree has uncommitted code edits | update.sh:39-49 |
| 2. Pull | git fetch + git pull --ff-only origin $BRANCH |
update.sh:53-69 |
| 3. Build | docker compose -f docker-compose.prod.yml up -d --build (rebuilds the hub image) |
update.sh:71-74 |
| 4. Backup | pg_dump of the keystone DB into ./backups/ before touching the schema |
update.sh:76-82 |
| 5. Migrate | alembic upgrade head — the only irreversible step |
update.sh:84-87 |
| 6. Health | Poll /readyz up to 15× (app process + DB reachable) |
update.sh:89-101 |
If step 6 fails, the script prints the exact restore command and tells you not to
continue the rollout (update.sh:103-107). That is your rollback. If the
box is already at origin's HEAD, it no-ops and exits (update.sh:64-67).
deploy.py)deploy.py is a thin, honest wrapper around Docker Compose — v0, no remote
orchestration, DNS, or TLS (its own docstring, deploy.py:5-8). Useful for first boot and
targeted actions:
| Command | Does | Code |
|---|---|---|
python3 deploy.py up |
Build + bring up the whole prod stack behind Caddy/TLS (first boot from a fresh snapshot) | deploy.py:58-61 |
python3 deploy.py migrate-prod |
alembic upgrade head against live Postgres via run --rm hub |
deploy.py:64-67 |
python3 deploy.py logs [service] |
Tail prod logs (-f --tail 100), optionally one service |
deploy.py:70-74 |
python3 deploy.py status |
List provisioned tenant instances from clients/registry.json |
deploy.py:77-91 |
deploy.py uprunsup -d --buildon every service but does not passGIT_SHA
and does not back up the DB before migrating — that's whyupdate.shis the day-to-day
path. Reach fordeploy.pyfor first boot and diagnostics, not routine deploys.
# Rebuild just the hub (product code changed, no control-plane change):
GIT_SHA=$(git rev-parse --short HEAD) \
docker compose -f docker-compose.prod.yml up -d --build hub
Always pass GIT_SHA (next section explains why). For a control-plane change, read the
gotcha first — a bare --build ops is a no-op.
The ops service — the corporate control plane on ops.gobuild.ca — has no build
context of its own. It reuses the keystone-hub image the hub service builds, with a
different command: (a different uvicorn entrypoint). Same for showcase.
# docker-compose.prod.yml
hub:
build: { context: ., args: { GIT_SHA: ${GIT_SHA:-unknown} } }
image: keystone-hub:latest # line 44 — the ONLY service that builds
ops:
image: keystone-hub:latest # line 203 — no build:, reuses the hub image
command: ["uvicorn", "controlplane.main:app", ...]
showcase:
image: keystone-hub:latest # line 229 — same story
Therefore, to ship a controlplane/* (or showcase/*) change you MUST rebuild hub
too:
GIT_SHA=$(git rev-parse --short HEAD) \
docker compose -f docker-compose.prod.yml up -d --build hub ops
--build ops alone silently does NOTHING to the code. ops has no Dockerfile/context,keystone-hub:latest — yourhub (whichops off it, in the same command. Same rule forshowcase.ops runs on the CP droplet (GoBuild-CP), not the app box —hub ops rebuild on the CP box. On a combined/dev box both liveGIT_SHA is mandatoryDockerfile bakes the SHA as a build arg → env var, deliberately as the last layer so
it never busts the dependency/code cache:
ARG GIT_SHA=unknown
ENV APP_GIT_SHA=${GIT_SHA} # Dockerfile:24-25
That env feeds Settings.app_git_sha (app/config.py:43), which the hub surfaces at
/_admin/version (app/routers/admin_api.py:142-151). The control plane reads it via
controlplane/hubclient.py:45-47 to drive the fleet version / drift column.
Omit
GIT_SHAand the image stampsunknown— the fleet Version column shows
unknownand drift detection is blind.deploy.py uphas this bug (it doesn't pass the
arg);update.shstep 3 passes it through Compose env. When rebuilding by hand, always
prefixGIT_SHA=$(git rev-parse --short HEAD).
After every box, confirm three things: process up, DB reachable, version matches.
app/routers/health.py)| Endpoint | Meaning | Auth |
|---|---|---|
GET /healthz |
Liveness — process is up. Deliberately leaks nothing that fingerprints the deployment. | none (health.py:11) |
GET /readyz |
Readiness — DB reachable (SELECT 1). This is what update.sh polls. |
none (health.py:18) |
# On the box, from inside the hub container (how update.sh does it):
docker compose -f docker-compose.prod.yml exec -T hub \
python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8000/readyz',timeout=5).status)"
# 200 == live + DB reachable
GET /_admin/version)The authoritative "is this box on the code + schema I think it is" check. It's control-plane
only (RS256-signed), so you read it through the CP / fleet dashboard, not with a raw curl.
It returns (admin_api.py:142-151):
git_sha — the deployed code SHA (from APP_GIT_SHA). Compare it to the SHA youGIT_SHA).alembic_current / alembic_heads / up_to_date — schema drift. up_to_date: falseThe probe is wrapped so it never 500s the fleet board (admin_api.py:148).
Core truth: code is reversible in seconds (git checkout); a half-applied DB migration
is not. The database is the only scary part of a deploy (CUSTOMER_DROPLET_OPS.md §6).
Three rules:
update.sh does./backups/hub-<stamp>-pre-<sha>.sql.CUSTOMER_DROPLET_OPS.md §6.3).Per-box order is always: backup → pull → build → migrate → health check → next box.
At ≤10 tenant orgs, sequential-with-a-human-watching beats clever auto-rollback. Keep it
boring.deploy.pydoes not yet orchestrate this across droplets — fleet rollout is
"SSH to each box and runupdate.sh" (deploy.py:5-8).
Code rolls back in seconds; the schema is the risk. The pre-migration pg_dump from
step 4 is your seatbelt — restore it and you're back on the old code + old schema.
When a deploy fails the health check, update.sh prints exactly this (update.sh:103-107):
# 1. Restore the pre-migration dump (undo the schema change):
cat backups/hub-<stamp>-pre-<newsha>.sql | \
docker compose -f docker-compose.prod.yml exec -T hub-db psql -U keystone keystone
# 2. Return to the old code and rebuild:
git reset --hard <OLD_SHA> && \
docker compose -f docker-compose.prod.yml up -d --build
Then investigate before retrying, and do not roll out to more boxes.
⚠️ Scope of the seatbelt. The pre-migration dump covers only the
keystoneDB,
only at deploy time, only on the droplet's local disk (no offsite copy). It does
not coverdocumenso/wikiDBs, MinIO PDFs, Chatwoot, OpenProject, or the control-plane
cp_data. Rolling back a control-plane deploy meansgit reset+ the hub ops
rebuild; the CP DB itself is not dumped
byupdate.sh. See the Infra & Ops Runbook §7 for the full backup-gap flag.
Caddy is only touched for TLS/domain changes, not routine code deploys (routine deploys
don't recreate it). Two things bite:
.env at container CREATE, not on reload. Changing a *_DOMAIN var andcaddy reload does nothing. Force a recreate:docker compose -f docker-compose.prod.yml up -d --force-recreate caddy
AAAA records. New droplets are v4-only; a leftover IPv6 AAAAops.gobuild.ca before).caddy_data holds the Let's Encrypt certs + ACME account — don't wipe it casually or every
host re-issues (rate-limit risk). Full TLS model in the Infra & Ops Runbook §3.
Email note (not SMTP): DigitalOcean blocks all outbound SMTP (25/465/587). The
hub and Documenso both send via Resend's HTTPS API (docker-compose.prod.yml:88-93).
If mail "silently stops" after a deploy, suspect a missing/expiredRESEND_API_KEY, not
a firewall or SMTP issue — SMTP is expected to be dead.
Copy-paste. Runs on each droplet in turn, app/canary box first.
# --- On the Dev box: publish the change ---
git add -A && git commit -m "feat(...): ..." # commit first (feeds /_admin/version)
git push # -> origin/main
# --- On the CANARY (app) box: deploy + verify ---
ssh gobuild-app
cd /home/construction-proj # repo root (has docker-compose.prod.yml)
bash scripts/update.sh # refuse-if-dirty -> pull -> build -> BACKUP -> migrate -> /readyz
# update.sh already health-checks. Extra confidence:
docker compose -f docker-compose.prod.yml exec -T hub \
python -c "import urllib.request;print(urllib.request.urlopen('http://localhost:8000/readyz',timeout=5).status)"
# In the fleet dashboard: confirm git_sha == the SHA you pushed, and up_to_date == true.
# Click around ~2 minutes.
# --- Only if the canary is healthy: each tenant-org box, ONE AT A TIME ---
ssh gobuild-<org> && cd /home/construction-proj && bash scripts/update.sh
# -> if any box FAILS: restore its dump, keep old code, HALT the rollout, fix by hand.
controlplane/* or showcase/*)The ops/showcase services reuse the hub image, so a bare --build ops is a no-op.
# --- On the Dev box: publish ---
git add -A && git commit -m "feat(cp): ..." && git push
# --- On the CONTROL-PLANE box (GoBuild-CP, where `ops` runs) ---
ssh gobuild-cp
cd /home/construction-proj
git pull --ff-only origin main
# Rebuild hub (which builds keystone-hub:latest) AND recreate ops off it, TOGETHER,
# with the SHA stamped in. --build ops ALONE ships nothing.
GIT_SHA=$(git rev-parse --short HEAD) \
docker compose -f docker-compose.prod.yml up -d --build hub ops
# (add `showcase` to the service list if the showcase app changed: ... --build hub ops showcase)
# --- Verify ---
docker compose -f docker-compose.prod.yml logs -f --tail 100 ops # boots clean?
# Fleet dashboard / GET /_admin/version on the hub: git_sha == pushed SHA.
# Log into ops.gobuild.ca and confirm the change is actually live.
Fast dev-only iteration (ephemeral — lost on the next recreate):
docker cp controlplane/. keystone-gobuild-ops-1:/app/controlplane/ && docker restart keystone-gobuild-ops-1.
Always finish with a real--build hub opsbake so the shipped image matches git.
These are the honest gaps in this playbook — resolve them before the 2am call.
pg_dump → psql restore path inupdate.sh doesn't dump cp_data before a CPops deploy beyond git reset + rebuild?main require green tests (pytest) + a successful*.dev.gobuild.ca, DEMO_MODE=1) the canary, the/_admin/version column forgit_sha or up_to_date: false, or is drift only noticed whenGIT_SHA=unknown trap. deploy.py up and any hand --build without the prefixunknown. Should the Dockerfile/compose fail loudly (or the fleet board alert)git_sha: unknown, instead of silently blanking the column?CUSTOMER_DROPLET_OPS.md §5)ops app, the RS256 Admin-API contract.docs/CUSTOMER_DROPLET_OPS.md — the clone-a-tenant runbook and the safe-deploy rules (§5–§7).scripts/update.sh — the single deploy command, annotated inline.