Status: 🟢 Live — read the flags. The production fleet is up and serving real
traffic (app.gobuild.ca). The moving parts documented here are all wired, but several
operational safety nets are assumed, not verified — backups, monitoring, DR and
secrets management each carry a ⚠️ below. This page is the 2am reference; the open
questions at the bottom are the things to nail down before the 2am call happens.
Audience: internal GoBuild ops. This page assumes SSH access to the droplets and
the private repo. Nothing here should be shared with clients.
Ground-truth sources: docker-compose.prod.yml, Caddyfile, Dockerfile, deploy.py,
.env.example, controlplane/do_client.py, controlplane/provisioning.py, and the ops
memory notes (fleet topology, deploy gotcha, credential wall-off).
GoBuild runs as role-separated DigitalOcean droplets, all in the default-nyc1 VPC
(10.116.0.0/20). The model is one Control Plane (CP) droplet governing many hub
droplets over a signed Admin API. Live layout as of 2026-07-08:
| Droplet | Public IP | Private IP | Size | Role / hostnames |
|---|---|---|---|---|
| GoBuild-App | 167.99.13.48 | 10.116.0.4 | s-8vcpu-16gb (~$96/mo) | PROD hub + showcase: app, showcase, sign, chat.gobuild.ca. Runs the full stack except ops. |
| GoBuild-CP | 137.184.207.137 | 10.116.0.6 | s-1vcpu-2gb (~$12/mo) | Control plane — ops.gobuild.ca. Runs only ops + caddy. Holds no client data. |
| GoBuild-Web | 157.230.7.149 | 10.116.0.7 | s-1vcpu-1gb (~$6/mo) | Marketing only — gobuild.ca + www. Plain caddy:2 serving static /srv/marketing; proxies /api/* + /site-assets/* to app.gobuild.ca. |
| GoBuild-Dev | 161.35.119.173 | 10.116.0.5 | — | Dev/staging — *.dev.gobuild.ca. Authoring box / source of truth for latest code. DEMO_MODE=1. |
| GoBuild-CRM | 142.93.200.90 | 10.116.0.2 | ~$32/mo | A locally-hosted CRM the owner keeps. Not managed/monitored by us — leave it alone. |
In words: the CP droplet is the single god-mode panel (ops.gobuild.ca). Each hub
droplet runs the tenant app — one hub per droplet (pooled = many orgs on one hub; siloed =
one org per droplet). Hubs do not run their own god-mode panel; the CP monitors and
provisions them. CP ↔ hub is the RS256-signed /_admin/* contract
(controlplane/hubclient.py; each hub is a FleetInstance row with a base_url). The CP
signs Admin-API tokens with secrets/cp_private.pem; each hub verifies with the matching
secrets/cp_public.pem.
⚠️ Fragile —
/_adminis not yet VPC-locked. The design target is CP → hub over
the VPC private network (base_url=http://10.116.x:8000). Today the CP reaches the prod
hub over public HTTPS (https://app.gobuild.ca);/_admin/*is internet-reachable and
protected only by the RS256 signature — the hub's:8000is not published privately, and
there is no firewall/Caddy VPC-lock. VPC-locking/_adminis an open hardening item
(evidence: fleet-topology memory note; also flagged in the go-live security audit §4).
DNS is NOT on DigitalOcean — it's on GoDaddy.
do_client.pyhasupsert_dns_a()for
siloed auto-provisioning, but the live fleet's records are managed by hand in GoDaddy.
See §6 DNS.
Every hub droplet runs the stack in docker-compose.prod.yml behind Caddy (compose project
name keystone-gobuild). Caddy-in-Docker terminates TLS and reverse-proxies by hostname.
Only Caddy publishes host ports (80/443) — everything else is internal to the docker
network.
| Service | Image | Internal port | Subdomain | What it's for |
|---|---|---|---|---|
caddy |
caddy:2 |
80 / 443 (published) | — | TLS termination + reverse proxy for every host. On-demand TLS for partner domains. |
hub |
keystone-hub:latest (built here) |
8000 | app.gobuild.ca (+ marketing /api/*, /site-assets/*) |
The product — FastAPI construction-management SaaS. |
hub-db |
postgres:16 |
5432 | internal | Main Postgres. Holds the keystone, documenso, and wiki databases. |
redis |
redis:7 |
6379 | internal | Hub cache / Celery queue. |
openproject |
openproject/openproject:15 (pinned) |
80 | internal | Invisible scheduling backend. Hub calls http://openproject:80. Users never see it. |
documenso |
documenso/documenso:v2.11.0 |
3000 | sign.gobuild.ca |
E-signature backend. Signs with a local PKCS#12 cert; stores PDFs in MinIO. |
minio |
minio/minio:RELEASE.2025-09-07… |
9000 (API) / 9001 (console) | internal | Self-hosted S3. Object store Documenso uploads PDFs to (bucket documenso). |
chatwoot-db |
pgvector/pgvector:0.8.2-pg16 |
5432 | internal | Chatwoot's own Postgres (needs pgvector). Separate from hub-db. |
chatwoot-redis |
redis:7 |
6379 | internal | Chatwoot's own Redis (persists with --save 60 1). |
chatwoot-rails |
chatwoot/chatwoot:v4.14.1-ce |
3000 | chat.gobuild.ca |
Omnichannel messaging/support web app. |
chatwoot-sidekiq |
chatwoot/chatwoot:v4.14.1-ce |
— (worker) | — | Chatwoot background jobs. No port. |
ops |
keystone-hub:latest (reuses hub image) |
8001 | ops.gobuild.ca |
Control plane (atlas + fleet). Own login, own DB, holds no client data. Runs on GoBuild-CP only. |
showcase |
keystone-hub:latest (reuses hub image) |
8002 | showcase.gobuild.ca |
Prospect mock-site viewer + feedback. Public viewer, no login. |
wiki |
ghcr.io/requarks/wiki:2 |
3000 | wiki.gobuild.ca |
This knowledge base (Wiki.js). Content in the wiki DB on hub-db (no data volume). |
| (marketing) | static, served by Caddy | — | gobuild.ca, www |
Files from /srv/marketing. On the live fleet this lives on GoBuild-Web, not the app box. |
Notes & flags:
ops and showcase share the hub image — this is the deploy gotcha in §4. Different command: (uvicorn entrypoints), same built image.reports.gobuild.ca block (evidence: Caddyfile:30, docker-compose.prod.yml has no metabase service). ⚠️ Stale reference: docs/onboarding_README.md and CUSTOMER_DROPLET_OPS.md §4 still list MB_EMBEDDING_SECRET_KEY — ignore it unless Metabase is re-enabled.ops (--no ops) — the CP runs on its own droplet. So on a combined/dev box ops is present; on the split prod fleet it is not.Caddy terminates all TLS. The global Caddyfile block wires on-demand TLS to the hub:
{ on_demand_tls { ask http://hub:8000/internal/tls-allow } }
Two ways a cert gets issued:
app, sign, chat, ops, showcase, wiki, marketing) — Caddyhttps:// block issues ahub:8000/internal/tls-allow?domain=… andapp/routers/landing_portal.py:237) approves: a pooled{label}.gobuild.ca landing/folder site. Everything else → 404, so Let's EncryptThe Caddyfile comments say certs are issued "via the
*.gobuild.cawildcard" — that's
loose wording. There is no wildcard TLS cert; Caddy issues a separate per-host
cert via HTTP-01. The wildcard is a DNS*.gobuild.caA-record on GoDaddy (→
GoBuild-App), which is what makes arbitrary subdomains resolve to the box so per-host
issuance can happen.
/internal/* is blocked at the edge — the app host and partner blocks both
respond 404 on /internal/* so the cron trigger and TLS oracle aren't publicly
reachable. Caddy's own on-demand ask hits hub:8000 directly on the docker network,
unaffected (evidence: Caddyfile:22, security audit §3.5/§4).
The Caddyfile and compose are byte-identical on every droplet; all per-droplet
identity comes from the git-ignored .env via {$VAR:default}. Set these to reconfigure a
box — never edit the tracked files:
| Var | Default | Used by |
|---|---|---|
APP_DOMAIN |
app.gobuild.ca |
hub |
SIGN_DOMAIN |
sign.gobuild.ca |
documenso |
CHAT_DOMAIN |
chat.gobuild.ca |
chatwoot |
OPS_DOMAIN |
ops.gobuild.ca |
control plane |
SHOWCASE_DOMAIN |
showcase.gobuild.ca |
showcase |
WIKI_DOMAIN |
wiki.gobuild.ca |
wiki |
MARKETING_DOMAINS |
gobuild.ca, www.gobuild.ca |
Caddy static site |
Plus the derived full-URL vars used inside containers: SIGN_URL, CHAT_URL,
MARKETING_URL, DOCUMENSO_FROM_ADDRESS.
⚠️
.env.exampleis incomplete. It listsAPP/SIGN/CHAT/OPS/MARKETINGdomains but
omitsSHOWCASE_DOMAINandWIKI_DOMAIN(both are read byCaddyfile), and omits
the god-mode/credential vars (CP_*,TENANT_SECRET_KEYFernet key — seedocs/god-mode/DEPLOY.md).
Missing vars silently fall back to thegobuild.cadefaults, which is wrong on a customer
clone (their showcase/wiki would answer on the gobuild.ca hostnames). Add them to.env
explicitly per droplet.
⚠️ Caddy reads
.envat container CREATE, not reload. Changing a*_DOMAINvar and
runningcaddy reloaddoes nothing. You must recreate the container:
docker compose -f docker-compose.prod.yml up -d --force-recreate caddy.
Cutover recipe: change DNS first, then restart that box's Caddy (clears ACME
staging-fallback/backoff). Watch for staleAAAArecords pointing at an old box's IPv6 —
delete them (new droplets are v4-only).
bash scripts/update.sh
update.sh does the whole safe lifecycle and stops the moment anything looks wrong:
refuse-if-dirty → git pull --ff-only → rebuild hub image → back up the DB to backups/
before migrating → alembic upgrade head → health-check /readyz. On failure it prints
the exact restore command and tells you not to continue the rollout.
Deploy order: home/canary droplet FIRST, then customer boxes one at a time. If a
migration breaks the canary, no customer was touched. When one fails a health check:
restore its DB dump, keep it on the old code, halt the rollout (evidence:
CUSTOMER_DROPLET_OPS.md §6).
The ops service has no build context of its own — it reuses the keystone-hub image
built by the hub service (docker-compose.prod.yml:203, image: keystone-hub:latest, no
build:). Same for showcase.
# Deploy a control-plane change (controlplane/*.py or controlplane/templates/*):
GIT_SHA=$(git rev-parse --short HEAD) \
docker compose -f docker-compose.prod.yml up -d --build hub ops
--build ops alone does NOT rebuild — ops has no Dockerfile/context, so it silentlyhubkeystone-hub:latest) and recreate ops off it. Same rule applies toshowcase.GIT_SHA. The Dockerfile bakes it as a build arg → APP_GIT_SHA env →/_admin/version, which drives the fleet Version/drift column. Without it theunknown and the dashboard goes blank. Commit before deploying so the SHAFast dev iteration (ephemeral, lost on recreate):
docker cp controlplane/. keystone-gobuild-ops-1:/app/controlplane/ && docker restart keystone-gobuild-ops-1. Always finish with a real--build hub opsbake.
python3 deploy.py up — first boot from a fresh snapshot (build + up the whole prod stack).python3 deploy.py migrate-prod — alembic upgrade head against live Postgres (run --rm hub).python3 deploy.py logs [service] — tail prod logs.⚠️
deploy.pyis v0 — a thin, honest wrapper around Docker Compose. It does not
do remote droplet orchestration, DNS, or TLS (its own docstring says so,deploy.py:5-8).
Fleet-wide rollout is still "SSH to each box and runupdate.sh."
A client's published website can move from {label}.gobuild.ca onto their own domain. Two
paths exist:
A. Tenant self-serve (in the hub portal) — app/routers/website.py:
POST /portal/website/domain — owner enters theircompany.com → stored asLandingPage.custom_domain, domain_status="pending".POST /portal/website/domain/check resolves the domain: if it resolves toapp.gobuild.ca's IP → verifying, then fetches https://{domain}/__up; a 200 flips itlive. Caddy issues HTTPS automatically once /internal/tls-allow approves the host.B. God-mode / operator attach — the CP does it for a folder-site prospect
(commit e530ea5):
POST /_admin/orgs/{id}/site/domain sets/clears the folder-site's custom_domain.POST /_provision/domain (admin-authed) → hubclient.set_site_domain.The routing that makes an attached domain serve the right page already existed:
resolve_site_host / resolve_folder_host match the incoming Host header, and
/internal/tls-allow gates the cert (landing_portal.py:67, 97, 237).
The only thing the client must do is add the A record. On-demand TLS handles the rest —
no per-customer Caddy edits.
gobuild.ca DNS is hosted at GoDaddy (nameservers ns*.domaincontrol.com).
DigitalOcean is not the DNS host for the live fleet — changing an A record means logging
into GoDaddy, not the DO panel.
Live record layout:
| Record | Points at |
|---|---|
@ (apex) |
GoBuild-Web (157.230.7.149) |
www |
CNAME → apex |
app, showcase |
GoBuild-App (167.99.13.48) — explicit A |
ops |
GoBuild-CP (137.184.207.137) — explicit A |
* (wildcard) |
GoBuild-App (167.99.13.48) — catches sign, chat, and partner domains |
⚠️ Stale
AAAAgotcha. Old boxes had IPv6; new droplets are v4-only. If a cutover
half-works (resolves to the wrong box over IPv6), look for a leftoverAAAArecord and
delete it. This has bittenops.gobuild.cacutover before (fleet-topology memory note).
controlplane/do_client.py:upsert_dns_a()writes A records via the DO API for siloed
auto-provisioning (provisioning.py:_s_set_dns, domain fromTENANT_DNS_DOMAIN). That
path assumes DNS is on DO — which contradicts the live GoDaddy setup. Siloed
auto-provisioning of DNS is therefore untested against the real DNS host (evidence:
provisioning.py:119-126vs the GoDaddy reality). Manual GoDaddy edits are the current truth.
DigitalOcean also blocks outbound SMTP (ports 25/465/587) — see §8.
docker-compose.prod.yml:276-288)| Volume | Contents | Backed by |
|---|---|---|
hub_pgdata |
Main Postgres — keystone + documenso + wiki DBs. The crown jewels. |
hub-db |
hub_uploads |
Hub file uploads (/data/uploads). |
hub |
op_data |
OpenProject assets. | openproject |
op_pgdata |
OpenProject's internal Postgres. | openproject |
minio_data |
MinIO object store — Documenso's signed PDFs (bucket documenso). |
minio |
chatwoot_pgdata |
Chatwoot's Postgres (conversations, contacts). | chatwoot-db |
chatwoot_redis |
Chatwoot Redis (persisted). | chatwoot-redis |
chatwoot_data |
Chatwoot ActiveStorage (/app/storage, local). |
rails + sidekiq |
cp_data |
Control-plane DB (sqlite fallback when CP_DATABASE_URL unset) — platform admins, fleet instances, incidents, showcase feedback. |
ops, showcase |
caddy_data |
Caddy's Let's Encrypt certs + ACME account. Losing this re-issues all certs (rate-limit risk). | caddy |
caddy_config |
Caddy autosave config. | caddy |
Bind mounts (host-side, not volumes): ./custom_pages (operator-authored pages, live-edited),
./secrets/cp_public.pem (hub, RO), ./secrets/cp_private.pem (ops, RO),
./certs/documenso.p12 (documenso signing cert, RO), ./clients (ops registry, RO),
./showcase/content (mock HTML).
update.sh runs pg_dump of the keystone DB into backups/docker compose -f docker-compose.prod.yml exec hub-db \
pg_dump -U keystone keystone > backup-$(date +%F-%H%M).sql
⚠️ This is the biggest open gap in this runbook. The pre-migration dump only covers
thekeystoneDB, only at deploy time, and only to the droplet's own disk (no offsite
copy). There is no confirmed scheduled backup of:documenso/wikiDBs, MinIO PDFs,
Chatwoot, OpenProject, or the control-planecp_data.do_client.create_droplet()sets
backups: False(do_client.py:46), and "enable DO automated backups on every
droplet" is still an unchecked TODO (CUSTOMER_DROPLET_OPS.md §9). Do not assume
anything is recoverable after a droplet loss. See the open questions.
DigitalOcean blocks all outbound SMTP (25/465/587), so SMTP auth just times out. Both
the hub and Documenso send via Resend's HTTPS API (port 443) instead:
NEXT_PRIVATE_SMTP_TRANSPORT: resend + NEXT_PRIVATE_RESEND_API_KEYdocker-compose.prod.yml:88-93).RESEND_API_KEY / RESEND_FROM.If email "silently doesn't send," the first suspect is a missing/expired RESEND_API_KEY,
not an SMTP/firewall issue — SMTP is expected to be dead. Per the credential wall-off
model, pooled orgs get a shared GoLive Resend key by default (dressed per-org via
From: "<org name>" + owner reply_to); an org can BYO its own key/domain.
chatwoot-rails + chatwoot-sidekiq + chatwoot-db + chatwoot-redis) — chat.gobuild.catmp/pids/server.pid behind → Rails refuses to boot ("server already running") → 502command runs rm -f tmp/pids/server.piddocker-compose.prod.yml:157). If you still see it, check for an OOM.mem_limit was raised (rails 2g, sidekiq 1200m) precisely because the oldrails db:chatwoot_prepare. The AI-agents ERROR duringsign.gobuild.cadocumenso DB) + the signing cert./certs/documenso.p12) + Resend. If document-create fails, check MinIO is up and thedocumenso bucket exists; if signing fails, check the .p12 mount + DOCUMENSO_SIGNING_PASSPHRASE.mem_limit: 1g.documenso, path-style). Data inminio_data. Console isn't proxied by Caddy — reach it by SSH port-forward if you need it.:15 — never latest (API/webhook shapes shift between releases). Hub talkshttp://openproject:80. Users never see it; it's not in the Caddyfile.op_pgdata before any version bump; re-validate the webhook consumer after.redis:7, no persistence configured (ephemeral is fine for the hub).| Item | Evidence | Note |
|---|---|---|
/_admin reachable over public HTTPS, not VPC-locked |
fleet-topology memory; security audit §4 | Signature-gated but internet-facing. Highest-value target. Hardening TODO. |
| No confirmed scheduled/offsite backups | do_client.py:46 (backups: False); CUSTOMER_DROPLET_OPS.md §9 TODO |
Only a pre-migration keystone dump to local disk. |
| Siloed DNS auto-provision assumes DO DNS | provisioning.py:119-126 vs GoDaddy reality |
Live DNS is GoDaddy; the DO upsert_dns_a path is untested against real DNS. |
deploy.py is v0 — no remote orchestration |
deploy.py:5-8 |
Fleet rollout = SSH + update.sh per box. |
| Siloed cloud-init is a placeholder | provisioning.py:73-82 (_cloud_init body is commented) |
Siloed droplet auto-bootstrap not wired end-to-end; pooled works. |
.env.example missing SHOWCASE_DOMAIN/WIKI_DOMAIN/CP_*/TENANT_SECRET_KEY |
.env.example; Caddyfile:56,64; god-mode/DEPLOY.md |
Silent fallback to gobuild.ca hostnames on clones. |
| Metabase retired but still referenced | Caddyfile:30; onboarding_README MB_EMBEDDING_SECRET_KEY |
Dead references; native Reports replaced it. |
god-mode/DEPLOY.md:88 claims tls-allow "only approves partner domains" |
vs landing_portal.py:248 (approves tenant subdomains) |
Doc is stale; the oracle was extended. |
Containers run as root (no USER) |
Dockerfile (no USER directive); security audit §4 |
Widens blast radius of any file-write bug. |
| No dependency lockfile | security audit §3.7 | Non-reproducible builds; wildcards floated. |
| chatwoot-rails restart loop seen on dev | god-mode/DEPLOY.md:94 |
Pre-existing on the dev box; check memory if messaging flaps. |
FleetInstance still named "GoBuild pooled hub" though it points at the real hub |
fleet-topology note | Cosmetic; rename when convenient. |
docs/CUSTOMER_DROPLET_OPS.md — clone-a-customer runbook, safe-deploy rules (§5/§6/§7).docs/god-mode/DEPLOY.md — stand up the control plane (CP keypair, Fernet key, first admin/MFA).docs/onboarding_README.md — secret-rotation buckets when cloning.docs/security_audit_go_live.md — go-live security findings + remediation status.docs/ARCHITECTURE.md §6 — deployment topology & fleet-management design.