Status: 🟢 Pooled path Live & fully automated · 🟡 Siloed path partly wired · ⚪ Siloed droplet bootstrap is DORMANT (placeholder) · Audience: internal ops only. This runs through the Control Plane (CP), which can read and write every Client's data. Not Client-facing.
What this covers: standing up a new Client — either as a lightweight org on the shared "pooled" hub, or as a dedicated "siloed" droplet — from the CP's /provision screen. Where the code stops and hands off to a human, this page says so with file:line evidence.
One-line mental model. One Control Plane droplet (ops.gobuild.ca) governs many hubs over a VPC-private, signed Admin API. "Provisioning" means: get the Client an org record on a hub, and — for siloed — first stand up the hub itself. Everything the CP does to a hub goes through the hub's /_admin/* surface, authenticated by a 60-second RS256 token (controlplane/hubclient.py, controlplane/security.py:84-95).
Status: 🟢 Live decision (a per-job kind)
This is the only real decision. It's stored as the ProvisioningJob.kind and picks which step list runs (controlplane/provisioning.py:24-26).
🟢 Pooled (kind="pooled") |
🟡 Siloed (kind="siloed") |
|
|---|---|---|
| What the Client gets | An Organization on the shared hub, reached at {slug}.app.gobuild.ca |
Its own DigitalOcean droplet running its own hub |
| Automation | Fully automated, runs inline — done in seconds | Not end-to-end today (see §4) — droplet boot is a placeholder |
Needs DIGITALOCEAN_TOKEN |
No — never touches DO (do_client.py:5-6) |
Yes — every step fails loudly without it (provisioning.py:88-89) |
| Steps | validate → create_org → activate (provisioning.py:24) |
validate → create_droplet → await_active → set_dns → register_instance → seed_org → activate (provisioning.py:25-26) |
| Credentials model | BYO per-tenant keys; no global fallback for Stripe/Twilio/BuildData | Legacy single-tenant fallback allowed (siloed orgs can read .env) |
| Use when | Almost everyone. Default. | Only if a Client genuinely needs their own private server |
The
/provisionand org-detail UIs already frame this correctly for operators: "Shared is right for almost everyone. Pick Dedicated only if they need their own private server." (controlplane/templates/provision.html:32,org_detail.html:37).
Status: 🟢 Live — fully automated, no infra, no manual steps.
POOLED_STEPS = ["validate", "create_org", "activate"] (provisioning.py:24). The job runs inline in the request handler (not on a background worker), so the operator gets the result on the same page, with a manual Retry button if a step errors.
_p_validate, provisioning.py:44-59) — require name / slug / owner_email; pick the target FleetInstance. If the job names an instance_id use it, else pick the oldest pooled hub. No pooled hub registered → hard error "no pooled hub is registered to place this org on"._p_create_org, provisioning.py:62-68) — POST /_admin/orgs on the shared hub via hubclient.create_org (hubclient.py:83-84). The hub creates the Organization + owner, seeds day-one cost-codes/templates, and returns the org + a temp password / magic-link invite._activate, provisioning.py:150-152) — mark the context active; the runner flips job.state = "done".That's it — the Client is live at {slug}.app.gobuild.ca (tenant resolution is host→slug→org, hub-side). Nothing else to do.
Status: 🟡 Steps exist and run, but the droplet they create never actually boots a hub (§4). Treat as a scaffold, not a button you can trust today.
SILOED_STEPS = ["validate", "create_droplet", "await_active", "set_dns", "register_instance", "seed_org", "activate"] (provisioning.py:25-26). Each step is idempotent — it no-ops when its ctx marker is already set — so a failed job resumes from the failed step on Retry without redoing work (provisioning.py:1-13, runner at provisioning.py:167-190).
_s_validate, provisioning.py:85-90) — require name/slug/owner_email; assert do_client.is_configured() (i.e. DIGITALOCEAN_TOKEN set), else "DIGITALOCEAN_TOKEN not set — siloed provisioning is unavailable"._s_create_droplet, provisioning.py:93-104) — POST /droplets (do_client.create_droplet, do_client.py:43-49):
gobuild-{slug}DO_REGION (default tor1 — ⚠️ see §5 region bug)DO_SIZE (default s-2vcpu-4gb)DO_IMAGE (default docker-20-04)user_data = the cloud-init script from _cloud_init — which is a placeholder (§4).["gobuild", "tenant"]. Stores ctx["droplet_id"]._s_await_active, provisioning.py:107-116) — get_droplet + droplet_ip; if not active with a public IPv4 yet, raise a retriable error "droplet is still booting — retry to continue". Re-running the job resumes here. This is deliberate — there is no internal wait/poll loop; the operator (or a caller) drives the retry._s_set_dns, provisioning.py:119-126) — upsert_dns_a(domain=TENANT_DNS_DOMAIN (default gobuild.ca), name="{slug}.app", ip=…). ⚠️ This targets the DigitalOcean DNS API, but the real DNS host is GoDaddy — see §3._s_register, provisioning.py:129-138) — insert a siloed FleetInstance with base_url = http://{ip}:8000. From now the CP considers this a hub and the monitor will start polling it._s_seed_org, provisioning.py:141-147) — POST /_admin/orgs on the new hub's Admin API to create the org + owner. This is where a placeholder droplet fails: there is no hub listening on :8000, so the Admin-API call errors._activate) — mark done.Status: ❌ Auto-DNS is a dead-end against the live setup · 🟢 manual GoDaddy edits are the real path
The code writes DNS via the DigitalOcean API:
# do_client.py:67-71 → POST /domains/{domain}/records (A record)
upsert_dns_a(domain="gobuild.ca", name="{slug}.app", ip=<droplet ip>)
But gobuild.ca DNS is hosted at GoDaddy, not DigitalOcean (nameservers ns*.domaincontrol.com — see infra/ops-runbook §6). DigitalOcean is not authoritative for the live fleet's zone, so do_client.upsert_dns_a writes into a DO-hosted zone that nothing resolves against. The siloed set_dns step is therefore untested against the real DNS host and, as written, does not create a record anyone can reach.
Real-world DNS for a new Client today is manual, in GoDaddy:
{slug}.app.gobuild.ca (or the chosen subdomain) → the droplet's public IP.*.gobuild.ca → GoBuild-App already catches sign, chat, and partner hosts, so pooled subdomains usually need no new record.AAAA gotcha — if a host half-resolves to the wrong box over IPv6, delete a leftover AAAA record (has bitten ops.gobuild.ca before).Second idempotency problem, even inside DO.
upsert_dns_ais misnamed — it is not idempotent: it just POSTs a new A record and the docstring says so plainly: "idempotency is the caller's job — match on an existing record before calling, or accept a duplicate then clean up" (do_client.py:67-69). The_s_set_dnsstep guards only on its ownctx["dns_done"]marker, so a fresh job for the same slug creates a duplicate A record (provisioning.py:119-126).
Status: ⚪ DORMANT — real bootstrap is commented out.
_cloud_init (provisioning.py:73-82) is an explicit placeholder. It writes one log line and leaves the actual work commented out:
# provisioning.py:73-82 (user_data placeholder)
echo 'gobuild tenant {slug} bootstrapping' >/var/log/gobuild-provision.log
# cd /opt/gobuild && git pull && docker compose -f docker-compose.prod.yml up -d --build
# docker compose exec -T hub alembic upgrade head
The docstring says the concrete script "lives with the droplet image (see docs)." That golden image does not exist / is out of band. The consequences chain:
docker-20-04 image with no GoBuild stack on it.register_instance (step 5) points a FleetInstance at http://{ip}:8000, where nothing listens.seed_org (step 6) calls that dead Admin API and fails. So an end-to-end siloed provision cannot currently complete from /provision alone.What a real siloed hub needs on the box (this is the missing golden-image content):
docker-compose.prod.yml stack (hub + hub-db + redis + Caddy + the embedded backends) — see CUSTOMER_DROPLET_OPS §2 (repo docs/CUSTOMER_DROPLET_OPS.md);.env (§6 below);cp_public.pem installed so the hub trusts the CP (§5).Until that image + bootstrap exists, the working way to birth a dedicated Client is the manual clone-per-Client runbook in CUSTOMER_DROPLET_OPS §7 (repo docs/CUSTOMER_DROPLET_OPS.md) (snapshot the home droplet → new droplet → reconfigure .env → scripts/update.sh → wipe demo data → hand over). The state machine is the intended future replacement for that runbook, not yet a substitute.
cp_public.pem (hub↔CP trust)Status: 🟢 Live for public hubs · plain HTTP on the CP↔hub hop by design
app.gobuild.ca, sign., chat., and per-Client subdomains) is fronted by Caddy, which issues Let's Encrypt certs automatically (on_demand_tls asks the hub whether a host is allowed). A siloed droplet gets its TLS from its own Caddy once the real stack is on it — another thing the placeholder cloud-init doesn't set up.base_url (http://{ip}:8000 or http://hub:8000) over the VPC-private network as plain HTTP — TLS is intentionally not used on this hop (hubclient.py:3-6; control-plane page notes the same). Security comes from the signed token, not transport TLS.tor1 vs the fleet's nyc1 VPCStatus: 🟡 Latent bug — set DO_REGION=nyc1 when provisioning siloed.
tor1 (provisioning.py:99)./servers provisioner defaults to nyc1, with the comment "nyc1 auto-joins the default-nyc1 VPC where the GoBuild fleet lives" (controlplane/main.py:1279).A siloed droplet born in tor1 lands outside the nyc1 VPC, so the CP's private http://{ip}:8000 Admin-API path can't reach it privately. Mitigation until fixed: set DO_REGION=nyc1 in the ops container env before running a siloed job.
cp_public.pem so the hub trusts the CPStatus: 🟢 Live mechanism · ⚪ not automated onto siloed droplets
Every hub must carry the CP's public key or its entire /_admin surface returns 403 (fail-closed — app/routers/admin_api.py). The keypair is generated once by scripts/cp_keygen.py:
cp_private.pem → mounted into the ops container only, signs tokens (CP_SIGNING_KEY_PATH, prod path /run/cp/cp_private.pem).cp_public.pem → mounted into every hub, verifies tokens (cp_public_key_path / CP_PUBLIC_KEY_PATH, prod path /run/cp/cp_public.pem).The token contract (iss=keystone-control-plane, aud=keystone-hub, RS256, 60s TTL) is duplicated in controlplane/security.py:49-51 and app/routers/admin_api.py — they must stay in lockstep.
For a new siloed hub,
cp_public.pemmust be placed on the droplet and mounted into its hub container — otherwiseseed_org(and all later god-mode) 403s. Nothing in the provisioning code installs it; it belongs in the (missing) golden image / cloud-init. This is a second reason siloed isn't end-to-end today.
.env per droplet — what must be uniqueStatus: 🟢 Live discipline (manual clone path) · ⚪ not templated by the state machine
Code is byte-identical on every droplet; secrets and identity live only in the git-ignored .env (CUSTOMER_DROPLET_OPS §3–§4 (repo docs/CUSTOMER_DROPLET_OPS.md), onboarding_README (repo docs/onboarding_README.md)). For any dedicated Client box you (today, by hand) must:
POSTGRES_PASSWORD, SECRET_KEY, CP_SECRET/CP_PASSWORD, the OpenProject/Documenso/Chatwoot/MinIO secrets, ALERTS_CRON_TOKEN, etc. openssl rand -hex 32 per value. Sharing them across Clients is a security hole.CLIENT_SLUG, APP_DOMAIN/SIGN_DOMAIN/CHAT_DOMAIN/OPS_DOMAIN, the matching *_URLs.DEMO_MODE=0 ⚠️ — it's 1 only on the home/demo box (the pitch deck). Never on a Client box.GEMINI/DEEPSEEK/DEEPGRAM) stay GoBuild's; email/SMS start as GoBuild's; verify SPACES_* is per-Client, not a shared bucket.
scripts/provision_client.pyis referenced as the "from-scratch (non-clone) provisioner that renders a fresh per-Client.env+ compose and registers the instance" (onboarding_READMERelated). That is a separate, script-driven path from the CP state machine — worth reconciling (open question).
| Concern | Pooled | Siloed |
|---|---|---|
| Create org + owner on a hub | 🟢 Automated (_admin/orgs) |
🟡 Coded but fails against a placeholder droplet |
| Create the droplet | — | 🟢 Automated (do_client.create_droplet) |
| Boot the hub stack on the droplet | — | ⚪ Placeholder — commented out (provisioning.py:73-82) |
Install cp_public.pem (CP trust) |
🟢 Pre-mounted on shared hub | ⚪ Not automated — golden-image job |
| DNS record | 🟢 Wildcard already covers subdomains | ❌ Code targets DO API; real host is GoDaddy → manual |
| Region / VPC | — | 🟡 Defaults tor1, should be nyc1 (bug) |
| TLS | 🟢 Caddy on shared hub | 🟡 Only once the real stack is on the box |
Per-droplet .env secrets |
🟢 N/A (shared hub) | ⚪ Manual (clone runbook) |
| Retry / resume | 🟢 Idempotent steps + Retry | 🟡 Idempotent steps, but DNS step can duplicate records |
Bottom line: Pooled is a real, one-click, automated path. Siloed is a well-structured scaffold that cannot complete end-to-end until the golden image (hub stack + cp_public.pem + real cloud-init), the GoDaddy DNS story, and the nyc1 region default are resolved. For a dedicated Client right now, use the manual clone runbook in CUSTOMER_DROPLET_OPS §7 (repo docs/CUSTOMER_DROPLET_OPS.md).
kind="siloed" from /provision? If not, should the button be gated/hidden until the golden image lands?_cloud_init says the concrete bootstrap "lives with the droplet image (see docs)" (provisioning.py:74-75) — does that image exist, what's baked into it (hub stack? cp_public.pem? a per-droplet .env template?), and who owns building/refreshing it?upsert_dns_a writes to DO (do_client.py:67-71 vs ops-runbook §6). Do we (a) move the zone to DO so auto-DNS works, (b) swap do_client for a GoDaddy API client, or (c) accept DNS as a manual step and delete the misleading auto-DNS step?upsert_dns_a idempotency. Should it match-then-update existing records (true upsert) rather than blind-POST a duplicate, and should _s_set_dns reconcile instead of guarding only ctx["dns_done"]? (do_client.py:67-71, provisioning.py:119-126)DO_REGION default from tor1 to nyc1 to match the fleet VPC (provisioning.py:99 vs main.py:1279), or is tor1 intentional? Any VPC-attach step needed beyond region choice?cp_public.pem installation for siloed. What's the mechanism to get the CP public key onto a new droplet and mounted into its hub — baked into the image, pushed by cloud-init, or a post-boot step? Without it every siloed Admin-API call 403s.scripts/provision_client.py (renders .env+compose, registers the instance — per onboarding_README) relate to the CP provisioning.py state machine? Are they meant to converge, and which is the intended future path?