Status: 🟢 Live in production (control-plane app + signed Admin API + fleet monitor) · Audience: GoBuild staff only. Not customer-facing. This app can read and edit every customer's data.
Two things are stubbed today — read Login & TOTP (2FA machinery exists but no enrollment path) and Provisioning (the siloed cloud-init is a placeholder). They are flagged inline with file:line evidence.
The Control Plane ("CP", branded GoBuild Admin, internally Keystone Control Plane) is a separate FastAPI app from the customer hub. It lives in controlplane/ and runs as the ops service at ops.gobuild.ca, reusing the hub Docker image with a different entrypoint (uvicorn controlplane.main:app, port 8001 — docker-compose.prod.yml:200-224).
It holds no client business data. Its own database (CP_DATABASE_URL, separate from every hub) stores only: platform-admin identities, the fleet registry, provisioning jobs, monitoring incidents, and an append-only audit log (controlplane/db.py:1-9, controlplane/models.py). All customer data is read/written live over each hub's signed Admin API.
Status: 🟢 Live
The CP is the "atlas + fleet" console — one pane of glass over the whole GoBuild estate:
controlplane/main.py.FleetInstance), DigitalOcean droplet lifecycle (/servers), provisioning, billing (/billing), usage (/usage), an integration-health board (/integrations), and live monitoring (/monitoring).Who uses it: GoBuild staff only. It runs on its own subdomain with its own login, entirely outside the tenant trust boundary. There is no customer access path to it. Operators are modeled as PlatformAdmin rows (models.py:25-38) with three roles:
| Role | Powers |
|---|---|
owner |
Everything, plus manage other CP admins (/admins, main.py:1095-1179) |
admin |
Full operations (provision, edit orgs, impersonate, servers) |
viewer |
Read-only — can view every page but every write is blocked |
Read-only is enforced at a single ASGI choke point (main.py:75-89): any POST/PUT/PATCH/DELETE (except /login, /logout) by a viewer returns 403, so no write route can forget the check.
The CP also exposes a few service endpoints used by the GoBuild Showcase to turn approved prospect mock-sites into live clients: /_provision/golive, /_provision/lead, /_provision/domain (main.py:148-259). These are authed by a forwarded platform-admin session or the showcase service token (svc-golive), both signed with the shared CP_SECRET.
Status: 🟠 Password login Live · TOTP machinery present but DORMANT (no enrollment path)
The CP has its own login, separate from any hub (main.py:262-296):
PlatformAdmin row; passwords are bcrypt (reused from the hub image — security.py:20).cp_session), signed with CP_SECRET, holding admin_id.issued_at.signature, 7-day max age, HttpOnly + SameSite=lax + Secure in production (security.py:56-77, main.py:287).main.py:102-111).CP_BOOTSTRAP_EMAIL + CP_BOOTSTRAP_PASSWORD are set, the first owner is seeded (bootstrap.py:28-44). Until then, sign-in is disabled and the login page says so.A full RFC-6238 TOTP implementation exists (controlplane/totp.py — secret generation, HOTP, verify with skew window, otpauth:// provisioning URI). Login will enforce it if an admin has a secret:
# main.py:279-281
ok = (admin is not None and admin.is_active
and security.verify_password(password, admin.password_hash)
and (not admin.totp_secret or totp.verify(admin.totp_secret, totp_code)))
But no admin can ever have a secret set:
bootstrap.py:41 and main.py:1120 both create admins with totp_secret="".controlplane/ (grep: totp_secret is only ever read or written as "").templates/login.html) has no TOTP input field — only email + password.So the second factor is effectively dead code: the gate reads "enforced only when a secret is set," and nothing sets one. bootstrap.py:39-40 and main.py:277-278 both comment this as intentional for the current dev instance. This is the single biggest security gap in the CP. See open questions.
Status: 🟢 Live
A hub is a running Keystone deployment serving the Admin API. Each is a FleetInstance row (models.py:41-54):
| Field | Meaning |
|---|---|
name |
Human label ("GoBuild pooled hub") |
kind |
pooled (shared hub, many orgs) or siloed (one org's own hub) |
base_url |
Internal address the CP calls, e.g. http://hub:8000 — never public; reached over the VPC-private network |
region |
DO region slug |
status |
active / … (only active hubs are monitored — monitor.py:116-117) |
last_health_at |
Last successful reachability check |
The pooled hub is seeded on first boot from HUB_INTERNAL_URL / HUB_NAME / HUB_KIND (bootstrap.py:47-59, default http://hub:8000). Siloed hubs are registered by the provisioning state machine.
/fleet (main.py:901-937) fans out over every instance and, per hub, calls the Admin API /_admin/health + /_admin/version, aggregating org/user/job counts and computing version drift (more than one distinct git_sha across reachable hubs → drift flag; hubs with up_to_date=false → pending-migration count).
There is also a legacy registry.json view (_legacy_fleet, main.py:1358-1368) — the pre-Phase-1 file-based registry, kept only to feed the dashboard tiles. Not the source of truth; FleetInstance rows are.
Status: 🟢 Pooled Live · 🟠 Siloed partially wired (droplet boot is a PLACEHOLDER — see below)
Provisioning is a resumable state machine (controlplane/provisioning.py). A ProvisioningJob (models.py:57-73) records kind, request params, a checkpointed step, and a result context. run() advances through the steps for that kind; each step is idempotent (no-ops when its ctx marker is set), so a retry resumes from the failed step. Jobs run inline in the request handler (main.py:779) — not on a background worker — with a manual Retry button (main.py:803-812).
POOLED_STEPS = [validate, create_org, activate]
name/slug/owner_email; pick the pooled FleetInstance.POST /_admin/orgs on the shared hub → creates the Organization + owner + seeds day-one cost-codes/templates; returns a magic-link invite URL (admin_api.py:484-541).SILOED_STEPS = [validate, create_droplet, await_active, set_dns, register_instance, seed_org, activate] — requires DIGITALOCEAN_TOKEN or every step fails loudly (do_client.py).
provisioning.py:85-90).POST /droplets via the DO API: name gobuild-{slug}, region DO_REGION (default tor1), size DO_SIZE (default s-2vcpu-4gb), image DO_IMAGE (docker-20-04), with cloud-init user_data (provisioning.py:93-104, do_client.py:43-49).active with a public IP yet, raise a retriable error ("still booting — retry to continue"). Re-running the job resumes here (provisioning.py:107-116).{slug}.app.{TENANT_DNS_DOMAIN} → droplet IP via the DO DNS API (provisioning.py:119-126).siloed FleetInstance pointing at http://{ip}:8000 (provisioning.py:129-138).POST /_admin/orgs on the new hub to create the org + owner (provisioning.py:141-147).¶ ⚠️ Siloed provisioning is not end-to-end today
- The droplet never actually boots a hub.
_cloud_init(provisioning.py:73-82) is an explicit placeholder: it writes a log line and leaves the real work (git pull && docker compose up -d --build && alembic upgrade head) commented out. The comment says "the concrete script lives with the droplet image (see docs)." So step 5'sregister_instancepoints at a droplet that isn't running Keystone, and step 6'sseed_orgwill fail its Admin-API call. Siloed provisioning depends on a golden image that must carry the compose stack andcp_public.pem; that image is out of band.- DNS is not idempotent.
do_client.upsert_dns_a(do_client.py:67-71) just POSTs a new A record; the docstring admits "idempotency is the caller's job," but_s_set_dnsguards only on its ownctx["dns_done"]marker, so a fresh job for the same slug can create a duplicate A record.- Region mismatch worth noting: the state machine defaults to
tor1(provisioning.py:99), while the manual/serversprovisioner defaults tonyc1with a "nyc1 auto-joins the default-nyc1 VPC where the GoBuild fleet lives" comment (main.py:1268-1281). New siloed tenants would land in the wrong region/VPC unlessDO_REGIONis set.
Also available: a manual DigitalOcean droplet panel at /servers (main.py:1182-1289) — reboot/power/snapshot/resize (CPU/RAM only, reversible)/destroy (type-to-confirm)/provision. It is hard-guarded to GoBuild droplets only: do_ignore_droplets (default smolmac,livv.ca) are invisible and do_protected_droplets (default GoBuild-App,GoBuild-CRM) are visible-but-locked, so the panel can never touch non-GoBuild or marketing servers.
Status: 🟢 Live · fail-closed on both ends
Every god-mode action crosses from CP to hub through the hub's /_admin/* surface (app/routers/admin_api.py), authenticated by a short-lived RS256 JWT that the CP signs with its private key and the hub verifies with the matching public key. The same contract is served whether the hub is pooled or siloed, so the CP never needs to know which.
Control Plane (ops) Hub
cp_private.pem ──signs──▶ RS256 JWT ──verify──▶ cp_public.pem
(CP_SIGNING_KEY_PATH) (60s TTL) (cp_public_key_path)
controlplane/security.py:84-95, hubclient.py): every Admin-API call mints a fresh token — nothing long-lived is stored or sent. Claims: iss=keystone-control-plane, aud=keystone-hub, sub=<admin email> (for the hub's own logs), iat, exp=iat+60s, unique jti. Signing key loaded inline from CP_SIGNING_KEY or from the file at CP_SIGNING_KEY_PATH (prod mounts cp_private.pem at /run/cp/cp_private.pem — docker-compose.prod.yml:223).admin_api.py:37-68): verifies the token against cp_public_key (inline) or cp_public_key_path (prod mounts cp_public.pem at /run/cp/cp_public.pem — docker-compose.prod.yml:66), checking algorithm/audience/issuer/expiry./_admin surface returns 403 (admin_api.py:55-56). If the CP has no signing key, hubclient raises before any call (security.py:87-88, hubclient.py:25-27).The token contract constants are duplicated in two files that must stay in lockstep: controlplane/security.py:49-51 and app/routers/admin_api.py:30-32.
| File / env | Where | Purpose |
|---|---|---|
cp_private.pem (CP_SIGNING_KEY / CP_SIGNING_KEY_PATH) |
CP only | Signs Admin-API tokens |
cp_public.pem (cp_public_key / cp_public_key_path) |
Every hub | Verifies CP tokens |
CP_SECRET |
CP + showcase | HMAC-signs CP session cookies and showcase service tokens (a different secret from the RS256 keypair) |
Note: authoritative audit lives on the CP (AuditLog), not the hub. The hub only uses the token sub for its own local logging. The internal base_url (e.g. http://hub:8000) is plain HTTP over the private VPC — TLS is not used on the CP↔hub hop.
The typed client wraps ~30 endpoints (hubclient.py): health, version, telemetry, vitals, usage, orgs CRUD + lifecycle (suspend/enable/delete/restore/purge/export), users CRUD + reset + activity, per-provider credentials, impersonate, site import/domain, and CRM lead creation.
Status: 🟢 Live (hub-side; the CP itself is single-DB)
Multi-tenancy is a hub concern (app/tenancy/), but the CP models and provisions against it, so it matters here. A request's tenant is resolved host → slug → Organization:
TenantMiddleware (middleware.py) — pure-ASGI, runs before routing with zero DB cost. It parses the Host header to a slug and publishes it as scope["tenant_slug"] + a ContextVar. In non-production only, an X-Tenant: header override is honored for tests/local dev (never in prod).tenant_slug_from_host (resolver.py:23-42) — acme.app.gobuild.ca → acme when tenant_base_domain == app.gobuild.ca. The bare zone, reserved labels (app, www, api, sign, chat, ops, reports, book, …), nested subdomains, and any host outside the zone all resolve to None.resolve_request_org (resolver.py:66-76):
{slug}.app.gobuild.ca) → that org.sole_org() returns the one org if exactly one exists (so an existing per-droplet deploy keeps working with no subdomain).None, so callers fail closed instead of guessing.An org's isolation (pooled vs siloed) is a per-org field editable from the CP. Live folder-sites (imported showcase mocks) get their own public subdomain {subdomain}.gobuild.ca and can attach a custom domain with on-demand TLS (admin_api.py:547-656).
Status: 🟢 Live
The wall-off is enforced in one place — app/services/credentials.py, the single source of truth for every tenant's integration keys. Every integration reads through the CredentialResolver; no call site touches get_settings().<key> directly.
Resolution order (CredentialResolver.resolve, credentials.py:194-203):
own (encrypted tenant_credentials) → legacy org.settings JSON → global .env
…but the last two steps apply only to siloed / single-tenant orgs. The rules:
None as "cannot send" and never fall back (credentials.py:219-223).MANAGED = OpenProject, Chatwoot, Documenso, AI keys — credentials.py:68) → GoBuild-hosted infra, data scoped per-org inside each service; every org uses the shared account automatically. These are always "on" and never count as an onboarding gap.SHARED_DEFAULT = {email} — credentials.py:75) → a deliberate exception: every org sends through the one shared GoLive Resend key/domain, dressed up to represent the org (display name = org name, reply-to = org owner — email_for, credentials.py:236-249). An org can still connect its own Resend key (BYO white-label). Sending email is one-way, so a shared key leaks no tenant data — unlike a shared Chatwoot inbox.Nuance on "no global fallback": it's exact for BYO providers (Stripe/Twilio/BuildData). For email and the managed providers, a shared/global account is used by design even for pooled orgs. The
/integrationsboard (main.py:939-970) surfaces the real gap: a pooled org missing its own email OR SMS capability is flaggedcant_send, so onboarding holes are visible instead of silent.
Credentials are stored encrypted (secretbox.encrypt_json → TenantCredential.ciphertext). God-mode edits every provider's raw keys from the CP via /_admin/orgs/{id}/credentials/* (admin_api.py:838-878); the CP UI is the per-org Integrations tab (main.py:544-600).
Status: 🟢 Live · CP-side audited
Two distinct god-mode mechanisms:
1. Admin-API access ("unlock /_admin"). Every CP→hub call is itself god-mode: a CP-signed RS256 token unlocks the hub's /_admin surface (see signed Admin API). Attributed to the operator's email in the token sub.
2. "Login as" — tenant impersonation (30-min session). Flow (main.py:725-749 → admin_api.py:734-759 → app/routers/impersonation.py):
POST /_admin/orgs/{org}/impersonate.create_impersonation_ticket, security.py:65-73, purpose=imp_ticket, carries impersonated_by) and returns a handoff URL on the tenant's own host: https://{slug}.{tenant_base_domain}/_impersonate/consume?t=….action="impersonate.start", target = the impersonated user + org, operator email, IP) before redirecting the browser./_impersonate/consume, which swaps the ticket for a 30-minute impersonation session cookie set on the hub's domain (not the CP's), carrying impersonated_by for the banner (impersonation.py:28-51)./_impersonate/exit clears the cookie.Audit trail: the authoritative record is the CP AuditLog — impersonate.start captures who impersonated whom, in which org, from which IP. Almost every mutating CP action writes an AuditLog row (org edits capture full before/after JSON) viewable at /audit (main.py:890-896).
Gaps to note: (a) there is no
impersonate.endaudit event — only the start is recorded; the exit just deletes the cookie (impersonation.py:54-59). Session duration isn't captured. (b) Impersonation is available to anyowneroradminrole (onlyvieweris blocked by the RBAC guard) — there is no separate, higher bar or per-org consent for entering a customer's account. See open questions.
Status: 🟢 Live (background loop, best-effort)
A background monitor (controlplane/monitor.py) is started in the app lifespan (main.py:59-62, gated by monitor_enabled, default on) and polls every active hub on an interval (monitor_interval_seconds, default 180s, floor 30s), after a 25s warmup so a co-deployed hub can boot. One bad check never wedges the loop.
Per cycle (run_checks, monitor.py:112-205), for each active hub:
/_admin/health with a confirm-retry (3s gap) so a transient blip doesn't page. Failure → server_down critical incident./_admin/version; if up_to_date is false (DB behind code, or multiple heads) → migration_drift warning./_admin/vitals disk %: ≥ monitor_disk_crit_pct (95) critical, ≥ monitor_disk_warn_pct (85) warning.vitals.sends_1h: ≥5 failures and ≥50% failure ratio in the last hour, per channel → send_failed warning.monitor_cert_domains; expired → critical, ≤ monitor_cert_warn_days (14) → warning.error state updated in the last 24h → provision_failed warning; auto-resolves when the job leaves the error state.Incidents (models.py:76-93) dedup on a key (e.g. {instance_id}:server_down): at most one open per key. The monitor alerts only on edges — open and resolve — never on every poll. Alerts (monitor.py:93-107): email always (alert_email, default gautam@gobuild.ca) via the shared Resend client; SMS only for critical incidents and only if alert_sms_to is set. The monitor authenticates its Admin-API calls as token subject monitor@controlplane.
/monitoring (main.py:973-1037) is the human view: open incidents (critical first), recently resolved, live per-server vitals, and — separately — all GoBuild droplets' power status pulled straight from DO (so a powered-off droplet that isn't even a registered hub is still visible). Operators can manually resolve an incident. A nav badge shows the open-incident count (monitor.open_incident_count).
| Var | Service | Purpose |
|---|---|---|
CP_SECRET |
ops, showcase | HMAC key for CP session cookies + showcase service tokens |
CP_SIGNING_KEY / CP_SIGNING_KEY_PATH |
ops | RS256 private key (cp_private.pem) to sign Admin-API tokens |
cp_public_key / cp_public_key_path |
each hub | RS256 public key (cp_public.pem) to verify CP tokens; unset ⇒ Admin API 403s |
CP_DATABASE_URL |
ops (+ showcase) | CP's own DB (sqlite fallback; point at Postgres in prod) |
CP_BOOTSTRAP_EMAIL / CP_BOOTSTRAP_PASSWORD |
ops | Seeds the first owner admin on boot |
HUB_INTERNAL_URL / HUB_NAME / HUB_KIND |
ops | Seeds the pooled FleetInstance |
DIGITALOCEAN_TOKEN |
ops | DO API for provisioning/servers/billing; unset ⇒ siloed + /servers + /billing disabled |
DO_REGION / DO_SIZE / DO_IMAGE / TENANT_DNS_DOMAIN |
ops | Siloed droplet defaults (note: tor1 vs /servers nyc1) |
do_ignore_droplets / do_protected_droplets |
ops | Hide non-GoBuild droplets / lock marketing droplets |
tenant_base_domain |
hub | Zone for {slug}.app.gobuild.ca resolution (default app.gobuild.ca) |
monitor_* / alert_email / alert_sms_to |
ops | Monitor cadence, thresholds, alert destinations |
| Gap | Evidence | Impact |
|---|---|---|
| TOTP 2FA cannot be enabled — no enrollment path, no login field | bootstrap.py:41, main.py:1120, templates/login.html, main.py:281 |
God-mode console is password-only |
| Siloed cloud-init is a placeholder — droplet never boots Keystone | provisioning.py:73-82 |
Siloed provisioning not end-to-end without a golden image |
| DNS not idempotent — duplicate A records possible on retry | do_client.py:67-71, provisioning.py:119-126 |
Retried siloed jobs can duplicate DNS |
Region default mismatch — tor1 vs nyc1 VPC |
provisioning.py:99 vs main.py:1279 |
New siloed tenants may miss the fleet VPC |
No impersonate.end audit — only start recorded |
impersonation.py:54-59 |
Impersonation session duration not audited |
| Provisioning runs inline in the request, not a worker | main.py:779 |
A slow/siloed provision blocks the request; relies on manual Retry |