Status: 🔴 GAP — not yet implemented. There is no scheduled, offsite, or
encrypted backup of the GoBuild production stack today. The only backup that runs
is a local-disk pre-migration dump of a single database insidescripts/update.sh,
and it only fires when a deploy actually changes the git SHA. Everything else — signed
contracts, the wiki, the control-plane audit log, and the encryption key that makes
tenant BYO credentials readable — would be lost with the droplet. This page documents
the truth, then the plan to fix it.
Audience: internal GoBuild ops only. This page names secret files and volumes.
Never share with clients.
Ground-truth sources: docker-compose.prod.yml, scripts/update.sh,
controlplane/do_client.py, app/config.py, app/services/storage.py,
app/services/secretbox.py, and the on-disk secrets (.env, secrets/*.pem,
certs/documenso.p12).
| Badge | Meaning |
|---|---|
| 🟢 | Backed up and recoverable |
| 🟡 | Partially covered — has a gap |
| ⚪ | Reconstructible without a backup (stateless / rebuildable) |
| ❌ | Not backed up — total loss on droplet failure |
| 🔴 | Whole-plan status: GAP, not implemented |
scripts/update.sh step 4 takes a pre-migration pg_dump of the keystone DB only,
to local disk on the same droplet:
# scripts/update.sh:78-82
mkdir -p backups
STAMP="$(date +%F-%H%M%S)"
BACKUP="backups/hub-${STAMP}-pre-${NEW_SHA}.sql"
"${COMPOSE[@]}" exec -T hub-db pg_dump -U keystone keystone > "$BACKUP"
This is a deploy seatbelt, not a backup strategy:
scripts/update.sh:64-67 exits early ifbackups/ on the same droplet (scripts/update.sh:78). Lose thebackups/ directory does notkeystone (scripts/update.sh:81). Not documenso, not wiki, notchatwoot, not the control-plane DB, not MinIO objects, not the secrets.DigitalOcean's own droplet-level automated backups are explicitly disabled for every
provisioned droplet: create_droplet(...) sends "backups": False
(controlplane/do_client.py:46). So there is no platform-level safety net underneath us
either.
Volumes are declared in docker-compose.prod.yml:276-287.
| Data store | Volume / location | What lives here | Status | Notes |
|---|---|---|---|---|
keystone DB |
hub_pgdata (:277) |
The product: orgs, jobs, money, CRM, documents metadata, tenant_credentials |
🟡 | Only a local pre-migration dump (update.sh:81). No offsite, no schedule. |
documenso DB |
hub_pgdata (:277, DB documenso per :86) |
E-signature envelopes, signing audit trail, recipients | ❌ | Never dumped anywhere. |
wiki DB |
hub_pgdata (:277, DB wiki per :253) |
This KB. All wiki content lives in Postgres (no data volume) | ❌ | Never dumped anywhere. |
| MinIO objects | minio_data (:283) |
Signed contract PDFs — documenso uploads to bucket documenso (:98-104) |
❌ | The actual executed contracts. Unprotected. |
| Hub uploads | hub_uploads (:282, STORAGE_DIR=/data/uploads :57) |
Client photos, plan sets (50–500 MB), permit/PO/change-order docs | ❌ | storage_backend defaults to local (app/config.py:183), so these bytes sit on the droplet, not in Spaces. Not backed up. |
| Control-plane DB | cp_data (:287, sqlite cp.db per :213) |
Fleet registry, platform_admins, audit log, showcase feedback |
❌ | The god-mode panel's own state + audit trail. Unprotected. |
| Chatwoot DB | chatwoot_pgdata (:284) |
Support/omnichannel conversations, contacts | ❌ | Separate chatwoot-db (pgvector) — never dumped. |
| Chatwoot storage | chatwoot_data (:286) |
Attachments (ActiveStorage local, :175) |
❌ | Not backed up. |
| Chatwoot Redis | chatwoot_redis (:285) |
Sidekiq queue (persisted, --save 60 1 :146) |
⚪ | Job queue — reconstructible; not worth backing up. |
| OpenProject data | op_data (:278) + op_pgdata (:279) |
Attachments + OpenProject's own Postgres | ❌ | Not dumped. |
| Caddy | caddy_data (:280) + caddy_config (:281) |
Let's Encrypt certs/ACME state | ⚪ | Certs re-issue automatically on a fresh box. |
| Redis (hub) | (unnamed) (:23-25) |
Cache / ephemeral | ⚪ | Stateless. |
These are git-ignored (they never enter the repo) and exist only on the droplet's
filesystem. Losing the droplet without an escrow copy = permanent loss, and some of
them cannot be regenerated:
| Secret | Location | Consequence if lost | Status |
|---|---|---|---|
TENANT_SECRET_KEY (Fernet) |
.env → app/config.py:67 |
All per-tenant BYO credentials become undecryptable. It encrypts tenant_credentials (BYO Stripe/Twilio/QBO/…) at rest via secretbox.py:28-33. Lose the key → every pooled org must re-enter every integration secret by hand. Ciphertext in the DB is dead weight. |
❌ |
.env (all other secrets) |
repo root (:45 env_file) |
POSTGRES_PASSWORD, SECRET_KEY, NEXTAUTH_SECRET, NEXT_PRIVATE_ENCRYPTION_KEY, Resend/Stripe/Twilio/Mapbox/Spaces keys, CP_SECRET, CHATWOOT_SECRET_KEY_BASE, OPENPROJECT_SECRET_KEY_BASE. Some are rotatable, some (Documenso encryption keys) will orphan encrypted rows if lost. |
❌ |
secrets/cp_private.pem |
docker-compose.prod.yml:223 |
Control-plane's Admin-API signing key. Regenerable, but every hub's cp_public.pem (:66) must be re-trusted in lockstep across the fleet. |
❌ |
certs/documenso.p12 |
docker-compose.prod.yml:106 |
PKCS#12 cert Documenso signs contracts with (:95, passphrase DOCUMENSO_SIGNING_PASSPHRASE :96). Lose it → new signatures chain to a different cert. |
❌ |
The single worst-case loss: droplet disk failure with no escrow. You lose the signed
contracts (MinIO), the DBs, andTENANT_SECRET_KEY. Even a heroic rebuild leaves
every tenant's stored integration credentials permanently unreadable.
Design goals: nightly, all databases, objects included, secrets escrowed,
pushed offsite, encrypted at rest, retained with rotation, and restore-tested.
.env already carries SPACES_KEY / SPACES_SECRET / SPACES_REGION / SPACES_BUCKET
/ SPACES_ENDPOINT (see app/config.py:185-189). Product file storage currently runs in
local mode (app/config.py:183), so the Spaces account is available. Use a
separate, private, versioned bucket for backups — e.g. gobuild-backups — not the
app bucket, and not on the same droplet. (Consider a different DO region than
nyc1/nyc3 so a regional incident doesn't take both.) An S3/B2 target works equally well.
hub-db — keystone, documenso, wiki:pg_dump -U keystone <db> for each. (One pg_dumpall --globals-only too, for roles.)docker compose exec -T chatwoot-db pg_dump -U chatwoot chatwoot.cp_data sqlite: copy cp.db (use sqlite3 .backup for apg_dump if CP_DATABASE_URL was ever pointed at Postgresdocker-compose.prod.yml:213).mc mirror the documenso bucket (signed PDFs), and hub_uploadsstorage_backend stays local. If MinIO is later fronted by Spaces, replicateop_pgdata dump + op_data attachments tarball (if OP data matters.env, secrets/cp_private.pem, certs/documenso.p12. BecauseTENANT_SECRET_KEY lives in .env, escrowing .env is what makes the DB restorable inage orgpg with a recipient key whose private half is not stored on the production droplet..env or plaintext DB dump.SPACES_KEY).SHA256SUMS alongside each night's set; the restore drill verifies it.Backing up .env next to the DB dumps means one compromised bucket key leaks both the
ciphertext and the key that decrypts it. Escrow secrets on a different trust boundary:
a password manager / dedicated secrets vault (1Password, Bitwarden, DO/Vault secret store),
or a separately-keyed age-encrypted blob in a different bucket/account. The age private
key for the data backups must itself be escrowed there too — otherwise the nightly job
produces artifacts nobody can open.
A single scripts/backup.sh on the GoBuild-App droplet, invoked by a systemd timer
or cron (~03:00 local). It must: dump → checksum → encrypt → s3 cp/mc mirror to the
backups bucket → prune per retention → emit success/failure to monitoring
(controlplane/monitor.py already exists for fleet health; wire a heartbeat so a silent
failure alerts — a backup job that stops running is the classic DR failure mode).
A backup you have never restored is a hypothesis. Run this quarterly on a throwaway
droplet or the Dev box — never first-time during a real outage.
Prep
GoBuild-Dev). Clone the repo, checkout the SHASHA256SUMS.age/gpg key. If you cannot get the key, the drill hasRestore data
docker compose -f docker-compose.prod.yml up -d hub-db chatwoot-db minio (DBs + objectcat keystone.sql | docker compose exec -T hub-db psql -U keystone keystone
cat documenso.sql | docker compose exec -T hub-db psql -U keystone documenso
cat wiki.sql | docker compose exec -T hub-db psql -U keystone wiki
cat chatwoot.sql | docker compose exec -T chatwoot-db psql -U chatwoot chatwoot
scripts/update.sh:105.)mc mirror the MinIO documenso bucket back; restore hub_uploads if captured.cp.db into cp_data.Restore secrets & bring up
.env, secrets/cp_private.pem, certs/documenso.p12 into place.TENANT_SECRET_KEY matches the one that encrypted the restored keystonedocker compose -f docker-compose.prod.yml up -d --build.Verify (acceptance)
/readyz returns 200 (same probe scripts/update.sh:92-94 uses).TENANT_SECRET_KEY round-trips via secretbox.py).app/routers/client_portal.py:499).wiki DB restored).ops.gobuild.ca; confirm the fleet registry + audit log are present.Numbers to ratify with the owner; the plan above is sized for the Proposed column.
| Metric | Meaning | Current reality | Proposed target |
|---|---|---|---|
| RPO (max data loss) | How much recent data we accept losing | Effectively total — last local keystone dump could be days old and dies with the droplet |
≤ 24h (nightly). Consider ≤ 1h WAL/PITR for keystone if signed-money data warrants |
| RTO (time to restore) | How long to be back up | Unbounded — no offsite artifact to restore from | ≤ 4h to a new droplet from offsite backups |
| Contract durability | Signed PDFs (legal record) | ❌ single copy in minio_data |
Zero-loss: versioned offsite replication |
| Secrets recoverability | Can we decrypt restored data | ❌ TENANT_SECRET_KEY single-copy on disk |
Escrowed in ≥2 locations, restore-tested |
Decision inputs needed: acceptable RPO for money/contract data vs. cost, RTO tolerance
(customer-facing hub down), and whether siloed customer droplets get the same policy or
just DO automated backups flipped on (do_client.py:46 currently False).
keystone dumpscripts/update.sh:81); DO backups off (do_client.py:46); signed contracts, wiki,TENANT_SECRET_KEY (app/config.py:67) makes tenant BYO.env (app/config.py:185-189) — the gap is ascripts/backup.sh + escrow discipline, not new infrastructure.