Status: 🟡 Real coverage, zero automation. There are ~93 in-process pytest
modules plus a live-hub e2e sweep, and they catch real regressions. But nothing
runs them for you — there is no CI, no pre-commit hook, no merge gate. Green is a
human decision. The owner's call is "document release as-is, no CI", so the
test-gate below is framed as a recommendation / roadmap, not current state.
Audience: engineers changing the hub. Every claim is grounded in a
file:lineyou can
open. When the code and this page disagree, the code wins — fix the page.
Cross-links: Engineering conventions ·
Deploy & release.
Two independent layers, run separately, sharing no fixtures.
| Layer | Where | Runner | Runs against | 🟢/🟡 |
|---|---|---|---|---|
| Unit / integration | tests/test_*.py (~93 modules) |
pytest |
app in-process, in-memory SQLite | 🟢 broad, fast, hermetic |
| End-to-end / demo-readiness | tests/e2e/ (3 test modules) |
scripts/demo-check.sh |
live hub (app.gobuild.ca) via Playwright |
🟢 exists 🟡 manual, live-only |
| CI / merge gate | — | — | — | 🟡 none — see The bar |
The bulk of the coverage. tests/conftest.py:1 runs the actual FastAPI app in-process
against an in-memory SQLite DB, so no Postgres or Docker is needed and the suite is fast.
Key fixtures (tests/conftest.py):
db_session (:61) — spins up sqlite:// with StaticPool, builds the schemaBase.metadata.create_all (not through Alembic — see migration caveat),PRAGMA foreign_keys=ON (:71) so tests catch insert-order FKget_db so the app and the testseeded (:96) — one org + owner + job + two schedule items, tierOrgTier.enterprise (all features on), plus email + Twilio credentials set viacredentials.set_credential so the resolve→send→log path is real (:122). Returns{org_id, user_email, password, job_id}.client (:134) — a TestClient(app).Hermetic by construction. conftest.py:7-39 blanks every outbound key at import —
Gemini, DeepSeek, Resend, Twilio, Stripe, OpenProject, Documenso, Chatwoot,
inbound-email. Tests therefore exercise the manual-fallback / logging paths and never
hit a live third party. It also forces KEYSTONE_ENV=local (:10) so session cookies
aren't Secure-only over the test's HTTP and the X-Tenant header is honored (see
below). Outbound HTTP that is exercised is mocked with httpx.MockTransport
(requirements.txt, note next to pytest-asyncio — respx is broken under httpx 0.28).
What's covered: auth, tenant scoping, the portal, billing pipeline, estimating/takeoff,
scheduling, comms/SMS, migration intake, entitlements, impersonation, credential
isolation, security headers, and dozens more — one module per feature area
(tests/test_*.py).
tests/e2e/ is a Playwright suite that drives the real deployed hub in a browser.
Three ordered layers (tests/e2e/pytest.ini):
test_01_reachability.py — every catalogued screen loads: HTTP < 400, not bouncedconsole.errors, or failed first-party requests (test_01_reachability.py:1). Atest_02_flows.py — cross-module journeys (Jobs↔Costing↔Schedule, Invoices↔Jobs,:95). The Stripe "pay online" leg is deliberately @pytest.mark.skip until test keys:118).test_03_visual.py — visual-regression against tests/e2e/baselines/.Signal hygiene lives in tests/e2e/conftest.py: a NOISE_SUBSTRINGS allowlist (:22)
drops benign third-party chatter (Stripe, Chatwoot, Mapbox, favicon, vite preload,
ERR_ABORTED cancellations) so only first-party errors fail a screen; a Collector
(:88) attaches to each page and captures console / pageerror / failed-request events.
One owner login per session is reused as Playwright storage_state so every test starts
authenticated (:66).
Markers you'll use (tests/e2e/pytest.ini:7): field (mobile field-PWA persona) and
visual (regression) — both excludable with -m "not …".
No root pytest.ini / pyproject.toml exists, so configuration is defaults; just invoke
pytest from the repo root:
cd /home/construction-proj
pytest -q # whole suite
pytest tests/test_smoke.py # one module
pytest tests/test_tenancy_isolation.py::test_session_replay_guard # one test
pytest -k costing # by keyword
⚠️
README.md:112says "7 tests" — that's stale; the suite is ~93 modules now. Trust
ls tests/test_*.py | wc -l, not the README.
scripts/demo-check.sh is the single command to run before any demo (:2). It builds
a Playwright Docker image, runs the e2e suite against E2E_BASE_URL
(default https://app.gobuild.ca), then triages and publishes the results.
scripts/demo-check.sh # full suite (reachability + console + flows + visual)
scripts/demo-check.sh -m "not visual" # skip visual regression
scripts/demo-check.sh -k flows # just functional flows
E2E_UPDATE_SNAPSHOTS=1 scripts/demo-check.sh -m visual # re-baseline screenshots
Credentials come from tests/e2e/.env.e2e (gitignored; see .env.e2e.example) —
E2E_OWNER_EMAIL / E2E_OWNER_PASSWORD are required (scripts/demo-check.sh:20).
Outputs:
tests/e2e/triage.py) — if ANTHROPIC_API_KEY is set, each failure isE2E_TRIAGE_MODEL, default claude-sonnet-4-6) as REAL /triage.py:13), so demo-check gates on it. Without a key, all failures are treatedtests/e2e/artifacts/dashboard.html, REPORT.md, report.html, and agobuild.ca/demo-check-<token>/ (demo-check.sh:46).Multi-tenancy is the property most worth a test, because a leak here is a data breach.
The canonical patterns:
Target a tenant without DNS. In non-production the X-Tenant header stands in for the
{slug}.app.gobuild.ca subdomain (tests/test_tenancy_isolation.py:7). Tests force
KEYSTONE_ENV=local so it's honored.
The shared-email fixture. two_orgs (test_tenancy_isolation.py:24) creates two orgs
whose owners share owner@shared.com with different passwords — the exact pooled-DB
collision the tenant foundation must survive. From it:
test_login_resolves_correct_tenant — the subdomain, not the email, decides which userorg claim matches (:64).test_login_rejects_other_tenants_password — Acme's user must not auth with Built's:75).test_session_replay_guard — a token valid on its own tenant host is 401 when:80).test_suspended_org_cannot_login — OrgStatus.suspended blocks login (:89).Scope every query by organization_id. The seeded fixture always stamps
organization_id on org/user/job/schedule rows (conftest.py:103), and read tests assert
the tenant only sees its own rows (e.g. test_jobs_scoped_to_tenant,
tests/test_smoke.py:30). When you add a model or route, add a test that a second org
cannot read/write it. See also test_credential_isolation.py,
test_isolation_fixes.py, test_impersonation.py.
Rule of thumb: any new query that touches tenant data needs both a "my org sees it" and
an "other org can't" assertion. The whole security model
rests on this.
Migrations are Alembic, in migrations/versions/ (~141 revisions;
alembic.ini → script_location = migrations). migrations/env.py:15 injects this
instance's DB URL from app settings — never hardcoded — and imports app.models.Base so
every table is registered for autogenerate.
The pytest suite builds its schema with Base.metadata.create_all (conftest.py:77),
not by running alembic upgrade head. So the model layer is tested, but the migration
scripts are not — a migration that diverges from the models (a missed column, a bad
data backfill, a broken downgrade) will pass every unit test and only fail on a real
Postgres upgrade. There is no test_*.py that applies migrations.
scripts/update.sh:84 takes apg_dump of hub-db before alembic upgrade head — the only irreversible step:86) — so a failed migrate can be restored. This lives in_migration_status (app/routers/admin_api.py:118) compares the DB'salembic_current against the script alembic_heads and returnsup_to_date (:138); surfaced via /version (:142) and asserted intests/test_admin_api.py:73. test_monitor.py:95 checks the fleet monitor flags/readyz (app/routers/health.py:18) confirms the DB is reachableupdate.sh polls it before declaring the box live.alembic revision --autogenerate -m "…" after editing models; read the generatedalembic heads → one) so up_to_date can go true.conftest.py:100) — match that shape.Automated tests are necessary, not sufficient. For any change with a runtime surface,
drive the actual flow and observe it — don't ship on green pytest alone. This is the
same principle the e2e write-flows encode (submit the real form, then assert the record
appears in the UI, test_02_flows.py:95).
/verify skill on non-trivial changes — it exercises the affected flowtest_01_reachability.py asserts).scripts/demo-check.sh and read REPORT.md.Not current policy. The owner has parked CI — "document release as-is, no CI."
This is the bar to aim for and the roadmap to get there, not a gate that exists today.
What SHOULD be green before you deploy:
pytest -q passes locally on your branch. (Available now — just run it.)/verify), console clean, no 500s.scripts/demo-check.sh green (no REAL failures) against a staging or the liveRoadmap to close the 🟡 gaps (suggested, not committed):
pytest -q on every push — the singletest_migrations.py that spins up Postgres and runs alembic upgrade head from emptydemo-check.sh (or at least reachability + flows, -m "not visual") into a| I want to… | Do this |
|---|---|
| Run the fast suite | pytest -q |
| Run one test | pytest tests/test_x.py::test_y |
| Test tenant isolation | model on test_tenancy_isolation.py; use X-Tenant + a second org |
| Pre-demo readiness | scripts/demo-check.sh → read artifacts/REPORT.md |
| Re-baseline screenshots | E2E_UPDATE_SNAPSHOTS=1 scripts/demo-check.sh -m visual |
| Check schema drift | /version → up_to_date (admin_api.py:138) |
| Verify a real change | run the hub, drive the flow, or use /verify |