Reliability and operations
Every production deploy goes blue-green: a new revision is health-checked with no traffic before traffic moves, and rollback is redeploying the previous revision against a schema that only ever expands. Recovery is rehearsed on a schedule — a weekly restore drill from backup and a weekly resilience drill that injects provider outages, slow dependencies, database failover and crashes. Capacity, timeouts and alerts are written down as numbers, and every scheduled job records its runs.
Deployment
- A disaster-recovery gate and a release-readiness gate run first.
- The live service definition is rendered from the repository and applied as a whole, so configuration drift between deploys is visible in review; a scheduled workflow reports drift that appears in between.
- The new revision is deployed under a tag with no traffic and smoke-tested on its health endpoint.
- Traffic moves to it; the previous revision stays available for immediate rollback.
Schema changes
300+ forward-only migrations are tracked in their own ledger, each declaring its phase: expand, migrate, verify or contract (ADR 0012). Contract runs only after two successful full-traffic releases, and a destructive statement needs an explicit approval with evidence. Deploys take a fresh backup and rehearse migrations on a production clone first. There is no generated down-SQL: rolling back means deploying the previous build against the expanded schema (ADR 0013).
Recovery targets
- Unhealthy revision
- Recovery time
- < 30 s
- Data loss
- None
- Mechanism
- Blue-green rollback
- Database primary failure
- Recovery time
- < 5 min
- Data loss
- < 5 min
- Mechanism
- Cloud SQL high-availability failover; 7 days of point-in-time recovery
- Data corruption
- Recovery time
- < 1 h
- Data loss
- Last good backup
- Mechanism
- Restore, then replay erasure tombstones
- Signing-key compromise
- Recovery time
- < 1 h
- Data loss
- None
- Mechanism
- Key-version rotation with JWKS overlap
- Stolen refresh token
- Recovery time
- Immediate
- Data loss
- None
- Mechanism
- Family revocation on replay
- Lost device
- Recovery time
- Minutes
- Data loss
- Unsynced draft only
- Mechanism
- Owner revokes the device from any other device
- Speech-recognition outage
- Recovery time
- Automatic
- Data loss
- None
- Mechanism
- Circuit breaker fails over to the secondary engine
| Scenario | Recovery time | Data loss | Mechanism |
|---|---|---|---|
| Unhealthy revision | < 30 s | None | Blue-green rollback |
| Database primary failure | < 5 min | < 5 min | Cloud SQL high-availability failover; 7 days of point-in-time recovery |
| Data corruption | < 1 h | Last good backup | Restore, then replay erasure tombstones |
| Signing-key compromise | < 1 h | None | Key-version rotation with JWKS overlap |
| Stolen refresh token | Immediate | None | Family revocation on replay |
| Lost device | Minutes | Unsynced draft only | Owner revokes the device from any other device |
| Speech-recognition outage | Automatic | None | Circuit breaker fails over to the secondary engine |
Drills
- Restore drill
- Cadence
- Weekly (Sunday)
- What it proves
- A backup restores and passes verification
- Resilience drill
- Cadence
- Weekly (Monday)
- What it proves
- Behaviour under provider outage, slow dependency, database failover, revision termination, worker crash and retry storm
- Load test
- Cadence
- On demand
- What it proves
- k6 scenarios for health, CRUD, AI streaming, server-sent-event limits and the rate limiter
| Drill | Cadence | What it proves |
|---|---|---|
| Restore drill | Weekly (Sunday) | A backup restores and passes verification |
| Resilience drill | Weekly (Monday) | Behaviour under provider outage, slow dependency, database failover, revision termination, worker crash and retry storm |
| Load test | On demand | k6 scenarios for health, CRUD, AI streaming, server-sent-event limits and the rate limiter |
Capacity envelope
- Application instances
- Value
- 1 – 5
- Concurrency per instance
- Value
- 15 requests
- Database pool
- Value
- 50 per instance, 250 total
- Database connection limit required
- Value
- 300
| Parameter | Value |
|---|---|
| Application instances | 1 – 5 |
| Concurrency per instance | 15 requests |
| Database pool | 50 per instance, 250 total |
| Database connection limit required | 300 |
Observability
- Structured logs with PII masking to Cloud Logging; request ids correlate logs, traces and audit rows.
- OpenTelemetry traces to Cloud Trace, sampled at 10 % in production.
- Errors recorded with scrubbing; every scheduled job writes a run log.
- Alerts for SLO burn rate, unhandled errors, cron failures, routing failures, upload-virus detections, live-voice SLOs and audit-chain verification.
- AI usage and cost per clinic from the decision record and token tracker.
.github/workflows/deploy.ymlscripts/render-active-cloud-run-service.mjsdocs/architecture/migration-v2.mdresilience/scenarios.jsoninfra/capacity-envelope.jsonloadtest/docs/runbooks/disaster-recovery.md