Business continuity
Backups, retention, recovery objectives, and the procedure for each disaster scenario.
Four independent copies of different things
| Asset | Backed up | Where | Retention |
|---|---|---|---|
| The analytical database — events, entities, observations, graph, API key digests | Daily automated snapshots; continuous (point-in-time) recovery is switched on for the project at the start of an enterprise agreement | Encrypted, stored separately from the primary | 7 days standard, 28 days on enterprise agreements |
| Bulk extracts and the satellite-scene cache | Written to object storage as produced | Cloudflare R2 — a different provider from the database | Rolling |
| Raw source payloads | Not yet archived. A content-addressed archive of every fetched payload, held with a second provider, is a written requirement and not a built system. | — | — |
| Schema and infrastructure definition | Every change, as a versioned migration in source control | GitHub, and every engineer's local clone | Permanent, full history |
| Application code and configuration | Every commit; each deployment is an immutable version | GitHub and the platform's version history | Permanent |
Backups are encrypted at rest and held separately from the primary, so a compromise of the running database is not a compromise of its history.
The record can also be rebuilt from source material: everything in the database derives from public feeds we can fetch again, and the schema and the code that fetch them are in source control with a second provider. Total loss of the database would be slow and expensive to recover from, but it would not be permanent loss of the product. What it would lose is any history a source has since withdrawn, which is why the raw archive above is a requirement.
Committed objectives, by tier
| Objective | Standard | Enterprise |
|---|---|---|
| RPO — maximum data loss | 24 hours | 5 minutes |
| RTO — time to restore service | 4 hours | 2 hours |
| Backup retention | 7 days | 28 days |
| Recovery granularity | Daily snapshot | Any point in the retention window |
Point-in-time recovery, which is what brings the recovery point objective to minutes, is a configuration change on the database platform: it is switched on for the project at the start of an enterprise agreement, not built. These are the objectives we commit to contractually.
The edge does not require recovery in the same sense. The API, Terminal and Atropos are stateless and globally distributed: a failed location is routed around automatically, and a faulty deployment is recovered by redeploying the previous immutable version, which takes about a minute.
Five scenarios and the response to each
| Scenario | Impact | Response | Expected |
|---|---|---|---|
| Faulty deployment | API errors or incorrect output | Redeploy the previous immutable version. No restore, no data involved. | Under 5 minutes |
| Edge location or region failure | None visible | The platform routes around it. No action required. | Automatic |
| Database unavailable | Cached reads continue; fresh reads and writes fail | Platform failover, or restore to a new instance and repoint the connection layer | Within RTO |
| Database corrupted or destroyed | Loss of the durable record | Restore to the recovery point, repoint, verify with the automated integrity checks, then re-ingest any gap from the public sources | Within RTO; any gap backfills behind it |
| Provider account compromised | Potentially total | Incident response: revoke every credential, rotate every secret, restore from backup into a clean project, rebuild the edge from source control. Everything required is in the repository. | Same day to service |
The published summary of the runbook
The full runbook is operational and is provided under NDA. Its structure:
- Declare. Classify the incident and start the notification deadlines in incident response. Nothing waits for the technical work to finish.
- Contain. Stop the harm before diagnosing it — disable the affected path, roll back, or revoke the credential.
- Restore. Recover the database to a new instance from the chosen recovery point, repoint the connection layer, and redeploy the edge from the last known good version.
- Verify before declaring recovery. Run the automated integrity checks against the restored system and read the output. A restored database that has lost an index, a trigger or a licence filter looks healthy until it is queried.
- Backfill the window between the recovery point and the failure from the public sources.
- Report. A written post-incident report within 5 business days.
What is tested and how often
- Rollback is the same mechanism as an ordinary deployment — redeploying an earlier immutable version — and a deliberate rollback to a prior version is part of the restore test, so that it is exercised rather than assumed.
- Restore from backup is tested to a scratch instance, at least annually and before the first enterprise agreement, and the result written up: what was restored, from which backup, how long it took, and what failed. The test plan exists; the first exercise has not yet been run, and its report joins the evidence pack when it has. Testing that a backup restores is a different thing from testing that it exists.
- The automated integrity checks are the verification step, which is what allows a restore to be trusted: they establish that the recovered system is correct rather than merely running.
- A full disaster-recovery exercise runs annually against the written test plan, the first before the first enterprise agreement, with the report shared with enterprise customers on request.
The key-person question
Vaethra is a small company, and the material continuity risk is people rather than infrastructure.
- Nothing depends on anyone's memory. Schema, configuration, site content and deployment are all in source control, and the repository carries the reasoning behind each decision as well as the change itself. That documentation is a continuity control.
- Credential recovery is not to depend on one person being reachable. Recovery codes for every platform exist today, on paper and in an end-to-end-encrypted drive, and a second holder is named in the access review. The key-person risk stays in the risk register, because a second holder is a mitigation and not a second engineer.
- Enterprise agreements can include source-code escrow through an escrow agent engaged for that agreement, releasing on defined insolvency or cessation events.
- The data is reproducible. Every record derives from a public source that still exists.