Backups & restore

The PDS keeps its records in one SQLite database and its blobs (media files) on the volume next to it, so backing those two things up is your disaster-recovery plan. Custos uses Litestream to stream the database to object storage continuously and restore it on boot, and its own blob mirror to do the same for blob files.

How it works

When the Litestream environment variables are set, the container runs the PDS under Litestream: it streams the SQLite write-ahead log to your bucket as writes happen (a continuous replica, not a nightly snapshot), and on boot it restores from that replica if the local database is missing. So a current restore point always exists.

The replica is defined in litestream.yml (committed in the repo) with force-path-style: false: virtual-hosted-style addressing, which Railway/Tigris-style buckets require.

Turning it on

Set these on the environment you want backed up (production; staging and local leave them unset and run the PDS directly):

Variable	Role
LITESTREAM_S3_BUCKET	The bucket the replica is written to. Setting this is what switches Litestream on.
LITESTREAM_S3_ENDPOINT	Object-storage endpoint (e.g. your Tigris/S3-compatible host).
LITESTREAM_ACCESS_KEY_ID	Access key for the bucket.
LITESTREAM_SECRET_ACCESS_KEY	Secret key for the bucket.

The bucket credentials are secrets

Store them in your platform’s secret manager, never in the repository or the image, the same discipline as the master key. Anyone with the replica has a full copy of your accounts’ data.

The replica is only half the picture

This backup protects the ciphertext. It does not protect the master key that decrypts it — back that up separately, in a different store than this replica. If the key is ever lost or compromised, follow the master-key disaster runbook instead of restoring from here alone.

Blobs: the bucket mirror

Litestream replicates only the SQLite database. Uploaded blobs (avatars, post images, video) live as files on the deployment volume, where losing the volume would destroy every account’s media. The blob mirror is the Litestream analogue for those files: a periodic sweep uploads every stored blob to an S3-compatible bucket, and on boot the server attempts to restore any file missing from the volume out of the bucket before it takes traffic. The restore is best-effort per blob: a blob whose bytes exist in neither place, or whose bucket copy fails content-hash verification, is logged loudly (per-CID error plus a boot summary count) and boot continues; that blob stays unavailable rather than blocking startup.

Turn it on by setting a bucket (unset means disabled):

Variable	Role
EZPDS_BLOB_MIRROR_BUCKET	Bucket the mirror writes to. Setting this is what switches the mirror on.
EZPDS_BLOB_MIRROR_ENDPOINT	S3-compatible endpoint URL.
EZPDS_BLOB_MIRROR_ACCESS_KEY_ID	Access key for the bucket.
EZPDS_BLOB_MIRROR_SECRET_ACCESS_KEY	Secret key for the bucket.

The sweep also propagates deletions, on a lag: after blob garbage collection or account deletion removes a blob’s database row, a subsequent sweep deletes the bucket object, so the mirror tracks the live blobstore rather than growing forever (the worst case of the lag is the bucket briefly retaining collected blobs). As a tripwire against acting on a wrong or empty database, a sweep that finds no blob rows at all while the bucket has objects skips delete propagation entirely. The full knob list (region, path-style addressing, key prefix, sweep interval) is in the configuration reference.

Same discipline as the database replica

Blob bytes are content-addressed but they are still your users’ data — keep the mirror bucket’s credentials in your secret manager, exactly like the Litestream bucket’s.

Blob integrity: the scrub sweep

A backup only helps if the bytes it protects are still good. A periodic scrub sweep re-hashes every stored blob against its recorded CID and size, and walks the blob directory for orphans in both directions: a database row whose file has gone missing, and a file that no row owns. Bitrot, a truncated write, or a bad restore surfaces as an operator alarm (the blob_scrub_* metrics and GET /v1/admin/health) months before a migration would trip over it.

When the bucket mirror is configured, the sweep can do more than report: a file that fails its hash or has gone missing is auto-healed from the mirror’s verified-good copy. Auto-heal is on by default and has no effect when the mirror is disabled: with no verified copy to pull from, a bad file is only ever flagged, never silently replaced. Both knobs (blob_scrub.interval_secs, default 6 hours; blob_scrub.auto_heal) are in the configuration reference.

Two guarantees close the loop around the sweep, so it catches genuine bitrot rather than gaps the write and read paths should have handled themselves:

• Writes are crash-durable. An uploaded blob is written to a temp file, fsynced, atomically renamed onto its content-addressed path, and the directory fsynced, before the database row is recorded. A crash or power loss can no longer leave truncated bytes at a valid path behind a row that already committed.
• Reads are verified. getBlob re-hashes a blob against its CID before serving it and returns a 404 (flagging the scrub alarm counter) on a mismatch, so a corrupted file is never handed to a downstream cache. A verified response carries the long-lived immutable cache header the blob spec recommends.

Restoring

To restore the database from the replica:

litestream restore <path-to-db>

Litestream pulls the latest state (or a point in time) from the bucket. On a fresh container the PDS does this automatically on boot when the local database is absent.

Rollback after a bad release

Schema migrations are forward-only; there is no down-path. Redeploying an earlier vX.Y.Z tag is safe only when the schema change was backward-compatible. If it wasn’t, roll back by restoring the database from the Litestream replica to a point before the promote, rather than by redeploying old code against a newer schema.

Inspect without rolling back

To look inside the replica without touching production, restore a throwaway copy and query it with sqlite3 — no rollback required. The repository’s operator debug kit documents this runbook.

Verify your restore path before you need it

A backup you’ve never restored is a hope, not a plan. Do a restore into a throwaway copy at least once so you know the bucket, credentials, and endpoint are right before an incident forces the question.