In a VPC deployment, Validio runs in your own Kubernetes cluster, so backups are yours to operate. This guide covers what to back up, how to restore it, and how to recover from a failed upgrade. For installation instructions, refer to VPC Deployment Installation.

Validio has no access to your cluster and does not back up your installation. For more information on what this means for your data, refer to Data Handling and Retention.

What to Back Up

Back up the following:

The Postgres persistent volume claim (PVC). It holds all platform data and configuration. Refer to Postgres service.
The Kubernetes Secrets, if the Helm chart created them. If you do not set existingSecret, the chart generates the secrets and they exist only in your cluster, so you must back them up alongside the PVC. If you pre-create the secrets through your own tooling, they are not part of the backup. That tooling must reproduce the original values rather than generate new ones, and must run before the installation starts.

You do not need to back up anything else. Redis is a cache and rebuilds itself, and every other resource is recreated when you reinstall Validio at the correct version.

🚧
The secrets must be the ones the backup was taken with
A restored Postgres volume carries its credentials inside it, and its encrypted fields can only be read with the encryption key that wrote them. In a fresh namespace the chart has nothing to look up, so it generates new values: a password that no longer matches the restored volume, and an encryption key that cannot decrypt its contents.
Nothing fails loudly when this happens. The installation starts normally and the encrypted fields are simply unreadable. Validio never has access to your encryption key and cannot recover those fields for you.

Choose a Backup Approach

Validio does not require a specific backup tool. Any approach works as long as it meets the following requirements:

Capture the Kubernetes resources and the volume data together. A backup needs both the namespaced resources, written out as manifests, and the contents of the persistent volumes. Capturing both in the same operation is what makes a namespace restorable as a working unit rather than a set of disconnected disk images. The volume data can be captured with cloud volume snapshots through CSI, or by copying the file contents out of the volume.
Scope backups by namespace and label. This lets you restore one Validio installation without affecting the rest of the cluster.
Back up whole namespaces rather than selected resources. It is easier to pick what you need at restore time than to discover that you never captured it.

We recommend the following:

Schedule a daily backup, and take an additional backup immediately before every upgrade.
Retain backups for 90 days in object storage, under a retention policy that blocks deletion and overwriting.
Record the Validio version alongside each backup. You need it at restore time, and working it out under pressure is avoidable.

Restore your Deployment

To restore an installation into the namespace it was backed up from, complete the following steps in order:

Stop the running installation
Delete the Postgres PVC
Restore the Postgres PVC from your backup
Verify the secrets
Resume the installation

These steps assume the validio namespace used in our examples.

Stop the running installation

If a continuous delivery or GitOps tool manages your installation, pause it first so that it does not recreate resources while you work. Then scale all deployments and statefulsets to zero:

kubectl -n validio scale deployment --all --replicas=0
kubectl -n validio scale statefulset --all --replicas=0

Confirm that no pods remain before you continue:

kubectl -n validio get pods

Delete the Postgres PVC

Delete the PVC by label selector, not by name. The name carries the Helm release prefix, which varies between installations.

kubectl -n validio delete pvc -l app.kubernetes.io/name=postgres

Restore the Postgres PVC from your backup

Restore the Postgres PVC using your backup tool.

The restored PVC may stay in the Pending state instead of binding straight away. This does not mean the restore failed. If your storage class uses the WaitForFirstConsumer volume binding mode, Kubernetes delays binding until a pod consumes the volume, which happens when you resume the installation in the final step. With a storage class that binds immediately, the PVC becomes Bound as soon as it is restored.

Verify the secrets

Confirm that the Kubernetes Secrets hold the same values they held when the backup was taken. In particular, verify the Postgres password and the encryption key. If they do not match, restore or recreate them with their original values before you continue.

Resume the installation

Scale the deployments and statefulsets back to their original replica counts. If a GitOps tool manages the installation, resume it instead and let it reconcile.

📘
If the namespace or the cluster is gone
When you are rebuilding from nothing rather than recovering a database, how you proceed depends on what your backup covers.
If you backed up the whole namespace, restore it in full. The secrets and the Postgres volume come back together and the workloads are recreated from the restored manifests, so the five steps above do not apply.
If your backup covers only the Postgres PVC and the secrets, restore the secrets first — a new installation has nothing to look up, so it generates fresh values instead. Then reinstall Validio at the version that was running when the backup was taken, and follow the five steps above.

Rollback is not Supported

Upgrades are one-way. Once you have upgraded, returning to an earlier Validio version is not supported, and you should not attempt a Helm rollback.

To recover from a failed upgrade, restore the backup that you took before the upgrade, into the same Validio version that the backup was taken at. Restoring a backup into a different version is not supported: the version you install must match the version recorded with the backup.

Because a restore is the only supported recovery path, take a backup immediately before every upgrade. A scheduled daily backup can be almost a full day stale by the time you upgrade. For upgrade instructions, refer to Upgrading VPC Deployments.

A Restored Deployment Starts Working Immediately

Sources return in whatever state they were in when the backup ran, which is normally Started. The restored installation therefore resumes processing against your real sources and destinations as soon as it comes up.

This is what you want in production, but it means you should run disaster recovery tests in a separate, non-production namespace rather than against your live installation.

If you would like Validio to review your backup configuration before you have to depend on it, contact us.