How do I back up and restore a Crossplane v2 control plane with Velero?

Goal

Restore a Crossplane v2 control plane from a Velero backup with the original managed resources adopted and no duplicate cloud resources.

Environment

• Crossplane or UXP v2.x, backed up with Velero

• Confirmed with provider-nop on kind (Kubernetes v1.37, Crossplane v2.4.2, Velero v1.18.4, UXP v2.4.2-up.1). Duplicate managed resources and types missing after a CRD change were also confirmed with provider-aws-ec2 and provider-aws-s3 v2.9.0 on EKS 1.33 (Velero v1.18.2)

• Not covered: Spaces control plane backups, Space-level backups and Azure Backup for AKS

XRDs are served under two versions, and backups store the wrong one for v1 XRDs:

Prerequisites

• Only for Velero: Velero on the source, and a velero CLI that can reach the object store

• Cluster admin on the target

• If the source control plane still runs (DR drill, migration): stop it reconciling first, or both control planes manage the same cloud resources. Turn off GitOps self-heal for its Crossplane and providers, scale Crossplane core to 0, then the provider Deployments; while core runs, it scales providers back up

• Only for the Git procedure: Velero on the target with the same backup storage location; Crossplane, providers, functions, XRDs and Compositions in Git or Helm, at the versions the source ran; a way to pause GitOps sync

• Only for the empty-target procedure: an empty cluster

• Only for UXP Backup/Restore: the UXP Standard plan or higher; BackupConfig, credentials and backup ClusterRole on the target as in the UXP docs; v1 XRDs in Git; a way to pause GitOps sync

Steps

We recommend UXP Backup/Restore over Velero: it skips package TLS Secrets and imports paused, so it avoids the TLS and duplicate-resource failures below; follow the UXP steps at the end. With Velero, never restore while Crossplane is running: keep it at 0 replicas during the restore, exclude pods and ReplicaSets so the backup cannot start an old Crossplane pod, then scale it back up.

Back up after Velero has seen every type

1. Restart Velero after installing a provider or creating an XRD

   Velero refreshes API discovery at startup and then every 5 minutes, and a backup does not refresh it. Restart it and wait for the backup storage location to report Available:

   kubectl -n velero rollout restart deployment/velero
   kubectl -n velero rollout status deployment/velero
   kubectl -n velero get bsl <bsl-name> -o jsonpath='{.status.phase}'

2. Back up every namespace and the cluster-scoped resources

   Claims, namespaced XRs and managed resources live in your own namespaces, so back up all of them, plus crossplane-system and cluster-scoped resources:

   velero backup create <backup> --include-cluster-resources=true \
     --exclude-namespaces velero,kube-system,kube-public,kube-node-lease --wait

   Check that your managed resource and XR types are in the item list of velero backup describe <backup> --details. Record the number of managed resources (kubectl get managed -A --no-headers | wc -l) for Verify.

Restore into a target with Crossplane from Git

1. Pause GitOps for claims and XRs before anything syncs

   Disable auto-sync and self-heal for anything that applies claims or XRs. Claims or XRs created while Crossplane runs compose new managed resources.

2. Install Crossplane and its packages on the target from Git

   Install Crossplane, providers, functions, XRDs and Compositions at the versions the source ran. Wait for packages Healthy and XRDs Established. This avoids the v1 XRD rejection and the package TLS mismatch under Notes.

3. Scale Crossplane to 0

   Disable self-heal for the Crossplane release first, or GitOps scales it back up.

   kubectl -n crossplane-system scale deployment crossplane --replicas=0

   Providers keep running and adopt restored managed resources by their crossplane.io/external-name. While Crossplane is at 0, its crossplane-no-usages webhook rejects deletes of objects labelled crossplane.io/in-use.

4. Restore without pods and ReplicaSets

   Velero restores pods and ReplicaSets before Deployments, so the backup's own Crossplane pod would otherwise run during the restore. Deployments recreate both.

   velero restore create <restore> --from-backup <backup> \
     --exclude-resources pods,replicasets.apps --wait
   velero restore describe <restore> --details

   Expect Completed, no errors, and already exists warnings for the objects from step 2.

5. Scale Crossplane back up and re-enable GitOps

   Use the previous replica count (1 by default):

   kubectl -n crossplane-system scale deployment crossplane --replicas=1

If the target cluster is empty: restore Crossplane from the backup too

This restores Crossplane, its packages and the matching root CA from the backup instead of Git. It needs a resource modifier that keeps Crossplane at 0 during the restore, and Crossplane's Helm release then comes from the backup rather than a fresh install. Pods and ReplicaSets are excluded, so pods that rely on Velero file-system volume restores need a separate restore.

1. Install Velero on the target with the same backup storage location

2. Create the resource modifier

   The first rule restores v1 XRDs at v1, the second restores the crossplane Deployment with 0 replicas:

   # crossplane-restore.yaml
   version: v1
   resourceModifierRules:
   - conditions:
       groupResource: compositeresourcedefinitions.apiextensions.crossplane.io
       matches:
       - path: /spec/scope
         value: LegacyCluster
     patches:
     - operation: replace
       path: /apiVersion
       value: apiextensions.crossplane.io/v1
   - conditions:
       groupResource: deployments.apps
       resourceNameRegex: "^crossplane$"
       namespaces:
       - crossplane-system
     patches:
     - operation: replace
       path: /spec/replicas
       value: "0"

   kubectl -n velero create configmap crossplane-restore --from-file=crossplane-restore.yaml

3. Restore with the modifier, without pods and ReplicaSets

   Velero restores pods and ReplicaSets before Deployments, so without the exclusion the backup's own Crossplane pod runs during the restore.

   velero restore create <restore> --from-backup <backup> \
     --exclude-resources pods,replicasets.apps \
     --resource-modifier-configmap crossplane-restore --wait
   velero restore describe <restore> --details

   Expect Completed, no errors, and already exists warnings for cluster defaults. Until step 4, deletes of objects labelled crossplane.io/in-use fail, as in the Git procedure.

4. Scale Crossplane up

   Use the previous replica count (1 by default):

   kubectl -n crossplane-system scale deployment crossplane --replicas=1

If you use UXP Backup/Restore: restore a pinned run with v1 XRDs applied

UXP Backup/Restore needs no scale-down, but two known issues each need a step here (see Notes).

1. Record the run name when the backup completes

   On the cluster that took the backup, right after it completes, and keep it outside the cluster:

   kubectl get backups.admin.uxp.upbound.io <backup-name> -o jsonpath='{.status.backupDirectory}'

   On a cluster that was itself restored, this can point to a partial run.

2. Pause GitOps for claims and XRs

   UXP does not pause GitOps for you. Disable auto-sync and self-heal for anything that applies claims or XRs.

3. Apply v1 XRDs on the target from Git

   Without them, a restore with LegacyCluster XRDs never completes.

4. Restore the recorded run

   apiVersion: admin.uxp.upbound.io/v1beta1
   kind: Restore
   metadata:
     name: <restore>
   spec:
     configRef:
       name: <backup-config>
     backupName: <backup-name>--<YYYYMMDD-HHMMSS>

   Expect the Restore to reach Completed.

Verify

Compare the inventory with the source

Claims bound, XRs Ready, and the number of managed resources you recorded at backup time (skip claim if no XRD defines claims):

kubectl get claim -A
kubectl get composite -A
kubectl get managed -A --no-headers | wc -l

A higher managed resource count means duplicates; see Notes.

Confirm composition works

About 3 minutes after the restore, check for composition errors:

kubectl get events -A --field-selector reason=ComposeResources

Warnings from the first minute after scale-up, such as does not have a valid function pipeline, clear once functions are ready. Repeating x509 errors are the package TLS mismatch under Notes; they start about 2 minutes after the restore.

After a UXP restore, confirm nothing is left paused

kubectl get managed,composite,claim -A -o json \
  | jq '[.items[] | select(.metadata.annotations["crossplane.io/paused"] == "true")] | length'

Expect only resources you paused yourself before the backup; the restore leaves those paused. If no XRD defines claims, drop ,claim: kubectl rejects the unknown type.

Notes

If a backup or restore misbehaves

Velero rejects v1 XRDs on restore

• Condition: an XRD created as apiextensions.crossplane.io/v1 with spec.scope: LegacyCluster (the v1 default), with or without claimNames, not present on the target.

• Symptom: restore PartiallyFailed with spec.scope: Unsupported value: "LegacyCluster": supported values: "Namespaced", "Cluster". Claims and XRs of that type are restored, but nothing reconciles them.

• Cause: the XRD CRD stores v1 and serves v2 without conversion. Velero backs up the preferred version, v2, whose schema rejects LegacyCluster.

• Fix: apply the XRDs from Git before the restore, as in the Git procedure (Velero skips existing objects), or restore them at v1 with the first rule of the empty-target resource modifier.

Restoring package TLS Secrets next to a new root CA breaks composition

• Condition: Crossplane is installed on the target before the restore, so it has its own crossplane-root-ca; providers and functions are not; the backup includes crossplane-system Secrets.

• Cause: Velero skips the existing crossplane-root-ca but restores <package>-tls-server (plus <package>-tls-client for providers) signed by the source root CA. Crossplane only issues a package certificate into an empty Secret and does not verify an existing one against the current root CA (crossplane#5456). Velero also restores the ProviderRevision without ownerReferences; until the Provider re-adopts it, Crossplane creates the provider's webhook configuration under the package's default name (for example validating-webhook-configuration) with the restored certificate as caBundle, and never updates it. It stays until that ProviderRevision is deleted, and breaks once the package Secrets are re-issued.

• Fix: install providers and functions before the restore (Velero skips existing Secrets), or use the empty-target procedure, where Velero restores the matching root CA. If already hit:

# 1. Delete the restored package TLS Secrets; Crossplane re-issues them
kubectl -n crossplane-system get secrets -l velero.io/restore-name=<restore> -o name \
  | grep -E -- '-tls-(server|client)$' | grep -v '^secret/crossplane-tls-' \
  | xargs kubectl -n crossplane-system delete
# 2. Delete webhook configurations created with the restored certificate
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o json \
  | jq -r '.items[] | select(any(.metadata.ownerReferences[]?; .kind == "ProviderRevision"))
      | select(.metadata.name | startswith("crossplane-") | not) | "\(.kind)/\(.metadata.name)"' \
  | xargs kubectl delete
# 3. Wait until the re-issued Secrets show DATA 3
kubectl -n crossplane-system get secrets | grep -E -- '-tls-(server|client) '
# 4. Restart all Deployments in crossplane-system except Crossplane core and the RBAC manager
kubectl -n crossplane-system get deploy -o name | grep -vE '/crossplane(-rbac-manager)?$' \
  | xargs kubectl -n crossplane-system rollout restart

Crossplane running during a restore duplicates managed resources

• Claims, whether the XRDs are pre-applied or restored in the same run: Velero restores types in alphabetical resource.group order, so a claim (networks) is usually restored before its XR (xnetworks). The claim creates the XR under its resourceRef name without spec.crossplane.resourceRefs, the XR composes new managed resources named from its new UID, and Velero then skips the backed-up XR as existing.

• XRs, v2 namespaced XRs included: an XR reconciled before its managed resources are restored composes new ones. Managed resource types whose resource.group sorts after the XR type are at risk; the more objects Velero restores in between, the wider the window.

The restore still ends Completed with 0 errors. New managed resources whose external name is generated can create duplicate cloud resources, and the originals are left without an owner. On EKS with provider-aws, a default restore created 3 extra VPCs and 2 extra subnets within 20 seconds, with every managed resource Ready; buckets, whose type sorts before the XR type, were adopted. Keep Crossplane at 0 during the restore.

Before deleting any of these managed resources, set deletionPolicy: Orphan on it, or the delete also removes its cloud resource. Don't delete the ownerless originals, they hold the real cloud resources. Deleting the new duplicates is not enough either: their XR composes them again. Contact support.

Backups taken within 5 minutes of a CRD change miss those types

Velero refreshes API discovery at startup and then every 5 minutes, and a backup does not trigger a refresh. A backup within 5 minutes of a provider install or XRD creation ends Completed with 0 errors and 0 warnings, without objects of the new types. Restart Velero first or wait 5 minutes.

A backup shortly after a type is deleted ends PartiallyFailed with the server could not find the requested resource. It still contains the other objects and can be restored.

Restored ProviderConfigUsages make providers log forbidden errors

Velero and UXP Backup/Restore both restore ProviderConfigUsages without their owner reference. Providers can then repeatedly report cannot delete ProviderConfigUsage: ... forbidden on their ProviderConfig, because their default RBAC has no delete on usages; a provider with broader rights deletes the orphans silently. To clean up, see ProviderConfigUsages stuck in deletion.

UXP Backup/Restore avoids the TLS and duplicate failures

In UXP v2.4.2-up.1 the import restores Secrets, XRDs and packages first, waits for packages Healthy and XRDs Established, imports claims, XRs and managed resources with crossplane.io/paused: "true", then unpauses XRs and claims, then managed resources. Names are preserved and nothing is duplicated. crossplane-root-ca and package TLS Secrets are not backed up; the target issues its own. GitOps is not paused for you; disable it for claims and XRs during the restore.

UXP Backup/Restore stalls on v1 XRDs

An archived LegacyCluster XRD not present on the target makes the Restore keep retrying, and nothing after the XRDs is imported. After 15 failed attempts it also reports Failed, but it keeps retrying:

cannot apply resource CompositeResourceDefinition/<name>: timed out waiting for the condition
Retrying restore after failed attempt <n>/15: ...

The XRD is archived at v2, like with Velero, and the admission error is masked by the retry timeout. Apply the v1 XRDs from Git before the restore or while it retries; the next attempt completes. A fix is in progress; contact support for status.

UXP restores by plain backup name can pick a partial, paused snapshot

The archive contains the Backup object (default), archived without status.phase. The restored copy runs again during the restore and writes a snapshot of the partially imported, still-paused objects under the same name. A later Restore with a plain backupName, from any cluster with the same BackupConfig, resolves to that newest snapshot. It reports Completed, but the target only gets those objects, all paused; the rest are missing, and a Restore object in that snapshot runs again. The first restore is not affected. Every restore, pinned or not, writes another partial run. Restore the run you recorded at backup time (UXP steps above). Contact support for status.

Related

• UXP backup and restore

• Velero restore resource modifiers

• ProviderConfigUsages stuck in deletion

• crossplane/crossplane#5456: TLS certificates not regenerated after a CA change