Crossplane/provider pods stuck in Pending or ContainerCreating after Spaces upgrade

Last updated: October 9, 2026

Issue

After a Spaces upgrade, one or more control planes stop reconciling:

  • Crossplane, provider, function, and/or rbac-manager pods are stuck in Pending or ContainerCreating, with no node assigned and no scheduling events.

  • ProviderRevisions/FunctionRevisions report:

    cannot get Crossplane service account: ServiceAccount "crossplane" not found

  • The crossplane and/or rbac-manager ServiceAccounts are missing from crossplane-system inside the control plane.

  • The ControlPlane resource still reports READY: True / Available, so the failure is not visible at the control plane level.

Environment

  • Spaces v1.13.x → v1.14.7 upgrade (confirmed trigger); also applies to v1.15.x, v1.16.x, and v1.17.x before their patch releases

  • Self-Hosted Spaces on EKS (and potentially other Kubernetes distributions)

  • UXP v1.19.0-up.1

  • Only a subset of control planes on a host is affected

Resolution

The fix is in Spaces v1.15.5, v1.16.2, v1.17.1, and v1.18.1. Upgrading to one of these prevents recurrence.

To recover an affected control plane now, force a clean reinstall of the Universal Crossplane (UXP) Helm release. This recreates the missing ServiceAccounts and their RBAC. No managed resources, CRDs, or Deployments are lost.

Do not recreate the ServiceAccounts manually. Doing so bypasses Helm's release state and can cause ownership and label drift on the next reconcile.

Run all commands below inside the affected control plane's context (for example up ctx <org>/<space>/<group>/<controlplane>), unless noted otherwise.

  1. Confirm the release name and the missing ServiceAccounts.

    helm -n crossplane-system list     # expect: universal-crossplane
kubectl -n crossplane-system get sa,pods

  2. Back up the UXP Helm release secrets.

    kubectl -n crossplane-system get secret \
  -l owner=helm,name=universal-crossplane -o yaml \
  > uxp-helm-secret-backup-$(date +%Y%m%d%H%M).yaml

  3. Delete all UXP Helm release secrets. (Destructive: this causes the Spaces controller to run a fresh install.) Delete every revision, not only the latest, so Helm finds no existing release.

    kubectl -n crossplane-system delete secret \
  -l owner=helm,name=universal-crossplane

  4. Wait for the Spaces controller to reinstall the release. To trigger it immediately, annotate the ControlPlane from its group context:

    kubectl annotate ctp/<controlplane> \
  internal.spaces.upbound.io/reconcile-requested-at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite

    Back in the control plane context, helm -n crossplane-system list shows universal-crossplane with a new UPDATED timestamp.

  5. Clear stuck revisions. Delete any Provider/FunctionRevisions still in backoff. Crossplane recreates them with fresh Deployments, ServiceAccounts, and TLS secrets.

    kubectl get providerrevision,functionrevision
kubectl delete providerrevision <name>
kubectl delete functionrevision <name>

  6. Verify recovery.

    kubectl -n crossplane-system get sa,pods
kubectl get providerrevision,functionrevision

    • crossplane and rbac-manager ServiceAccounts exist

    • All crossplane-system, provider, and function pods are Running

    • A sample managed resource shows READY and SYNCED

If you are not able to perform these steps yourself, contact Upbound Support for assistance.

Cause

During a Spaces upgrade, the virtual cluster behind each control plane is also upgraded. On a few control planes, that upgrade failed because an internal webhook was briefly unavailable while its pod was replaced. The recovery path for the failed upgrade removed the crossplane and rbac-manager ServiceAccounts and nothing restored them. Without them, Crossplane core can't start.

This is your cause if:

  1. The symptom appeared right after a Spaces upgrade.

  2. Only some control planes on the host are affected.

  3. The crossplane ServiceAccount is missing from crossplane-system.

  4. Other UXP resources (Deployments, ClusterRoles, ClusterRoleBindings) still carry their pre-upgrade creation timestamps. Only the ServiceAccounts are missing or newly created.