Provider pods stuck in ImagePullBackOff (401 from a private registry) when the provider uses a custom ServiceAccount from a DeploymentRuntimeConfig

Last updated: November 20, 2026

Issue

  • After the nodes of the host cluster are replaced (for example during an EKS version upgrade with a new node group), provider pods in a control plane go into ImagePullBackOff

  • Image pulls fail with a 401 from the private registry, for example:

Failed to pull image "<registry>/<image>:<tag>": ... failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://<registry>/...: 401
  • Restarting the Crossplane workload does not help

  • Only some pods are affected. Pods that have imagePullSecrets keep running; every failing pod has no imagePullSecrets

  • The affected provider pods all use the same custom ServiceAccount, set through a DeploymentRuntimeConfig


Environment

  • Crossplane control plane (reported on Upbound Spaces 1.17.3) pulling provider images from a private registry that requires authentication

  • Provider installed with spec.runtimeConfigRef pointing to a DeploymentRuntimeConfig that uses a custom ServiceAccount

  • Seen after all nodes were replaced during an EKS 1.35 upgrade (new AL2023 node group)


Resolution

Run all commands below against the control plane (for example after up ctx <org>/<space>/<group>/<controlplane>), not against the host cluster.

  1. List the failing pods and the ServiceAccount each one uses:

kubectl -n crossplane-system get pods --field-selector=status.phase=Pending \
  -o custom-columns=NAME:.metadata.name,SA:.spec.serviceAccountName

Find the pull secret to use. Crossplane's own ServiceAccount already has it:

kubectl -n crossplane-system get sa crossplane -o jsonpath='{.imagePullSecrets}'

To see which Providers use a runtime config and which package pull secrets they have:

kubectl get providers.pkg.crossplane.io,functions.pkg.crossplane.io \
  -o custom-columns=NAME:.metadata.name,PULLSECRETS:.spec.packagePullSecrets,RUNTIME:.spec.runtimeConfigRef.name
  1. Check whether the custom ServiceAccount has a pull secret. If this returns nothing, this article applies:

kubectl -n crossplane-system get sa <custom-sa> -o jsonpath='{.imagePullSecrets}'
  1. Confirm the pull secret contains credentials for your registry:

kubectl -n crossplane-system get secret <pull-secret> \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths | keys[]'
  1. Add the pull secret to the custom ServiceAccount:

kubectl -n crossplane-system patch sa <custom-sa> --type merge \
  -p '{"imagePullSecrets":[{"name":"<pull-secret>"}]}'
  1. Delete the stuck pods that use the custom ServiceAccount. Kubernetes copies a ServiceAccount's imagePullSecrets into a pod only when the pod is created, so existing pods do not pick up the patch. The pods are recreated with the pull secret:

kubectl -n crossplane-system delete pods \
  --field-selector=spec.serviceAccountName=<custom-sa>,status.phase=Pending

Validated: after the pull secret was added to the ServiceAccount and the pods were restarted, the provider pods pulled their images and started.

Steps 4 and 5 are a workaround. For a permanent fix, use one of these options:

  • Add imagePullSecrets to the custom ServiceAccount where it is defined (for example in your Helm chart, GitOps repository or Terraform). This also keeps the manual patch from being reverted.

  • Set spec.packagePullSecrets on the Provider.

  • Use an ImageConfig with pullSecretRef. Crossplane adds this pull secret to the provider's runtime Deployment as well, not only to package pulls. Plan this across your whole estate before rolling it out.

Note: adding imagePullSecrets to the deploymentTemplate of the DeploymentRuntimeConfig does not work, because Crossplane overwrites it with the Provider's packagePullSecrets.


Cause

  • Crossplane copies the pull secrets of its own ServiceAccount onto the ServiceAccounts it creates for providers. When a DeploymentRuntimeConfig points the provider to an existing ServiceAccount through serviceAccountName, Crossplane leaves that ServiceAccount unchanged, so it does not get the pull secret.

  • The custom ServiceAccount existed but had no imagePullSecrets, so its pods could not authenticate to the private registry, while pods using the ServiceAccounts Crossplane creates for each provider revision were unaffected.

  • The problem most likely only appeared after the nodes were replaced because the old nodes still had the provider images cached. With the default imagePullPolicy: IfNotPresent, a node that already has the image does not pull it again, so the missing pull secret only causes a 401 on fresh nodes.