Provider pods stuck in ImagePullBackOff (401 from a private registry) when the provider uses a custom ServiceAccount from a DeploymentRuntimeConfig
Last updated: November 20, 2026
Issue
After the nodes of the host cluster are replaced (for example during an EKS version upgrade with a new node group), provider pods in a control plane go into
ImagePullBackOffImage pulls fail with a 401 from the private registry, for example:
Failed to pull image "<registry>/<image>:<tag>": ... failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://<registry>/...: 401Restarting the Crossplane workload does not help
Only some pods are affected. Pods that have
imagePullSecretskeep running; every failing pod has noimagePullSecretsThe affected provider pods all use the same custom ServiceAccount, set through a
DeploymentRuntimeConfig
Environment
Crossplane control plane (reported on Upbound Spaces 1.17.3) pulling provider images from a private registry that requires authentication
Provider installed with
spec.runtimeConfigRefpointing to aDeploymentRuntimeConfigthat uses a custom ServiceAccountSeen after all nodes were replaced during an EKS 1.35 upgrade (new AL2023 node group)
Resolution
Run all commands below against the control plane (for example after up ctx <org>/<space>/<group>/<controlplane>), not against the host cluster.
List the failing pods and the ServiceAccount each one uses:
kubectl -n crossplane-system get pods --field-selector=status.phase=Pending \
-o custom-columns=NAME:.metadata.name,SA:.spec.serviceAccountNameFind the pull secret to use. Crossplane's own ServiceAccount already has it:
kubectl -n crossplane-system get sa crossplane -o jsonpath='{.imagePullSecrets}'To see which Providers use a runtime config and which package pull secrets they have:
kubectl get providers.pkg.crossplane.io,functions.pkg.crossplane.io \
-o custom-columns=NAME:.metadata.name,PULLSECRETS:.spec.packagePullSecrets,RUNTIME:.spec.runtimeConfigRef.nameCheck whether the custom ServiceAccount has a pull secret. If this returns nothing, this article applies:
kubectl -n crossplane-system get sa <custom-sa> -o jsonpath='{.imagePullSecrets}'Confirm the pull secret contains credentials for your registry:
kubectl -n crossplane-system get secret <pull-secret> \
-o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths | keys[]'Add the pull secret to the custom ServiceAccount:
kubectl -n crossplane-system patch sa <custom-sa> --type merge \
-p '{"imagePullSecrets":[{"name":"<pull-secret>"}]}'Delete the stuck pods that use the custom ServiceAccount. Kubernetes copies a ServiceAccount's
imagePullSecretsinto a pod only when the pod is created, so existing pods do not pick up the patch. The pods are recreated with the pull secret:
kubectl -n crossplane-system delete pods \
--field-selector=spec.serviceAccountName=<custom-sa>,status.phase=PendingValidated: after the pull secret was added to the ServiceAccount and the pods were restarted, the provider pods pulled their images and started.
Steps 4 and 5 are a workaround. For a permanent fix, use one of these options:
Add
imagePullSecretsto the custom ServiceAccount where it is defined (for example in your Helm chart, GitOps repository or Terraform). This also keeps the manual patch from being reverted.Set
spec.packagePullSecretson the Provider.Use an
ImageConfigwithpullSecretRef. Crossplane adds this pull secret to the provider's runtime Deployment as well, not only to package pulls. Plan this across your whole estate before rolling it out.
Note: adding imagePullSecrets to the deploymentTemplate of the DeploymentRuntimeConfig does not work, because Crossplane overwrites it with the Provider's packagePullSecrets.
Cause
Crossplane copies the pull secrets of its own ServiceAccount onto the ServiceAccounts it creates for providers. When a
DeploymentRuntimeConfigpoints the provider to an existing ServiceAccount throughserviceAccountName, Crossplane leaves that ServiceAccount unchanged, so it does not get the pull secret.The custom ServiceAccount existed but had no
imagePullSecrets, so its pods could not authenticate to the private registry, while pods using the ServiceAccounts Crossplane creates for each provider revision were unaffected.The problem most likely only appeared after the nodes were replaced because the old nodes still had the provider images cached. With the default
imagePullPolicy: IfNotPresent, a node that already has the image does not pull it again, so the missing pull secret only causes a 401 on fresh nodes.
Related
Crossplane/provider pods stuck in Pending or ContainerCreating after Spaces upgrade: https://help.upbound.io/articles/6761262300