Control planes fail with 401 pulling Helm charts from a private registry, but images pull fine

Last updated: October 28, 2026

Applies to: Self-hosted Spaces where registry points at a private registry (Artifactory, Harbor, ACR, Nexus, …) and imagePullSecrets references upbound-pull-secret. Verified against Spaces 1.17 and 1.18.

Symptom

  • New control planes don't become ready, or existing ones stop reconciling. The control plane conditions or the spaces-controller logs in upbound-system show an error like:

failed to pull chart with name "vcluster-chart" and version "<version>": ... 401 Unauthorized

  • Container images from the same registry, using the same pull secret, pull without problems.

  • It often appears right after a Spaces upgrade or a spaces-controller restart, even though the pull secret "hasn't changed" in a while.

Cause

Spaces deploys some per-control-plane components (for example vcluster) as Helm charts. The Spaces controller pulls those charts from your registry itself; the kubelet isn't involved. To authenticate, Spaces:

  1. Reads the pull secret configured in imagePullSecrets (upbound-pull-secret), which it copies into each control plane's host namespace (mxp-<id>-system) as upbound-system-pull-secret.

  2. Finds the auths entry that matches the registry.

  3. Logs in with that entry's username and password fields.

Spaces doesn't decode the auth field. The kubelet accepts either auth or username/password, so a secret that contains only auth works for container images. For chart pulls, Spaces finds no credentials, pulls anonymously and gets a 401.

Secrets created with kubectl create secret docker-registry contain all three fields, so they work. Secrets produced by other tooling often contain only auth: operators that copy a cluster-wide pull secret into every namespace, External Secrets templates, CI scripts.

Why it appears "out of nowhere"

The Spaces controller caches pulled charts locally and only pulls a chart again when its version changes (a Spaces upgrade) or the cache is gone (the controller pod restarted). If the secret was replaced with an auth-only version after the chart was cached, nothing fails until the next upgrade or restart.

Fix

Make sure the auths entry for your registry host contains username and password. Keep auth as well, because the kubelet uses it for images. Several registries are fine: give each host its own entry, each with all three fields.

apiVersion: v1
kind: Secret
metadata:
  name: upbound-pull-secret
  namespace: upbound-system
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: |
    {
      "auths": {
        "my-registry.example.com": {
          "username": "<user>",
          "password": "<password-or-token>",
          "auth": "<base64 of user:password-or-token>"
        },
        "my-other-registry.example.com": {
          "username": "<other-user>",
          "password": "<other-password-or-token>",
          "auth": "<base64 of other-user:other-password-or-token>"
        }
      }
    }

The simplest way to get the right shape for a single registry is:

kubectl create secret docker-registry upbound-pull-secret \
  -n upbound-system \
  --docker-server=my-registry.example.com \
  --docker-username=<user> \
  --docker-password=<password-or-token>

If a separate tool manages your cluster-wide pull secret and only writes auth, have it write a dedicated upbound-pull-secret in upbound-system in the shape above, rather than reusing the generic secret.

Verify

Check which fields each entry of the copy in a control plane's host namespace has. Every relevant host should list auth, password and username:

kubectl get secret upbound-system-pull-secret -n mxp-<id>-system \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq '.auths | map_values(keys)'

How the registry entry is matched

  • Spaces compares each auths key against the registry Helm value by prefix, ignoring any https:// scheme. my-registry.example.com, https://my-registry.example.com/ and my-registry.example.com/mirror all match registry: my-registry.example.com/mirror. A key with a path that isn't a prefix of registry doesn't match.

  • Avoid several keys that are all prefixes of the same registry value (for example my-registry.example.com and my-registry.example.com/mirror). Spaces uses the first match it finds, not the longest one, and the order isn't guaranteed. Make sure overlapping entries carry the same credentials, or keep only one.

  • Entries for other hosts are ignored, so a multi-registry secret is fine.