Bootstrapping Control Planes with GitOps

When a ControlPlane is created, there is a gap between when the resource is applied and when the virtual cluster is reachable. Manifests that must land inside the control plane — RBAC bindings, Crossplane provider installs, and similar — need to wait until the API server is healthy.

This guide describes the recommended GitOps pattern for sequencing that work: an app-of-apps root, the Spaces argocd-plugin as the readiness gate, and an ApplicationSet that bootstraps each control plane automatically.

Cluster topology

Identify which topology applies before wiring destinations:

• Co-located — Argo CD runs on the same cluster as Upbound Spaces. All resources (the ControlPlane, connection secrets, Argo CD Applications) are local.

• Remote — Argo CD runs on a separate hub cluster. Spaces runs on a management cluster. Each control plane is a third endpoint:

Remote topology affects where connection secrets are readable, where PostSync Jobs run, and how cluster C is registered on hub A. The argocd-plugin's externalCluster mode (below) is the simplest path when Argo CD is remote.

Readiness gate

Spaces sets conditions on the ControlPlane when provisioning completes:

For bootstrapping RBAC and most package installs, gate on Ready: True. Use Healthy: True only when bootstrap resources depend on providers being operational.

The connection secret (spec.writeConnectionSecretToRef) is populated with a kubeconfig key once the control plane is ready. GitOps tools use it to target the control plane cluster.

Recommended pattern: app-of-apps + argocd-plugin + ApplicationSet

The git repo is the source of truth. A root Application (app-of-apps) syncs argocd/apps/ from git; everything below it is managed declaratively from the repo.

root  (app-of-apps)  ──►  syncs argocd/apps/ from git
  │
  ├─ control-planes (Application)     control-planes/ ──► host cluster      creates CTPs
  │                                          │
  │                                          ▼  CTP Ready → argocd-plugin registers it
  │                                             as an Argo CD cluster (readiness gate)
  │
  └─ ctp-bootstrap (ApplicationSet)   matches the registered cluster by label
                                             │
                                             ▼  generates a bootstrap Application →
                                                bootstrap/ ──► CTP, in wave order

Organize your repo around three content areas plus the Argo CD control objects that wire them together:

my-repo/
├── argocd/apps/          # Argo CD Applications and ApplicationSets (synced by the root)
├── control-planes/       # ControlPlane manifests (synced to the Spaces cluster)
└── bootstrap/            # Sync-wave manifests (synced into each control plane)

Root Application (app-of-apps)

The root Application reconciles the Argo CD control objects in argocd/apps/ from git:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/my-repo
    targetRevision: main
    path: argocd/apps
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Remote topology: When Argo CD runs on hub A, the root Application lives on A. Any Application that targets Spaces resources on cluster B must set destination.server to B's API endpoint — not https://kubernetes.default.svc, which refers to the hub.

control-planes Application

Syncs ControlPlane manifests to the Spaces (host) cluster:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: control-planes
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/my-repo
    targetRevision: main
    path: control-planes
  destination:
    server: https://kubernetes.default.svc   # Spaces cluster (co-located)
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Each ControlPlane carries a label the ApplicationSet selects. The argocd-plugin propagates that label onto the registered cluster secret:

apiVersion: spaces.upbound.io/v1beta1
kind: ControlPlane
metadata:
  name: my-control-plane
  namespace: default
  labels:
    bootstrap.spaces.upbound.io/enabled: "true"
spec:
  writeConnectionSecretToRef:
    name: my-control-plane-kubeconfig
    namespace: default   # MUST match the ControlPlane namespace

ctp-bootstrap ApplicationSet

Generates one bootstrap Application per registered control plane that carries the bootstrap label:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ctp-bootstrap
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            bootstrap.spaces.upbound.io/enabled: "true"
  template:
    metadata:
      name: "bootstrap-{{nameNormalized}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/my-org/my-repo
        targetRevision: main
        path: bootstrap
      destination:
        name: "{{name}}"   # plugin-registered cluster (e.g. default/my-control-plane)
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

The ApplicationSet cannot generate an Application until the argocd-plugin registers the cluster secret — and the plugin only registers once the control plane is Ready. Registration is the readiness gate; no custom health check is required.

Bootstrap sync waves

Annotate bootstrap manifests with argocd.argoproj.io/sync-wave. Argo CD applies each wave, waits for health, then proceeds:

Example (wave 1 RBAC):

apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:name: platform-adminsannotations:
    argocd.argoproj.io/sync-wave: "1"subjects:- kind: Group
    name: oidc:platform-admins
    apiGroup: rbac.authorization.k8s.ioroleRef:kind: ClusterRolename: controlplane-admin     # NOT cluster-admin — see RBAC note belowapiGroup: rbac.authorization.k8s.io

Enable the argocd-plugin

The plugin is an alpha/preview feature. It and externalCluster mode ship in every Spaces chart 1.6.0 → 1.17.0; useUIDFormatForCTPSecrets needs ≥ 1.8.0.

features:
  alpha:
    argocdPlugin:
      enabled: true
      useUIDFormatForCTPSecrets: true   # UID secret names; avoids cross-namespace conflicts
      target:
        secretNamespace: argocd         # co-located: namespace where Argo CD runs

The plugin watches ControlPlane objects and writes an Argo CD cluster secret for each one once it reports Ready: True.

See Use Argo CD with Spaces for full plugin configuration.

External (remote) Argo CD

When Argo CD runs on a separate hub cluster, point the plugin at it with externalCluster settings and give it a kubeconfig to reach the hub:

features:
  alpha:
    argocdPlugin:
      enabled: true
      useUIDFormatForCTPSecrets: true
      target:
        externalCluster:
          enabled: true
          secret:
            name: my-argo-cluster      # Secret in upbound-system on the Spaces cluster
            key: kubeconfig            # kubeconfig with access to hub A

Equivalent Helm flags:

--set "features.alpha.argocdPlugin.enabled=true"
--set "features.alpha.argocdPlugin.useUIDFormatForCTPSecrets=true"
--set "features.alpha.argocdPlugin.target.externalCluster.enabled=true"
--set "features.alpha.argocdPlugin.target.externalCluster.secret.name=my-argo-cluster"
--set "features.alpha.argocdPlugin.target.externalCluster.secret.key=kubeconfig"

Create the referenced Secret in upbound-system (where the Spaces controller runs). The plugin writes each control plane's cluster secret onto hub A, so the app-of-apps root, the control-planes Application, and the ctp-bootstrap ApplicationSet all live on the hub — same pattern as co-located, with control-planes Application destination.server pointing at the Spaces cluster on B.

The registered endpoint is the externally reachable spaces-router address, not the in-cluster service.

Verifying external Argo CD

After enabling externalCluster, confirm the full loop from hub A through Spaces cluster B into control plane C:

Prerequisites

1. Hub cluster (A) with Argo CD ≥ 3.5 (or Pod resource exclusions on ≤ 3.4 — see pitfalls).

2. Spaces cluster (B) with argocd-plugin + externalCluster enabled and the hub kubeconfig Secret in upbound-system.

3. Network path from hub A to each control plane's spaces-router endpoint.

4. Git repo with the app-of-apps layout above; control-planes Application destination.server set to cluster B.

Checks

# Cluster secret appears on hub A after the control plane is Ready
kubectl --context <hub> get secrets -n argocd \
  -l argocd.argoproj.io/secret-type=cluster

# Root and children reconcile on hub A
kubectl --context <hub> -n argocd get applications.argoproj.io
kubectl --context <hub> -n argocd get applicationset

# ControlPlane reached Ready on B
kubectl --context <spaces> wait controlplane <name> -n <ns> \
  --for=condition=Ready=True --timeout=600s

# Bootstrap Application is Synced/Healthy on hub A
kubectl --context <hub> -n argocd get application bootstrap-<normalized-name> \
  -o jsonpath='{.status.sync.status}/{.status.health.status}{"\n"}'

# Sync waves applied inside the control plane (creation timestamps non-decreasing)
kubectl --context <ctp> get ns <bootstrap-ns> \
  -o jsonpath='wave 0: {.metadata.creationTimestamp}{"\n"}'
# repeat for wave 1–3 resources

Expected outcome

• Cluster secret on hub A labelled bootstrap.spaces.upbound.io/enabled: "true".

• root, control-planes, and bootstrap-<name> Applications Synced/Healthy on hub A.

• Bootstrap manifests present inside the control plane in non-decreasing wave order.

Alternative patterns

Use these when the argocd-plugin is unavailable or bootstrapping is a one-time action.

App-of-apps with a custom health check

A single parent Application creates the ControlPlane (wave 0) and a child bootstrap Application (wave 1). Add a Lua health check on spaces.upbound.io/ControlPlane to argocd-cm so Argo CD knows when wave 0 is healthy:

resource.customizations.health.spaces.upbound.io_ControlPlane: |
  hs = {}
  if obj.status and obj.status.conditions then
    for _, c in ipairs(obj.status.conditions) do
      if c.type == "Ready" then
        if c.status == "True" then
          hs.status = "Healthy"
          return hs
        else
          hs.status = "Progressing"
          return hs
        end
      end
    end
  end
  hs.status = "Progressing"
  return hs

Cluster registration for the child Application still requires the argocd-plugin or a PostSync hook.

PostSync hook for cluster registration

When the plugin is off, a PostSync Job on the Spaces cluster reads the connection secret locally and calls argocd cluster add against the hub API. This is more imperative but avoids a standing registration mechanism. See Use Argo CD with Spaces for a full Job manifest.

PostSync hooks suit one-off bootstrapping; the app-of-apps + ApplicationSet pattern is better for ongoing GitOps management.

Common pitfalls

Connection secret namespace

The secret from spec.writeConnectionSecretToRef is created on the Spaces cluster. It must be in the same namespace as the ControlPlane. Pick a namespace your GitOps tooling and bootstrap Jobs can read consistently.

Bootstrap idempotency

Write bootstrap manifests so re-sync succeeds silently. Use declarative apply semantics; avoid create-only resources.

ApplicationSet generates no Application

The argocd-plugin must propagate bootstrap.spaces.upbound.io/enabled onto the cluster secret. Verify:

kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster \
  -o jsonpath='{.items[*].metadata.labels}'

If the label is missing, adjust the ApplicationSet selector to match a label the plugin sets.

Argo CD ≤ 3.4 crash-loops against control planes

Argo CD v3.2–v3.4 panics in populatePodInfo on the control plane vcluster's LimitRange (upstream argo-cd#26529; fixed in Argo CD 3.5). The application-controller crash-loops; sync and terminate hang. Workaround until upgrade:

# argocd-cm
resource.exclusions: |
  - apiGroups: ['']
    kinds: ['Pod']

AddOns hit a hubRBAC ceiling

Crossplane Provider installs bootstrap cleanly. AddOns install via upbound-controller-manager, which under authorization.hubRBAC=true cannot receive all RBAC a chart requires — and the bootstrap identity has no escalate verb to grant it. Keep bootstrap packages to Providers and RBAC-light AddOns, or apply per-chart helm-values workarounds.

Remote: in-cluster targets the hub, not Spaces

When Argo CD is remote, destination.server: https://kubernetes.default.svc refers to hub A. Any Application managing Spaces resources must set destination.server to cluster B explicitly.

Remote: connection secret stays on the Spaces cluster

Anything that needs the kubeconfig — bootstrap Jobs, provider-kubernetes ProviderConfig — must run on cluster B or sync the secret to where it runs. Do not assume it is available on the hub.

Compatibility