ProviderConfigUsages stuck in deletion: high volume of audit log events with "cannot delete resource providerconfigusages"

Last updated: October 15, 2026

Issue

After enabling audit logging on Spaces through SharedTelemetryConfig, a high amount of logs is generated for resource deletion in a loop. The ProviderConfigUsages appear to be stuck in deletion state. Example log event:

providerconfigusages.aws.upbound.io "<uid>" is forbidden: User "system:serviceaccount:crossplane-system:<provider-service-account>" cannot delete resource "providerconfigusages" in API group "aws.upbound.io" at the cluster scope

No system is down and no workflow is affected. The impact is log volume and log ingestion cost.

The same loop can also leave a ProviderConfig stuck in deletion: if you delete a ProviderConfig that has these usages, it never finishes deleting and the error keeps repeating.

Environment

  • Crossplane or UXP control planes, including control planes in Upbound Spaces

  • Cluster-scoped ProviderConfigUsages. Denied deletes were confirmed for provider-family-aws and provider-helm. Orphaned usages were also found for provider-family-gcp and provider-kubernetes. The API group in the error depends on the provider, for example helm.crossplane.io.

  • Most visible when audit logging is enabled, for example via SharedTelemetryConfig in Spaces

  • Recent change: the control plane's resources were migrated or imported from another control plane, or restored from a backup. The loop can go unnoticed until audit logging is enabled.

Resolution

This is a confirmed issue. Contact support for the current status of a fix. The steps below are the workaround.

Run these steps in each affected control plane, as a control plane admin. They cover cluster-scoped ProviderConfigUsages only.

  1. Set the context of the affected control plane and find the ProviderConfigs that are in the loop:

CTX=<control-plane-context>
kubectl --context "$CTX" get events -A --field-selector reason=UsageAccounting

  1. List the usages that have no controller owner reference and count them:

for crd in $(kubectl --context "$CTX" get crd -o name | cut -d/ -f2 | grep '^providerconfigusages\.'); do
  kubectl --context "$CTX" get "$crd" -A -o json | jq -r --arg t "$crd" '.items[]
    | select([.metadata.ownerReferences[]? | select(.controller==true)] | length == 0)
    | "\($t) \(.metadata.namespace // "-") \(.metadata.name) -> \(.resourceRef.apiVersion) \(.resourceRef.kind)/\(.resourceRef.name)"'
done | tee pcu-orphans.txt
wc -l < pcu-orphans.txt

  1. Check one of them. A usage is named after the UID of the resource that created it. Compare that name with the current UID of the resource it points to, using the kind and the group from the apiVersion in the step 2 output:

kubectl --context "$CTX" get <kind>.<group> <resource-name> -o jsonpath='{.metadata.uid}{"\n"}'

If the UIDs differ, the usage is a stale copy and the resource tracks its usage under its current UID. If the resource no longer exists, the usage is also stale.

  1. Run the deletion as a dry run, saving the output to a file:

for crd in $(kubectl --context "$CTX" get crd -o name | cut -d/ -f2 | grep '^providerconfigusages\.'); do
  kubectl --context "$CTX" get "$crd" -o json | jq -r '.items[]
    | select([.metadata.ownerReferences[]? | select(.controller==true)] | length == 0)
    | .metadata.name' |
  xargs -r kubectl --context "$CTX" delete "$crd" --wait=false --dry-run=server
done > pcu-dryrun.txt 2>&1

Check the result before going on:

grep -c ' deleted (server dry run)$' pcu-dryrun.txt
grep -v ' deleted (server dry run)$' pcu-dryrun.txt

The first command should print the same number as the wc -l count in step 2. Each line in the file should read providerconfigusage.<group> "<usage-name>" deleted (server dry run). If the count is different or the second command shows any error, do not continue.

  1. Destructive: delete the orphaned usages.

for crd in $(kubectl --context "$CTX" get crd -o name | cut -d/ -f2 | grep '^providerconfigusages\.'); do
  kubectl --context "$CTX" get "$crd" -o json | jq -r '.items[]
    | select([.metadata.ownerReferences[]? | select(.controller==true)] | length == 0)
    | .metadata.name' |
  xargs -r kubectl --context "$CTX" delete "$crd" --wait=false
done

This removes only ProviderConfigUsage objects, not managed resources or cloud resources. Any usage that is still needed is recreated with the correct owner reference on the resource's next reconcile. A paused managed resource recreates its usage only after it is unpaused. Until then, that resource does not count as a user of its ProviderConfig and does not block the ProviderConfig from being deleted.

  1. Verify. Step 2 should list no usages and print 0. Within a minute, step 1 should show no new warnings and the audit log events should stop.

Cause

Each managed resource records its use of a ProviderConfig in a ProviderConfigUsage that it owns. When a usage loses its owner reference, for example after a migration or a restore, the provider treats it as stale and tries to delete it. The provider has no delete permission on ProviderConfigUsages, so the request is denied. It is retried about every 30 seconds for each affected ProviderConfig, and each denied request is written to the audit log. The usages do not recover on their own. The log volume depends on the number of affected ProviderConfigs, not on the number of orphaned usages. The number of orphaned usages can also be far higher than the number of managed resources: in one case, a control plane with 49 managed resources had 7,240 orphaned usages, left over from resources that had been moved to other control planes.

Cause test: the error above repeats in audit logs or events, step 2 lists usages, and their names do not match the current UIDs of the resources they point to.