Prevent error-state resources from starving reconciliation — configure ProviderConfig reconciliationPolicy (exponential backoff)

Last updated: July 21, 2026

Issue

  • At scale, managed resources stuck in a permanent error state keep retrying and consume the shared reconciliation budget

  • This starves healthy/new managed resources — they stop reconciling, so new create/update changes are not applied to the cloud

  • Common in large multi-tenant environments with many MRs (e.g. 100+ database instances), where error-state resources are owned by other teams and not fixed quickly

  • Increasing the reconcile rate / poll interval as a workaround causes cloud-provider API throttling

Environment

  • Crossplane / UXP with official (upjet-based) providers

  • reconciliationPolicy API available in provider families v2.6.0+ (AWS, GCP, Azure) and provider-azuread v2.3.0+

  • Available on the v2 release branches of the official providers only

Resolution

  • Configure a ProviderConfig reconciliationPolicy.exponentialFailureRateLimiter to back off error-state MRs more aggressively, freeing reconcile workers for healthy/new resources

  • Example:

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: long-delay
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: creds
  reconciliationPolicy:
    exponentialFailureRateLimiter:
      baseDelay: "1m"
      maxDelay: "60m"

  • Point the relevant managed resources at that ProviderConfig:

spec:
  providerConfigRef:
    name: long-delay

  • Constraints: baseDelay minimum 1s, maxDelay minimum 60s

  • You can define multiple ProviderConfigs (e.g. a dedicated long-delay policy) and reference them from specific MRs

  • Requires official providers on the v2 branch — upgrade to v2 to use this API

  • Verify: error-state MRs retry less frequently and healthy/new MRs reconcile without starvation (error-state count should drop as workers are freed)

Cause

  • Official providers use a binary exponential backoff for failing MRs, but the maximum delay was effectively capped/hardcoded, so error-state MRs kept retrying often

  • With a limited number of reconcile workers, many constantly-failing MRs consume the budget and block healthy/new MRs — a fairness/starvation problem

  • The new reconciliationPolicy API makes the backoff configurable, so error-state MRs can be de-prioritized without raising the global reconcile rate (which would trigger cloud API throttling)