Crossplane function pod gRPC connectivity errors: "DeadlineExceeded" / "i/o timeout"

Last updated: July 21, 2026

Overview

You may encounter errors similar to the following when Crossplane attempts to run a Composition pipeline step:

cannot compose resources: cannot run Composition pipeline step "patch-and-transform": cannot run Function "crossplane-contrib-function-patch-and-transform": rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing: dial tcp <IP>:9443: i/o timeout"

This error indicates that Crossplane is attempting to connect to a stale gRPC endpoint for a function pod — typically after the pod has restarted or been rescheduled — and is unable to reach it, causing composition reconciliation to fail.

Root Cause

This issue is caused by Crossplane caching an outdated IP address for a function pod. After the pod restarts or is rescheduled, the cached endpoint no longer exists, resulting in connection timeouts. In managed Spaces environments, this condition can be triggered by an internal Kubernetes CA rotation or during a Spaces upgrade when a vcluster is uninstalled and reinstalled.

Impact

Affected control planes will fail to reconcile Compositions that use the impacted function (e.g., function-patch-and-transform). Resources may appear as "Not Synced" in the Upbound console even if they have been deployed in the backend.

Workaround (Self-Managed Control Planes)

If you manage your own control planes and have the necessary permissions, you can resolve this by performing the following steps. Order is important.

  1. Ensure your kubeconfig is pointed at the affected Managed Control Plane (MCP), not the Spaces host.

  2. Delete the stale function pod first:

    kubectl delete pod -n crossplane-system -l pkg.crossplane.io/function=function-patch-and-transform

  3. Then restart the Crossplane controller:

    kubectl rollout restart deployment crossplane -n crossplane-system

Deleting the function pod first ensures a healthy endpoint is available before Crossplane reconnects, preventing it from caching the stale endpoint again.

Managed / Dedicated Spaces Environments

If your control planes are managed by Upbound (Dedicated Spaces), you will not have the permissions required to perform the above steps yourself. In this case, please contact Upbound Support and provide the names of the affected control planes. The support team will:

  • Identify and remediate all affected control planes on your behalf.

  • Proactively check for additional control planes exhibiting the same behavior beyond those you have identified.

Permanent Fix

Upbound has addressed the underlying causes of this issue:

  • The helm provisioner bug that triggered the Kubernetes CA rotation has been fixed and deployed to production, with backports available for Spaces versions 1.15, 1.16, and 1.17.

  • The edge case where a vcluster could be uninstalled during a Spaces upgrade (which could re-trigger this condition) has also been resolved in recent patch releases.

  • Alerting has been added to notify Upbound's on-call engineers if this class of issue occurs again, enabling proactive detection and response.

If you are running an affected Spaces version, ensure you have upgraded to a patched release to prevent recurrence.

After Remediation

Once the affected control planes have been remediated, composition reconciliation should resume normally. Resources that were showing as "Not Synced" in the Upbound console should catch up automatically. If you continue to see unexpected behavior after remediation, please reach out to Upbound Support.