Postgres disk fills up rapidly after a few days with Upbound Hub connected control planes

Issue

After running Upbound Hub for a few days with connected control planes, the Postgres database runs out of disk space and enters a CrashLoopBackOff. Inspecting the database reveals that inbox_messages_* weekly partition tables have grown to several gigabytes — nearly filling the entire PersistentVolumeClaim — with millions of rows all marked as processed. The growth rate can be 4–5 GB/day per a small number of connected control planes.

When Postgres becomes unavailable, hub-core is unable to process requests, which can cascade to connectors on connected control planes. Connectors may report errors such as invalid_grant: subject_token is invalid or expired, which can appear to be a token issue but is actually caused by hub-core being unable to respond while Postgres is down.

Environment

Upbound Hub v1.1.1 · BYO Postgres · EKS · Kubernetes 1.36

Observed with 2 or more connected control planes syncing a large number of resources.

Resolution

This is a confirmed issue. Contact Upbound support for the current status of a fix.

Immediate recovery steps if Postgres has already run out of disk:

Expand the Postgres PersistentVolumeClaim to restore database availability.
Drop the oversized processed partition to reclaim space. This is a destructive operation — confirm the partition contains only processed rows before dropping it. You can verify with:
```
SELECT status, count(*) FROM inbox_messages_<year>_<week> GROUP BY status;
```
Once Postgres is back online, hub-core will resume processing and connector errors should resolve automatically.

Useful queries for monitoring inbox partition growth:

Partition sizes:

SELECT tablename, pg_size_pretty(pg_total_relation_size('public.'||tablename)) as size
FROM pg_tables WHERE tablename LIKE 'inbox_messages_%' ORDER BY tablename;

Row count and status breakdown for a specific partition:

SELECT status, count(*) FROM inbox_messages_<year>_<week> GROUP BY status;

Overall database size:

SELECT pg_size_pretty(pg_database_size('hub')) as total_db_size;

While waiting for a fix, follow the Postgres sizing guidance linked in the Related section below to provision sufficient storage for your number of connected control planes and resources.

Cause

The Hub connector writes a message to the inbox_messages table for every resource change observed on connected control planes. In the affected versions, weekly inbox partitions are used instead of the intended finer-grained partitioning, and processed messages are not cleaned up. This causes the partitions to accumulate indefinitely, consuming disk at a rate proportional to the number of connected control planes and the number of managed resources.

A patch to restore the intended partitioning and cleanup behaviour is in progress. Contact Upbound support for the current status.