The "hcloud-cloud-controller-manager" (HCCM) is the bridge between your Kubernetes cluster and the Hetzner Cloud and Robot APIs. It implements the interfaces defined upstream in Kubernetes’ "k8s.io/cloud-provider" package, which lets cloud providers plug their own logic into a shared set of controllers. In HCCM, four of these controllers do the work.
The four controllers behind HCCM
The node controller initializes each Kubernetes node, attaching metadata such as the instance type and, crucially, the provider ID. That provider ID is the ID of your Hetzner Cloud or Robot server, prefixed with "hcloud://" or "hrobot://". It is immutable and serves as the unique link between a Kubernetes node and its counterpart in the API. Beyond initialization, the node controller also runs a status-update loop every five minutes by default and is tunable via "--node-status-update-frequency". An uninitialized node makes no API call, but each already-initialized node triggers one call to "InstanceV2.InstanceMetadata".
The node lifecycle controller watches Kubernetes nodes that have stopped being "ready". By default it checks every five seconds whether such a node still exists in the cloud provider and whether it has been shut down. If a node is shut down, the controller applies a corresponding taint. If the associated Hetzner Cloud or Robot Server no longer exists, the node object is deleted.
The routes controller adds one route per node to the Private Network, mapping each Kubernetes node’s Pod CIDR to its private IP. For example, if node-1 has private IP "10.0.0.2" and Pod CIDR "10.244.0.0/24", HCCM creates the route "10.244.0.0/24 → 10.0.0.2". Traffic destined for any pod on node-1 is then routed directly to that node over the Private Network. This avoids the overhead for an overlay network like VXLAN.
Finally, the service controller creates and configures Hetzner Cloud Load Balancers for every Kubernetes "Service" of type "LoadBalancer".
Each of these controllers talks to the Hetzner APIs, and that traffic counts against your account’s rate limit. The work described below was aimed at cutting unnecessary requests without sacrificing correctness.
A shared cache for the node controllers
First we turned to the node and node lifecycle controllers. Both are built on the cloud provider’s InstanceV2 interface, which exposes a node’s metadata ("InstanceMetadata"), whether it has been shut down ("InstanceShutdown"), and whether it still exists ("InstanceExists").
These controllers get especially noisy during scaling events. The node controller generates a lot of traffic during scale-up and scale-down, and the node lifecycle controller adds to it. The lifecycle controller can be particularly aggressive when a group of nodes is shut down: every five seconds it checks whether each node still exists ("InstanceExists") and, if so, whether it is shut down ("InstanceShutdown"). Since those are two sequential method calls, every check costs two API requests. That repeated pattern is exactly where a cache earns its keep.
A scale-up also needs information about every Kubernetes node, which suggested a particular design: a cache that pulls all servers in a single request, holds them for a configurable interval, and refreshes them together rather than one at a time. The routes controller already had a cache of this kind, so we reworked that cache to fit our needs and made it shareable across the controllers.
The default TTL is ten seconds and can be adjusted. The caching strategy is configurable too: the cache can pull and store each server individually instead of all at once. We fetch all servers by default, because a single batched request is what cuts API usage significantly.
The result is that bursts of activity that previously translated into a flood of individual API calls now resolve against a single shared, periodically refreshed view of the cluster’s servers, keeping HCCM well within your rate limits even when the cluster is in motion.
Starting with the routes controller
The routes controller was the next target. It reconciles routes on a fixed timer. The upstream default is every ten seconds, though HCCM has used a 30-second interval since v1.14.2. The problem is that every reconcile, change or no change, forces a call to the Hetzner Cloud API to fetch the Private Network and enumerate its routes. On a stable cluster where nothing is happening, the controller still burns at least one API request every 30 seconds for no benefit.
The routes controller now reconciles only when there’s a reason to: when nodes are added or removed, or when a defined set of node properties changes. For the many clusters that are mostly static and don’t use autoscaling, this removes the steady drip of pointless requests entirely.
As a safety net, we also run a periodic reconcile every 12h to 24h. This interval is chosen randomly at the controller’s start time.
Working upstream
The routes controller change came with a complication: the code isn’t ours. The routes controller lives in shared upstream Kubernetes under SIG Cloud Provider, the special interest group responsible for the interfaces every cloud integration builds on. Changing how it behaves meant changing it for everyone that plugs into the same controllers — and that is not something you land with a pull request and a thumbs-up.
We joined the SIG’s biweekly meeting to make the case, and the reception was warm. Other controllers had already moved to an event-based approach, so the idea was familiar, and the members were glad to see someone willing to take the topic on. The proposal was accepted with two conditions: it needed a Kubernetes Enhancement Proposal, and the new behavior had to sit behind a feature gate.
Both conditions are worth understanding, because the reasoning behind them is the reasoning behind how Kubernetes manages change at all. A KEP is the project’s mechanism for proposing and coordinating new work: a structured document covering motivation, design, a test plan, and graduation criteria, reviewed in the open before any code is written. The feature gate requirement is about blast radius. Event-driven reconciliation changes the behavior of code that dozens of providers depend on, and a gate means it can ship disabled, be enabled cautiously, and be switched back off if something goes wrong — without forcing the new behavior on anyone who isn’t ready for it.
The real friction wasn’t the idea but the paperwork around it: learning the templates and tracking down the documents that describe each process. Still, the proposal became KEP-5237, which has since merged in the alpha stage. With it accepted, we implemented the change, and since HCCM v1.29.0 the feature gate has been enabled by default.
What carried the process through was staying present. Joining the biweekly meetings and giving regular updates on the KEP kept the SIG members engaged and made it far easier to get feedback, reviews, and the final approval. They were generous with guidance and support the whole way — a reminder that upstream work is as much about showing up consistently as it is about the code.
Measuring the impact
The measurements below exclude the "GET /actions" calls to the Hetzner Cloud API, since these endpoints depend on the latency of other resources — like creating a route in a Private Network.
Two scenarios were measured: scaling from zero to ten workers alongside a single control-plane node, and a six-minute window on an already-running cluster of one control-plane and ten workers. The six-minute window was chosen deliberately, so that the node controller’s five-minute status-update loop fires at least once within the measurement.
The two optimizations pay off in different places. On a static cluster, the routes controller’s timer-driven reconciles are the dominant source of traffic, so the feature flag alone cuts steady-state requests from 31 to 12. During a scale-up the flag barely moves the number (114 to 109), because that burst comes from the node controllers rather than the routes controller. The shared cache is what handles that case, collapsing the per-node lookups into a single batched view and bringing scale-up traffic down to 27.
Wrapping up
The two changes target the biggest sources of avoidable traffic: the routes controller’s timer-driven reconciles on idle clusters, and the node controllers’ per-node lookups during scaling. The feature flag keeps static clusters quiet; the shared cache absorbs scaling bursts. Together they cut scale-up requests against the Hetzner Cloud API by roughly 76% and steady-state requests by 94%, without changing what the controllers guarantee.
If you want this, you can use HCCM version v1.33 to have all the features mentioned here.
For more information about HCCM, check out the GitHub project.
