Why One Kubernetes Node Hit 96% Utilization While Another Sat at 48%

We were investigating a production Kubernetes application that started slowing down during periods of higher traffic.

The cluster was healthy, autoscaling was enabled, and capacity was still available. But when we checked the worker nodes, something immediately stood out.

• Node 1 — 96% utilization
• Node 2 — 78% utilization
• Node 3 — 48% utilization

One worker was almost saturated while another was using less than half of its available capacity.

The cluster could already autoscale between 2 and 10 nodes, so simply increasing the node count didn't explain what was happening.

Why was one worker reaching 96% utilization while another still had significant capacity available?

We Started With Pod Placement

Our first step was to find out where the application replicas were actually running.

kubectl get pods -n <namespace> -o wide

This showed us something node-level metrics alone couldn't: which application replicas were running on each worker.

Having three replicas does not necessarily mean Kubernetes will place one replica on each of three nodes.

For example:

replicas: 3

Without an explicit distribution rule, multiple replicas can be scheduled onto the same worker if the scheduler considers that placement valid.

Under normal traffic, that may not immediately cause a problem. But when several busy replicas are concentrated on the same worker, that worker can become a hotspot while other nodes still have available capacity.

It also creates a resilience concern. If several replicas of the same application are running on one worker, losing that worker can affect multiple replicas at the same time.

We Compared Requests With Actual Usage

Next, we compared node and pod utilization.

kubectl top nodes
kubectl top pods -n <namespace>

We also reviewed the CPU and memory requests configured for the application.

For example:

resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1"
memory: "1Gi"

These values are only examples. Production requests and limits should be based on the actual behavior of the workload.

Resource requests matter because Kubernetes uses them as an important input when deciding where a new pod can be scheduled.

The scheduler doesn't simply see:

Node 1 = 96%

Node 3 = 48%

and automatically move a running pod from Node 1 to Node 3.

Healthy running pods generally remain where they were scheduled.

So we needed to understand two different things:

1. What resources Kubernetes thought the workloads required.
2. What the workloads were actually consuming under production traffic.

Current CPU utilization and Kubernetes scheduling capacity are related, but they are not the same thing.

What We Found

The cluster wasn't simply running out of capacity.

The bigger issue was how application replicas were being distributed across the capacity already available.

There were three different Kubernetes mechanisms involved:

• HPA scales application replicas.
• Kubernetes Scheduler decides where new pods run.
• Cluster Autoscaler adds or removes worker capacity when required.

These mechanisms solve different problems.

HPA can create additional replicas when application load increases, but it doesn't decide which worker node those replicas should run on.

Cluster Autoscaler can add another worker when Kubernetes needs additional node capacity, but it doesn't continuously move healthy running pods away from a busy worker simply because another worker currently has lower utilization.

The scheduler controls pod placement.

That distinction was important.

We didn't just have a capacity question. We had a workload-placement problem.

How We Addressed the Workload Distribution

Instead of treating additional node capacity as the only solution, we introduced an explicit scheduling strategy for the application replicas.

For this type of workload, Topology Spread Constraints can be used to tell Kubernetes to consider replica distribution across worker nodes.

For a backend Deployment, that can look like this:

spec:
template:
metadata:
labels:
app: backend
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: backend

The same approach can be applied independently to frontend replicas using their own labels.

Why kubernetes.io/hostname?

The following setting is important:

topologyKey: kubernetes.io/hostname

It tells Kubernetes that individual worker nodes are the topology domains we care about when distributing matching replicas.

Without a distribution strategy, replicas may end up concentrated like this:

Node 1 Node 2 Node 3
● ● ● ● -

With an appropriate topology spread policy, the scheduler can work toward a distribution such as:

Node 1 Node 2 Node 3
● ● ● ●

This doesn't mean every worker will have identical CPU utilization.

It means the scheduler now has an explicit rule telling it to consider how matching application replicas are distributed across available workers.

Why maxSkew: 1?

We use:

maxSkew: 1

This limits how uneven the distribution of matching pods should become across eligible topology domains, according to the selected scheduling policy.

The objective is to avoid unnecessary concentration of replicas on a particular worker.

Why ScheduleAnyway?

The configuration uses:

whenUnsatisfiable: ScheduleAnyway

This makes the distribution preference softer.

Kubernetes should try to improve the spread, but the topology rule shouldn't unnecessarily prevent a pod from being scheduled when the ideal distribution isn't possible.

A stricter option such as DoNotSchedule can be useful for some workloads, but it can also leave pods Pending when the topology requirement cannot be satisfied.

The right choice depends on the workload and its availability requirements.

Resource Requests Still Matter

Topology spread is only one part of the scheduling picture.

CPU and memory requests also need to represent realistic workload requirements.

If requests are much lower than normal consumption, Kubernetes can schedule several workloads onto a worker based on an unrealistic picture of their resource requirements.

If requests are unnecessarily high, pods may struggle to schedule even when runtime metrics appear to show available capacity.

So the approach shouldn't be:

Add one YAML field and the problem is solved.

A better scheduling strategy combines:

• realistic CPU and memory requests;
• appropriate replica distribution;
• application autoscaling;
• cluster capacity autoscaling.

Each solves a different part of the problem.

Rolling Out the Change

After updating the workload configuration, the Deployment can be rolled out through the environment's normal deployment process.

For a manifest-based deployment:

kubectl apply -f deployment.yaml

The rollout can then be checked with:

kubectl rollout status deployment/<deployment-name> -n <namespace>

Once the new pods are running, the important check isn't simply whether their status is Running.

We also need to check where they are running.

kubectl get pods -n <namespace> -o wide

This allows us to verify whether newly scheduled replicas are being distributed across the available workers as expected.

Validating the Result

After changing scheduling behavior, we continue monitoring both the nodes and application pods.

kubectl top nodes
kubectl top pods -n <namespace>

We also watch what happens when application traffic increases and HPA creates additional replicas.

The complete flow becomes:

Traffic increases → HPA creates replicas → Scheduler places pods → Topology rules influence distribution → Cluster Autoscaler provides additional capacity when required

Understanding this flow is important because Cluster Autoscaler alone cannot solve every uneven-utilization problem.

Balanced Doesn't Mean Equal

One important point when looking at node utilization is that the goal isn't to transform:

96% / 78% / 48%

into:

70% / 70% / 70%

A healthy Kubernetes cluster does not require every worker to show exactly the same CPU or memory utilization.

Different pods handle different amounts of traffic and perform different types of work. Some variation between workers is completely normal.

What we want to avoid is a persistent hotspot where one worker repeatedly approaches resource exhaustion while other suitable workers still have significant available capacity.

That is the difference between normal utilization variance and a workload-distribution problem.

What This Incident Taught Us

At first glance, seeing a worker at 96% utilization makes additional capacity look like the obvious solution.

Add another node. Increase the instance size. Increase the autoscaling limit.

But that can hide the actual problem.

Before changing infrastructure capacity, it is worth understanding the responsibilities of the different Kubernetes components:

• HPA decides when the application needs more replicas.
• Scheduler decides where new pods should run.
• Topology Spread Constraints influence how matching replicas are distributed.
• Cluster Autoscaler provides additional worker capacity when Kubernetes needs it.

Once these responsibilities are separated, uneven node utilization becomes much easier to investigate.

Adding capacity may reduce immediate pressure.

It does not necessarily fix the reason workloads are becoming concentrated on particular workers.

The Takeaway

When one Kubernetes worker is approaching saturation while another still has plenty of available capacity, don't immediately assume that you need bigger nodes or more nodes.

Start by checking:

1. Where are the application replicas actually running?
2. What CPU and memory have those pods requested?
3. What resources are they actually consuming?
4. How is HPA behaving when traffic increases?
5. Does the workload have a scheduling strategy for replica distribution?
6. Is Cluster Autoscaler solving a capacity problem while the actual issue is workload placement?

Understanding those answers will tell you whether you have a capacity problem, a scheduling problem, or both.

More capacity can hide a scheduling problem. Better scheduling addresses it.