Kubernetes OOMKilled: Debugging Exit Code 137 Without Just Raising the Limit

The 137 that everyone fixes wrong

OOMKilled with exit code 137 is the most casually mis-fixed failure in Kubernetes. The status is unambiguous — the kernel’s out-of-memory killer sent SIGKILL to your container because it went over its cgroup memory limit — and the reflex is equally unambiguous: double the limit, redeploy, move on. That works about half the time. The other half, you have just scheduled the same outage for next Tuesday, with more memory wasted per replica in the meantime.

The question worth thirty seconds of your time before you touch a manifest is simple: was the limit wrong, or is the application wrong? Everything else in this article is about answering that quickly.

Confirm it was actually an OOMKill

Exit code 137 means SIGKILL, and SIGKILL has other senders. A node draining, a kubectl delete pod --force, or a runtime shutdown timeout can all produce 137 without any memory involvement. Read the termination reason rather than the code:

kubectl get pod <pod> -n <ns> \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

If any line reads OOMKilled, that container is the one. This form matters in multi-container pods, where the killed container is often a sidecar — a log shipper buffering to memory during a backpressure event, or a mesh proxy holding connections — and not the application everyone is staring at.

Then get the shape of the failure from events and the restart count:

kubectl describe pod <pod> -n <ns> | sed -n '/Last State/,/Ready/p'
kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[*].restartCount}'

A restart count that climbs steadily on a fixed cadence is a leak announcing itself. A count that jumps only during traffic peaks is a limit that is too tight for peak, not for baseline.

Container limit or node pressure? They are not the same failure

Two very different events get discussed as “the pod ran out of memory.”

Container-level OOMKill. The container exceeded its own resources.limits.memory. The kernel kills the offending process inside that cgroup. The pod stays scheduled on the node and restarts in place. Status: OOMKilled.

Node-level memory pressure. The node itself is running out of allocatable memory, and the kubelet starts evicting pods to protect it. Status: Evicted, with a message about memory pressure, and the pod is rescheduled elsewhere. Eviction order follows QoS: BestEffort pods go first, then Burstable pods using more than their requests, and Guaranteed pods (requests equal to limits) last.

Check which one you are in:

kubectl get events -n <ns> --field-selector reason=Evicted
kubectl describe node <node> | grep -A5 'Conditions:\|Allocated resources'

If you are seeing evictions, no amount of tuning one workload’s limit will help. The real problem is that requests across the cluster do not reflect real usage, so the scheduler is overcommitting the node. That is a capacity and rightsizing problem, and it is the same problem that shows up on the bill — the Kubernetes cost optimization guide covers the rightsizing side of it.

Leak or undersized limit: read the curve

This is the whole diagnosis, and it takes one graph. Plot working-set memory for the container over the full lifetime of a pod, from start to kill:

container_memory_working_set_bytes{namespace="<ns>", pod=~"<pod>.*", container!="", container!="POD"}

There are only two shapes that matter.

A plateau. Memory rises during warmup, flattens, and sits at a stable level — which happens to be above your limit, or close enough that a traffic peak pushes it over. The application is behaving. The limit is wrong. Raise it to the observed plateau plus 20-30% headroom and you are done.

A ramp. Memory rises and keeps rising for as long as the pod lives, with no plateau, and the time-to-kill scales with the limit. That is a leak, an unbounded cache, or a queue nobody drains. Raising the limit changes nothing except how long you wait. Chase it in the application: heap dump on the JVM, --inspect and a heap snapshot on Node, pprof on Go.

The sawtooth in between — rises, drops sharply, rises again — is usually garbage collection working correctly against a heap ceiling that is set too close to the container limit. Fix the runtime configuration rather than the limit.

If you have no Prometheus, kubectl top gives you a poor but non-zero substitute, sampled by hand:

watch -n 30 'kubectl top pod <pod> -n <ns> --containers'

Working set is not RSS, and page cache counts

A recurring source of confusion: the number the kubelet enforces against is not the resident set size you see in ps. It is the cgroup’s working set — roughly total charged memory minus inactive file-backed pages that the kernel can reclaim under pressure.

The practical consequence is that file I/O can OOMKill a container that has a perfectly healthy heap. A job that writes a large file, or a service that reads a lot of data off disk, accumulates page cache charged to its cgroup. Under normal conditions the kernel reclaims it happily. Under a sharp allocation spike it may not reclaim fast enough, and the OOM killer fires against a container whose application memory never moved.

You can read the raw truth from the cgroup inside the container. On cgroup v2 nodes:

kubectl exec <pod> -n <ns> -c <container> -- sh -c \
  'cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current; cat /sys/fs/cgroup/memory.events'

memory.events carries an oom_kill counter, which is the least ambiguous evidence available that this cgroup has been killed for memory, and how many times. On older cgroup v1 nodes the equivalents are memory.limit_in_bytes and memory.usage_in_bytes under /sys/fs/cgroup/memory/.

The runtimes that ignore your limit

Managed runtimes size their heap against what they believe the machine has. If that belief comes from /proc/meminfo rather than the cgroup, a container with a 512Mi limit on a 64Gi node will happily plan to use several gigabytes of heap and get killed long before it ever collects.

Modern versions of most runtimes are container-aware, but the defaults are still frequently wrong for a tight limit. Set them explicitly:

# JVM: heap as a percentage of the *container* limit, leaving room for
# metaspace, thread stacks, direct buffers and the JVM itself.
env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=70 -XX:+UseContainerSupport"

# Node.js: old-space ceiling in MiB, well under the container limit.
  - name: NODE_OPTIONS
    value: "--max-old-space-size=768"

# Go 1.19+: soft memory limit that makes the GC work harder as you approach it.
  - name: GOMEMLIMIT
    value: "900MiB"

The rule of thumb that has saved me the most incidents: the heap ceiling should be 70-80% of the container limit, never 100%. The remaining 20-30% is not waste, it is the space where thread stacks, native allocations, JIT code caches, and page cache live. Set the heap equal to the limit and the runtime will do exactly what you told it to and get killed for it.

Python has no heap ceiling to set, so the lever there is allocator behaviour. Glibc’s per-thread arenas can balloon RSS in threaded workloads; MALLOC_ARENA_MAX=2 is a cheap first thing to try before you go looking for a leak that may not exist.

Requests, limits and the QoS class you did not choose

Setting a limit without a matching request is how a workload ends up in the Burstable class and first in line for eviction under node pressure. If a workload genuinely must not be killed — a stateful component, a leader, a queue consumer holding a lease — give it Guaranteed QoS by setting requests equal to limits:

resources:
  requests:
    memory: "1Gi"
    cpu: "500m"
  limits:
    memory: "1Gi"
    cpu: "2"

Note the asymmetry: memory request equals memory limit, CPU limit is higher than the request or absent entirely. Memory is incompressible — exceed it and you die — so a matched pair is protection. CPU is compressible; a tight CPU limit throttles rather than kills, and CPU throttling during startup is a classic way to turn a healthy container into a probe-failure crash loop. That exact interaction is the one covered in the CrashLoopBackOff debugging playbook, and it showed up for real in a CrashLoopBackOff production recovery where a new node group made startup slow enough to trip a probe that had been marginal for months.

Why blanket-raising limits costs you twice

Memory limits do not directly bill you — requests do, because requests are what the scheduler reserves and therefore what determines how many nodes you run. But limits and requests move together in practice, because a team that has been burned by OOMKills raises both, and nobody ever lowers them again.

The result is a cluster where every workload reserves two to four times what it uses, bin-packing gets worse, node count goes up, and the monthly bill grows without any traffic growth to explain it. On the AWS bill reduction engagement a meaningful slice of the ~42% saving was exactly this: requests that had drifted upward one incident at a time, rightsized against 30 days of real usage with sane headroom rather than fear.

The disciplined version is to let data set the numbers. Run the Vertical Pod Autoscaler in recommendation mode, where it observes and suggests but changes nothing:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"   # recommend only; you stay in control of the manifest

Read its recommendations after a couple of weeks, including a peak period, and set limits from the P99 of observed usage plus headroom. That is a number with evidence behind it rather than a number someone doubled at 2am.

Catch OOMKills before a human notices

Kill events are silent unless you watch for them. Two alerts cover most of it, both from kube-state-metrics:

- alert: ContainerOOMKilled
  expr: |
    increase(kube_pod_container_status_restarts_total[15m]) > 0
    and on(namespace, pod, container)
      kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
  for: 0m
  labels: { severity: warning }
  annotations:
    summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} was OOMKilled"

- alert: ContainerMemoryNearLimit
  expr: |
    container_memory_working_set_bytes{container!="", container!="POD"}
      / on(namespace, pod, container)
        kube_pod_container_resource_limits{resource="memory"} > 0.9
  for: 15m
  labels: { severity: warning }

The second one is the useful half. It fires before the kill, on the workload that is quietly walking toward the ceiling, which is when a fix is cheap and nobody is paged.

The checklist for the next OOMKill

1. Confirm the reason is OOMKilled, and identify which container — sidecars count.
2. Check whether the pod was Evicted instead. If so, this is node capacity, not this workload.
3. Plot container_memory_working_set_bytes for the pod’s full lifetime. Plateau or ramp?
4. Plateau: raise the limit to observed peak plus 20-30% headroom, and raise the request with it.
5. Ramp: it is a leak. Get a heap dump or profile. Do not raise the limit and call it fixed.
6. Check the runtime’s heap ceiling is 70-80% of the container limit, not equal to it.
7. If the workload must not be killed, give it Guaranteed QoS (memory request equals memory limit).
8. Look at whether heavy file I/O is charging page cache to the cgroup.
9. Add the near-limit alert so the next one is caught before the kill.

Most OOMKills are step 3. The teams that stay out of this loop are the ones that treat the memory curve as the diagnosis and the limit as the last thing they change, not the first.

If you are staring at a cluster full of restarts right now and want a second pair of eyes, that is infrastructure debugging and incident response work I do regularly. If the OOMKills have already been “fixed” by raising every limit and your bill is now the problem, that is cloud cost optimization.