Kubernetes · Field notes
KubeGlance Blog
Practical Kubernetes troubleshooting, kubectl deep-dives, and cluster operations — from the team building KubeGlance.
Giving an AI coding agent read-only access to your Kubernetes cluster
Give an AI coding agent its own read-only Kubernetes identity: a ServiceAccount config bound to a get/list/watch ClusterRole, checked with kubectl auth can-i.
PVC stuck in Pending: the four messages, and the one that is not an error
A PVC stuck in Pending says why in its events: no storage class is set, waiting for first consumer, waiting for pod to be scheduled, or ExternalProvisioning.
Init:0/2, Init:CrashLoopBackOff, Init:ErrImagePull: the Init: prefix in kubectl's STATUS column
Init:ErrImagePull, Init:CrashLoopBackOff and Init:0/2 all mean an init container is blocking the pod. How the Init: status is built, and how to read it.
Pod stuck in ContainerCreating: what the kubelet is still waiting for
A pod stuck in ContainerCreating is waiting on a volume, the CNI plugin or an image pull. How to tell which, plus the new v1.37 SELinux cause.
What kubectl rollout history CHANGE-CAUSE is really showing you
A free-text annotation on the Deployment, copied to whichever ReplicaSet is current when you set it. It can describe a revision it was never part of.
Kubernetes requests vs usage: finding reserved but idle CPU and memory
Compare what pods reserve with what they use over a week, in PromQL: the rate() trap, p95 vs peak, safe requests, and which pods get evicted first.
Query Prometheus through the Kubernetes API server, without port-forward
Reach Prometheus, VictoriaMetrics or Thanos with kubectl get --raw through the API server's service proxy: the URL, the RBAC, and two misleading 503s.
What a kubeconfig actually grants, and how to scope one down
A kubeconfig grants nothing by itself. It carries an identity, and the cluster decides what that identity can do. How to find out, and how to hand out less.
Can your cluster lose a node? Three checks before you drain
A drain that hangs and a replacement pod stuck Pending have the same two causes: a disruption budget that can never allow one, and requests nobody uses.
Your GPU pod is CrashLoopBackOff and Kubernetes will not say the device failed
A failed GPU surfaces as an ordinary restart loop with no indication the device is the cause. How to diagnose it today, and what KEP-4680 adds to pod status.
Why kubectl top disagrees with your dashboard
kubectl top prints a rate over a window that is not fixed, rounded to whole units, from a source that keeps no history. Your dashboard does none of that.
What RESTARTS 3 (5m ago) is actually counting
The RESTARTS column is a sum across every container in the pod, and the time in brackets is the most recent restart of any of them. Neither is per-container.
CPU requests vs limits: what each one writes to the cgroup
Requests become cpu.weight, limits become cpu.max. One is a share of a contended CPU, the other is a hard ceiling enforced whether or not the node is busy.
CreateContainerConfigError and CreateContainerError: the four messages
CreateContainerConfigError, or Init:CreateContainerConfigError: a missing ConfigMap or Secret key, almost always. The message names the missing object.
Pod stuck in Terminating: what is actually blocking the delete
A pod stuck in Terminating is one of four things: a grace period, an unreachable node, a finalizer, or a volume that will not detach. How to tell which.
x509: certificate signed by unknown authority, and kubectl's other connection errors
Four kubectl errors, four different layers. Work out which one failed before you change anything, because the fixes are not interchangeable.
OOMKilled and exit code 137: what actually happened
OOMKilled means your container hit its own memory limit, not that the node ran out. Three different kills look alike in kubectl. How to tell them apart.
0/3 nodes are available: how to fix a Pending pod
The FailedScheduling message is a per-node tally, not one reason. Here is how to read it, and the fix for every clause it can contain.
How kubectl decides what to put in the STATUS column
kubectl get pods never shows status.phase. Here is the real algorithm behind the STATUS column, and why a pod reads Running while a container crash-loops.
RBAC for a read-only Kubernetes user (and why view isn't enough)
The built-in view role has three surprises: no cluster-scoped resources, no secrets, and it is aggregated — so your cloud provider may have quietly widened it.
How KUBECONFIG merges multiple files (first wins, not last)
Set KUBECONFIG to several files and kubectl merges them by a rule that surprises most people: the first file to define a name wins, and later files are ignored.
How to fix CrashLoopBackOff in Kubernetes
CrashLoopBackOff is not an error, it is a waiting state. The exit code tells you what actually happened. Here is how to read it and fix each cause.
ImagePullBackOff and ErrImagePull: what they mean and how to fix them
ErrImagePull is the failure. ImagePullBackOff is the waiting. The real error is in the pod's Events, and there are only about seven things it ever says.
Your clusters, on every screen you own
KubeGlance is a native Kubernetes client for iPhone and iPad — the real dashboard, not a companion — with a full Mac app on the same core. Pods, workloads, logs and events, straight from your kubeconfig.
Download KubeGlance