443 dead FUSE mounts wedged kubelet: a production postmortem

TL;DR. Pods on one of our agent nodes had been leaking FUSE mounts for about four weeks. A dead FUSE mount answers every access with ENOTCONN ("transport endpoint is not connected"), and kubelet's volume teardown cannot get past that error, so it retries forever. 443 dead mounts had piled up, k3s-agent was burning most of a core at idle, and cold pod boots went from about a second to between 20 and 105 seconds. Lazy-detaching the corpses fixed the node within minutes; a 60-second janitor in our CSI node driver now reaps them automatically. Most of the wall-clock time went into three traps that each lie to you during diagnosis, and they are the reason to read this.

Background

plori runs AI agents on their own cloud computers; each computer's persistent disk is a FUSE mount served by orlop, our open-source storage plane. The pods behind those computers are disposable: a warm pool pre-boots them so agents wake fast, idle ones get reaped, deploys replace them. Two paths can let a mount outlive its pod: warm-pool pods get their disk mounted by a sidecar after startup, so no CSI teardown is ever called for that mount, and a CSI plugin restart kills the FUSE processes it spawned. Either way the userspace side of the mount dies while the kernel's mount record survives on the host, and every access to it returns ENOTCONN from then on. No agent data is at risk (the data lives in the storage plane); the corpse is purely a host-local liability.

We knew individual mounts could leak; it was in our runbook as a known limitation. What we had not understood was what hundreds of them would do to kubelet.

The incident

On 2026-07-23 our alerting reported warm pod boot failures: context deadline exceeded against a 120-second boot timeout. Every pod boot on that node was taking 20 to 105 seconds with zero active agent runs, and all the time was in container startup, so the node itself was slow. There was no user impact; the pod that tripped the timeout was a pool-replacement pod, and user-facing agent claims kept hitting the warm pool normally.

On the node, k3s-agent (the k3s process that bundles kubelet) was averaging 0.83 cores at idle. /proc/mounts explained why: 443 dead orlop FUSE mounts belonging to long-dead pods, pinning about 450 orphaned pod directories and leaked cgroups, the oldest four weeks old. The journal was one long loop of UnmountVolume.TearDown failed ... transport endpoint is not connected.

Why kubelet can never clean this up

ENOTCONN on a FUSE path means the userspace filesystem process behind the mountpoint is gone while the kernel's mount record remains. The mount cannot recover; the only useful operation left is unmounting it. kubelet's volume teardown does filesystem calls on the path first, gets ENOTCONN, errors, and re-queues the operation. Per volume, every ~2 minutes, forever. The pod's directories and cgroups are never released, and kubelet and cAdvisor keep walking the growing pile. This is long-standing upstream behavior, not something unique to our stack: kubernetes#96361 and kubernetes#129550 both describe FUSE mounts kubelet can never clean up.

Each dead mount is individually harmless. The failure is the accumulation: the node degraded gradually over weeks, with no single triggering event, until a timeout finally fired.

The cleanup

Lazily detach every dead mount, from the host mount namespace, enumerating from /proc/mounts:

# inside the HOST mount namespace (nsenter -t 1 -m; see trap 1)
awk '$1 == "orlop" { print $2 }' /proc/mounts   # enumerate; do not glob (trap 2)
umount -l "$mountpoint"                          # repeat until gone (trap 3)

We detached all 443. Kubelet drained its own backlog with no restart needed: within about five minutes k3s-agent dropped from 0.85 to 0.18 cores and the teardown errors stopped.

The three traps that cost the most wall-clock time

Why doesn't umount work from kubectl debug node?

kubectl debug node/... --profile=sysadmin followed by chroot /host feels like a root shell on the node. For mount operations it is not. The debug pod receives a copy of the host's mount table inside its own mount namespace, so umount there succeeds, returns zero, and changes nothing on the host. It is a silent no-op, and we spent real time believing mounts were gone that were not.

Rule: mutating the host mount table requires entering the host mount namespace, nsenter -t 1 -m, not chroot /host.

Why can't ls or shell globs find a dead FUSE mount?

Glob expansion lstats candidate paths. On a dead FUSE mountpoint, lstat returns ENOTCONN, and the shell treats the path as nonexistent. A command like umount /var/lib/kubelet/pods/*/volumes/*/agent-disk quietly skips exactly the mounts you are trying to kill, while matching any healthy ones. find behaves no better.

Rule: enumerate dead FUSE mounts from /proc/mounts or /proc/self/mountinfo, never from directory listings or globs.

Why is the path still broken after a successful umount?

Mounts stack. The same path can carry two or more mount records, for example when a replacement pod mounted over a predecessor's corpse. One umount pops the top record and reveals the one beneath it, which is just as dead.

Rule: unmount in a loop until the path no longer appears in /proc/mounts.

The durable fix: a janitor in the CSI node driver

A one-off cleanup would leave the leak in place, and the next four quiet weeks would rebuild the pile. So our CSI node driver DaemonSet now sweeps every 60 seconds (plus once at startup, so a plugin restart cleans up the mounts its own death orphaned): read /proc/self/mountinfo, collect orlop mounts under /var/lib/kubelet/pods/, stat each one, lazy-detach any that answer ENOTCONN. The DaemonSet mounts that path with Bidirectional propagation, so its detaches propagate back to the host.

The safety rule is the part worth copying: reap only on ENOTCONN. A dead endpoint holds no unflushed state, so lazy detach is safe by construction. A live mount stats fine and is never touched. A hung-but-alive mount blocks the stat (probes run in a goroutine per path) and is deliberately left alone, because a janitor guessing about hung-but-alive filesystems is how you cause the outage you were preventing. We did not try to fix kubelet upstream: kubelet cannot know an arbitrary wedged filesystem is safe to force-detach, but we know it for our own mounts, and a janitor scoped to our mount source shipped in a day.

The second bug hiding in the same alert

The same alert stream held an unrelated bug with the identical signature: deleting an agent immediately after creating it raced the pre-warm boot, leaving a pod booting against already-revoked disk access until the same 120-second timeout fired. One symptom, two causes; it is the postmortem cliché that keeps being true. If we had deduplicated alerts by signature we would have shipped the janitor and kept paging ourselves.

What to check when this happens to you

1. Journal for UnmountVolume.TearDown failed ... transport endpoint is not connected.
2. grep fuse /proc/mounts on the node, and compare mount entries against actually-running pods.
3. Compare directories under /var/lib/kubelet/pods and cgroups under kubepods.slice against scheduled pods.
4. Remediate with nsenter -t 1 -m, enumerate from /proc/mounts, umount -l in a loop.
5. Then automate it, or the pile comes back.

FAQ

What does "transport endpoint is not connected" mean on a FUSE mount? The userspace FUSE server process behind the mountpoint has exited (crashed, was killed, or its pod died) while the kernel mount record remains. Every filesystem operation on the path returns ENOTCONN. The mount cannot recover; the only remaining useful operation is unmounting it. Your data is whatever the backing store holds; the dead mount itself holds nothing.

Why do Kubernetes pods get stuck terminating with FUSE mounts in emptyDir volumes? kubelet's volume teardown performs filesystem operations on the volume path. On a dead FUSE mount those fail with ENOTCONN, so teardown errors and retries indefinitely, and the pod's directories and cgroups are never released. See kubernetes/kubernetes issues #96361 and #129550.

Is umount -l (lazy detach) safe on a dead FUSE mount? If a stat on the mountpoint returns ENOTCONN, yes: the server side is gone, there is nothing left to flush, and lazy detach removes the mount record even if something still holds the path open. Do not lazy-detach mounts that merely hang; a hung mount may have a live server with unflushed state.

How do I find dead FUSE mounts on a node? Read /proc/mounts (or /proc/self/mountinfo) and stat each FUSE mountpoint; ENOTCONN marks a corpse. Do not rely on ls, find, or shell globs, which skip ENOTCONN paths as if they did not exist.