What: Five eBPF probes that surface what DCGM exporter can't see on a Kubernetes GPU node — TCP RTT, scheduler latency, TC drops, shaped bytes, and retransmits, all per pod.
Where: A Talos Linux worker with an NVIDIA 4070 Super on Thunderbolt 4, passed through Proxmox. Cilium is the CNI, with bandwidth manager and BBR turned on.
Why: DCGM will happily show you 92% GPU utilization while inference latency triples, because the bottleneck is not the GPU — it's the Thunderbolt link, the kernel scheduler, or a noisy neighbor pod eating bandwidth on the same node.
When you need it: any time your DCGM dashboard says the GPU is fine but the user-facing latency says it isn't.
Who it's for: engineers running self-hosted inference on commodity hardware, not A100s on a hyperscaler's PCIe Gen5 backbone.
Table of Contents
- The visibility gap DCGM leaves behind
- The setup: Thunderbolt GPU on a Talos node
- 5 eBPF metrics that fill the DCGM gap
- Wiring it together with Cilium and BBR
- What "working" looks like in Grafana
- Common mistakes when you roll this yourself
- FAQ
The visibility gap DCGM leaves behind
DCGM exporter is built around the NVIDIA driver. It reports utilization, memory, temperature, power, PCIe TX/RX counters, and a handful of profiling metrics. That's everything inside the GPU. It cannot see:
- How long a process waited on the runqueue before it ever touched the GPU
- Whether TCP retransmits are stretching out an inference request that crosses the pod boundary
- Whether the kubelet, kube-proxy, or another DaemonSet is throttling the link the GPU node depends on
- Whether the Thunderbolt 4 controller is renegotiating link speed under thermal load
eBPF closes those gaps because it attaches to the same kernel paths the GPU traffic already walks through. No kernel patches, no workload changes.
The setup: Thunderbolt GPU on a Talos node
The node under test is an NVIDIA 4070 Super in an external Thunderbolt enclosure, passed through Proxmox to a Talos Linux VM (mlops-worker-00). The VM joins the cluster as a GPU node. DCGM exporter runs as a DaemonSet on every GPU node. Everything else in this post sits on top of that.
┌─────────────────────┐ ┌────────────────────────────┐ │ Proxmox Host │ │ Kubernetes Node (Talos) │ │ ┌───────────────┐ │ │ ┌──────────────────────┐ │ │ │ eBPF Agent │◀─┼── bridge ─┼─▶│ DaemonSet: eBPF │ │ │ │ (systemd │ │ vmbr0 │ │ gpu + net probes │ │ │ │ container) │ │ │ └──────────────────────┘ │ │ └───────────────┘ │ │ ┌──────────────┐ │ │ │ │ │ Prometheus │ │ │ │ │ │ node_exporter│ │ │ │ │ └──────────────┘ │ │ │ │ ▲ │ │ │ Thunderbolt│ Scrape │ │ │ │ 3/4 │ │ │ │ │ │ ┌─────┴─────┐ │ │ │ │ │ Grafana │ │ │ │ │ └────────────┘ │ └─────────────────────┘ └────────────────────────────┘
Talos is immutable. You can't shell in and apt install bcc or run perf against the host. The eBPF stack has to ship as DaemonSets with hostNetwork: true and the kernel capabilities the probes need (CAP_BPF, CAP_PERFMON, plus CAP_SYS_ADMIN on older kernels). Most homelab tutorials skip this because they assume you control /usr/lib/modules. On Talos you don't.
5 eBPF metrics that fill the DCGM gap
Five metric series the eBPF DaemonSet exports. Each one answers a question DCGM can't.
1. ebpf_tcp_rtt_microseconds_sum
Per-pod TCP round-trip time, pulled from kernel TCP socket state. Pair it with ebpf_tcp_rtt_microseconds_count to get average RTT per pod, and break it down by destination CIDR to see which workloads share the GPU node's network path. When inference latency rises and GPU utilization stays flat, this is the first panel I check.
2. ebpf_sched_latency_microseconds_sum
Kernel scheduler latency — how long a task sat on a runqueue before getting CPU time. Inference pipelines on a single node fight for the same CPU when batch sizes change. A spike here while DCGM shows the GPU idle is the canonical sign your dataloader is starving the GPU.
3. ebpf_tc_dropped_packets_total
Packets dropped by the Linux traffic-control layer, segmented by pod. Bandwidth-managed traffic that exceeds its annotation shows up here long before retransmits do. If this counter is climbing, either your ceiling is too tight or the workload is bursting wider than you planned.
4. ebpf_tc_bytes_total
Bytes shaped by tc per pod, ingress and egress. This is the metric that proves bandwidth manager is actually doing its job — if your annotations cap a pod at 500 Mbps and the rate of this counter parks at ~62.5 MB/s under load, you have evidence of enforcement.
5. ebpf_tcp_retransmissions_total
Retransmits per pod. With BBR enabled, retransmits to a properly provisioned peer should sit close to zero. A non-zero baseline is either a bandwidth ceiling problem, a Thunderbolt link problem, or a peer that's itself overloaded.
A normal DCGM dashboard answers "is the GPU busy?". These five answer "is the GPU getting fed in time?" — which is the question that actually matters when you're running LLM inference or model-serving workloads.
Wiring it together with Cilium and BBR
Cilium is the eBPF dataplane I run for the cluster. Two features matter here: the bandwidth manager and BBR. Bandwidth manager makes per-pod annotations enforceable. BBR (Bottleneck Bandwidth and Round-trip propagation time) is the TCP congestion control algorithm that gives you the latency wins the eBPF RTT metric will eventually surface.
Enable both in the Cilium Helm values:
bandwidthManager:
enabled: true
bbr: true
Then cap per-pod bandwidth with the standard annotations. Anything in the namespace without these annotations runs uncapped, which is almost always the wrong default on a homelab node:
metadata:
annotations:
kubernetes.io/egress-bandwidth: "500M"
kubernetes.io/ingress-bandwidth: "500M"
The toolchain behind the dashboards looks like this:
┌──────────────────────────────┐ │ Talos Linux Worker │ │ ┌──────────────────────────┐ │ │ │ NVIDIA GPU (Thunderbolt) │ │ │ └──────────────────────────┘ │ │ ↑ PCIe/DMA ↑ Network ↓ │ │ ┌──────────────────────────┐ │ │ │ eBPF Agent DaemonSet │───► Prometheus (TCP RTT, IRQ latency) │ └──────────────────────────┘ │ │ ┌──────────────────────────┐ │ │ │ DCGM Exporter DaemonSet │───► Prometheus (GPU util, PCIe TX/RX) │ └──────────────────────────┘ │ │ ┌──────────────────────────┐ │ │ │ Parca Agent (eBPF CO-RE) │───► Parca Server (CPU flamegraphs) │ └──────────────────────────┘ │ └──────────────────────────────┘
Cilium and Hubble handle network QoS, BBR, and the per-pod tc enforcement. The eBPF agent owns the kernel-scheduler and TCP RTT probes DCGM can't see. DCGM exporter still owns everything inside the GPU — utilization, PCIe throughput, the driver-side numbers. Parca sits on top of the same eBPF infrastructure to give you CPU flamegraphs via CO-RE, which is how you connect a scheduler-latency spike back to a specific call stack.
If you already run Cilium — and if you followed testing cluster mesh you do — flipping the two Helm values and shipping the eBPF agent DaemonSet is all that's left.
What "working" looks like in Grafana
The metrics above feed three panels I keep on a single Grafana row next to the standard DCGM dashboard. The PromQL is short on purpose — these are starting queries, not the final ones you'd ship to a multi-tenant production cluster.
# Average TCP RTT per pod (lower under BBR)
rate(ebpf_tcp_rtt_microseconds_sum[1m])
/ rate(ebpf_tcp_rtt_microseconds_count[1m])
# Bandwidth manager enforcement check (should pin near the annotation)
rate(ebpf_tc_bytes_total[1m]) * 8
# Scheduler latency p99 per pod (correlate with DCGM utilization dips)
histogram_quantile(0.99,
rate(ebpf_sched_latency_microseconds_bucket[5m]))
When BandwidthManager and BBR are configured right, jitter on intra-cluster traffic flattens out — on the workloads I run, that's usually a 30–60% reduction in p99 RTT spread. Retransmits to healthy peers drop to roughly zero, which is the easiest signal that BBR is doing something useful. And ebpf_tc_bytes_total stops overshooting the annotation ceiling under load and parks against it instead. Once those three behaviors hold, an inference latency spike DCGM can't explain stops being a mystery — it shows up as scheduler latency, dropped packets, or a Thunderbolt-induced PCIe TX/RX dip on the panels you already have.
Common mistakes when you roll this yourself
A few things will burn an afternoon if you're not expecting them:
- Running the eBPF agent without
hostNetwork: true. The TCP probes attach to socket state on the host network namespace. Inside a pod network namespace you see nothing. - Trusting DCGM PCIe counters in isolation. Thunderbolt link renegotiation shows up as a dip in PCIe TX/RX without any error metric. Cross-reference with
dmesgparsed through Vector or an eBPF tracepoint onpci_bus_error. - Skipping
bbr: truebecause "it's just congestion control." BBR is the difference between RTT variance you can see and RTT variance you can act on. Without it, theebpf_tcp_rtt_microseconds_sumpanel moves — it just moves noisily enough to be useless. - Forgetting to set both ingress and egress annotations.
kubernetes.io/ingress-bandwidthis enforced separately from egress. A pod that bursts on the side you forgot will drive retransmits on the side you capped. - Letting Parca agent and the eBPF agent fight for
perf_event_open. On older kernels they will, silently. Set CPU sample rates explicitly on both, or run Parca on a subset of nodes.
If you're setting up the rest of the homelab platform underneath this, homelab part one and homelab part two cover the Proxmox and Talos pieces this post assumes.
FAQ
Does this replace DCGM exporter?
No. DCGM still owns everything inside the GPU. The eBPF probes are additive — they answer host-kernel and network-path questions DCGM was never designed to surface. Run both.
Is the eBPF agent safe to run in production?
The probes here attach to stable tracepoints (sched/sched_wakeup, tcp/tcp_probe, tc hooks) and are read-only. The risk is overhead, not stability. On the 4070 node above the agent costs roughly 0.4% CPU at idle and ~1.5% under sustained inference load. Benchmark on your own kernel before generalizing — kernel version matters more than people expect.
Does Talos support eBPF programs out of the box?
Yes. Talos ships a modern kernel with BTF enabled, which is what CO-RE eBPF programs need. No extensions or system extensions required to run Cilium, Parca, or a custom eBPF DaemonSet. You do need to grant the DaemonSet the right capabilities — see the setup section.
Next steps
The lowest-cost move on a Cilium-on-Talos cluster is to flip bandwidthManager.enabled and bbr on, add the per-pod annotations, and let the eBPF agent confirm enforcement before you trust the numbers. Once those five metrics are flowing into Prometheus, the next layer worth building is the correlation between Parca flamegraphs and the scheduler-latency panel — that's where you find the call stacks responsible for starving the GPU, not just the symptom.
For the rest of the platform underneath all of this, the homelab series walks through the Proxmox and Talos foundations end-to-end.