NIC Diagnostics for XDP

Comprehensive reference for diagnosing, configuring, and monitoring NICs in AF_XDP / XDP workloads.

1. Driver Detection & NIC Identification

Identify the NIC and driver in use

bash

# PCI device listing — find your NIC's bus address
lspci

# Driver name, version, firmware, bus-info
ethtool -i <iface>

# Link state, speed, duplex negotiation
ethtool <iface>
ethtool <iface> | egrep -i 'link|speed|duplex'

# Confirm interface is up and check for attached XDP program
ip link show dev <iface>
ip link show dev <iface> | grep xdp

Why it matters

XDP zero-copy support is driver-specific. Only certain drivers (ice, i40e, mlx5, etc.) support

XDP_DRV

mode and AF_XDP zero-copy. Knowing the driver tells you what features are available and what quirks to expect.

2. Hardware Queue Configuration

View and set combined queue count

bash

# Show current and max queue count
ethtool -l <iface>

# Set combined queues (must match or exceed XDP queue IDs you bind to)
ethtool -L <iface> combined <N>

# List all queues exposed by the NIC
ls -1 /sys/class/net/<iface>/queues

Ring buffer sizes

bash

# Show current and max ring buffer depths (rx/tx)
ethtool -g <iface>

Why it matters

Each AF_XDP socket binds to a specific hardware queue. You need enough queues for your workload, and the ring buffer depth affects burst absorption. Too few queues = contention; too shallow rings = drops under burst.

3. Offload Control — GSO, GRO, TSO, LRO

Inspect current offload state

bash

ethtool -k <iface>
ethtool -k <iface> | grep -E 'generic-receive|large-receive|scatter-gather|tcp-segmentation'

Disable offloads for XDP

bash

# XDP requires offloads disabled — aggregated/segmented frames break XDP processing
ethtool -K <iface> gro off lro off tso off gso off

Why it matters

GRO/LRO aggregate multiple packets into a single large buffer. TSO/GSO do the same on the TX side. XDP programs operate on individual frames at the driver level — aggregated super-frames will either be rejected or cause undefined behavior. Always disable these before attaching XDP programs.

4. VLAN Offload Control

Inspect and toggle VLAN offloads

bash

ethtool -k <iface> | grep -i vlan

# Disable VLAN tag stripping (keep tags in packet data for XDP inspection)
ethtool -K <iface> rxvlan off
ethtool -K <iface> txvlan off

# Or via the longer form
ethtool -K <iface> rx-vlan-offload off
ethtool -K <iface> rx-vlan-filter off

# Re-enable if needed
ethtool -K <iface> rxvlan on
ethtool -K <iface> rx-vlan-filter on

Why it matters

When VLAN offload is on, the NIC strips the 802.1Q tag before the packet reaches XDP. If your XDP program needs to inspect or filter on VLAN IDs, you must disable

rxvlan

so the tag remains in the Ethernet header.

5. Flow Director (FDIR) / ntuple Rules

Check ntuple support

bash

ethtool -k <iface> | grep -i ntuple

View existing flow rules

bash

ethtool -n <iface>
ethtool -u <iface>

Add FDIR rules — steer traffic to a specific hardware queue

bash

# Steer UDP traffic matching a 5-tuple → queue 3
sudo ethtool -U <iface> flow-type udp4 \
  src-ip <src> dst-ip <dst> dst-port <port> action 3

# Steer TCP traffic to a specific dst-port → queue 0
sudo ethtool -U <iface> flow-type tcp4 \
  src-ip 0.0.0.0 dst-ip <dst> dst-port <port> action 0

Why it matters

FDIR rules let you pin specific flows to specific hardware queues. Since each AF_XDP socket binds to one queue, FDIR is how you guarantee that your target traffic lands on the queue your XDP program is listening on. Without FDIR, RSS hashing distributes packets unpredictably across all queues.

6. Private Flags & Loopback

bash

# Show driver-specific private flags
ethtool --show-priv-flags <iface>

# Enable hardware loopback (useful for testing without a second machine)
sudo ethtool --set-priv-flags <iface> loopback on
sudo ethtool -s <iface> loopback on

# Check loopback support
sudo ethtool --show-features <iface> | grep loopback

7. Detecting Idle CPU Cores for Pinning

Map the CPU topology

bash

# Full CPU topology: CPU ID, physical core, socket, NUMA node
lscpu -e=CPU,CORE,SOCKET,NODE,ONLINE

Find NUMA node for the NIC

bash

# Critical: pin XDP threads to cores on the same NUMA node as the NIC
cat /sys/class/net/<iface>/device/numa_node

Find which cores are busy vs idle

bash

# Per-CPU utilization — look for cores near 0% usage
mpstat -P ALL 1 5

# Check what's pinned to each core already
ps -eo pid,comm,psr --sort=psr | awk '{count[$3]++; procs[$3]=procs[$3] " " $2} END {for (c in count) print "CPU " c ": " count[c] " procs:" procs[c]}'

# Check IRQ affinity — which cores handle which NIC interrupts
cat /proc/interrupts | grep <iface>
awk '/<iface>-TxRx/{print $1,$NF}' /proc/interrupts | sed 's/://'
grep . /proc/irq/*/smp_affinity_list

Disable irqbalance (prevents the OS from moving IRQs to your pinned cores)

bash

sudo systemctl stop irqbalance
systemctl status irqbalance

Pin IRQs to specific cores

bash

# Check current IRQ CPU affinity
cat /proc/irq/<irq_num>/smp_affinity_list

# Pin an IRQ to a specific core
echo <core_id> | sudo tee /proc/irq/<irq_num>/smp_affinity_list

MSI-X IRQ vector enumeration

bash

# List all MSI-X vectors for the NIC's PCI device
ls /sys/devices/pci<domain>/<bus>/<device>/msi_irqs

Why it matters

XDP busy-poll loops are CPU-bound. Pinning them to an idle core on the correct NUMA node eliminates cross-node memory latency and prevents the scheduler from preempting your hot loop. Always:

Find the NIC's NUMA node
Pick idle cores on that node
Stop irqbalance
Pin NIC IRQs away from your XDP cores
Pin your XDP threads to the chosen cores

8. Hardware Queue & Drop Monitoring

Live NIC statistics

bash

# Full stats dump
ethtool -S <iface>

# XDP/XSK specific counters
ethtool -S <iface> | grep -i xdp

# Filter for drops, errors, misses
ethtool -S <iface> | egrep -i 'rx|drop|err|xdp|xsk' | head -n 50

# Per-queue packet counts
ethtool -S <iface> | grep -E "rx_queue"
ethtool -S <iface> | grep "rx_queue_<N>_packets:"

Live monitoring dashboards

bash

# Watch queue counters in real time
watch -n 1 'ethtool -S <iface> | grep -E "rx_queue"'

# Watch drops and errors
watch -n1 "ethtool -S <iface> | grep -E 'rx_packets|rx_dropped|rx_queue'"

# Combined NIC + XDP socket status
watch -n 1 "echo '=== NIC ===' && ethtool -S <iface> | grep -iE 'drop|miss|err|full' && echo '=== XDP ===' && cat /proc/net/xdp 2>/dev/null"

# Full drop monitoring loop
IFACE=<iface>; QUEUE=<N>
while true; do
    echo "--- $(date) ---"
    echo "NIC Drops:"
    ethtool -S $IFACE 2>/dev/null | grep -E "drop|miss|error|discard" | head -10
    echo -e "\nQueue $QUEUE:"
    ethtool -S $IFACE 2>/dev/null | grep -i "queue_${QUEUE}"
    echo -e "\nXDP Sockets:"
    cat /proc/net/xdp 2>/dev/null || echo "No XDP sockets found"
    echo -e "\nInterface Totals:"
    cat /proc/net/dev | awk -v iface=$IFACE '$1 ~ iface {print "RX pkts:", $2, "RX drop:", $5}'
    sleep 5
done

AF_XDP socket status

bash

# Kernel's view of active XDP sockets
cat /proc/net/xdp
cat /proc/net/xdp 2>/dev/null || ss -ax | grep -i xdp

9. BPF Program Inspection & Profiling

bpftool — inspect loaded XDP/BPF programs and maps

bash

# List all loaded BPF programs
bpftool prog show

# Details on a specific program
bpftool prog show id <prog_id>

# Profile a BPF program (cycles, instructions over 5 seconds)
bpftool prog profile id <prog_id> duration 5 cycles instructions

# List all BPF maps
bpftool map show
bpftool map show | grep -i xsk

# Dump map contents (debug maps, XSK maps)
bpftool map dump name <map_name>
bpftool map dump pinned /sys/fs/bpf/xsks_map
bpftool map dump id <map_id>

Why it matters

bpftool

is the primary way to verify your XDP program is loaded, attached to the right interface, and that its maps (especially XSK maps) are populated correctly. Map dumps let you verify that socket file descriptors are registered in the XSKMAP.

10. Kernel Tracing — ftrace / trace_pipe

Capture XDP trace events from the kernel

bash

# Stream trace output (Ctrl+C to stop)
cat /sys/kernel/debug/tracing/trace_pipe

# Background capture to a file
cat /sys/kernel/debug/tracing/trace_pipe > /tmp/xdp_trace.log &

# Read the log
cat /tmp/xdp_trace.log

# Tail live
tail -f /sys/kernel/debug/tracing/trace_pipe

# Stop background trace capture
pkill -f trace_pipe
fuser -k /sys/kernel/debug/tracing/trace_pipe

Watch dmesg for XDP events

bash

watch -n1 "dmesg | grep xdp"

Why it matters

When XDP programs call

bpf_trace_printk()

, output goes to

trace_pipe

. This is the primary printf-style debugging mechanism for eBPF/XDP programs. Use it to trace packet paths, verify filter logic, and debug drop reasons.

11. perf — CPU-Level Performance Profiling

Stat counters (lightweight, no recording)

bash

# Core hardware counters for your XDP process
sudo perf stat -e cycles,instructions,cache-misses,LLC-load-misses,branches,branch-misses \
  -p $(pgrep <process>) -- sleep 10

# Extended counters (-d -d -d = most detail)
sudo perf stat -d -d -d -p $(pgrep <process>)

Record + flamegraph workflow

bash

# Record with DWARF call graphs (most accurate stacks)
sudo perf record --call-graph dwarf -e cycles \
  -p $(pgrep <process>) -- sleep 10

# Record on a specific CPU core
sudo perf record -F 997 -g --call-graph dwarf -C <core> -o perf.data -- sleep 60

# Record multiple event types
sudo perf record -e cycles,stalled-cycles-frontend,stalled-cycles-backend,cache-misses,branch-misses \
  -g -p $(pgrep <process>)

# Interactive report
sudo perf report

# Generate flamegraph (requires inferno)
sudo perf script -i perf.data | inferno-collapse-perf | inferno-flamegraph > flamegraph.svg

# Live top-like view
sudo perf top -p $(pgrep <process>) -g

Why it matters

perf stat

answers "is my hot loop efficient?" — watch IPC (instructions per cycle), cache miss rates, and branch mispredictions.

perf record

+ flamegraphs show exactly where CPU time is spent in your rx/tx loop, revealing bottlenecks in ring operations, packet parsing, or syscalls.

12. IRQ-to-Queue-to-Core Mapping

Full mapping workflow

bash

# 1. Find which IRQs belong to your NIC
cat /proc/interrupts | grep <iface>
awk '/<iface>-TxRx/{print $1,$NF}' /proc/interrupts | sed 's/://'

# 2. Check current CPU affinity for each IRQ
cat /proc/irq/<irq_num>/smp_affinity_list

# 3. Pin queue IRQs to specific cores (avoid your XDP poll cores)
echo <core_id> | sudo tee /proc/irq/<irq_num>/smp_affinity_list

Why it matters

Each NIC queue generates interrupts on a specific IRQ line. If the kernel delivers that IRQ to the same core running your XDP busy-poll, you get contention. Map out which IRQ serves which queue, then pin IRQs to cores that are not running your XDP threads.

13. Quick Diagnostic Checklist

Run this sequence when setting up or debugging an XDP workload:


ethtool -i <iface>
ethtool -l <iface>
ethtool -g <iface>
ethtool -k <iface> | grep -E 'gro|tso|gso|lro'
ethtool -k <iface> | grep ntuple
ethtool -n <iface>
cat /sys/class/net/<iface>/device/numa_node
lscpu -e=CPU,CORE,SOCKET,NODE,ONLINE
systemctl status irqbalance
cat /proc/interrupts | grep <iface>
bpftool prog show
cat /proc/net/xdp
ethtool -S <iface> | grep -i drop

Step	Command	Looking For
1	`ethtool -i <iface>`	Driver supports XDP (ice, i40e, mlx5)
2	`ethtool -l <iface>`	Enough combined queues
3	`ethtool -g <iface>`	Ring buffer depth adequate
4	`ethtool -k <iface> \| grep -E 'gro\|tso\|gso\|lro'`	All OFF
5	`ethtool -k <iface> \| grep ntuple`	ntuple ON for FDIR
6	`ethtool -n <iface>`	FDIR rules steering to correct queues
7	`cat /sys/class/net/<iface>/device/numa_node`	NUMA node for core selection
8	`lscpu -e=CPU,CORE,SOCKET,NODE,ONLINE`	Available cores on correct NUMA
9	`systemctl status irqbalance`	Should be STOPPED
10	`cat /proc/interrupts \| grep <iface>`	IRQs pinned away from XDP cores
11	`bpftool prog show`	XDP program loaded and attached
12	`cat /proc/net/xdp`	AF_XDP sockets active
13	`ethtool -S <iface> \| grep -i drop`	Zero or stable drop counters

nic-xdp-diagnostics

NPX Install

Tags

SKILL.md Content

NIC Diagnostics for XDP

1. Driver Detection & NIC Identification

Identify the NIC and driver in use

Why it matters

2. Hardware Queue Configuration

View and set combined queue count

Ring buffer sizes

Why it matters

3. Offload Control — GSO, GRO, TSO, LRO

Inspect current offload state

Disable offloads for XDP

Why it matters

4. VLAN Offload Control

Inspect and toggle VLAN offloads

Why it matters

5. Flow Director (FDIR) / ntuple Rules

Check ntuple support

View existing flow rules

Add FDIR rules — steer traffic to a specific hardware queue

Why it matters

6. Private Flags & Loopback

7. Detecting Idle CPU Cores for Pinning

Map the CPU topology

Find NUMA node for the NIC

Find which cores are busy vs idle

Disable irqbalance (prevents the OS from moving IRQs to your pinned cores)

Pin IRQs to specific cores

MSI-X IRQ vector enumeration

Why it matters

8. Hardware Queue & Drop Monitoring

Live NIC statistics

Live monitoring dashboards

AF_XDP socket status

9. BPF Program Inspection & Profiling

bpftool — inspect loaded XDP/BPF programs and maps

Why it matters

10. Kernel Tracing — ftrace / trace_pipe

Capture XDP trace events from the kernel

Watch dmesg for XDP events

Why it matters

11. perf — CPU-Level Performance Profiling

Stat counters (lightweight, no recording)

Record + flamegraph workflow

Why it matters

12. IRQ-to-Queue-to-Core Mapping

Full mapping workflow

Why it matters

13. Quick Diagnostic Checklist