Back to Details

kernel-debugging

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Kernel Debugging

内核调试

Purpose

用途

Guide agents through debugging the Linux kernel: kgdb over serial or USB, kdb built-in debugger commands, ftrace and 
trace-cmd
, kprobes and kretprobes, dynamic debug (
dyndbg
), crash dump analysis with crash and makedumpfile, dmesg log levels, and 
/proc/sys
runtime tuning for debugging.
指导开发者进行Linux内核调试：通过串口或USB使用kgdb、kdb内置调试器命令、ftrace与
trace-cmd
、kprobes与kretprobes、动态调试（
dyndbg
）、使用crash和makedumpfile分析崩溃转储、dmesg日志级别，以及用于调试的
/proc/sys
运行时调优。

When to Use

适用场景

  • Kernel panic or oops with unclear stack trace
  • Live debugging a kernel module or driver with kgdb
  • Tracing function calls without recompiling (ftrace, kprobes)
  • Enabling verbose driver logging with dyndbg
  • Analyzing production crash dumps from kdump
  • Tuning kernel runtime parameters for debug sessions
  • 内核panic或oops且堆栈跟踪不清晰
  • 使用kgdb对内核模块或驱动进行在线调试
  • 无需重新编译即可追踪函数调用（ftrace、kprobes）
  • 使用dyndbg启用详细的驱动日志
  • 分析来自kdump的生产环境崩溃转储
  • 为调试会话调优内核运行时参数

Workflow

工作流程

1. dmesg and log levels

1. dmesg与日志级别

bash
undefined
bash
undefined

Kernel ring buffer

Kernel ring buffer

dmesg -T -l err,warn dmesg -w # follow live
dmesg -T -l err,warn dmesg -w # follow live

Log level (0=emerg .. 7=debug)

Log level (0=emerg .. 7=debug)

cat /proc/sys/kernel/printk
cat /proc/sys/kernel/printk

current default minimum boot-default

current default minimum boot-default

4 4 1 7

4 4 1 7

Temporarily increase verbosity

Temporarily increase verbosity

echo 8 > /proc/sys/kernel/printk

Kernel `printk` levels: `KERN_EMERG` through `KERN_DEBUG`. Driver `dev_dbg()` requires `dyndbg` or `DEBUG` define.
echo 8 > /proc/sys/kernel/printk

内核`printk`级别：`KERN_EMERG`至`KERN_DEBUG`。驱动的`dev_dbg()`需要`dyndbg`或`DEBUG`宏定义。

2. kgdb over serial

2. 串口kgdb

Kernel config: 
CONFIG_KGDB
, 
CONFIG_KGDB_SERIAL_CONSOLE
.
bash
undefined
内核配置：
CONFIG_KGDB
、
CONFIG_KGDB_SERIAL_CONSOLE
。
bash
undefined

Boot parameter (ttyS0, 115200)

Boot parameter (ttyS0, 115200)

kgdboc=ttyS0,115200 kgdbwait
kgdboc=ttyS0,115200 kgdbwait

Or at runtime

Or at runtime

echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc echo g > /proc/sysrq-trigger # break into kgdb

GDB host side:

```bash
echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc echo g > /proc/sysrq-trigger # break into kgdb

GDB主机端：

```bash

Connect to remote serial

Connect to remote serial

gdb vmlinux (gdb) set serial baud 115200 (gdb) target remote /dev/ttyUSB0 (gdb) bt (gdb) list *panic+0x20

USB variant: `CONFIG_KGDB_KDB` with `kgdboc=usb` on supported hardware.
gdb vmlinux (gdb) set serial baud 115200 (gdb) target remote /dev/ttyUSB0 (gdb) bt (gdb) list *panic+0x20

USB变体：在支持的硬件上，启用`CONFIG_KGDB_KDB`并设置`kgdboc=usb`。

3. kdb commands

3. kdb命令

When 
CONFIG_KGDB_KDB
enabled, kdb provides in-kernel shell:
kdb commands (at SysRq break)
├── bt          — stack backtrace
├── lsmod       — loaded modules
├── md <addr>   — memory display
├── ps          — process list
├── id <cpu>    — CPU registers
├── go          — continue execution
└── cpu <n>     — switch CPU context
bash
undefined
启用
CONFIG_KGDB_KDB
后，kdb提供内核内shell：
kdb commands (at SysRq break)
├── bt          — stack backtrace
├── lsmod       — loaded modules
├── md <addr>   — memory display
├── ps          — process list
├── id <cpu>    — CPU registers
├── go          — continue execution
└── cpu <n>     — switch CPU context
bash
undefined

Enter kdb

Enter kdb

echo k > /proc/sysrq-trigger
undefined
echo k > /proc/sysrq-trigger
undefined

4. ftrace

4. ftrace

bash
undefined
bash
undefined

Enable function tracer

Enable function tracer

echo function > /sys/kernel/debug/tracing/current_tracer echo 1 > /sys/kernel/debug/tracing/tracing_on
echo function > /sys/kernel/debug/tracing/current_tracer echo 1 > /sys/kernel/debug/tracing/tracing_on

Filter to specific function

Filter to specific function

echo schedule > /sys/kernel/debug/tracing/set_ftrace_filter echo ':probe_' >> /sys/kernel/debug/tracing/set_ftrace_notrace
echo schedule > /sys/kernel/debug/tracing/set_ftrace_filter echo ':probe_' >> /sys/kernel/debug/tracing/set_ftrace_notrace

Capture and read

Capture and read

cat /sys/kernel/debug/tracing/trace_pipe
cat /sys/kernel/debug/tracing/trace_pipe

Disable

Disable

echo 0 > /sys/kernel/debug/tracing/tracing_on echo nop > /sys/kernel/debug/tracing/current_tracer

```bash
echo 0 > /sys/kernel/debug/tracing/tracing_on echo nop > /sys/kernel/debug/tracing/current_tracer

```bash

trace-cmd (userspace recorder)

trace-cmd (userspace recorder)

trace-cmd record -p function -l schedule,do_page_fault trace-cmd report trace-cmd.dat # kernelshark GUI
undefined
trace-cmd record -p function -l schedule,do_page_fault trace-cmd report trace-cmd.dat # kernelshark GUI
undefined

5. kprobes and kretprobes

5. kprobes与kretprobes

bash
undefined
bash
undefined

Dynamic kprobe via debugfs

Dynamic kprobe via debugfs

echo 'p:myprobe do_sys_open %ax %si' > /sys/kernel/debug/tracing/kprobe_events echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable cat /sys/kernel/debug/tracing/trace
echo 'p:myprobe do_sys_open %ax %si' > /sys/kernel/debug/tracing/kprobe_events echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable cat /sys/kernel/debug/tracing/trace

kretprobe

kretprobe

echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events
echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events

Cleanup

Cleanup

echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events

In-kernel kprobe (module):

```c
#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = my_pre_handler,
};

static int __init mod_init(void)
{
    return register_kprobe(&kp);
}
echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events

内核内kprobe（模块）：

```c
#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = my_pre_handler,
};

static int __init mod_init(void)
{
    return register_kprobe(&kp);
}

6. Dynamic debug (dyndbg)

6. 动态调试（dyndbg）

bash
undefined
bash
undefined

Enable all debug messages for a module

Enable all debug messages for a module

echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control

Enable specific file:line

Enable specific file:line

echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control
echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control

Query current settings

Query current settings

grep mydriver /sys/kernel/debug/dynamic_debug/control

Boot-time: `dyndbg="module mydriver +p"` on kernel command line.
grep mydriver /sys/kernel/debug/dynamic_debug/control

启动时配置：在内核命令行添加`dyndbg="module mydriver +p"`。

7. kdump and crash analysis

7. kdump与崩溃分析

bash
undefined
bash
undefined

Setup kdump (crash kernel reserved memory)

Setup kdump (crash kernel reserved memory)

/etc/default/grub: crashkernel=256M

/etc/default/grub: crashkernel=256M

sudo kdump-config show
sudo kdump-config show

After crash, vmcore in /var/crash/

After crash, vmcore in /var/crash/

ls /var/crash/*/vmcore
ls /var/crash/*/vmcore

Analyze with crash utility

Analyze with crash utility

crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore
crash> bt crash> log crash> ps crash> kmem -i crash> mod

```bash
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore
crash> bt crash> log crash> ps crash> kmem -i crash> mod

```bash

Compress vmcore

Compress vmcore

makedumpfile -c -d /proc/vmcore /tmp/vmcore.lzo
undefined
makedumpfile -c -d /proc/vmcore /tmp/vmcore.lzo
undefined

8. /proc/sys runtime tuning for debugging

8. 用于调试的/proc/sys运行时调优

bash
undefined
bash
undefined

Panic on oops (reproduce in VM)

Panic on oops (reproduce in VM)

sysctl kernel.panic_on_oops=1
sysctl kernel.panic_on_oops=1

Soft lockup detection

Soft lockup detection

sysctl kernel.softlockup_panic=1
sysctl kernel.softlockup_panic=1

Watchdog

Watchdog

sysctl kernel.nmi_watchdog=1
sysctl kernel.nmi_watchdog=1

Slab debugging

Slab debugging

Boot: slub_debug=P,pagealloc

Boot: slub_debug=P,pagealloc

undefined
undefined

9. Debugging decision tree

9. 调试决策树

Kernel issue type?
├── Panic/oops → dmesg; crash vmcore; CONFIG_DEBUG_INFO_BTF
├── Logic bug in driver → dyndbg; ftrace function_graph
├── Performance regression → perf record -g -a; trace-cmd
├── Intermittent → kprobes on suspect path
└── Module crash → kgdb; try_module_get audit
Kernel issue type?
├── Panic/oops → dmesg; crash vmcore; CONFIG_DEBUG_INFO_BTF
├── Logic bug in driver → dyndbg; ftrace function_graph
├── Performance regression → perf record -g -a; trace-cmd
├── Intermittent → kprobes on suspect path
└── Module crash → kgdb; try_module_get audit

Common Problems

常见问题

SymptomCauseFix
kgdb won't connectWrong tty/baudMatch 
kgdboc
to serial adapter
ftrace empty tracetracing_off or nop tracer
echo function > current_tracer
; enable tracing_on
kprobe registration failsMissing CONFIG_KPROBES or inlined functionUse tracepoint or static_key
dyndbg has no effectCONFIG_DYNAMIC_DEBUG disabledRebuild kernel with option
crash can't read vmcoreWrong vmlinux debug symbolsInstall 
linux-image-$(uname -r)-dbg
SysRq doesn't work
kernel.sysrq
disabled
echo 1 > /proc/sys/kernel/sysrq
症状原因解决方法
kgdb无法连接错误的tty/波特率将
kgdboc
与串口适配器匹配
ftrace追踪为空tracing_off或nop追踪器
echo function > current_tracer
；启用tracing_on
kprobe注册失败缺少CONFIG_KPROBES或函数被内联使用tracepoint或static_key
dyndbg无效果CONFIG_DYNAMIC_DEBUG未启用重新编译内核并开启该选项
crash无法读取vmcore错误的vmlinux调试符号安装
linux-image-$(uname -r)-dbg
SysRq无法工作
kernel.sysrq
被禁用
echo 1 > /proc/sys/kernel/sysrq

Related Skills

相关技能

  • skills/low-level-programming/linux-kernel-modules
    — module development being debugged
  • skills/kernel/device-drivers
    — driver-specific probe/IRQ issues
  • skills/kernel/kernel-internals
    — subsystem knowledge for interpreting traces
  • skills/debuggers/gdb
    — GDB commands shared with kgdb
  • skills/profilers/linux-perf
    — perf for kernel hotspots
  • skills/observability/ebpf
    — non-invasive kernel tracing alternative
  • skills/low-level-programming/linux-kernel-modules
    — 待调试的模块开发
  • skills/kernel/device-drivers
    — 驱动特定的探针/IRQ问题
  • skills/kernel/kernel-internals
    — 用于解读追踪信息的子系统知识
  • skills/debuggers/gdb
    — kgdb共享的GDB命令
  • skills/profilers/linux-perf
    — 用于内核热点分析的perf
  • skills/observability/ebpf
    — 非侵入式内核追踪替代方案