Core Dumps

Core Dumps（核心转储）

Purpose

用途

Guide agents through enabling, collecting, and analysing core dumps for post-mortem crash investigation without rerunning the buggy program.

指导相关人员完成Core Dump的启用、收集和分析，无需重新运行出错程序即可进行事后崩溃调查。

Triggers

触发场景

"My program crashed in production — how do I analyse the core?"
"How do I enable core dumps on Linux?"
"I have a core file but no symbols / source"
"How do I use debuginfod to get symbols for a core?"
"coredumpctl show me the crash"

“我的程序在生产环境崩溃了——我该如何分析core文件？”
“如何在Linux上启用Core Dump？”
“我有core文件但没有符号/源码”
“如何使用debuginfod获取core文件的符号？”
“coredumpctl帮我查看崩溃信息”

Workflow

操作流程

1. Enable core dumps (Linux)

1. 在Linux上启用Core Dump

bash

undefined

bash

undefined

Per-session (lost on logout)

会话级生效（登出后失效）

ulimit -c unlimited

Persistent (add to /etc/security/limits.conf)

永久生效（添加到/etc/security/limits.conf）

soft core unlimited
hard core unlimited

soft core unlimited
hard core unlimited

Check current limit

检查当前限制

ulimit -c

Set core pattern (where and how cores are named)

设置core文件命名规则（存储位置和命名方式）

Default: 'core' in CWD — often not useful

默认：当前工作目录下的'core'文件——通常实用性不强

sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t

%e = executable, %p = PID, %t = timestamp

%e = 可执行文件名, %p = 进程ID, %t = 时间戳

Persistent (add to /etc/sysctl.d/99-core.conf)

永久生效（添加到/etc/sysctl.d/99-core.conf）

kernel.core_pattern=/tmp/core-%e-%p-%t kernel.core_uses_pid=1

undefined

kernel.core_pattern=/tmp/core-%e-%p-%t kernel.core_uses_pid=1

undefined

2. systemd/coredumpctl (modern Linux)

2. systemd/coredumpctl（现代Linux系统）

If systemd manages core dumps (common on Ubuntu 20+, Fedora, Arch):

bash

undefined

如果系统由systemd管理Core Dump（Ubuntu 20+、Fedora、Arch等系统常见）：

bash

undefined

List recent crashes

列出近期崩溃记录

coredumpctl list

Show details of the latest crash

显示最新崩溃的详细信息

coredumpctl info

Load latest crash in GDB

在GDB中加载最新崩溃的core文件

coredumpctl gdb

Load specific PID crash

加载指定进程ID的崩溃core文件

coredumpctl gdb 12345

Export core file

导出core文件

coredumpctl dump -o myapp.core PID


Core storage location: `/var/lib/systemd/coredump/`.

coredumpctl dump -o myapp.core PID


Core文件存储位置：`/var/lib/systemd/coredump/`。

3. Enable core dumps (macOS)

3. 在macOS上启用Core Dump

bash

undefined

bash

undefined

macOS uses /cores by default (must be root-writable)

macOS默认存储在/cores目录（需具备root写入权限）

ulimit -c unlimited

Check

检查core文件

ls /cores/

launchd-launched services: set in plist

launchd启动的服务：需在plist文件中配置

<key>HardResourceLimits</key>

<dict><key>Core</key><integer>9223372036854775807</integer></dict>

undefined

undefined

4. Analyse a core with GDB

4. 使用GDB分析core文件

bash

undefined

bash

undefined

Load binary and core

加载二进制文件和core文件

gdb ./prog core.12345

If the binary was stripped, provide the unstripped copy

如果二进制文件已被剥离符号，提供未剥离符号的副本

gdb ./prog-with-symbols core.12345

Essential first commands

核心初始命令

(gdb) bt # call stack (gdb) bt full # stack + locals (gdb) info registers # CPU state at crash (gdb) frame 2 # jump to interesting frame (gdb) info locals # local variables in frame (gdb) print ptr # inspect a pointer

(gdb) bt # 调用栈 (gdb) bt full # 调用栈+局部变量 (gdb) info registers # 崩溃时的CPU状态 (gdb) frame 2 # 跳转到指定栈帧 (gdb) info locals # 当前栈帧的局部变量 (gdb) print ptr # 查看指针内容

All threads (multi-threaded crash)

所有线程信息（多线程崩溃场景）

(gdb) thread apply all bt full

undefined

(gdb) thread apply all bt full

undefined

5. Analyse a core with LLDB

5. 使用LLDB分析core文件

bash

lldb ./prog -c core.12345

bash

lldb ./prog -c core.12345

Or

或者

lldb (lldb) target create ./prog --core core.12345

Commands

常用命令

(lldb) bt (lldb) thread backtrace all (lldb) frame select 2 (lldb) frame variable

undefined

(lldb) bt (lldb) thread backtrace all (lldb) frame select 2 (lldb) frame variable

undefined

6. Missing symbols: debuginfod

6. 缺失符号：使用debuginfod

debuginfod

serves debug symbols from a central server, mapping build IDs to DWARF data.

bash

undefined

debuginfod

从中央服务器提供调试符号，将构建ID映射到DWARF数据。

bash

undefined

Install client (Debian/Ubuntu)

安装客户端（Debian/Ubuntu）

sudo apt install debuginfod

Enable (add to ~/.bashrc or /etc/environment)

启用（添加到~/.bashrc或/etc/environment）

export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org"

GDB auto-fetches symbols when DEBUGINFOD_URLS is set

设置DEBUGINFOD_URLS后，GDB会自动获取符号

gdb ./prog core

Manually query

手动查询

debuginfod-find debuginfo <build-id> debuginfod-find source <build-id> /path/to/file.c

undefined

debuginfod-find debuginfo <build-id> debuginfod-find source <build-id> /path/to/file.c

undefined

7. Missing symbols: manual approach

7. 缺失符号：手动处理方式

bash

undefined

bash

undefined

Check if binary has a build ID

检查二进制文件是否包含构建ID

readelf -n ./prog | grep Build

Find the correct debug package

查找对应的调试包

Debian: apt install prog-dbg or prog-dbgsym

Debian系统：apt install prog-dbg或prog-dbgsym

RPM: dnf install prog-debuginfo

RPM系统：dnf install prog-debuginfo

Point GDB to debug symbols directory

为GDB指定调试符号目录

(gdb) set debug-file-directory /usr/lib/debug

Or use eu-readelf to dump build ID, then find .debug file

或者使用eu-readelf导出构建ID，然后查找.debug文件

eu-readelf -n ./prog find /usr/lib/debug -name "*.debug" | xargs eu-readelf -n 2>/dev/null | grep <build-id>

undefined

eu-readelf -n ./prog find /usr/lib/debug -name "*.debug" | xargs eu-readelf -n 2>/dev/null | grep <build-id>

undefined

8. Strip binaries and keep symbols

8. 剥离二进制文件符号并保留符号文件

Best practice: build with symbols, strip for distribution, keep an unstripped copy.

bash

undefined

最佳实践：编译时保留符号，剥离符号后用于分发，保留未剥离符号的副本。

bash

undefined

Build

编译

gcc -g -O2 -o prog main.c

Separate debug info

分离调试信息

objcopy --only-keep-debug prog prog.debug objcopy --strip-debug prog prog.stripped

Add a debuglink so GDB finds the debug file automatically

添加debuglink，让GDB自动找到调试文件

objcopy --add-gnu-debuglink=prog.debug prog.stripped

Deploy prog.stripped; keep prog.debug in a symbols store indexed by build-id

部署prog.stripped；将prog.debug存储在按构建ID索引的符号仓库中

undefined

undefined

9. Quick triage from core without full debug session

9. 无需完整调试会话快速排查core文件

bash

undefined

bash

undefined

Print backtrace non-interactively

非交互式打印调用栈

gdb -batch -ex 'bt full' -ex 'thread apply all bt full' ./prog core 2>&1 | tee crash.txt

Print registers

打印寄存器信息

gdb -batch -ex 'info registers' ./prog core

Check signal that caused crash

检查导致崩溃的信号

gdb -batch -ex 'info signal' ./prog core


For a full cheatsheet covering core pattern tokens, coredumpctl, GDB/LLDB commands, debuginfod servers, and strip/symbol workflows, see [references/cheatsheet.md](references/cheatsheet.md).

gdb -batch -ex 'info signal' ./prog core


如需涵盖core文件命名规则、coredumpctl、GDB/LLDB命令、debuginfod服务器、符号剥离/保留流程的完整速查表，请查看[references/cheatsheet.md](references/cheatsheet.md)。

core-dumps

Original

Translation

Core Dumps

Core Dumps（核心转储）

Purpose

用途

Triggers

触发场景

Workflow

操作流程

1. Enable core dumps (Linux)

1. 在Linux上启用Core Dump

Per-session (lost on logout)

会话级生效（登出后失效）

Persistent (add to /etc/security/limits.conf)

永久生效（添加到/etc/security/limits.conf）

Check current limit

检查当前限制

Set core pattern (where and how cores are named)

设置core文件命名规则（存储位置和命名方式）

Default: 'core' in CWD — often not useful

默认：当前工作目录下的'core'文件——通常实用性不强

%e = executable, %p = PID, %t = timestamp

%e = 可执行文件名, %p = 进程ID, %t = 时间戳

Persistent (add to /etc/sysctl.d/99-core.conf)

永久生效（添加到/etc/sysctl.d/99-core.conf）

2. systemd/coredumpctl (modern Linux)

2. systemd/coredumpctl（现代Linux系统）

List recent crashes

列出近期崩溃记录

Show details of the latest crash

显示最新崩溃的详细信息

Load latest crash in GDB

在GDB中加载最新崩溃的core文件

Load specific PID crash

加载指定进程ID的崩溃core文件

Export core file

导出core文件

3. Enable core dumps (macOS)

3. 在macOS上启用Core Dump

macOS uses /cores by default (must be root-writable)

macOS默认存储在/cores目录（需具备root写入权限）

Check

检查core文件

launchd-launched services: set in plist

launchd启动的服务：需在plist文件中配置

<key>HardResourceLimits</key>

<key>HardResourceLimits</key>

<dict><key>Core</key><integer>9223372036854775807</integer></dict>

<dict><key>Core</key><integer>9223372036854775807</integer></dict>

4. Analyse a core with GDB

4. 使用GDB分析core文件

Load binary and core

加载二进制文件和core文件

If the binary was stripped, provide the unstripped copy

如果二进制文件已被剥离符号，提供未剥离符号的副本

Essential first commands

核心初始命令

All threads (multi-threaded crash)

所有线程信息（多线程崩溃场景）

5. Analyse a core with LLDB

5. 使用LLDB分析core文件

Or

或者

Commands

常用命令

6. Missing symbols: debuginfod

6. 缺失符号：使用debuginfod

Install client (Debian/Ubuntu)

安装客户端（Debian/Ubuntu）

Enable (add to ~/.bashrc or /etc/environment)

启用（添加到~/.bashrc或/etc/environment）

GDB auto-fetches symbols when DEBUGINFOD_URLS is set

设置DEBUGINFOD_URLS后，GDB会自动获取符号

Manually query

手动查询

7. Missing symbols: manual approach

7. 缺失符号：手动处理方式

Check if binary has a build ID