最近在 Ubuntu 18.04 上升级了内核版本到 v5.3.15,使用 crash 工具时发现无法分析 dump 文件,日志如下。
log:
root@node:/var/crash# crash vmlinux /var/crash/201912251143/dump.201912251143
crash 7.2.7++
Copyright (C) 2002-2019 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu"...
crash: page excluded: kernel virtual address: ffff0000124d4e58 type: "shadow_timekeeper xtime_sec"
crash: page excluded: kernel virtual address: ffff000012094a78 type: "pv_ops"
crash: page excluded: kernel virtual address: ffff801fb9de0058 type: "IRQ stack pointer"
crash: page excluded: kernel virtual address: ffff801fb9fc2058 type: "IRQ stack pointer"
crash: page excluded: kernel virtual address: ffff801fba1a4058 type: "IRQ stack pointer"
crash: page excluded: kernel virtual address: ffff801fba386058 type: "IRQ stack pointer"
crash: page excluded: kernel virtual address: ffff801fba568058 type: "IRQ stack pointer"
crash: page excluded: kernel virtual address: ffff801fba74a058 type: "IRQ stack pointer"
crash: page excluded: kernel virtual address: ffff801fba92c058 type: "IRQ stack pointer"
....
crash: page excluded: kernel virtual address: ffff80af9ba1e058 type: "IRQ stack pointer"
crash: page excluded: kernel virtual address: ffff000012531478 type: "high_memory"
我将内核版本恢复到v5.0.0,手动触发crash,用crash分析dump文件,一切正常。不知道原因,帮帮忙。
environment:
root@node:/var/crash# lsb
lsblk lsb_release
root@node:/var/crash# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
root@node:/var/crash# file /bin/ls
/bin/ls: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=de05fcef79d88af9cf9a71ed38e73af0b179bfb2, stripped
root@k8snode:/var/crash# dpkg -l | grep kexe
ii kexec-tools 1:2.0.16-1ubuntu1.1 arm64 tools to support fast kexec reboots
root@node:/var/crash#
crash with -d8 It seems that there is a problem with parsing.
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 24d4e58/24d4 crash: page excluded: kernel virtual address: ffff0000124d4e58 type: "shadow_timekeeper xtime_sec" xtime timespec.tv_sec: fffffa943b10: Wed Jan 20 03:10:08 8921554 read_diskdump: paddr/pfn: 1e17dec/1e17 -> cache physical page: 1e17000 utsname: sysname: Linux nodename: k8snode release: 5.3.15-050315.2019121301-generic version: #appstream SMP PREEMPT Tue Dec 24 16:06:52 CST 2019 machine: aarch64 domainname: (none) base kernel version: 5.3.15 ....
debug the living the v5.3.15 , the crash tool work well. root@node:/var/crash# crash vmlinux /proc/kcore
crash 7.2.7++ Copyright (C) 2002-2019 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "aarch64-unknown-linux-gnu"...
KERNEL: vmlinux
DUMPFILE: /proc/kcore
CPUS: 64
DATE: Thu Jan 2 16:20:24 2020
UPTIME: 02:02:33
LOAD AVERAGE: 0.57, 0.28, 0.21 TASKS: 1121 NODENAME: k8snode RELEASE: 5.3.15-050315.2019121301-generic VERSION: #appstream SMP PREEMPT Tue Dec 24 16:06:52 CST 2019 MACHINE: aarch64 (unknown Mhz) MEMORY: 255.7 GB PID: 5588 COMMAND: "crash" TASK: ffff809f97551b00 [THREAD_INFO: ffff809f97551b00] CPU: 63 STATE: TASK_RUNNING (ACTIVE)
crash> ... I think kexec-tools's version is probably old. I install kexec-tools v2.0.20 from https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git. But the same error reporting.