makedumpfile issues many readpage_elf: Attempt to read non-existent page

Discussion:

Louis Bouchard

2016-09-22 13:30:03 UTC

Hello,

I am investigating an issue with makedumpfile and kernel 4.8 where makedumpfile
(1.6.0) exits on error with the following message :

get_mem_map: Can't distinguish the memory type.

I found commit 2c21d4656e8d3c2af2b1e14809d076941ae69e96 in the upstream
development branch that is supposed to fix this :

[PATCH v2] Support _count -> _refcount rename in struct page

_count member was renamed to _refcount in linux commit 0139aa7b7fa12
("mm: rename _count, field of the struct page, to _refcount") and this
broke makedumpfile. The reason for making the change was to find all users
accessing it directly and not through the recommended API. I tried
suggesting to revert the change but failed, I see no other choice than to
start supporting both _count and _refcount in makedumpfile.

Though, when I apply the patch and test on either Ubuntu's 4.8.0-11 kernel, or
kernel.org's mainline 4.8.0-040800rc7 kernel, I get the following repeated

makedumpfile -c -d 31 /proc/vmcore /var/crash/201609221517/dump-incomplete
[ 7.513337] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
[ 7.524186] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78000, size:16
[ 7.528440] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78000).
[ 7.536562] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
[ 7.544356] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78010, size:16
[ 7.552422] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78010).
[ 7.560317] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
[ 7.568422] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78020, size:16
[ 7.572296] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78020).

I also tested with all the commits in the upstream/development branch with no
luck, I still get the same behavior.

Does someone have an idea of where this could come from ?

TIA,

Louis

--
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61

Atsushi Kumagai

2016-09-23 12:04:42 UTC

Permalink

Post by Louis Bouchard
Hello,
I am investigating an issue with makedumpfile and kernel 4.8 where makedumpfile
get_mem_map: Can't distinguish the memory type.
I found commit 2c21d4656e8d3c2af2b1e14809d076941ae69e96 in the upstream
[PATCH v2] Support _count -> _refcount rename in struct page
_count member was renamed to _refcount in linux commit 0139aa7b7fa12
("mm: rename _count, field of the struct page, to _refcount") and this
broke makedumpfile. The reason for making the change was to find all users
accessing it directly and not through the recommended API. I tried
suggesting to revert the change but failed, I see no other choice than to
start supporting both _count and _refcount in makedumpfile.
Though, when I apply the patch and test on either Ubuntu's 4.8.0-11 kernel, or
kernel.org's mainline 4.8.0-040800rc7 kernel, I get the following repeated

I also tested with all the commits in the upstream/development branch with no
luck, I still get the same behavior.
Does someone have an idea of where this could come from ?

Thanks for your report, but I'm afraid that I'm going on
vacation till Oct 2.
I hope someone helps you until I'm back.

Regards,
Atsushi Kumagai

Post by Louis Bouchard
TIA,
Louis
--
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61
_______________________________________________
kexec mailing list
http://lists.infradead.org/mailman/listinfo/kexec

Louis Bouchard

2016-09-29 13:34:53 UTC

Permalink

Hello,

Post by Atsushi Kumagai

Post by Louis Bouchard
Does someone have an idea of where this could come from ?

Thanks for your report, but I'm afraid that I'm going on
vacation till Oct 2.
I hope someone helps you until I'm back.
Regards,
Atsushi Kumagai

As a follow up to my question, I've been investigating the issue. The source of
the problem seems to come from readpage_elf() which tries to read to an
inexistant page :

readpage_elf: Attempt to read non-existent page at 0x1a7b7fff5000.

The error happens when readpage_elf() is called from section_mem_map_addr with
pgaddr, which is derived from paddr. This one comes from :

paddr = vaddr_to_paddr(addr)

When tracing vaddr_to_paddr, I see a difference in the calculation. on a 4.4
kernel :

info->phys_base = 0

On 4.8 :
info->phys_base = 0xffffffffe3800000

So the paddr values used to define pgaddr are :

for 4.4 kernels : 0x3fff4000
for 4.8 kernels : 0x1a7b7fff5000

Running makedumpfile --dump-dmesg in debug mode leads to :

4.4 kernel :

sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
phys_start : 1000000
phys_end : 2224000
virt_start : ffffffff81000000
virt_end : ffffffff82224000
LOAD (1)
phys_start : 1000
phys_end : 9fc00
virt_start : ffff880000001000
virt_end : ffff88000009fc00
LOAD (2)
phys_start : 100000
phys_end : 2b000000
virt_start : ffff880000100000
virt_end : ffff88002b000000
LOAD (3)
phys_start : 33000000
phys_end : 3fffe000
virt_start : ffff880033000000
virt_end : ffff88003fffe000
Linux kdump
page_size : 4096

max_mapnr : 3fffe

Buffer size for the cyclic mode: 65535

num of NODEs : 1

Memory type : SPARSEMEM_EX

mem_map (0)
mem_map : ffffea0000000000
pfn_start : 0
pfn_end : 8000
mem_map (1)
mem_map : ffffea0000200000
pfn_start : 8000
pfn_end : 10000
mem_map (2)
mem_map : ffffea0000400000
pfn_start : 10000
pfn_end : 18000
mem_map (3)
mem_map : ffffea0000600000
pfn_start : 18000
pfn_end : 20000
mem_map (4)
mem_map : ffffea0000800000
pfn_start : 20000
pfn_end : 28000
mem_map (5)
mem_map : ffffea0000a00000
pfn_start : 28000
pfn_end : 30000
mem_map (6)
mem_map : ffffea0000c00000
pfn_start : 30000
pfn_end : 38000
mem_map (7)
mem_map : ffffea0000e00000
pfn_start : 38000
pfn_end : 3fffe
mmap() is available on the kernel.

log_buf : ffffffff8210521c
log_end : 0
log_buf_len : 262144
log_first_idx : 0
log_next_idx : 141176

The dmesg log is saved to /tmp/titi.

makedumpfile Completed.

For 4.8 kernels :
sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
phys_start : 7200000
phys_end : 847d000
virt_start : ffffffffa3a00000
virt_end : ffffffffa4c7d000
LOAD (1)
phys_start : 1000
phys_end : 9fc00
virt_start : ffff880000001000
virt_end : ffff88000009fc00
LOAD (2)
phys_start : 100000
phys_end : 2b000000
virt_start : ffff880000100000
virt_end : ffff88002b000000
LOAD (3)
phys_start : 33000000
phys_end : 3fffe000
virt_start : ffff880033000000
virt_end : ffff88003fffe000
Linux kdump
page_size : 4096

max_mapnr : 3fffe

Buffer size for the cyclic mode: 65535
The kernel version is not supported.
The makedumpfile operation may be incomplete.

num of NODEs : 1

Memory type : SPARSEMEM_EX

mem_map (0)
mem_map : 0
pfn_start : 0
pfn_end : 8000
mem_map (1)
mem_map : 0
pfn_start : 8000
pfn_end : 10000
mem_map (2)
mem_map : 0
pfn_start : 10000
pfn_end : 18000
mem_map (3)
mem_map : 0
pfn_start : 18000
pfn_end : 20000
mem_map (4)
mem_map : 0
pfn_start : 20000
pfn_end : 28000
mem_map (5)
mem_map : 0
pfn_start : 28000
pfn_end : 30000
mem_map (6)
mem_map : 0
pfn_start : 30000
pfn_end : 38000
mem_map (7)
mem_map : 0
pfn_start : 38000
pfn_end : 3fffe
mmap() is available on the kernel.

log_buf : ffffffffa4b3c19c
log_end : 0
log_buf_len : 262144
log_first_idx : 0
log_next_idx : 43160

The dmesg log is saved to /tmp/toto.

makedumpfile Completed.

I don't know if it is to be expected with kernels 4.8 to have all mem_map
addresses set to 0, unlike with k4.4 :

mem_map (0)
mem_map : 0
pfn_start : 0
pfn_end : 8000

Any comment appreciated while I continue to investigate.

Kind regards,

...Louis

--
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61

Louis Bouchard

2016-10-24 14:54:02 UTC

Permalink

Hello,

Post by Louis Bouchard
Hello,

Post by Atsushi Kumagai

Post by Louis Bouchard
Does someone have an idea of where this could come from ?

Thanks for your report, but I'm afraid that I'm going on
vacation till Oct 2.
I hope someone helps you until I'm back.
Regards,
Atsushi Kumagai

As a follow up to my question, I've been investigating the issue. The source of
the problem seems to come from readpage_elf() which tries to read to an
readpage_elf: Attempt to read non-existent page at 0x1a7b7fff5000.
The error happens when readpage_elf() is called from section_mem_map_addr with
paddr = vaddr_to_paddr(addr)
When tracing vaddr_to_paddr, I see a difference in the calculation. on a 4.4
info->phys_base = 0
info->phys_base = 0xffffffffe3800000
for 4.4 kernels : 0x3fff4000
for 4.8 kernels : 0x1a7b7fff5000
sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
phys_start : 1000000
phys_end : 2224000
virt_start : ffffffff81000000
virt_end : ffffffff82224000
LOAD (1)
phys_start : 1000
phys_end : 9fc00
virt_start : ffff880000001000
virt_end : ffff88000009fc00
LOAD (2)
phys_start : 100000
phys_end : 2b000000
virt_start : ffff880000100000
virt_end : ffff88002b000000
LOAD (3)
phys_start : 33000000
phys_end : 3fffe000
virt_start : ffff880033000000
virt_end : ffff88003fffe000
Linux kdump
page_size : 4096
max_mapnr : 3fffe
Buffer size for the cyclic mode: 65535
num of NODEs : 1
Memory type : SPARSEMEM_EX
mem_map (0)
mem_map : ffffea0000000000
pfn_start : 0
pfn_end : 8000
mem_map (1)
mem_map : ffffea0000200000
pfn_start : 8000
pfn_end : 10000
mem_map (2)
mem_map : ffffea0000400000
pfn_start : 10000
pfn_end : 18000
mem_map (3)
mem_map : ffffea0000600000
pfn_start : 18000
pfn_end : 20000
mem_map (4)
mem_map : ffffea0000800000
pfn_start : 20000
pfn_end : 28000
mem_map (5)
mem_map : ffffea0000a00000
pfn_start : 28000
pfn_end : 30000
mem_map (6)
mem_map : ffffea0000c00000
pfn_start : 30000
pfn_end : 38000
mem_map (7)
mem_map : ffffea0000e00000
pfn_start : 38000
pfn_end : 3fffe
mmap() is available on the kernel.
log_buf : ffffffff8210521c
log_end : 0
log_buf_len : 262144
log_first_idx : 0
log_next_idx : 141176
The dmesg log is saved to /tmp/titi.
makedumpfile Completed.
sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
phys_start : 7200000
phys_end : 847d000
virt_start : ffffffffa3a00000
virt_end : ffffffffa4c7d000
LOAD (1)
phys_start : 1000
phys_end : 9fc00
virt_start : ffff880000001000
virt_end : ffff88000009fc00
LOAD (2)
phys_start : 100000
phys_end : 2b000000
virt_start : ffff880000100000
virt_end : ffff88002b000000
LOAD (3)
phys_start : 33000000
phys_end : 3fffe000
virt_start : ffff880033000000
virt_end : ffff88003fffe000
Linux kdump
page_size : 4096
max_mapnr : 3fffe
Buffer size for the cyclic mode: 65535
The kernel version is not supported.
The makedumpfile operation may be incomplete.
num of NODEs : 1
Memory type : SPARSEMEM_EX
mem_map (0)
mem_map : 0
pfn_start : 0
pfn_end : 8000
mem_map (1)
mem_map : 0
pfn_start : 8000
pfn_end : 10000
mem_map (2)
mem_map : 0
pfn_start : 10000
pfn_end : 18000
mem_map (3)
mem_map : 0
pfn_start : 18000
pfn_end : 20000
mem_map (4)
mem_map : 0
pfn_start : 20000
pfn_end : 28000
mem_map (5)
mem_map : 0
pfn_start : 28000
pfn_end : 30000
mem_map (6)
mem_map : 0
pfn_start : 30000
pfn_end : 38000
mem_map (7)
mem_map : 0
pfn_start : 38000
pfn_end : 3fffe
mmap() is available on the kernel.
log_buf : ffffffffa4b3c19c
log_end : 0
log_buf_len : 262144
log_first_idx : 0
log_next_idx : 43160
The dmesg log is saved to /tmp/toto.
makedumpfile Completed.
I don't know if it is to be expected with kernels 4.8 to have all mem_map
mem_map (0)
mem_map : 0
pfn_start : 0
pfn_end : 8000
Any comment appreciated while I continue to investigate.
Kind regards,
...Louis

I have now completed the kernel bisection between 4.7.8 and 4.8-rc1 and

Post by Louis Bouchard
commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
Date: Tue Jun 21 17:47:03 2016 -0700
x86/mm: Enable KASLR for physical mapping memory regions
Add the physical mapping in the list of randomized memory regions.
The physical memory mapping holds most allocations from boot and heap
allocators. Knowing the base address and physical memory size, an attacker
can deduce the PDE virtual address for the vDSO memory page. This attack
https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/blob/master/Presentation/CanSec2016_Presentation.pdf
(See second part of the presentation).
The exploits used against Linux worked successfully against 4.6+ but
https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/tree/master/Demos/Linux/exploits
Similar research was done at Google leading to this patch proposal.
Variants exists to overwrite /proc or /sys objects ACLs leading to
elevation of privileges. These variants were tested against 4.6+.
The page offset used by the compressed kernel retains the static value
since it is not yet randomized during this boot stage.

Post by Louis Bouchard
-#define __PAGE_OFFSET _AC(0xffff880000000000, UL)
+#define __PAGE_OFFSET_BASE _AC(0xffff880000000000, UL)
+#ifdef CONFIG_RANDOMIZE_MEMORY
+#define __PAGE_OFFSET page_offset_base
+#else
+#define __PAGE_OFFSET __PAGE_OFFSET_BASE
+#endif /* CONFIG_RANDOMIZE_MEMORY */

I'll try to see if I can fix that.

Kind regards,

...Louis

--
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61

Louis Bouchard

2016-10-25 08:59:23 UTC

Permalink

Good morning,

Post by Louis Bouchard
Hello,

Post by Atsushi Kumagai

Post by Louis Bouchard
Does someone have an idea of where this could come from ?

Thanks for your report, but I'm afraid that I'm going on
vacation till Oct 2.
I hope someone helps you until I'm back.
Regards,
Atsushi Kumagai

I have now completed the kernel bisection between 4.7.8 and 4.8-rc1 and

I'll try to see if I can fix that.
Kind regards,
...Louis

Some more *important* information in this mostly monologue thread : Pratyush
Anand has pushed a patch to the list earlier today that apparently fixes this
issue :

[PATCH Makedumpfile 1/4] x86_64: Calculate page_offset from pt_load[1]

HTH,

Kind regards,

...Louis

[1] https://www.mail-archive.com/***@lists.infradead.org/msg16628.html

--
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61

Atsushi Kumagai

2016-10-25 09:19:40 UTC

Permalink

Hello,

Post by Louis Bouchard

Post by Louis Bouchard
I have now completed the kernel bisection between 4.7.8 and 4.8-rc1 and

https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/blob/master/Prese
ntation/CanSec2016_Presentation.pdf

Post by Louis Bouchard

Post by Louis Bouchard
(See second part of the presentation).
The exploits used against Linux worked successfully against 4.6+ but

https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/tree/master/Demos
/Linux/exploits

Post by Louis Bouchard

Post by Louis Bouchard
Similar research was done at Google leading to this patch proposal.
Variants exists to overwrite /proc or /sys objects ACLs leading to
elevation of privileges. These variants were tested against 4.6+.
The page offset used by the compressed kernel retains the static value
since it is not yet randomized during this boot stage.

I'll try to see if I can fix that.
Kind regards,
...Louis

Some more *important* information in this mostly monologue thread : Pratyush
Anand has pushed a patch to the list earlier today that apparently fixes this
[PATCH Makedumpfile 1/4] x86_64: Calculate page_offset from pt_load[1]
HTH,
Kind regards,

Yeah, It appears so. I'm reviewing the patches,
please wait for that.
I appreciate your investigation for this issue.

Thanks,
Atsushi Kumagai

Post by Louis Bouchard
...Louis
--
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61