makedumpfile: support for newer kernels [v4.9 onwards]

[Abhishek Shah]

Hi All,

I am using makedumpfile utility to compress the ramdump taken by kexec/kdump,
i.e. /proc/vmcore, on arm64 platform with kernel v4.12-rc3.

/proc/vmcore works fine with crash utility without passing it though
makedumpfile, but the file generated by makedumpfile is not getting
processed by crash utility; I am getting following error :

crash: seek error: kernel virtual address: ffff8008000c16a0 type:
"memory section"

I am using kernel v4.12-RC3; makedumpfile currently seems to support
kernel up to 4.8 as per documentation.
Is there plan to support newer kernel? can you provide some pointers
to modifications required for newer kernel support?


Regards,
Abhishek

[Pratyush Anand]

Hi Pratyush,
I have posted the same question on the list as well....
Maybe I will copy your response there and continue discussion...

Ah..I was on PTO, and just chasing with volume of mails.
I see your mail at list :-)

I am using yocto(2.3 version) to build makedumpfile, which fetches v1.6.1
makedumpfile and also applys some patches including arm64 specific patches.

What are these ARM64 specific patches? I believe, you do not need any topup in
makedumpfile for ARM64 if you use v1.6.1.

We have VA_BITS set as 48 bits... I will get other information and post on the
list.

OK.

Regards,
Abhishek
Hi Abhishek,
Hi Atsushi,
I am using makedumpfile utility to compress the ramdump taken by
kexec/kdump,
i.e. /proc/vmcore, on arm64 platform.
/proc/vmcore works fine with crash utility without passing it though
makedumpfile, but the file generated by makedumpfile is not getting
processed
"memory
section"
Makedumpfile currently seems to support kernel up to 4.8 as per
documentation.
Is there plan to support newer kernel? can you provide some pointers to
modifications required for newer kernel support, if I can help?
Are you using makedumpfile-1.6.1? I expect that it should work with latest
ARM64 kernel as well. If it does not work, can you please share your
kernel configuration specially about,
VA_BITS, PAGE_SHIFT, RANDOMIZE_BASE etc.
..and please keep mailing list in CC so that you can get answer from more
people. Also, other can get benefit from this conversation.
~Pratyush

~Pratyush

[Abhishek Shah]

Hi Pratyush,

Post by Pratyush Anand

I am using yocto(2.3 version) to build makedumpfile, which fetches v1.6.1
makedumpfile and also applys some patches including arm64 specific patches.

What are these ARM64 specific patches? I believe, you do not need any topup
in makedumpfile for ARM64 if you use v1.6.1.

Sorry, I got confused with yocto kexec recipe.
There are NO arm64 specific patches in yocto for makedumpfile.

Post by Pratyush Anand

We have VA_BITS set as 48 bits... I will get other information and post on the
list.

OK.

Kernel config:
ARM64_VA_BITS_48 [=y]
ARM64_PAGE_SHIFT [=12]
RANDOMIZE_BASE [=n]
ARM64_4K_PAGES [=y]

Regards,
Abhishek

[Pratyush Anand]

Hi Abhishek,

Hi Pratyush,

Post by Pratyush Anand

I am using yocto(2.3 version) to build makedumpfile, which fetches v1.6.1
makedumpfile and also applys some patches including arm64 specific patches.

What are these ARM64 specific patches? I believe, you do not need any topup
in makedumpfile for ARM64 if you use v1.6.1.

Sorry, I got confused with yocto kexec recipe.
There are NO arm64 specific patches in yocto for makedumpfile.

Post by Pratyush Anand

We have VA_BITS set as 48 bits... I will get other information and post on the
list.

OK.

ARM64_VA_BITS_48 [=y]
ARM64_PAGE_SHIFT [=12]
RANDOMIZE_BASE [=n]
ARM64_4K_PAGES [=y]

These are the only variable configs on which arm64 makedumpfile depends. I
tested with above configuration and things were working fine.

I used 4.11 based kernel and crash-7.1.9.

BTW, which version of crash utility do you use. If you are not at latest can
you please try with latest.

~Pratyush

CCed Dave if he has some input.

/proc/vmcore works fine with crash utility without passing it though
makedumpfile, but the file generated by makedumpfile is not getting
crash: seek error: kernel virtual address: ffff8008000c16a0 type: "memory
section"

[Abhishek Shah]

Hi Pratyush/Dave,

Hi Abhishek,

Hi Pratyush,
ARM64_VA_BITS_48 [=y]
ARM64_PAGE_SHIFT [=12]
RANDOMIZE_BASE [=n]
ARM64_4K_PAGES [=y]

These are the only variable configs on which arm64 makedumpfile depends. I
tested with above configuration and things were working fine.
I used 4.11 based kernel and crash-7.1.9.
BTW, which version of crash utility do you use. If you are not at latest can
you please try with latest.

I was using crash 7.1.9++ with head commit
5c52842a58a2602dba81de71831af98b2b53c6e0;
I have upgraded my kernel and built the crash utility again; and
following is my setup details now:

kernel: 4.12- rc5
yocto based rootfs: Pyro[2.3] which has kexec 2.0.14 and makedumpfile 1.6.1.
crash: 7.1.9++ [Head: 4d517ad28acd845fe6e91360e645cf0446a4757b]

The output generated by makedumpfile gets processed by crash utility
if I use following command:
makedumpfile -E /proc/vmcore vmcore

But, the output generated by following command (which I used while
reporting the issue) gives the "seek error" while being processed by
crash.
makedumpfile /proc/vmcore vmcore

(As a result?), output from following command gives "seek error" as
well while being processed by crash:
makedumpfile -c /proc/vmcore vmcore

makedumpfile Manual suggests:
"-E
Create DUMPFILE in the ELF format.
This option cannot be specified with -c option, because the ELF format
does not support compressed data."
So only elf formatted output from makedumpfile gets processed by crash.

@Pratyush, Thanks for testing and verifying the kernel configuration
for arm64 makedumpfile.
When you say things were working fine for you, what exact
makedumpfile command did you use??

Regards,
Abhishek

[Atsushi Kumagai]

Hello,

I would like to inform you that I can't reply on this thread next week
because of vacation, Thanks for your cooperation.

Post by Abhishek Shah
Is there plan to support newer kernel?

I'm going to release the next version in July, it will support kernel 4.11.

Thanks,
Atsushi Kumagai

Post by Abhishek Shah
Hi Pratyush/Dave,

Hi Abhishek,

Hi Pratyush,
ARM64_VA_BITS_48 [=y]
ARM64_PAGE_SHIFT [=12]
RANDOMIZE_BASE [=n]
ARM64_4K_PAGES [=y]

These are the only variable configs on which arm64 makedumpfile depends. I
tested with above configuration and things were working fine.
I used 4.11 based kernel and crash-7.1.9.
BTW, which version of crash utility do you use. If you are not at latest can
you please try with latest.

I was using crash 7.1.9++ with head commit
5c52842a58a2602dba81de71831af98b2b53c6e0;
I have upgraded my kernel and built the crash utility again; and
kernel: 4.12- rc5
yocto based rootfs: Pyro[2.3] which has kexec 2.0.14 and makedumpfile 1.6.1.
crash: 7.1.9++ [Head: 4d517ad28acd845fe6e91360e645cf0446a4757b]
The output generated by makedumpfile gets processed by crash utility
makedumpfile -E /proc/vmcore vmcore
But, the output generated by following command (which I used while
reporting the issue) gives the "seek error" while being processed by
crash.
makedumpfile /proc/vmcore vmcore
(As a result?), output from following command gives "seek error" as
makedumpfile -c /proc/vmcore vmcore
"-E
Create DUMPFILE in the ELF format.
This option cannot be specified with -c option, because the ELF format
does not support compressed data."
So only elf formatted output from makedumpfile gets processed by crash.
@Pratyush, Thanks for testing and verifying the kernel configuration
for arm64 makedumpfile.
When you say things were working fine for you, what exact
makedumpfile command did you use??
Regards,
Abhishek
_______________________________________________
kexec mailing list
http://lists.infradead.org/mailman/listinfo/kexec

[Pratyush Anand]

Post by Abhishek Shah
@Pratyush, Thanks for testing and verifying the kernel configuration
for arm64 makedumpfile.
When you say things were working fine for you, what exact
makedumpfile command did you use??

makedumpfile -l --message-level 1 -d 31 /proc/vmcore /var/crash/vmcore

~Pratyush

[Abhishek Shah]

Hi Pratyush,

Post by Pratyush Anand

Post by Abhishek Shah
@Pratyush, Thanks for testing and verifying the kernel configuration
for arm64 makedumpfile.
When you say things were working fine for you, what exact
makedumpfile command did you use??

makedumpfile -l --message-level 1 -d 31 /proc/vmcore /var/crash/vmcore

I am getting "seek error" as reported earlier if I use above command
to generate dump in my setup:
crash: seek error: kernel virtual address: ffff8008000c2020 type:
"memory section"


Regards,
Abhishek

[Abhishek Shah]

Hi All,

I was finally able to root cause the issue that I was facing in the
kdump compressed output
generated by makedumpfile.

While debugging "write_kdump_pages_and_bitmap_cyclic"; I came to know that
somehow "info->max_mapnr" was modified/ reduced.

Seeing further, I could see that the "get_mem_map" function at the end
[if (!is_xen_memory())]
is attempting to change "info->max_mapnr" value based on "mem=" option.
-- in particular commit ebe2fa3a5c1e2bfac3778c0fbeca4287a934cc58.
And in my kernel command line, "mem=2G" was present, as I intended to
take dump of 2G RAM,
resulting into reduced "info->max_mapnr".

So, I commented out this portion and confirmed that "info->max_mapnr"
remains same
throughout. And I was able to get kdump compressed vmcore process-able
by crash utility.

Just to summarize:
Following was the problem statement: crash was giving "seek error"
while makedumpfile generated kdump-compressed vmcore was getting analyzed.

Post by Abhishek Shah
The output generated by makedumpfile gets processed by crash utility
makedumpfile -E /proc/vmcore vmcore
But, the output generated by following command (which I used while
reporting the issue) gives the "seek error" while being processed by
crash.
makedumpfile /proc/vmcore vmcore
(As a result?), output from following command gives "seek error" as
makedumpfile -c /proc/vmcore vmcore

Post by Abhishek Shah
ARM64_VA_BITS_48 [=y]
ARM64_PAGE_SHIFT [=12]
RANDOMIZE_BASE [=n]
ARM64_4K_PAGES [=y]

kernel: 4.12- rc5
yocto based rootfs: Pyro[2.3] which has kexec 2.0.14 and makedumpfile 1.6.1.
crash: 7.1.9++ [Head: 4d517ad28acd845fe6e91360e645cf0446a4757b]

Thanks,
Abhishek

[Atsushi Kumagai]

Hello Abhishek,

Thanks for your investigation.

Post by Abhishek Shah
I was finally able to root cause the issue that I was facing in the
kdump compressed output
generated by makedumpfile.
While debugging "write_kdump_pages_and_bitmap_cyclic"; I came to know that
somehow "info->max_mapnr" was modified/ reduced.
Seeing further, I could see that the "get_mem_map" function at the end
[if (!is_xen_memory())]
is attempting to change "info->max_mapnr" value based on "mem=" option.
-- in particular commit ebe2fa3a5c1e2bfac3778c0fbeca4287a934cc58.
And in my kernel command line, "mem=2G" was present, as I intended to
take dump of 2G RAM,
resulting into reduced "info->max_mapnr".

Basically, this adjusting info->max_mapnr must be harmless since
it just intends to ignore unused memory regions.
Did you find out why adjusting info->max_mapnr causes the problem ?

Post by Abhishek Shah
So, I commented out this portion and confirmed that "info->max_mapnr"
remains same
throughout. And I was able to get kdump compressed vmcore process-able
by crash utility.

This sounds like there are some valid data beyond the adjusted info->max_mapnr,
is that true ?

Thanks,
Atsushi Kumagai

Post by Abhishek Shah
Following was the problem statement: crash was giving "seek error"
while makedumpfile generated kdump-compressed vmcore was getting analyzed.

Post by Abhishek Shah
The output generated by makedumpfile gets processed by crash utility
makedumpfile -E /proc/vmcore vmcore
But, the output generated by following command (which I used while
reporting the issue) gives the "seek error" while being processed by
crash.
makedumpfile /proc/vmcore vmcore
(As a result?), output from following command gives "seek error" as
makedumpfile -c /proc/vmcore vmcore

Post by Abhishek Shah
ARM64_VA_BITS_48 [=y]
ARM64_PAGE_SHIFT [=12]
RANDOMIZE_BASE [=n]
ARM64_4K_PAGES [=y]

kernel: 4.12- rc5
yocto based rootfs: Pyro[2.3] which has kexec 2.0.14 and makedumpfile 1.6.1.
crash: 7.1.9++ [Head: 4d517ad28acd845fe6e91360e645cf0446a4757b]

Thanks,
Abhishek

[Abhishek Shah]

Post by Atsushi Kumagai

Post by Abhishek Shah
Seeing further, I could see that the "get_mem_map" function at the end
[if (!is_xen_memory())]
is attempting to change "info->max_mapnr" value based on "mem=" option.
-- in particular commit ebe2fa3a5c1e2bfac3778c0fbeca4287a934cc58.
And in my kernel command line, "mem=2G" was present, as I intended to
take dump of 2G RAM,
resulting into reduced "info->max_mapnr".

Basically, this adjusting info->max_mapnr must be harmless since
it just intends to ignore unused memory regions.
Did you find out why adjusting info->max_mapnr causes the problem ?

Operation of "write_kdump_pages_and_bitmap_cyclic" depends on
info->max_mapnr value.
That is the reason why we have problem with kdump compressed dump, but
not with elf formatted dump.

From the memory regions that I have, the last region does not seem to
be mapped (which contains the address sought by crash);
because of which, info->max_mapnr is getting adjusted to some reduced
value (compared to the previous info->max_mapnr of 880203).

Following is the memory regions that I have:
mem_map (0)
mem_map : ffff8008000c6280
pfn_start : 0
pfn_end : 40000
mem_map (1)
mem_map : 0
pfn_start : 40000
pfn_end : 80000
mem_map (2)
mem_map : 0
pfn_start : 80000
pfn_end : c0000
mem_map (3)
mem_map : 0
pfn_start : c0000
pfn_end : 100000
mem_map (4)
mem_map : 0
pfn_start : 100000
pfn_end : 140000
mem_map (5)
mem_map : 0
pfn_start : 140000
pfn_end : 180000
mem_map (6)
mem_map : 0
pfn_start : 180000
pfn_end : 1c0000
mem_map (7)
mem_map : 0
pfn_start : 1c0000
pfn_end : 200000
mem_map (8)
mem_map : 0
pfn_start : 200000
pfn_end : 240000
mem_map (9)
mem_map : 0
pfn_start : 240000
pfn_end : 280000
mem_map (10)
mem_map : 0
pfn_start : 280000
pfn_end : 2c0000
mem_map (11)
mem_map : 0
pfn_start : 2c0000
pfn_end : 300000
mem_map (12)
mem_map : ffff800045136a80
pfn_start : 300000
pfn_end : 340000
mem_map (13)
mem_map : ffff800046136480
pfn_start : 340000
pfn_end : 380000
mem_map (14)
mem_map : ffff800047136c00
pfn_start : 380000
pfn_end : 3c0000
mem_map (15)
mem_map : 0
pfn_start : 3c0000
pfn_end : 400000
mem_map (16)
mem_map : 0
pfn_start : 400000
pfn_end : 440000
mem_map (17)
mem_map : 0
pfn_start : 440000
pfn_end : 480000
mem_map (18)
mem_map : 0
pfn_start : 480000
pfn_end : 4c0000
mem_map (19)
mem_map : 0
pfn_start : 4c0000
pfn_end : 500000
mem_map (20)
mem_map : 0
pfn_start : 500000
pfn_end : 540000
mem_map (21)
mem_map : 0
pfn_start : 540000
pfn_end : 580000
mem_map (22)
mem_map : 0
pfn_start : 580000
pfn_end : 5c0000
mem_map (23)
mem_map : 0
pfn_start : 5c0000
pfn_end : 600000
mem_map (24)
mem_map : 0
pfn_start : 600000
pfn_end : 640000
mem_map (25)
mem_map : 0
pfn_start : 640000
pfn_end : 680000
mem_map (26)
mem_map : 0
pfn_start : 680000
pfn_end : 6c0000
mem_map (27)
mem_map : 0
pfn_start : 6c0000
pfn_end : 700000
mem_map (28)
mem_map : 0
pfn_start : 700000
pfn_end : 740000
mem_map (29)
mem_map : 0
pfn_start : 740000
pfn_end : 780000
mem_map (30)
mem_map : 0
pfn_start : 780000
pfn_end : 7c0000
mem_map (31)
mem_map : 0
pfn_start : 7c0000
pfn_end : 800000
mem_map (32)
mem_map : 0
pfn_start : 800000
pfn_end : 840000
mem_map (33)
mem_map : 0
pfn_start : 840000
pfn_end : 880000
mem_map (34)
mem_map : 0
pfn_start : 880000
pfn_end : 880203

Post by Atsushi Kumagai

Post by Abhishek Shah
So, I commented out this portion and confirmed that "info->max_mapnr"
remains same
throughout. And I was able to get kdump compressed vmcore process-able
by crash utility.

This sounds like there are some valid data beyond the adjusted info->max_mapnr,
is that true ?

I am not sure about it, but crash does look for data in mem_map(34),
I haven't yet look into this in-detail. I will do that soon.


Regards,
Abhishek

[Dave Anderson]

----- Original Message -----

Hi Abhishek,

Hi Pratyush,

Post by Pratyush Anand

I am using yocto(2.3 version) to build makedumpfile, which fetches v1.6.1
makedumpfile and also applys some patches including arm64 specific patches.

What are these ARM64 specific patches? I believe, you do not need any topup
in makedumpfile for ARM64 if you use v1.6.1.

Sorry, I got confused with yocto kexec recipe.
There are NO arm64 specific patches in yocto for makedumpfile.

Post by Pratyush Anand

We have VA_BITS set as 48 bits... I will get other information and post
on the list.

OK.

ARM64_VA_BITS_48 [=y]
ARM64_PAGE_SHIFT [=12]
RANDOMIZE_BASE [=n]
ARM64_4K_PAGES [=y]

These are the only variable configs on which arm64 makedumpfile depends. I
tested with above configuration and things were working fine.
I used 4.11 based kernel and crash-7.1.9.
BTW, which version of crash utility do you use. If you are not at latest can
you please try with latest.
~Pratyush
CCed Dave if he has some input.

/proc/vmcore works fine with crash utility without passing it though
makedumpfile, but the file generated by makedumpfile is not getting
crash: seek error: kernel virtual address: ffff8008000c16a0 type: "memory
section"

A "seek error" means that the page for the physical address that was translated
from ffff8008000c16a0 could not be found in the dumpfile. The output from
"crash --d ..." might yield some additional clues.

However, given that crash apparently works with the 4.12 /proc/vmcore that the
dumpfile was generated from, it seemingly points to makedumpfile. That's always
the first thing I ask when seeing this kind of bug, i.e., does crash work with the
same kernel on a live system, or even better, with the initial ELF vmcore file
from which it was generated. If it does, then it's usually a makedumpfile issue,
because the virtual address gets translated to a physical address by the upper-level
crash code, and then passed down into whatever the underlying memory source is.
If it could be read from /proc/vmcore, then it should accessible from makedumpfile's
compressed kdump.

That being said, I haven't tested ARM64 on any 4.12-rc kernels.

Dave