AptioMemoryFix

[Pene]

Unfortunately, both new variants hanged for Linux (all of them worked for OSX, by the way).

By the way, the kernel panic you've been sent is for this situation (AptioMemoryFix before linux)?

 

And indeed, thanks for the workaround. If you have anything further to test, I'd be glad to help.

 

Edited May 27, 2018 by Pene

[vit9696]

That's astounding :/ Yet, even if I contact Linux team it won't be fix in most of the distributions in any foreseeable future :/

The only possible approach could be to try some heuristics here probably, which means a lot of mess but *sigh*

 

Could you try some of these, whether they work or not. Sorry for a lot of tests, but I am really at a loss in finding a suitable solution, when hibernation on macOS is broken from the root and works thanks to mere luck of handy addresses… 

EFI_PHYSICAL_ADDRESS RtShims = BASE_1GB;
  Status = gBS->AllocatePages (
    AllocateMaxAddress,
    EfiRuntimeServicesCode,
    EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart),
    &RtShims
    );
  gRtShims             = (VOID *)(UINTN)RtShims;

or

EFI_PHYSICAL_ADDRESS RtShims = BASE_2GB;
  Status = gBS->AllocatePages (
    AllocateMaxAddress,
    EfiRuntimeServicesCode,
    EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart),
    &RtShims
    );
  gRtShims             = (VOID *)(UINTN)RtShims;

or

EFI_PHYSICAL_ADDRESS RtShims = BASE_4GB;
  Status = gBS->AllocatePages (
    AllocateMaxAddress,
    EfiRuntimeServicesCode,
    EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart),
    &RtShims
    );
  gRtShims             = (VOID *)(UINTN)RtShims;

EFI_PHYSICAL_ADDRESS RtShims = BASE_1GB;
  Status = AllocatePagesFromTop (
    AllocateMaxAddress,
    EfiRuntimeServicesCode,
    EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart),
    &RtShims,
    FALSE
    );
  gRtShims             = (VOID *)(UINTN)RtShims;

or

EFI_PHYSICAL_ADDRESS RtShims = BASE_2GB;
  Status = AllocatePagesFromTop (
    AllocateMaxAddress,
    EfiRuntimeServicesCode,
    EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart),
    &RtShims,
    FALSE
    );
  gRtShims             = (VOID *)(UINTN)RtShims;

or

EFI_PHYSICAL_ADDRESS RtShims = BASE_4GB;
  Status = AllocatePagesFromTop (
    AllocateMaxAddress,
    EfiRuntimeServicesCode,
    EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart),
    &RtShims,
    FALSE
    );
  gRtShims             = (VOID *)(UINTN)RtShims;

Quote

By the way, the kernel panic you've been sent is for this situation (AptioMemoryFix before linux)?

Yes, it is like that:

[ 0.011000] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
...
[    0.011000] Call Trace:
[    0.011000]  ? efi_call+0x58/0x90
[    0.011000]  ? virt_efi_set_variable.part.6+0x79/0xf0
[    0.011000]  ? efi_delete_dummy_variable+0x5e/0x80
[    0.011000]  ? efi_enter_virtual_mode+0x4cd/0x4ef
[    0.011000]  ? start_kernel+0x481/0x529

 

Edited May 27, 2018 by vit9696

[Pene]

The first variant worked here for Linux only with BASE_1GB:

EFI_PHYSICAL_ADDRESS RtShims = BASE_1GB;
  Status = gBS->AllocatePages (
    AllocateMaxAddress,
    EfiRuntimeServicesCode,
    EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart),
    &RtShims
    );
  gRtShims             = (VOID *)(UINTN)RtShims;

 

In the second variant I assume you had a typo (AllocateMaxAddress is not needed and results in too many arguments), so assuming you meant the following, it worked here for Linux with all 3 cases (BASE_1GB, BASE_2GB, BASE_4GB):

EFI_PHYSICAL_ADDRESS RtShims = BASE_1GB; // works also with BASE_2GB and BASE_4GB
  Status = AllocatePagesFromTop (
    EfiRuntimeServicesCode,
    EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart),
    &RtShims,
    FALSE
    );
  gRtShims             = (VOID *)(UINTN)RtShims;

 

For OSX, all 6 variants worked.

Edited May 27, 2018 by Pene

[vit9696]

Yes, thanks for correcting me. AllocateMaxAddress behaviour is simply crazy, because AllocatePagesFromTop will allocate the pages at least as high gBS->AllocatePages, and it is simply insane that it with the increase of the top value AllocatePages starts to fail. Well, the spec does not prescribe the desired behaviour, so I suppose it is kind of ok to just assume that is broken.

 

Thanks for the report… I will see what can be done and will hopefully push a fix in a considerate future.

[Pene]

Yes, indeed broken behaviour.

 

Thank you for the collaboration with this issue.

If you need any further tests for determining a fix, feel free to mention.

[vit9696]

Updated at 8:00 MSK.

 

Ok, I have a workaround but I think may still want some tests just in case. The fix is committed here:

https://github.com/vit9696/AptioFixPkg/commit/82fffac9146cf6bd0f1acb9bf6fb8c5132520e0b

 

I attach a binary with the prints uncommented. These prints will show an address where RtShims got allocated.

 

I will need:

1. The address printed (and does it change across the reboots) - ideally as a screenshot.

2. Whether macOS/Windows/Linux boots

3. Whether hibernation works (if it did not work previously, skip this) — better test twice in a row (boot → hibernate → wake → hibernate → wake → reboot).

4. Whether sleep works (if it did not work previously, skip this)

5. Whether nvram works (skip, if it did not work previously), do not forget to check after hibernate wake.

 

Systems of most interest are X99, X299, Z68, Z77, but others are fine as well.

AptioMemoryFix-R19-PRNTS.efi

Edited May 28, 2018 by vit9696

[Pene]

Hello,

 

Your new workaround works fine here (Z370) on all 3 operating systems (Linux/Windows/OSX). Sleep/Nvram works fine also (Not sure about hibernation, not using it currently). Output follows:

RtShims of 1 pages allocate mode 0 attempt at 0x7FFFF000 - Not Found
RtShims of 1 pages allocate mode 0 attempt at 0x3FFFF000 - Success

The addresses are consistent across multiple reboots.

 

Thanks again.

Edited May 28, 2018 by Pene

[vit9696]

Ugh. I got a number of reports about broken hibernation once again. Sorry guys, but I removed all the hacks related to hibernation until a proper solution is found. While it does sound unpleasant, it is for your own good, as the existing hibernation code (which is all the same in AptioFixV2, V3, AptioMemoryFix) is a bomb that may randomly fail and/or corrupt the booted operating system resulting in unrecoverable damage.

 

Personally I cannot even try 'fixing' it, as Clover remains incapable of hibernating a FV2 volume (cc @Sherlocks, I rechecked, and yes it is still broken), but I will welcome all the research and fixes. CC @apianti, @Download-Fritz, @lvs1974, @nms, @Slice, @TheRacerMaster, @Zenith432, sorry, if I forgot anyone, but I hope you are monitoring.

 

This is the relevant commit: https://github.com/vit9696/AptioFixPkg/commit/f5c552b77e22ad80c472b34d570e69f9d8a54dba

And it contains most of the comments explaining the problem.

 

Basically the root issue is that one cannot pass the memory map when waking from hibernation. Almost certainly it needs some tiny/basic patching, yet to figure it out one will have to do some debugging sessions. The original "solution" was to ignore the new memory map and let XNU reuse the original one, this results in numerous memory corruptions: instant reboots after waking, hangs, visual glitches, broken nvram, crashing apps. In most cases it worked "fine"… until XNU memory subsystem overwrites some vital efi pointer or an SMM driver overwrites some XNU memory.

 

Everything else was trying to workaround the thing, for example, by allocating some lucky pointer for RtShims (which are necessary for APTIO V boards to get working NVRAM). The lucky pointer overlapped with KASLR and additionally broke some slides, so custom slide engine had to workaround it… for safe boot with enforced slide=0 there is a patch for boot.efi to force-enable KASLR, and now it broke Linux just because some pages did not allocate well @_@.

 

That's a question to address in the nearest future, as the hacks get almost incomprehensible. So I am kind of asking for help and attention from the others. Note, that the hibernation code in Clover may also want to be rewritten to support Bds-decoded IORTC* variables instead of boot-switch-vars, which modern boot.efi relies on.

[Pavo]

14 minutes ago, meaganmargaret said:

Okay, something in AptioMemoryFix is broken (I'm on 4458).

 

When I try to boot, the booting process gets stuck at:

 

Start RandomSeed

End RandomSeed

Error allocating 0x11d60 pages at 000000001531a000 alloc type 2

Couldn't allocate runtime area

.......(and at this point, its dead)....

 

I have an X99 with a 6950x processor, and to get my system to boot I had to go back OSX free 2000 and EMU.  I also tried the AptioMemoryFix from Clover 4509, and it did the same.  Does anyone have any idea what this error means or why with the newer versions of AptioMemoryFix it's showing up?

 

Anyone have the latest AptioMemoryFix and AptioInput (compiled versions)?

Here you go

AptioInputFix.efi

AptioMemoryFix.efi

[vit9696]

Using Free2000 turns your system in a bomb that may randomly crash.

Using 4G encoding makes your firmware less broken (as it stops allocating too much over xnu-reserved memory).

Pay attention to these recommendations. https://github.com/vit9696/AptioFixPkg#aptiomemoryfix

Edited June 2, 2018 by vit9696

[Pavo]

1 hour ago, vit9696 said:

Using Free2000 turns your system in a bomb that may randomly crash.

Using 4G encoding makes your firmware less broken (as it stops allocating too much over xnu-reserved memory).

Pay attention to these recommendations. https://github.com/vit9696/AptioFixPkg#aptiomemoryfix

@vit9696 would you know why when I enabled 4G Encoding I lose my audio device on Asus Z9PE-D8 WS motherboard?

Edited June 2, 2018 by Pavo

[vit9696]

Unfortunately not, might be some bug in the drivers, but I do not work in AMI. On some ASUS boards 4G Encoding works exactly in an opposite way, so science cannot theorise the hey they did.

[apianti]

11 hours ago, meaganmargaret said:

 

Hey, thanks!  Any idea what causes this?  Why does OSX Free 2000 alway work and AptioMemoryFix always seem to have this problem?  any thoughts?

 

The free2000 driver is a plague on my existance and I hate myself everyday for ever even suggesting it when I knew exactly what would happen.... People would use it without knowing that it literally just frees random memory to make room for the kernel. THIS IS THE WORST POSSIBLE THING YOU COULD EVER DO! There is no possibility to get fully working system without just complete luck.

 

10 hours ago, meaganmargaret said:

 

So, for anyone else with an X99 or maybe even an X299 motherboard with Thunderbolt who is having this problem:  turning on enabling 4G encoding allowed the system to boot.

 

Anybody got any details why, or what other fixes there might be, please post....

 

Anyone who has X99/X199/X299/X399 board will have the most problems with memory. The reason is because these boards have a large amount of onboard devices, especially PCIe and NVMe. The peripherals that will be connected to these buses usually need large memory mapped buffers to operate, because the firmware for some reason purposefully fragments the memory map so that there is not enough continuous free space to load the kernel cache. Also the more devices you have the more likely you will encounter memory errors, especially multiple GPUs and SSDs, as these have the largest memory mapped resources.

 

1 hour ago, Pavo said:

@vit9696 would you know why when I enabled 4G Encoding I lose my audio device on Asus Z9PE-D8 WS motherboard?

 

Because it probably allocates the runtime memory region above 4G barrier for macOS addressing like you enabled....? macOS can only use 32bit addresses when it starts.

 

EDIT: Oh wait 4G encoding? Is it not called above 4G decoding?

Edited June 2, 2018 by apianti

[Pavo]

1 hour ago, apianti said:

Oh wait 4G encoding? Is it not called above 4G decoding?

Yeah my bad its called Above 4G Decoding.

[genzai]

4 hours ago, apianti said:

Anyone who has X99/X199/X299/X399 board will have the most problems with memory.

There is an X199?

 

I have a loaded up x99 that does not even have a 4G decode option in bios (maybe its always forced on?) that is fully loaded up with PCIe cards and it boots fine. I have a z370 board with less hardware that needs 4G decoding enabled to boot. All definitely comes down to how much hardware you have enabled and some hard to understand specifics of each motherboard/firmware design. Thunderbolt, iGPU, dGPU and some RAID cards seem to be the worst culprits. Aptiomemoryfix has been the hands down best solution though and i really thank you for continuing its development.

 

About macOS only using 32bit addresses when it starts, would a future version that can do 64bit addresses not be the holy grail that finally solves this once and for all? And why can you load up a real mac with as much hardware as it can possible support and never run into memory allocation issues on boot?

 

Thanks,

g\

[apianti]

57 minutes ago, genzai said:

There is an X199?

 

Oh, technically, no, I believe that Intel abandoned it because of Ryzen forcing them to increase their timeline or something because it came out around the same time and the X299 was released only a few months later. There's only one motherboard made by Asus, called X99-E for some reason...

 

57 minutes ago, genzai said:

I have a loaded up x99 that does not even have a 4G decode option in bios (maybe its always forced on?) that is fully loaded up with PCIe cards and it boots fine. I have a z370 board with less hardware that needs 4G decoding enabled to boot. All definitely comes down to how much hardware you have enabled and some hard to understand specifics of each motherboard/firmware design. Thunderbolt, iGPU, dGPU and some RAID cards seem to be the worst culprits. Aptiomemoryfix has been the hands down best solution though and i really thank you for continuing its development.

 

You can check your memory map to see if there are allocations above 0x100000000, then you will know it is enabled. The firmware also depends on the board manufacturer. And the memory allocation error comes down to luck as well, that your firmware just happens to not suck on one but does on the other... There are some ways I'm trying to minimize the amount of fragmented data to prevent this problem.

 

57 minutes ago, genzai said:

About macOS only using 32bit addresses when it starts, would a future version that can do 64bit addresses not be the holy grail that finally solves this once and for all? And why can you load up a real mac with as much hardware as it can possible support and never run into memory allocation issues on boot?

 

I imagine the 32bit addresses are kinda stuck even though they have dropped 32bit support, they are built into the boot arguments structure that is passed from boot.efi. They would have to change the whole act of jumping, which is entirely possible, into the kernel. A real mac can load up as much as memory will allow as well, the difference is that the firmware of a mac is designed to be moved anywhere, so all of that stuff is actually relocated into the kernel space. Aptiofix1 and aptiofix2 allow some of this relocation to happen, Aptiofix3 and AptioMemoryFix do not allow the relocation. Therefore, there is always some part of the firmware that is not moved and is in the way of where the kernel wants to go. This is more often the case if you do not use AptioMemoryFix as the slide is more likely to not be available (since it does a slide randomization with only available slides), where on a mac the memory there will just be relocated anyway so it has no problem. I hope that answered your question, I could go on forever but that's probably good.

Edited June 2, 2018 by apianti

[mhaeuser]

The kernel is loaded before the RT pages are defragmented, the moving does and would not free any area "where the kernel wants to go".

[djgusto]

On 5/28/2018 at 11:24 AM, vit9696 said:

Updated at 8:00 MSK.

 

Ok, I have a workaround but I think may still want some tests just in case. The fix is committed here:

https://github.com/vit9696/AptioFixPkg/commit/82fffac9146cf6bd0f1acb9bf6fb8c5132520e0b

 

I attach a binary with the prints uncommented. These prints will show an address where RtShims got allocated.

 

I will need:

1. The address printed (and does it change across the reboots) - ideally as a screenshot.

2. Whether macOS/Windows/Linux boots

3. Whether hibernation works (if it did not work previously, skip this) — better test twice in a row (boot → hibernate → wake → hibernate → wake → reboot).

4. Whether sleep works (if it did not work previously, skip this)

5. Whether nvram works (skip, if it did not work previously), do not forget to check after hibernate wake.

 

Systems of most interest are X99, X299, Z68, Z77, but others are fine as well.

AptioMemoryFix-R19-PRNTS.efi

 

I have a Z68 system and I'm keen to help were I can. Gigabyte GA-Z68MA-D2H-B3 rev-1.3. The BIOS is rather limited and I do not believe there as menu option to Enable 4G encoding.

 

I have encountered the "does printf work" error when trying to boot the High Sierra installer from a USB.

 

However I can successfully boot my already installed versions of Sierra and Windows 10 using AptioMemoryFix (I think the version I've tried are R9, R12/14???, R16, R19). I have not tried booting any linux distros. Sleep works however I do not have hibernation enabled and haven't tested it. NVRAM works.

 

Attempting to boot the High Sierra installer from USB returns the below message before I get the "does printf work?" error message after selecting what to boot.

RtShims of 1 pages allocate mode 0 attempt at 0x7FFFF000 - Success

The address above remains the same upon multiple boot attempts. See attached screen shot.

[locomacg6]

Hi Guys maybe some of you can help me. i got stuck on this screen booting from the hard disk

but if i boot from the usb it boots fine, using aptiomemoryfix, can someone please give me a hand on this . Thanks and have a nice day

[apianti]

On 6/3/2018 at 3:42 AM, Download-Fritz said:

The kernel is loaded before the RT pages are defragmented, the moving does and would not free any area "where the kernel wants to go".

 

The kernel maps and relocates runtime pages only but does not relocate the pages that are marked to be MMIO, remember?? The pages are marked differently by aptiofix, then reverted later, so they are not moved by the kernel, remember????? Therefore, they could be overwritten by the kernel, "where the kernel wants to go." Jesus. That's why you need a slide where the memory is free or will be - but that's not really reliable either since there might still be some boot time stuff that runs after the kernel is loaded, like kernel and kext patching.

[mhaeuser]

41 minutes ago, apianti said:

The pages are marked differently by aptiofix, then reverted later, so they are not moved by the kernel, remember????? Therefore, they could be overwritten by the kernel, "where the kernel wants to go." Jesus.

 

X) boot.efi loads the kernel. It uses UEFI allocation services.

Y, Y > X) boot.efi does or would (if the type hasn't been changed for all but one page) defragment the runtime data.

 

Can we please agree on the concept of linear time? "Jesus??????"

 

[Slice]

kernel not uses UEFI allocation and change memory map by himself.

[mhaeuser]

kernel not uses UEFI allocation and change memory map by himself.

But boot.efi does, which is what I said

[cecekpawon]

Yo @vit9696 did you noticed "UnlockSlideSupportForSafeMode" has changed in 361.60.5 (10.13.5, maybe also newer)? How can we deal with this? Megathanks

[apianti]

21 hours ago, Download-Fritz said:

 

X) boot.efi loads the kernel. It uses UEFI allocation services.

Y, Y > X) boot.efi does or would (if the type hasn't been changed for all but one page) defragment the runtime data.

 

Can we please agree on the concept of linear time? "Jesus??????"

 

 

The kernel does this not boot.efi. You should know this since the code to do so is in the kernel and we talked about it. I said that the runtime pages need to be preserved, boot.efi is not doing that. The kernel does just like slice said:

 

9 hours ago, Slice said:

kernel not uses UEFI allocation and change memory map by himself.

 

 

8 hours ago, Download-Fritz said:

9 hours ago, Slice said:

kernel not uses UEFI allocation and change memory map by himself.

But boot.efi does, which is what I said

 

No, it doesn't. Look at the code for the kernel. Again. Then study up. Then don't talk because you're still going to be wrong, apparently.

Edited June 10, 2018 by apianti