AptioMemoryFix

[gujiangjiang]

The latest Clover still contain AptioFixR16 but no R17 so can anyone give the latest AptioFix R17 binary efi.

thanks for vit9696’a work and for all man works for this project!


从我的 iPhone 发送，使用 Tapatalk

[MaLd0n]

files.zip

[Gymnae]

On 5/12/2018 at 1:57 PM, vit9696 said:

Hello,

 

Firstly, it is not AptioMemoryFix allocating pages but boot.efi.

 

Secondly, slide argument is not ignored, it is just the value that is improper. You may set 0 here and rebuild AptioMemoryFix to see the actually set slide in ioreg in IODevice: https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/Config.h#L52

 

Thirdly, the memory map UEFI Shell command does not show the memory map as it changes over boot.efi loading process. It allocates quite a lot, and even AptioMemoryFix cannot catch it all. To get a more reliable memory map you could try using -aptiodump boot argument after setting 1 here: https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/Config.h#L76

But I would say even that is not perfect.

 

 

Thank you for your feedback. I think the way AptioMemoryFix picks the slide is fine for me, the iGPU is initialized with 64MB Pre-alloc

 

Another thing: I seem to not enjoy native NVRAM with either AptioMemoryFix nor OsxAptioFix3Drv-64 with my H370 based system and have to use EmuVariableUefi-64.efi together with RC scripts for NVRAM. I checked this by adding "MyVar" to NVRAM. It is not persisted without EmuVariableUefi-64.efi and RC scripts.

Edited May 15, 2018 by Gymnae

[apianti]

16 hours ago, Gymnae said:

 

Thank you for your feedback. I think the way AptioMemoryFix picks the slide is fine for me, the iGPU is initialized with 64MB Pre-alloc

 

Another thing: I seem to not enjoy native NVRAM with either AptioMemoryFix nor OsxAptioFix3Drv-64 with my H370 based system and have to use EmuVariableUefi-64.efi together with RC scripts for NVRAM. I checked this by adding "MyVar" to NVRAM. It is not persisted without EmuVariableUefi-64.efi and RC scripts.

 

You restarted after you set the test NVRAM variable? You should have working NVRAM unless you have the whitelist issue.

[Gymnae]

3 minutes ago, apianti said:

 

You restarted after you set the test NVRAM variable? You should have working NVRAM unless you have the whitelist issue.

Yes, I did the following:

1. Installed the AptioMemoryFix.efi build from source
2. Ran `sudo nvram MyVar=TestValue`
3. Rebooted
4. Checked with `nvram -p`
5. MyVar is missing
6. Installed EmuVariableUefi-64.efi and RC scripts via Clover installer (4458)
7. Rebooted
8. Re-ran 2. - 4.
9. MyVar present

 

My Motherboard:

Gigabyte H370 HD3 Rev.1 Bios F2

Edited May 16, 2018 by Gymnae
added mobo

[Pavo]

@vit9696 Having build issues with your latest commit on your AptioFixPkg, see below:

In file included from /root/AptioFix/UDK/Build/AptioFixPkg/DEBUG_GCC5/X64/AptioFixPkg/Platform/AptioMemoryFix/AptioMemoryFix/DEBUG/AutoGen.h:16:0,
                 from <command-line>:0:
/root/AptioFix/UDK/AptioFixPkg/Platform/AptioMemoryFix/UnicodeCollation/UnicodeCollationEng.c: In function 'InitializeUnicodeCollationEng':
/root/AptioFix/UDK/MdePkg/Include/Base.h:48:49: error: unused variable '_VerifySizeofUnicodeLanguages' [-Werror=unused-variable]
 #define VERIFY_SIZE_OF(TYPE, Size) extern UINT8 _VerifySizeof##TYPE[(sizeof(TYPE) == (Size)) / (sizeof(TYPE) == (Size))]
                                                 ^
/root/AptioFix/UDK/AptioFixPkg/Platform/AptioMemoryFix/UnicodeCollation/UnicodeCollationEng.c:140:5: note: in expansion of macro 'VERIFY_SIZE_OF'
     VERIFY_SIZE_OF (UnicodeLanguages, 6);
     ^
cc1: all warnings being treated as errors
make: *** [/root/AptioFix/UDK/Build/AptioFixPkg/DEBUG_GCC5/X64/AptioFixPkg/Platform/AptioMemoryFix/AptioMemoryFix/OUTPUT/UnicodeCollation/UnicodeCollationEng.obj] Error 1


build.py...
 : error 7000: Failed to execute command
	make tbuild [/root/AptioFix/UDK/Build/AptioFixPkg/DEBUG_GCC5/X64/AptioFixPkg/Platform/AptioMemoryFix/AptioMemoryFix]


build.py...
 : error F002: Failed to build module
	/root/AptioFix/UDK/AptioFixPkg/Platform/AptioMemoryFix/AptioMemoryFix.inf [X64, GCC5, DEBUG]

Was not having any issues before building using GCC until you switched to UDK2018. Any ideas on how to fix this?

[vit9696]

A fix is to buy yourself a proper compiler from llvm.org , but the issue should be fixed anyway. I committed the necessary changes upstream.

[Pavo]

18 hours ago, vit9696 said:

A fix is to buy yourself a proper compiler from llvm.org , but the issue should be fixed anyway. I committed the necessary changes upstream.

@vit9696 lol true that but now I am getting errors on the make -C BaseTools command lol

GenFvInternalLib.c:24:23: fatal error: uuid/uuid.h: No such file or directory
 #include <uuid/uuid.h>
                       ^
compilation terminated.
make[2]: *** [GenFvInternalLib.o] Error 1
make[2]: Leaving directory `/root/AptioFix/UDK/BaseTools/Source/C/GenFv'
make[1]: *** [GenFv] Error 2
make[1]: Leaving directory `/root/AptioFix/UDK/BaseTools/Source/C'
make: *** [Source/C] Error 2
make: Leaving directory `/root/AptioFix/UDK/BaseTools'

 

[vit9696]

This is neither a problem of AptioFix buildsystem, nor a problem of UDK buildsystem. Your OS has no libuuid headers installed, and this is obviously an error on your side. Most likely some kind of libuuid-dev or libuuid-devel package is needed. Googling prior to bugreporting may work better sometimes 

[Pavo]

41 minutes ago, vit9696 said:

This is neither a problem of AptioFix buildsystem, nor a problem of UDK buildsystem. Your OS has no libuuid headers installed, and this is obviously an error on your side. Most likely some kind of libuuid-dev or libuuid-devel package is needed. Googling prior to bugreporting may work better sometimes 

Yeah I'm an idiot lol, I forgot to install required packages prior to running the new setup.sh lol, its all good now sorry about that.

[Pavo]

Ok docker builder working again https://hub.docker.com/r/pavo/aptiofix_builder/

[Pene]

Hello vit9696,

 

I noticed on my system that I cannot boot to linux when AptioMemoryFix (latest) is loaded before grub starts.

Grub (UEFI) manages to start properly, but the system hangs when it boots the linux distro (Ubuntu in my case).

When AptioMemoryFix is not loaded, Ubuntu starts without problems.

(I'm unsure if this happens only on my system, or only with ubuntu, or if this is more widespread. I'm on a pure UEFI system, no CSM)

 

As Clover loads UEFI drivers before booting any OS, this results in Ubuntu being unbootable from the Clover menu (Windows and OSX boot fine).

Do you have any idea what might be going on?

 

P.S. nice work with AptioMemoryFix!

Edited May 27, 2018 by Pene

[vit9696]

Hi,

The issue indeed is (was) unknown, yet I suppose it should not happen.

 

Firstly, may I ask you to provide more details about your installation? I.e. Ubuntu version, architecture installation method, why did you use grub (I thought Linux supports native EFI loading?), anything you may find important. It will be great to have a virtual machine (e.g. VMware) where the issue could be reproduced (i.e. by loading AptioMemoryFix from UEFI Shell and then starting grub). Also, do you see any messages from your Linux distro? It should also have some kind of panic log (via serial or swap), which might be helpful if you can get it.

 

Secondly, I could suggest you to do some debugging that may possibly hint where the problem is. Could you go to AptioMemoryFix.c and 

https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/AptioMemoryFix.c#L109 and try commenting the shim installation lines?

  InstallRtShims (GetVariableCustomSlide);
  InstallBsOverrides ();
  InstallRtOverrides ();

That would break macOS, but it should pretty certainly fix Linux. If it does, try finding which exact line breaks it (i.e. by uncommenting).

 

Similarly after you find the culprit you could step further and narrow it down to the exact function:

https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/RtShims.c#L126

    gRT->GetVariable               = (EFI_GET_VARIABLE)((UINTN)gRtShims              + ((UINTN)&RtShimGetVariable          - (UINTN)&gRtShimsDataStart));
    gRT->SetVariable               = (EFI_SET_VARIABLE)((UINTN)gRtShims              + ((UINTN)&RtShimSetVariable          - (UINTN)&gRtShimsDataStart));
    gRT->GetNextVariableName       = (EFI_GET_NEXT_VARIABLE_NAME)((UINTN)gRtShims    + ((UINTN)&RtShimGetNextVariableName  - (UINTN)&gRtShimsDataStart));
    gRT->GetTime                   = (EFI_GET_TIME)((UINTN)gRtShims                  + ((UINTN)&RtShimGetTime              - (UINTN)&gRtShimsDataStart));
    gRT->SetTime                   = (EFI_SET_TIME)((UINTN)gRtShims                  + ((UINTN)&RtShimSetTime              - (UINTN)&gRtShimsDataStart));
    gRT->GetWakeupTime             = (EFI_GET_WAKEUP_TIME)((UINTN)gRtShims           + ((UINTN)&RtShimGetWakeupTime        - (UINTN)&gRtShimsDataStart));
    gRT->SetWakeupTime             = (EFI_SET_WAKEUP_TIME)((UINTN)gRtShims           + ((UINTN)&RtShimSetWakeupTime        - (UINTN)&gRtShimsDataStart));
    gRT->GetNextHighMonotonicCount = (EFI_GET_NEXT_HIGH_MONO_COUNT)((UINTN)gRtShims  + ((UINTN)&RtShimGetNextHighMonoCount - (UINTN)&gRtShimsDataStart));
    gRT->ResetSystem               = (EFI_RESET_SYSTEM)((UINTN)gRtShims              + ((UINTN)&RtShimResetSystem          - (UINTN)&gRtShimsDataStart));

https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/ServiceOverrides.c#L124

  gBS->AllocatePages      = MOAllocatePages;
  gBS->GetMemoryMap       = MOGetMemoryMap;
  gBS->ExitBootServices   = MOExitBootServices;
  gBS->HandleProtocol     = MOHandleProtocol;
  gBS->StartImage         = MOStartImage;

https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/ServiceOverrides.c#L143

gRT->SetVirtualAddressMap = MOSetVirtualAddressMap;

Once we know the details I should hopefully be able to provide some help. Also, does Windows work fine?

[Pene]

Hi,

 

Thanks.

I am using the latest Ubuntu Desktop 18.04 LTS (ubuntu-18.04-desktop-amd64), with linux kernel 4.15.0.22.

Ubuntu, as many other UEFI Linux distributions are using grubx64.efi, which then loads the linux kernel.

It is displaying:

Loading Linux 4.15.0-22-generic ...
Loading initial ramdisk ...

and hangs. That doesn't really provide any useful information, as that is the stage before linux kernel starts. It may try to load the linux kernel and hang. 

Referring to your question about Windows, yes, Windows managed to load without problems.

 

Debugging as you suggested, I managed to narrow down the problem to this trio:

//gRT->GetVariable               = (EFI_GET_VARIABLE)((UINTN)gRtShims              + ((UINTN)&RtShimGetVariable          - (UINTN)&gRtShimsDataStart));
//gRT->SetVariable               = (EFI_SET_VARIABLE)((UINTN)gRtShims              + ((UINTN)&RtShimSetVariable          - (UINTN)&gRtShimsDataStart));
//gRT->GetNextVariableName       = (EFI_GET_NEXT_VARIABLE_NAME)((UINTN)gRtShims    + ((UINTN)&RtShimGetNextVariableName  - (UINTN)&gRtShimsDataStart));

When all 3 are commented out, linux kernel starts successfully.

 

Edited May 27, 2018 by Pene

[vit9696]

Thanks for a quick test, that was very helpful. Hmmm… With that in mind it is a little difficult matter.

Basically we cannot be sure whether Linux uses any other Runtime Services interfaces other than these 3 (even though it definitely can, e.g. an RTC driver: drivers/rtc/rtc-efi.c has been present for years).

 

If you do not mind making a few more tests...

 

My first check would be to ensure that even uncommenting gRT->GetNextVariableName assignment leads to a hang. It is the simplest of them all, and if it still hangs then it is pretty definitely not the recent code additions to GetVariable/SetVariable.

 

Next, if it does not work, I would try to simplify it even further by replacing the GetNextVariableName shim just to this:

https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/X64/AsmRtShims.nasm#L134

global ASM_PFX(RtShimGetNextVariableName)
ASM_PFX(RtShimGetNextVariableName):
    mov        rax, qword [ASM_PFX(gGetNextVariableName)]
    jmp        rax

With this we can be sure that if the pointers are virtualised (and if InstallRtOverrides do get called they must be):

https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/ServiceOverrides.c#L653

 

To make sure that the pointers are virtualised I would revert all the commented lines out and put a hlt here:

https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/RtShims.c#L196

if (FixedCount != ARRAY_SIZE (mShimPtrArray)) {
  __asm__ volatile ("hlt");
}

If it does not go further, then our pointer virtualisation code is flawed.

 

At this step I would probably stop and hear the results, because it is very hard to image what could go wrong. Thanks

Edited May 27, 2018 by vit9696

[Pene]

First test - uncommenting gRT->GetNextVariableName (other 2 commented out) - indeed leads to Linux hang.

Second test - simplfying ASM_PFX(RtShimGetNextVariableName) jmp to 'jmp rax' - Linux still hangs (with GetNextVariableName commented out).

Third test - untouched source code, only code with hlt added - OSX loads properly, Linux obviously still hangs (you meant to test this with OSX, right?)

 

 

Edited May 27, 2018 by Pene

[vit9696]

That's quite something.

 

Quote

Second test - simplfying ASM_PFX(RtShimGetNextVariableName) jmp to 'jmp rax' - still hangs (with GetNextVariableName commented out).

Sorry, is that a typo? I.e. did you mean GetNextVariableName is UNcommented but it still hangs? Because otherwise I cannot see how it could hang if commenting the assignment previously fixed the problem. 

 

Quote

Third test - untouched source code, only hlt added - OSX loads properly, Linux obviously still hangs (you meant to test this with OSX, right?)

Sorry. I got you confused. The test was meant to check whether Linux-provided memory map is sane. Actually I had a suspect about it, and wanted to see whether it hangs much earlier or not with this change added. Perhaps it is better to try putting a hlt unconditionally to be sure that it can be distinguished from a normal hang?

 

Also… it may be stupid but… could you try booting to Linux without Clover? I.e. add AptioMemoryFix via the UEFI Shell (something alike if I remember correctly):

bcfg driver add 0 fsX:\EFI\something\AptioMemoryFix.efi AptioMemoryFix

And then use the boot select menu to load straight to grub…

A friend has an APTIO V board, and Linux works fine for him, so I have some doubts…

 

Edited May 27, 2018 by vit9696

[Pene]

Hi again.

 

About second test - that obviously was a typo, I meant GetNextVariableName uncommented (the other two commented out).

 

About third test - I added an unconditional hlt now (which resulted in Osx hanging as well before the kernel starts) - but for linux I cannot see the difference. With nothing commented out, and the only change in source code being the added hlt, Linux seems to hang exactly at the same place  (before the linux kernel manages to start).

 

I tested without Clover also. It indeed hangs exactly the same, with only the additional driver AptioMemoryFix driver loaded before grub starts.

 

Edited May 27, 2018 by Pene

[vit9696]

Hmmm… Ok, we could use a cold reboot I guess. Would you replace hlt with this?

IoWrite8 (0xCF9, 0x06);
while (1);

 

[Pene]

Ok, with cold reboot added (had to include IoLib also), and no other source code modifications (nothing commented out), Linux reboots before kernel starts.

[vit9696]

Oho. Sorry, I might have gotten confused a little again. Did you try to guard it by FixedCount?

if (FixedCount != ARRAY_SIZE (mShimPtrArray)) {
 IoWrite8 (0xCF9, 0x06);
}

If it still reboots, then the virtualisation code must be broken much earlier… 

 

If not… hmm… let's check if we can execute any code from Linux kernel at all?

I.e. let's just leave only the RtShimGetNextVariableName shim uncommented, and put an error there? Like this:

global ASM_PFX(RtShimGetNextVariableName)
ASM_PFX(RtShimGetNextVariableName):
    ; Until boot.efi virtualizes the pointers we use a custom wrapper.
    mov        rax, qword [ASM_PFX(gGetVariableOverride)]
    test       rax, rax
    jnz        .USE_CURRENT_FUNC
    mov        rax, 0x8000000000000002
    ret
.USE_CURRENT_FUNC:
    mov        rax, qword [ASM_PFX(gGetNextVariableName)]
    jmp        short FourArgsShim

If it hangs, then the shim area is invalid/not executable. This would leave us with broken pointer virtualisation once again, but it should not be our own code at least.

[Pene]

With cold reboot guarded by FixedCount, Linux doesn't reboot, just hangs as before.

 

With the modification you specified above in X64/AsmRtShims.nasm and the situation below for RtShims.c, Linux still hangs:

    //gRT->GetVariable               = (EFI_GET_VARIABLE)((UINTN)gRtShims              + ((UINTN)&RtShimGetVariable          - (UINTN)&gRtShimsDataStart));
    //gRT->SetVariable               = (EFI_SET_VARIABLE)((UINTN)gRtShims              + ((UINTN)&RtShimSetVariable          - (UINTN)&gRtShimsDataStart));
    gRT->GetNextVariableName       = (EFI_GET_NEXT_VARIABLE_NAME)((UINTN)gRtShims    + ((UINTN)&RtShimGetNextVariableName  - (UINTN)&gRtShimsDataStart));
    gRT->GetTime                   = (EFI_GET_TIME)((UINTN)gRtShims                  + ((UINTN)&RtShimGetTime              - (UINTN)&gRtShimsDataStart));
...

(Just to double check, I also tried again commenting out GetNextVariableName in the code above, and it did not hang anymore).

 

Edited May 27, 2018 by Pene

[vit9696]

Wow. But that means that Linux violates the spec and does not map(?) the Runtime Code memory.

Just to quadruple check you could try:

global ASM_PFX(RtShimGetNextVariableName)
ASM_PFX(RtShimGetNextVariableName):
    mov        rax, 0x8000000000000002
    ret

And check if it still hangs?

 

If it does, it could be reasonable to play with the memory type? Try changing the type to EfiRuntimeServicesData or maybe EfiReservedMemoryType here:

https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/RtShims.c#L92

 

Last attempt, while I am not positive about it, perhaps could be to try using the lower code that does not use the pool allocator but instead allocates a page by putting:

#define APTIOFIX_ALLOCATE_POOL_GIVES_STABLE_ADDR 0

here: https://github.com/vit9696/AptioFixPkg/blob/master/Platform/AptioMemoryFix/Config.h#L29

Edited May 27, 2018 by vit9696

[Pene]

Yes it still hangs with the super simplified code for RtShimGetNextVariableName.

 

Changing memory types to both types you specified also didn't help.

 

With the #define changed to APTIOFIX_ALLOCATE_POOL_GIVES_STABLE_ADDR 0, Linux loads properly (had to add also #include "MemoryMap.h" in RtShims.c for this to compile).

 

Edited May 27, 2018 by Pene

[vit9696]

I am glad to hear we found a workaround. In fact I have just been sent a kernel panic, that shows that it actually is an attempt to execute non-executable memory by Linux kernel. And goddammit, it is a serious spec violation by either the firmware or the linux kernel >__<.

 

I am unsure what to do next, because on some firmwares using the allocation from top breaks hibernation (or nvram after hibernating twice). Perhaps allocating "any" address could behave better? I will have to run a lot of tests to be sure. Could you check whether this code also works for you? If it does, there is a small chance we could find a borderline between Linux glitches and macOS hibernation issues.

  EFI_PHYSICAL_ADDRESS RtShims = 0;
  Status = gBS->AllocatePages (
    AllocateAnyPages,
    EfiRuntimeServicesCode,
    EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart),
    &RtShims
    );
  gRtShims             = (VOID *)(UINTN)RtShims;

Another way would be to try to allocate a page-aligned chunk from pool:

  Status = gBS->AllocatePool (
    EfiRuntimeServicesCode,
    EFI_PAGES_TO_SIZE (EFI_SIZE_TO_PAGES ((UINTN)&gRtShimsDataEnd - (UINTN)&gRtShimsDataStart)),
    &gRtShims
    );