EFI Variable Store on Aptio V (Haswell-E and up)

[goodwin_c]

Why i'm asking - after macos boot i can see that at least one RT_Data and one ACPI_NVS are remapped into upper memory (0xffffff***) and their phys addr <> virt addr. Also, all MMIO has phys addr <> virt addr (and as i can see clover aptio fix is marking some rt_data as mmio). So i'm wondering if that can't be reason of non working nvram

This is mine memory map https://paste.ee/p/NzRWi

[mhaeuser]

Why i'm asking - after macos boot i can see that at least one RT_Data and one ACPI_NVS are remapped into upper memory (0xffffff***) and their phys addr <> virt addr. Also, all MMIO has phys addr <> virt addr (and as i can see clover aptio fix is marking some rt_data as mmio). So i'm wondering if that can't be reason of non working nvram

This is mine memory map https://paste.ee/p/NzRWi

 

No, that is to FIX NVRAM access. RT_data gets relocated by boot.efi and thus the SMM-side of the driver doesn't write to the CommBuffer. MMIO pages do not get relocated. Only one RT_data page remains RT_data, and that's the SystemTable, as it needs to be relocated for the kernel not to crash.

[goodwin_c]

No, that is to FIX NVRAM access. RT_data gets relocated by boot.efi and thus the SMM-side of the driver doesn't write to the CommBuffer. MMIO pages do not get relocated. Only one RT_data page remains RT_data, and that's the SystemTable, as it needs to be relocated for the kernel not to crash.

 

May I ask few questions?

 - do we really need to reallocate all RT_data and MMIO?

 - does kernel purge all rt_data blocks?

 - is there any way to prevent original RT_Data from reallocation? (just tried to comment out rt defragmentation block in aptiofix2 - and kernel is not booting - just stuck in "++++")

 - so, if kernel is not purging block marked as MMIO - maybe we can try to just mark RT_Data as MMIO without real reallocation? Or this has no sense?

What i currently see that all RT_Data are marked as MMIO and then ALL MMIO are reallocated. Also one RT_Code block is reallocated, marked as ACPI_NVS and attribute is changes from 800000000000000f to just f

And made my system booting with original aptiofix2 just providing slide=32 boot arg - this forcibly moves kernel allocation to free block in memoty - i have always free memory starting from 4010000 addr (1DB46 pages - more than enough), and with slide=32 kernel is allocated from 4100000

[mhaeuser]

1) MMIO is not reallocated.

2) Nothing RT_* or MMIO is purged... that kinda defeats the purpose.

3) Marking RT_data as MMIO IS preventing relocation.

4) Marking RT_data as MMIO without relocation is what is happening right now.

[goodwin_c]

1) MMIO is not reallocated.

2) Nothing RT_* or MMIO is purged... that kinda defeats the purpose.

3) Marking RT_data as MMIO IS preventing relocation.

4) Marking RT_data as MMIO without relocation is what is happening right now.

 

What i can say as for now:

- after unpacking my uefi - i can tell 100% that nvram is SMM based - there are nvram smm drivers inside

- in total, nvram is working

- nvram is working just before macOS will boot. I can access all nvram storage from clover UEFI shell. I can make changes - for ex., i can delete variables

- clover is writing a lot of variables successfully into nvram. I can see variables for last boot partition, some smc keys etc.

- after boot process nvram is not available anymore

My main suggestion is that AptioFix drivers are breaking things.

After some memmap comparison - i can see that my RT_Code region is changed by aptiofix to ACPI_NVS and has removed runtime flag (all RT_Data regions are changed to MMIO as expected, and they still has RT flag) . In Linux same RT_Code region has its RT flag. But trying to preserve that changes in aptiofix - my system is not booting, just freeze after aptiofix, kernel is not loading. What do you think? Can this be a cause of broken nvram?

Also, another thing that i can't understand. As i can see, aptiofix is copying efi sys table to separate rt_data region. Is it really needed? Why we can't use original RT_Data without changing things?

[goodwin_c]

1) MMIO is not reallocated.

2) Nothing RT_* or MMIO is purged... that kinda defeats the purpose.

3) Marking RT_data as MMIO IS preventing relocation.

4) Marking RT_data as MMIO without relocation is what is happening right now.

 

I have one more question. Is it possible that SMM part is stored not in RT_Data but in RT_Code part? If I'm trying to secure my RT_Code from kernel in any way (not removing rt flag, renaming to MMIO or even leaving it w/o changes - trying all 3 ways) - I have boot freeze in most cases. Sometimes i have kernel panic with something like EFIRuntime of nvram in output (very rare, have no full info yet). But mostly it is just boot freeze approximately few lines after FakeSMC header line. It is strange, bcz all RT_Data are secured with MMIO renaming, and only RT_Code has rt flag unset - so most probably purged - and in result nvram is not working. In Linux RT_Code is in safe, and in Clover before booting boot.efi - in all that cases nvram is working.

[mhaeuser]

Can this be a cause of broken nvram?

 

No.

 

As i can see, aptiofix is copying efi sys table to separate rt_data region. Is it really needed? Why we can't use original RT_Data without changing things?

 

Same reason as for CommBuffer.

 

Is it possible that SMM part is stored not in RT_Data but in RT_Code part?

 

The CommBuffer? No.

 

If I'm trying to secure my RT_Code from kernel in any way (not removing rt flag, renaming to MMIO or even leaving it w/o changes - trying all 3 ways) - I have boot freeze in most cases

 

Secure by removing the code to secure??

 

bcz all RT_Data are secured with MMIO renaming, and only RT_Code has rt flag unset - so most probably purged - and in result nvram is not working.

 

Data and Code are secured.

[goodwin_c]

But as i can see from AptioFix code - only RT_Data is secured through marking it as MMIO regions. And RT_Code is processed with Defragment function from BootFixes - it is copied to "reserved area inside kernel boot image" and later marked as ACPI_NVS without RT flag. As i can understand with my poor level of knowledge - it can be cleaned later by kernel. Or ACPI_NVS without RT flag is secured from cleanup?

Also, very interesting comment in code:

// we need to remove RT_code and RT_data flags since they causes GPF on some UEFIs.
// OSX maps RT_code as Read+Exec only while faulty frivers writes to their
// static vars which are in RT_code

Maybe this is our case?

[mhaeuser]

But as i can see from AptioFix code - only RT_Data is secured through marking it as MMIO regions. And RT_Code is processed with Defragment function from BootFixes - it is copied to "reserved area inside kernel boot image" and later marked as ACPI_NVS without RT flag. As i can understand with my poor level of knowledge - it can be cleaned later by kernel. Or ACPI_NVS without RT flag is secured from cleanup?

Also, very interesting comment in code:

// we need to remove RT_code and RT_data flags since they causes GPF on some UEFIs.
// OSX maps RT_code as Read+Exec only while faulty frivers writes to their
// static vars which are in RT_code

Maybe this is our case?

 

Please check the entire control flow, you are not understading what's happening... yes, these fixes do exist, but they are present due to the 'NVRAM fix' coming later. As part of protecting data to be relocated, all regions with the RUNTIME attribute are converted to MMIO, this includes RT_Data and RT_code.

 

EDIT: Modded local code, ignore.

[goodwin_c]

Please check the entire control flow, you are not understading what's happening... yes, these fixes do exist, but they are present due to the 'NVRAM fix' coming later. As part of protecting data to be relocated, all regions with the RUNTIME attribute are converted to MMIO, this includes RT_Data and RT_code.

 

Maybe i don't see something, but:

 

VOID
ProtectRtDataFromRelocation(
  IN UINTN MemoryMapSize,
  IN UINTN DescriptorSize,
  IN UINT32 DescriptorVersion,
  IN EFI_MEMORY_DESCRIPTOR *MemoryMap
  )
{
UINTN NumEntries;
UINTN Index;
EFI_MEMORY_DESCRIPTOR *Desc;
// UINTN BlockSize;
 
Desc = MemoryMap;
NumEntries = MemoryMapSize / DescriptorSize;
DBG("FixNvramRelocation\n");
DBGnvr("FixNvramRelocation\n");
 
for (Index = 0; Index < NumEntries; Index++) {
// BlockSize = EFI_PAGES_TO_SIZE((UINTN)Desc->NumberOfPages);
 
if ((Desc->Attribute & EFI_MEMORY_RUNTIME) != 0) {
if (Desc->Type == EfiRuntimeServicesData && Desc->PhysicalStart != gSysTableRtArea)
{
DBG(" RT data %lx (0x%x) -> MemMapIO\n", Desc->PhysicalStart, Desc->NumberOfPages);
Desc->Type = EfiMemoryMappedIO;
}
}
 
Desc = NEXT_MEMORY_DESCRIPTOR(Desc, DescriptorSize);
}
}

 

And it is clear that it checkes only "Desc->Type == EfiRuntimeServicesData". And as expected, in result after system boot I can clearly see that my original RT_Code region is NOT CHANGED to MMIO. It is marked as ACPI_NVS. This is fact after comparing memmap before and after boot - if you don't believe me I can give you two my memmaps to compare.

Ok, there are my two memmaps - one made from clover shell, second - from darwin dumper https://drive.google.com/open?id=0B1xZ4VKT4JKZeEhDUDU1OGhfSVE

Also, this is my freeze when i'm securing my RT_Code in "DefragmentRuntimeServices" with

 

if (Desc->Type == EfiRuntimeServicesCode) {
Desc->Type = EfiMemoryMappedIO;
Desc = NEXT_MEMORY_DESCRIPTOR(Desc, DescriptorSize);
continue;
}

so it is not get copied and renamed into ACPI_NVS

https://goo.gl/photos/rUWCyVzaVN6k96en6

[mhaeuser]

Oh yikes, my local code is modded. Sorry.

[goodwin_c]

P.S. Also, just was checking virtual addresses - and from my point of view they are looking absolutely wrong. I'm not sure whose fault is that - boot.efi, kernel or apiofix. As i understand - virtual addresses for RT_Data should be like Physical Address + 0xffffff8000000000. But this is not true in my case - you can just check my memmap. It is made with original untoched AptioFix2 and slide=32 bootarg

And sorry if i'm writing too much not very logical things. I have a lot of experience in system administration or development, but not in such low level things as EFI, bootloader etc. So i'm just learning and trying to understand how things are working to find reason of nvram problem.

[mhaeuser]

P.S. Also, just was checking virtual addresses - and from my point of view they are looking absolutely wrong. I'm not sure whose fault is that - boot.efi, kernel or apiofix. As i understand - virtual addresses for RT_Data should be like Physical Address + 0xffffff8000000000. 

 

Nah, it's alright. Virtual is Physical Address + 0xffffff8000000000 when data is relocated. They have a pointer to the last virtual address used and assign the new physical location (where data would be relocated to, if it was relocated). Means when the data is relocated, Physical is Virtual & ~0xffffff8000000000. See: https://github.com/Piker-Alpha/macosxbootloader/blob/El-Capitan/src/boot/MemoryMap.cpp#L212

Only RT_code and RT_data are relocated: https://github.com/Piker-Alpha/macosxbootloader/blob/El-Capitan/src/boot/MemoryMap.cpp#L277

[goodwin_c]

Hmm, things are going even more weird. Made one interesting experiment - wrote "NvTest" variable on AptioFix driver init. As expected - variable was written to physical nvram (i can see it from shell with dmpstore). But after reverting AptioFix to original without my custom code - i can see that var with "nvram"command in macOS!!! That means that nvram is not fully broken, but write access is somehow not working. Made another experiment - booted to Linux and tried to delete variable with efivars - and you know what? Permission denied and error in kernel logs. Unfortunately I don't know how to test nvram and how to make memmap from Windows, but looks like in both Linux and MacOS nvram write is locked after ExitBootService, but it is working before OS will take control.

P.S. Ok, in Linux i can delete variable. Looks like in Linux all variables are immutable by default to protect system from accident removal of some critical nvram variables (linux gives access to all nvram variables). But on MacOS it is how i described - i can read existing variables but can't modify them after system boot.

[goodwin_c]

Nah, it's alright. Virtual is Physical Address + 0xffffff8000000000 when data is relocated. They have a pointer to the last virtual address used and assign the new physical location (where data would be relocated to, if it was relocated). Means when the data is relocated, Physical is Virtual & ~0xffffff8000000000. See: https://github.com/Piker-Alpha/macosxbootloader/blob/El-Capitan/src/boot/MemoryMap.cpp#L212

Only RT_code and RT_data are relocated: https://github.com/Piker-Alpha/macosxbootloader/blob/El-Capitan/src/boot/MemoryMap.cpp#L277

Ok, then with our original RT_Data it is clear - only one RT_Data is being relocated, and this is new 1page region with system table copy for kernel. But what about RT_Code and Defragment function?

Also, how it can be that we have read access to nvram but don't have write access?

[goodwin_c]

Do you know, NVRAM is using SMM mechanism only for writing variables? Or for reading as well?

[mhaeuser]

Read and write. For some new Aptio V versions, you can take EDK2 Variable code as reference code, though it's not used by all vendors.

[goodwin_c]

Hm, do you have any idea why I have read access and not write access? It is definitely SMM based NVRAM - i have NvramSMM and NvramSMI drivers in my UEFI ( it is InsydeH2O in Dell laptop)

And write access is not working only in macOS. For ex., in Linux both read and write are working. In clover before macOS read and write are working. But after booting macOS - only read is working. Weird...

If Smm is used for both operations, and read is working fine - then Smm is not broken.

[mhaeuser]

Hm, do you have any idea why I have read access and not write access? It is definitely SMM based NVRAM - i have NvramSMM and NvramSMI drivers in my UEFI ( it is InsydeH2O in Dell laptop)

And write access is not working only in macOS. For ex., in Linux both read and write are working. In clover before macOS read and write are working. But after booting macOS - only read is working. Weird...

If Smm is used for both operations, and read is working fine - then Smm is not broken.

macOS exposes the Runtime services in IOReg I think... try calling SetVariable on your own and see what it returns.

[goodwin_c]

Do you mean writing user-land app to call SetVariable? Hmm, never tried to do that, will try to learn tomorrow.

Looks like i found source for original nvram tool https://opensource.apple.com/source/system_cmds/system_cmds-735.20.1/nvram.tproj/nvram.c

Let's see how it is working...

[Slice]

It just write into device-tree/options assuming some driver in macOS will send it to NVRAM.

[goodwin_c]

It just write into device-tree/options assuming some driver in macOS will send it to NVRAM.

Yeah, it just calls SetOFVariable function from external (most probably some kext bcz i can't find it in kernel source)

P.S. AppleEFIRuntime has plugin AppleEFINVRAM. I think this guy is making all hard job.

P.P.S. Also, calling SetVariable from initiation phase of AptioFix driver work fine as i wroter before, but i have idea - i will try to call rt->setvariable on different stages of AptioFix - before exitbootservices, after all RT fixes etc. Let see if it will break on some stage if it is called from efi driver - this will give us at least understanding if problem is in aptio fixes itself, or somewhere OS related

[goodwin_c]

With each step things are more interesting. Last my results using AptioFix2, playing with rt->setvariable:

 - function KernelEntryPatchJumpBack, before return -> no variable in memory - just after booting modified driver and after second reboot no custom variable available

 - function OvrSetVirtualAddressMap -> nvram write ok. Added few variables in different places - on beginning, before return, in the middle after each fix. All are written to physical nvram

 - func OsxAptioFixDrvEntrypoint -> write ok

 - func MOStartImage, before RunImageWithOverrides call -> write ok

 - func DefragmentRuntimeServices -> write ok on beginning and before return

 - func RunImageWithOverrides -> write ok on begginning, and no write before return! Looks like things are broken by some changes inside of that function

 - func FixBootingWithoutRelocBlock -> no write, things are already broken

 

P.S. All steps was made with sequence that guarantee working physical write. Boot with modified aptiofix driver, reboot to create variables in memory, rollback to unmodified aptiofix, one more reboot to be sure that we are reading variables from physical nvram storage but not created in memory by aptiofix. All variables has unique name like NvTest+some digit.

[goodwin_c]

P.P.S. About RunImageWithOverrides i made wrong changes - it shouldn't reach end of procedure bcz of starting boot.efi before. Added more variables - and all variables are written correctly until gStartImage entry. So looks ok there. So, looks like everything is OK before boot.efi and broken after boot.efi. Somebody has knowledge what can break things inside of boot.efi?

[mhaeuser]

 - func RunImageWithOverrides -> write ok on begginning, and no write before return! Looks like things are broken by some changes inside of that function

RunImageWithOverrides is not supposed to return.

 

EDIT: Sorry, overread the last message...  that OvrSetVirtualAddressMap works fine proofs that the issue is not related to the original SMM buffer issue (physical and virtual write to the same buffer here). Also, KernelEntryPatchJumpBack is not called with AptioFix2 - call it and try again?