Insanely fast virtual Mac (QEMU, OVMF, Clover and native graphics)

[Peter_sm]

Ok, so I shall do this in a VM :-) not in my hackintosh

[waschbenzin]

You can use a real usb drive and dump it later using dd. 

But you need to make sure, that you have 2 partitions on the usb drive (Mac OS X might do this automatically if it is big enough). The EFI partition (ESP) is usually invisible, but as said you can mount it with Clover Configurator.

[waschbenzin]

How does mac osx calculate its speed?

 

The speed calculation can be found at http://www.opensource.apple.com/source/xnu/xnu-2782.40.9/osfmk/i386/tsc.cin the method tsc_init. 

 

There seem to be two options:

 

At first there is a check for a virtual cpu, if everything is correct here, the values are just read from the cpuid (special processor command the qemu/kvm can emulate). I tried modifying my kvm kernel module to get this working correctly, but I couldn't do it. I modified arch/x86/kvm/cpuid.c. There is a huge switch-case in __do_cpuid_ent that handles the different functions. I modified (0x4000000/KVM_CPUID_SIGNATURE/) to include "leafs" up to 0x4000010 and modified this function to return the correct speeds for my system in the processor registers. While I could verify that the change was in place it was never called :-(.

 

The second option is used, if the speeds can not be read from the vm's cpu. In this case the EFI is asked for Bus and TSC speeds. The value received here are from the DataHubDxe used in Clover. Clover sets these values in DataHubCpu.c. While the osx-kernel does read the TSC speed (the speed that makes our clocks go wrong), it simply doesn't use it. Instead it determines the tscGranularity from the cpu. There are a few MSR (model-specific registers) that are used to read this values (I'm pretty sure Clover can't modify these values). I am able to modify the kvm kernel module to return a modified busratio (add case 0xCE to arch/x86/kvm/x86.c -> kvm_get_msr_common) . Allowing me to run my system at a Busspeed of 100MHz with 4000Ghz TSC. 

 

This could be the end of the story, but having a tainted kernel, that I have to modify with each new kernel version doesn't seem to be an ideal solution. I would prefer to propose some changes into Clover:

- Refactor Clover to allow a 64-bit value for the BusRatio. Currently a signed 32-bit value seems to be used.

- Remove the checks in DataHubCpu.c which limits the Busspeed to 1GHz (if QEMU is set). BTW: The osx kernels sanity checks are 90Mhz to 10Ghz.

 

This way we would have complete freedom for the BusSpeed setting in Clover.

 

Other findings:

OSX seems to be supported on more QEMU cpus. Aside from core2duo, I got Penryn to IvyBridge working (unfortunately here the busspeed isn't multiplied by 4, making the VM run even faster ;-) ). 

[evilsephiroth]

Didn't kvm ignore msrs for booting OS X? Can you detail better the changes in your module? I hope doesn't break other things...

[waschbenzin]

It only ignores unsupported msrs (you can have a look at the dmesg kernel log). 

 

My changes  to arch/x86/kvm/x86.c -> kvm_get_msr_common:

case 0xce:
msr_info->data = (((uint64_t)40ULL) << 40)  | (((uint64_t)40ULL) << 8);
break;

The 40ULL values are my "busratio", to get a 4GHz TSC from a 100MHz Bus. Here I modify the min (<< 40) and max value ( << 8) of the flexible bus ratio to a fixed value of 40. As I don't know the side effects of the settings (for Linux and Windows Guests especially) I don't like this solution at all.

[evilsephiroth]

 

It only ignores unsupported msrs (you can have a look at the dmesg kernel log). 

 

My changes  to arch/x86/kvm/x86.c -> kvm_get_msr_common:

case 0xce:
msr_info->data = (((uint64_t)40ULL) << 40)  | (((uint64_t)40ULL) << 8);
break;

The 40ULL values are my "busratio", to get a 4GHz TSC from a 100MHz Bus. Here I modify the min (<< 40) and max value ( << 8) of the flexible bus ratio to a fixed value of 40. As I don't know the side effects of the settings (for Linux and Windows Guests especially) I don't like this solution at all.

 

downloaded kernel source, modified source code, recompiled only module or whole kernel ? 

 

installed without uncommon procedures?

[waschbenzin]

This is what I did (may contain typos). 

apt-get source linux-source-4.2.0 linux-headers-$(uname -r)
#copy to desired location
make oldconfig
make prepare
make scripts
cp -v /usr/src/linux-headers-$(uname -r)/Module.symvers .
# edit source files
make M=arch/x86/kvm modules
modprobe -r kvm_intel
modprobe -r kvm
insmod arch/x86/kvm/kvm.ko
insmod arch/x86/kvm/kvm-intel.ko

I put a few debug outputs in and I could see my module was in place using dmesg.

 

But as said, I don't think this is the road we want to go. I hope I have a bit of time to have a closer look at a fix in Clover soon. 

[evilsephiroth]

Did you try busratio option in clover?

[waschbenzin]

Yes, but I think Clover can't pass the bus ratio to the mac kernel. At least not to the part where the tsc timing is calculated. I think this is unrelated to the speed problem.

 

So I finally looked at the calculations in Clover and I just couldn't make my way through all of it. So I decided to change the value at the last possible moment, which isn't an ideal solution either. The following code will set the frontsidebus solely from the tsc calibration clover does. For core2duo cpu it will divide by 4 otherwise just use the value as it is.

 if (gSettings.QEMU) {
    FrontSideBus = gCPUStructure.TSCFrequency;
    switch (gCPUStructure.Model) {
      case CPU_MODEL_DOTHAN:
      case CPU_MODEL_YONAH:
      case CPU_MODEL_MEROM:
        FrontSideBus = DivU64x32(FrontSideBus, 4);
        break;
      default:
        break;
    }
    DBG("Using QEMU FrontSideBus=%d\n", FrontSideBus);
  }

This works for me using either core2duo or IvyBridge as processor in the vm. It even works directly after rebooting, which the previous solution didn't (didn't seem to be deterministic in any way, but when it worked, it worked until the next reboot ;-) )

 

I included a diff, so you can easily modify your own clover build or slice could implement this into the official clover (making any compiling unneccessary). It also includes a Clover-Version with the fix. Replace EFI/BOOT/BOOTX64.efi and EFI/CLOVER/CLOVERX64.efi in the EFI partition of your Clover image with this file and speedproblems in qemu hopefully come to an end...

clover-qemu-fsb.zip

[evilsephiroth]

Yes, but I think Clover can't pass the bus ratio to the mac kernel. At least not to the part where the tsc timing is calculated. I think this is unrelated to the speed problem.

 

So I finally looked at the calculations in Clover and I just couldn't make my way through all of it. So I decided to change the value at the last possible moment, which isn't an ideal solution either. The following code will set the frontsidebus solely from the tsc calibration clover does. For core2duo cpu it will divide by 4 otherwise just use the value as it is.

 if (gSettings.QEMU) {
    FrontSideBus = gCPUStructure.TSCFrequency;
    switch (gCPUStructure.Model) {
      case CPU_MODEL_DOTHAN:
      case CPU_MODEL_YONAH:
      case CPU_MODEL_MEROM:
        FrontSideBus = DivU64x32(FrontSideBus, 4);
        break;
      default:
        break;
    }
    DBG("Using QEMU FrontSideBus=%d\n", FrontSideBus);
  }

This works for me using either core2duo or IvyBridge as processor in the vm. It even works directly after rebooting, which the previous solution didn't (didn't seem to be deterministic in any way, but when it worked, it worked until the next reboot ;-) )

 

I included a diff, so you can easily modify your own clover build or slice could implement this into the official clover (making any compiling unneccessary). It also includes a Clover-Version with the fix. Replace EFI/BOOT/BOOTX64.efi and EFI/CLOVER/CLOVERX64.efi in the EFI partition of your Clover image with this file and speedproblems in qemu hopefully come to an end...

well, honestly I am stuck at clover boot...

 

I attached a cd to the vm with virt-manager like this to install on my logical Volume -device ide-hd,bus=ide.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 -drive file=/mnt/archData/virtualMisc/ElCapitan.iso,if=none,media=cdrom,id=drive-sata0-0-2,readonly=on,format=raw.

 

When I boot into the cd (with -v -x to see output) with clover generated by your procedure, I have this 

Error allocating 0x800 pages at 0x0000000000200000 alloc type 2

Error loading kernel cache (0xe)

 

The iso has been done with DiskMaker from a real Mac.

[waschbenzin]

I haven't had much luck with ISO images in this setup either. You could to make a usb installer according to https://support.apple.com/HT201372, but you seem to have no problem with mounting the image. I too noticed, that booting without the prepared cached kernel can be troublesome. Maybe dropping -x from the boot-args helps booting the kernel normally...

 

You can tried to add OsxAptioFixDrv, OsxAptioFix2Drv or OsxLowMemFixDrv to your Clover image. They are supposed to fix this kind of issue (I think). You can easily modify the clover image in linux, when you mount it through a loop device (figure out offset and sizelimit with disk):

mount -o loop,offset=1048576,sizelimit=10469376 path/to/image path/where/to/mount

[evilsephiroth]

 

I haven't had much luck with ISO images in this setup either. You could to make a usb installer according to https://support.apple.com/HT201372, but you seem to have no problem with mounting the image. I too noticed, that booting without the prepared cached kernel can be troublesome. Maybe dropping -x from the boot-args helps booting the kernel normally...

 

You can tried to add OsxAptioFixDrv, OsxAptioFix2Drv or OsxLowMemFixDrv to your Clover image. They are supposed to fix this kind of issue (I think). You can easily modify the clover image in linux, when you mount it through a loop device (figure out offset and sizelimit with disk):

mount -o loop,offset=1048576,sizelimit=10469376 path/to/image path/where/to/mount

tried without -x but the output is a bunch of "+" signs... will try adding the relevant fix.  You used an usb installer?

 

EDIT: sadly El capitan is only on my real mac so I will install on a real usb and then dump with dd.

tried without -x but the output is a bunch of "+" signs... will try adding the relevant fix.  You used an usb installer?

 

EDIT: sadly El capitan is only on my real mac so I will install on a real usb and then dump with dd.

with the fix it boots but then after a infamous "missing bluetooth controller transport" and nvidia official it reboots... 

Thinking to go with qemu for now and later translate in a feasible libvirt guest

[nexx892]

 

How does mac osx calculate its speed?

 

The speed calculation can be found at http://www.opensource.apple.com/source/xnu/xnu-2782.40.9/osfmk/i386/tsc.cin the method tsc_init. 

 

There seem to be two options:

 

At first there is a check for a virtual cpu, if everything is correct here, the values are just read from the cpuid (special processor command the qemu/kvm can emulate). I tried modifying my kvm kernel module to get this working correctly, but I couldn't do it. I modified arch/x86/kvm/cpuid.c. There is a huge switch-case in __do_cpuid_ent that handles the different functions. I modified (0x4000000/KVM_CPUID_SIGNATURE/) to include "leafs" up to 0x4000010 and modified this function to return the correct speeds for my system in the processor registers. While I could verify that the change was in place it was never called :-(.

 

The second option is used, if the speeds can not be read from the vm's cpu. In this case the EFI is asked for Bus and TSC speeds. The value received here are from the DataHubDxe used in Clover. Clover sets these values in DataHubCpu.c. While the osx-kernel does read the TSC speed (the speed that makes our clocks go wrong), it simply doesn't use it. Instead it determines the tscGranularity from the cpu. There are a few MSR (model-specific registers) that are used to read this values (I'm pretty sure Clover can't modify these values). I am able to modify the kvm kernel module to return a modified busratio (add case 0xCE to arch/x86/kvm/x86.c -> kvm_get_msr_common) . Allowing me to run my system at a Busspeed of 100MHz with 4000Ghz TSC. 

 

This could be the end of the story, but having a tainted kernel, that I have to modify with each new kernel version doesn't seem to be an ideal solution. I would prefer to propose some changes into Clover:

- Refactor Clover to allow a 64-bit value for the BusRatio. Currently a signed 32-bit value seems to be used.

- Remove the checks in DataHubCpu.c which limits the Busspeed to 1GHz (if QEMU is set). BTW: The osx kernels sanity checks are 90Mhz to 10Ghz.

 

This way we would have complete freedom for the BusSpeed setting in Clover.

 

Other findings:

OSX seems to be supported on more QEMU cpus. Aside from core2duo, I got Penryn to IvyBridge working (unfortunately here the busspeed isn't multiplied by 4, making the VM run even faster ;-) ). 

 

How did you get IvyBridge working? Also has anybody figured out a method for audio?

[waschbenzin]

I can't remember doing anything special apart from replacing core2duo with IvyBridge in the command line. 

 

Sound options:

- USB audio (this is what I'm doing)

- Virtual audio device

- HDMI sound of the pass through graphics card

- passing through a sound card

 

My experience with getting sound working apart from USB sound is pretty limited. You can still inject kexts to get sound working. There should be plenty of information on the board

[evilsephiroth]

well, so far the vm is working nice aside from 2 problems...

 

1)No hdmi audio. Gonna investigate and eventually switch to usb dongle

 

2)Multiple screens....

I have 2 screens. gpu 970 gtx is connected by hdmi cable to one and dvi-d cable to the other. no adaptor or something like that

 

When I start the vm, ovmf and clover are displayed on hdmi screen at 1920x1080.then when booting resolution is reduced to 800x600 cut (even if I set in clover plist 1920x1080). After post and boot screen,

only dvi screen is displayed and hdmi monitor goes in standby mode...

 

I'm not using adaptors nor strange cables. My gpu has also display port and I'm gonna try with a display port => hdmi with the other combinations (dp=> hdmi + hdmi OR dp=> hdmi + DVI-D).

Booting with nvda_drv=1

 

 

Any hint or experience?

[waschbenzin]

1) HDMI audio is not working in my setup either.

 

2) I think Mac OS X is supposed to boot at 800x600, since otherwise I end up with a mess on screen. My triple screen setup is working fine. This probably depends on how compatible your graphics card is with mac osx. Have you tried a real Hackintosh? Is it any different?

 

I'm not using any special options to get the graphics card working. 

 

I've now set CsrActiveConfig to 0x0 to enable the System Integrity Protection (SIP), which is working just fine .

[evilsephiroth]

1) HDMI audio is not working in my setup either.

 

2) I think Mac OS X is supposed to boot at 800x600, since otherwise I end up with a mess on screen. My triple screen setup is working fine. This probably depends on how compatible your graphics card is with mac osx. Have you tried a real Hackintosh? Is it any different?

 

I'm not using any special options to get the graphics card working. 

 

I've now set CsrActiveConfig to 0x0 to enable the System Integrity Protection (SIP), which is working just fine .

seems strange. My card has 4 outputs. DVI-I, DVI-D, cdmi and display port and can t output 2 screens in the same time... 

 

GTX 970 Asus DirectCu Mini. Sadly, I haven't got space for another gpu in my micro atx motherboard... 

Trying Yosemite could solve my doubts?

[evilsephiroth]

1) HDMI audio is not working in my setup either.

 

2) I think Mac OS X is supposed to boot at 800x600, since otherwise I end up with a mess on screen. My triple screen setup is working fine. This probably depends on how compatible your graphics card is with mac osx. Have you tried a real Hackintosh? Is it any different?

 

I'm not using any special options to get the graphics card working. 

 

I've now set CsrActiveConfig to 0x0 to enable the System Integrity Protection (SIP), which is working just fine .

what smbios are you using? boot args? gpu? just to know if I'm missing something

what smbios are you using? boot args? gpu? just to know if I'm missing something

auto solved. smbios iMac14,2 was using MacPro3.1

 

EDIT: second monitor seems somewhat blurry... It has been detected as svga display. First instead is working fine...

 

EDIT2: hdmi + dvi-i combination worked

1) HDMI audio is not working in my setup either.

 

2) I think Mac OS X is supposed to boot at 800x600, since otherwise I end up with a mess on screen. My triple screen setup is working fine. This probably depends on how compatible your graphics card is with mac osx. Have you tried a real Hackintosh? Is it any different?

 

I'm not using any special options to get the graphics card working. 

 

I've now set CsrActiveConfig to 0x0 to enable the System Integrity Protection (SIP), which is working just fine .

just one thing. I've got clover in a separated image. Is there a way to tell clover to boot right away without showing boot menu?

[waschbenzin]

Of course, you can boot without the boot screen! Have a look at http://clover-wiki.zetam.org/Configuration/Boot. DefaultVolume and Timeout (possibly Fast?) are the values you are looking for.

 

Had the system running for 10 hours now. The clock gathered an additional 15 minutes :-\. Well I'm from 1000% speed down to 102,5% (without any hardcoded busspeed), not too bad...

[andonuts]

Can't get this to work. Here's my setup and what happens:

 

h/w

CPU: Intel Xeon E3-1226

GPU: AMD R9 290X

 

s/w

Clover: 2.3k_r3320

DSDT: q35-acpi-dsdt.aml

boot-args: debug=0x08

boot media: OS X 10.11.1 Installer (created using createinstallmedia)

 

qemu

qemu-system-x86_64 -enable-kvm -m 2048 -M q35 -cpu core2duo,vendor=GeniuneIntel -smp 2,sockets=1,cores=2,threads=1 -bios /usr/share/ovmf/OVMF_CODE.fd -usb -smbios type=2 -net none -device isa-applesmc,osk="osk-removed" -drive id=Clover,if=none,file=clover.img -device ide-drive,bus=ide.1,drive=Clover -drive id=MacHDD,if=none,file=/dev/sdf -device ide-drive,bus=ide.2,drive=MacHDD -device vfio-pci,host=01:00.0,bus=pcie.0 -device vfio-pci,host=01:00.1,bus=pcie.0 -device vfio-pci,host=07:00.0,bus=pcie.0 -vga none -serial stdio -display none

I can get into Clover, which finds the OS X installer drive. When I try to boot OS X, I get a blank screen and this serial output:

!!!! X64 Exception Type - 06(#UD - Invalid Opcode)  CPU Apic ID - 00000000 !!!!
RIP  - 00000000000E0000, CS  - 0000000000000038, RFLAGS - 0000000000010246
RAX  - 0000000000000000, RCX - 0000000000000000, RDX - 000000007DA91B20
RBX  - 000000007CCE2018, RSP - 000000007FF597C8, RBP - 000000007DA80332
RSI  - 000000007DA91B20, RDI - 0000000000004545
R8   - 000000007DA97180, R9  - 0000000000000004, R10 - 000000007DA50D36
R11  - 000000007EE9E55D, R12 - 000000007EE9A018, R13 - 000000007DA87D79
R14  - 000000007DA566C9, R15 - 000000007DA44D5D
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080000033, CR2 - 0000000000000000, CR3 - 000000007FEF8000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 000000007FED6F18 0000000000000047, LDTR - 0000000000000000
IDTR - 000000007F696018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 000000007FF59420
!!!! Can't find image information. !!!!

[waschbenzin]

Did you enable ignoring invalid MSRs?

echo 1 > /sys/module/kvm/parameters/ignore_msrs

[Slice]

 

Did you enable ignoring invalid MSRs?

echo 1 > /sys/module/kvm/parameters/ignore_msrs

Is it a linux's command?

-bash: /sys/module/kvm/parameters/ignore_msrs: No such file or directory

I have same panic as andonuts

 

What I did:

1. Created ISO.

2. Installed Clover (with QEMU patch!)

3. Put some system files on Mackintosh partition of the ISO. But yes, it is not complete installation. There are files from 10.7.5.

4. QEMU 2.2.0

start with

qemu-system-x86_64 -L ~/Desktop/QemuTest/QEMU -m 2048 -cpu core2duo -bios OVMF.fd  -machine q35 -vga std -usb -device usb-kbd -device usb-mouse -hda /Users/slice/Desktop/QemuTest/QEMU-test2.img

and got this

 

 

The system begins to load

 

But then the panic.

[evilsephiroth]

Is it a linux's command?

-bash: /sys/module/kvm/parameters/ignore_msrs: No such file or directory

I have same panic as andonuts

 

What I did:

1. Created ISO.

2. Installed Clover (with QEMU patch!)

3. Put some system files on Mackintosh partition of the ISO. But yes, it is not complete installation. There are files from 10.7.5.

4. QEMU 2.2.0

start with

qemu-system-x86_64 -L ~/Desktop/QemuTest/QEMU -m 2048 -cpu core2duo -bios OVMF.fd -machine q35 -vga std -usb -device usb-kbd -device usb-mouse -hda /Users/slice/Desktop/QemuTest/QEMU-test2.img

and got this

Снимок экрана 2015-11-05 в 16.08.45.png

 

Снимок экрана 2015-11-05 в 16.09.43.png

 

The system begins to load

Снимок экрана 2015-11-05 в 16.10.49.png

 

But then the panic.

Hmm do you have kvm qemu and libvirt installed in your Linux distro? Kernel version?

[Slice]

Hmm do you have kvm qemu and libvirt installed in your Linux distro? Kernel version?

No, I am tested in Mac OSX 10.7.5. 

kvm is somehow impossible and libvirt absent.

 

I usually used QEMu for testing Clover. It is fast and safe method.

 

The patch sent to 3322

[waschbenzin]

He isn't using linux at all, but QEMU under Mac OSX (see /Users ... decoration in screenshots...). Therefore the command to ignore msr in kvm doesn't help, since he isn't using kvm at all. This changes everything . I really didn't expect to run QEMU outside of linux, since without kvm its terribly slow.

 

Nevertheless I used the following command line to boot Mac OS without KVM:

qemu-system-x86_64 \

-device ahci,id=hdbus,bus=pcie.0 \

-device ide-drive,bus=hdbus.1,drive=Clover \

-drive id=Clover,if=none,file=Clover-3320.raw,format=raw  \

-m 2048 \

-cpu core2duo \

-bios OVMF-pure-efi.fd  \

-machine q35 \

-vga std \

-usb -device usb-kbd -device usb-mouse \

-device isa-applesmc,osk="blablabla" \

-hda /raid/Temp/mac-disk.img

 

With slices command line I couldn't even get to Clover, since OVMF was unable to find the partition. 

 

I will give this a try on my mac and report back.