zebity Posted May 23, 2018 Share Posted May 23, 2018 (edited) Hi Hypervisors, EDITED - 9 JUNE 2018 - as all other issues resolved but am now stuck with QEMU Error: "PFLASH: Possible BUG - Write Block Confirm" details on testing lower in thread (to follow). I have been working to move my Late 2009 Xserve MacOS Server onto Ubuntu 18.04 LTS QEMU/KVM based virtual machine using OVMF and Clover. After much effort and testing based on the following information from: Kraxel's, Kholia, Gordon Turner and Clover site: https://www.kraxel.org/blog/2017/09/running-macos-as-guest-in-kvm/ https://github.com/kholia/OSX-KVM https://gist.github.com/gordonturner/2a2e5ecde5e7860b52e2 https://clover-wiki.zetam.org/Home I have now managed to get OVMF/Clover boot and install of MacOS Sierra. I started with Ubuntu 16.04 initially but this required download and compile of QEMU to get update of machines to: pc-q35-2.9 or better, so I moved to Ubuntu 18.04, which has pc-q35.2.11 available as standard. So I am now working with standard Ubuntu 18.04 LTS based systems with following packages: kvm/qemu/libvirt/bridge-utils/ovmf/virt-manager As I want to use PCIe Passthrough for a number of PCIe card currently installed in the server (SmallTree 10GbE & Areca ARC-1883 SAS RAID) I set up my machine with linux kernel boot configuration (/etc/default/grub) of: GRUB_CMDLINE_LINUX_DEFAULT="iommu=1 intel_iommu=on" (as my machine is intel VT-d based HW virtualisation) This resulted in creation of a number of iommu groups (see /sys/kernel/iommu_group directory for this and this posting for information: https://forum.level1techs.com/t/ubuntu-17-04-vfio-pcie-passthrough-kernel-update-4-14-rc1/119639 , noting that Ubuntu 18.04 LTS has kernel version: 4.15.0-22-generic so there is no need to do kernel update for iommu to work). Using bare minimal Clover config.plist: Spoiler <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Boot</key> <dict> <key>Arguments</key> <string></string> <key>DefaultVolume</key> <string>clover</string> <key>Debug</key> <false/> <key>Secure</key> <false/> <key>Timeout</key> <integer>3</integer> </dict> <key>GUI</key> <dict> <key>Scan</key> <dict> <key>Entries</key> <true/> <key>Tool</key> <true/> </dict> <key>ScreenResolution</key> <string>1024x768</string> <key>Theme</key> <string>embedded</string> </dict> <key>RtVariables</key> <dict> <key>BooterConfig</key> <string>0x28</string> <key>CsrActiveConfig</key> <string>0x3</string> </dict> <key>SMBIOS</key> <dict> <key>Trust</key> <false/> </dict> <key>SystemParameters</key> <dict> <key>InjectKexts</key> <false/> <key>InjectSystemID</key> <true/> </dict> </dict> </plist> And following Clover UEFI drivers: Spoiler apfs.efi AppleImageCodec-64.efi AppleKeyAggregator-64.efi AppleUITheme-64.efi DataHubDxe-64.efi FirmwareVolume-64.efi FSInject-64.efi OsxFatBinaryDrv-64.efi PartitionDxe-64.efi SMCHelper-64.efi UsbKbDxe-64.efi UsbMouseDxe-64.efi VBoxHfs-64.efi I can boot MacOS, but it is very very slow... it sits on the apple boot logo for over a minutes before finally moving onto the progress bar. The other problem I have is that no PCI or Network devices appear in the "About This Mac" System Information Report and I cannot get any network connectivity from bridged E1000 network device. Here is my virtlib.xml dump for the virtual machine: Spoiler <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>NAME-server-macos-10.12</name> <uuid>38986f0a-d1fd-476d-8478-3220afa50422</uuid> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <vcpu placement='static' current='2'>8</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.11'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/home/USER/Documents/virtual-machines/NAME.server.macos-10.12/OVMF_VARS.fd</nvram> <bootmenu enable='yes'/> </os> <features> <acpi/> <vmport state='off'/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Penryn</model> <vendor>GenuineIntel</vendor> <topology sockets='2' cores='4' threads='1'/> <feature policy='require' name='vme'/> <feature policy='require' name='x2apic'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='aes'/> <feature policy='require' name='xsave'/> <feature policy='require' name='avx'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='avx2'/> <feature policy='require' name='smep'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/kvm-spice</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/USER/Documents/virtual-machines/NAME.server.macos-10.12/NAME-hd1-01.qcow2'/> <target dev='sdb' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/home/USER/Documents/virtual-machines/images/macos-sierra-install.img'/> <target dev='sdc' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/USER/Documents/virtual-machines/images/clover-boot-24k-r4458.qclow2'/> <target dev='sdd' bus='sata'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <model name='i82801b11-bridge'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <model name='pci-bridge'/> <target chassisNr='2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:4b:c6:e8'/> <source bridge='br50'/> <model type='e1000'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='keyboard' bus='usb'> <address type='usb' bus='0' port='2'/> </input> <input type='mouse' bus='ps2'/> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='3'/> </input> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'> <listen type='address'/> <image compression='off'/> </graphics> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </sound> <video> <model type='vga' vram='65536' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='5'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='1'/> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='none' model='apparmor'/> <qemu:commandline> <qemu:arg value='-readconfig'/> <qemu:arg value='/home/USER/Documents/virtual-machines/NAME.server.macos-10.12/macos.cfg'/> </qemu:commandline> </domain> Can any one advise of whether I need to update the CLOVER config.plist to add extra items in to allow network and PCI Passthrough to work. Thank you. Regards, Zebity Edited June 9, 2018 by zebity Change Topic to reflect: PFLASH: Possible BUG block Link to comment Share on other sites More sharing options...
zebity Posted May 24, 2018 Author Share Posted May 24, 2018 (edited) Hi QEMU Hypervisors, reviewed my set up and saw that I had network model as: <model type='e1000'/> While all the guideance had more specific: <model type='e1000-82545em'/> . I falsely assumed that e1000 would be ok. On changing XML to reflect more specific case, my machine still booted very very slowly but this time my networking come up ok. So I optimistically downloaded the SmallTree 10GbE drivers for my SmallTree 10GbE card (connected via PCI passthrough to VM) and rebooted. Machine is now hanging on the Apple logo boot screen. To try to fix this I tried loaded the dsdt data from Kholia GitHub site and corresponding config.plist that refers to this and put these into my CLOVER setup, but this did not fix problem. So it looks like I need to go back to square one and do fresh install of MacOS Sierra to clean up from the SmallTree test. Any suggestions on approach to getting SmallTree 10GbE card would be appreciated. EDIT: ok I have now rebuilt VM with much less trouble than before (gets easier with practice ;-) ). Notes on rebuilt: 1: Now I am running from virsh I no longer have -monitor stdio flag used in cmd line case, which appears to have impact on crashing. 2: still super slow with initial boot, but runs ok once booted, possible problems could be: efi driver selection, config.plist settings. Thoughts to progress: 1. Remove all driver efis except bare core and see if this improves things, then add others if boot fails... 2. As part of 1 turn on Debug ( not sure where this writes to ... I did have it on at one point and impact was just getting set of ++++ progress indicators rather than Apple logo 3. Replace vboxhfs-64.efi with latest Apple one (I have full Darwindumper, dump from my MacBook Pro Mid 2015 with High Sierra and UEFITool which appears to make it easy to do extract if you know where to look.... hint hint) 4. Thinking out loud ... to get existing Mac EFI compatible PCIe cards to work with clover, I need to get kexts from my existing installation and copy these to my Clover EFI partition and then add add inject items into config.plist, as well as having PCI pass through enabled. No one appears to have posted on getting either SmallTree 10GbE or Areca ARC-1883 raid cards running, so this is unexplored territory I believe. 5. Further thinking out load... in QEMU environment, why would I need any of the various OsxAptioFixXXDrv... drivers, as these are related to AMI bios machines? Isn’t OVMF based on Intel code and therefore these are not required or does core Tiano code require these (or some specific subset them). In any case with QEMU shouldn’t all VMs be pretty much identical when it comes to drivers and dsdt unless you use PCI pass through which brings with it hw variation ? Finally one of areas I had trouble with on virsh XML was with permissions and command line addition as per bottom of my posted XML (based on Kraxel news post to get appple smc “OSK” string added). The work around was add to security label disablement tag: <seclabel type='none' model='apparmor'/> Note: this is Ubuntu specific as AppArmor is ubuntu specific. I did try to add dynamic tagging directive, but this did not work. again, any tips hints would be helpful. Cheers, Zebity Busy learning: KVM, QEMU, Clover and other new technologies. Edited May 27, 2018 by zebity Fix wording Link to comment Share on other sites More sharing options...
zebity Posted May 27, 2018 Author Share Posted May 27, 2018 (edited) Hi QEMUers/Hypervisors, seems I am my own helper at the moment ;-) , so here to report on a successful operation of SmallTree 10GbE via PCI passthrough on QEMU based set up. As per earlier post, I did a bunch of tests with Clover and various driver options installed and at the same time run the Gnome System Monitor application to allow me to see (at very high level) what was going on. Firstly, still very slow to boot, so my ratio to testing to configuring is about 80/20, as I am spending lots and lots of time just waiting for machine to start. I observed that when I used the OsxAptioFixXXDrv drivers that I would end up with a CPU thread locked in infinite loop (CPU usage shoots up to 100% and does not come back down). So I have removed the OsxAptioFixXXDrv drivers from my configuration. I did a quick browse of code for these and they appear to relate to relocating code to make room for OS when you have low memory configuration. When I started trying to get VM running I did have crashes relating to memory problems. This was solved simply by providing more memory to the VM (hence current 16GB). This seemed reasonable as I have at least 16GB in all my physical Mac's and have alway's found that Mac's run much better when given plenty of memory, so would not expect VM to be any different. Here is my more streamlined set of EFI drivers I am now using: Spoiler AppleImageCodec-64.efi AppleKeyAggregator-64.efi AppleUITheme-64.efi DataHubDxe-64.efi FirmwareVolume-64.efi FSInject-64.efi OsxFatBinaryDrv-64.efi PartitionDxe-64.efi SMCHelper-64.efi VBoxHfs-64.efi As I said boot is still slow, but once up and running performance is ok. Having landed on EFI drivers setup, next step was to update config.plist so PCIe passthrough would work for SmallTree 10GbE (starting point) and Areca ARC-1883 (next target). I started by trying to hand configure the SMBIOS settings, using inputs from a couple of DarwinDumper reports I took from MacPro5,1 and MacBook11.5 I took. I had no luck with my hand crafting so resorted to using "Clover Configurator". I have chosen to go with MacPro6,1 based config, as it has a more current hardware profile than my Mac Pro (I have never upgrade this to nMP, due to lack of field upgrade as per cMP). In addition to MacPro6,1 configuration I also had success with Clover Configurator iMac option. As I am trying to keep it simple as learning all these new parts is steep learning curve so have completely avoided the DSDT stuff. This looks like a massive amount of effort, which I might not need, as it relates to ACPI and I am running on a power managed server, so there is no value in having VM look after this stuff as well. The good news is that Clover Configurator generated configs worked straight up. Huge time saver !! Thank you to who ever created that, but wish some code was available to read ;-) . I tested with Inject off and on and found that with Inject (but no kexts EFI/CLOVER directory) that everything appeared to work. So here is my simple MacPro6,1 generated config.plist (no ACPI/DSDT stuff): Spoiler <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Boot</key> <dict> <key>Debug</key> <false/> <key>DefaultVolume</key> <string>clover</string> <key>Secure</key> <false/> <key>Timeout</key> <integer>3</integer> </dict> <key>GUI</key> <dict> <key>Scan</key> <dict> <key>Entries</key> <true/> <key>Tool</key> <true/> </dict> <key>ScreenResolution</key> <string>1024x768</string> <key>Theme</key> <string>embedded</string> </dict> <key>RtVariables</key> <dict> <key>BooterConfig</key> <string>0x28</string> <key>CsrActiveConfig</key> <string>0x3</string> </dict> <key>SMBIOS</key> <dict> <key>BiosReleaseDate</key> <string>02/02/2018</string> <key>BiosVendor</key> <string>Apple Inc.</string> <key>BiosVersion</key> <string>MP61.88Z.0122.B00.1802021814</string> <key>Board-ID</key> <string>Mac-XXXXXXXXXXXXXXXXX</string> <key>BoardManufacturer</key> <string>Apple Inc.</string> <key>BoardSerialNumber</key> <string>XXXXXXXXXXXXXXXXX</string> <key>BoardType</key> <integer>11</integer> <key>BoardVersion</key> <string>1.0</string> <key>ChassisAssetTag</key> <string>Pro-Enclosure</string> <key>ChassisManufacturer</key> <string>Apple Inc.</string> <key>ChassisType</key> <string>0x02</string> <key>Family</key> <string>MacPro</string> <key>FirmwareFeatures</key> <string>0xE80FE137</string> <key>FirmwareFeaturesMask</key> <string>0xFF1FFF3F</string> <key>LocationInChassis</key> <string>Part Component</string> <key>Manufacturer</key> <string>Apple Inc.</string> <key>Mobile</key> <false/> <key>PlatformFeature</key> <string>0x04</string> <key>ProductName</key> <string>MacPro6,1</string> <key>SerialNumber</key> <string>XXXXXXXXXXXX</string> <key>SmUUID</key> <string>XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX</string> <key>Trust</key> <true/> <key>Version</key> <string>1.0</string> </dict> <key>SystemParameters</key> <dict> <key>InjectKexts</key> <string>Yes</string> <key>InjectSystemID</key> <true/> </dict> </dict> </plist> You can see that I have "InjectKexts" set to Yes.. of which more later This config was created by simply loading the super simple one originally from Kraxel news posting and also copied on Kholia postings (as referenced in first posting). So now comes the big moment, testing the SmallTree 10GbE with PCI passthrough. After much reading and looking at what others had done, I decided to re-try the dump "I know nothing approach" used earlier. Most of the examples posted are trying to get non Mac HW to work with Apple SW. In my case I had HW with OSX specific EFI firmware and SW and I am trying to get this to work via QEMU/Hypervisor. This meant that it made more sense to let drivers get installed in standard System/Library/Extensions folder than do manual copy to EFI partition. So result...... It worked from the get go. I now have QEMU MacOS Server with 10GbE. Next stop is Area ARC-1883. As final test I did GeekBench 3 64bit benchmark test to compare how my QEMU machine performed compared to physical Late 2009 Xserve. Again result was good: As you can see the 8 Core QEMU based VM is actually performing better than my physical 8 Core XServe. So things are going in a happy direction. Once I get ARC-1883 I will post further update and will also do disk and network throughput testing. I am also currently working on how to covert physical OSX machine into virtual one and testing moving VMWare Fusion based OSX machines to QEMU. Cheers from Australia, Zebity. Edited May 27, 2018 by zebity fix english. Link to comment Share on other sites More sharing options...
zebity Posted June 4, 2018 Author Share Posted June 4, 2018 (edited) Hi not quite insane as using virtualisation ;-) mac'ers , further progress to report... I have now got Apple Mac compatible ARECA ARC-1883 SAS RAID Card working on my QEMU Mac Server setup. Again this was very simple to set up: 1. Add PCI Card resource to QEMU Hardware for the via Virt-Manager, to get new PCI Passthrough resource 2. Install Areca downloaded driver using standard native MacOS installation process 3. Reboot See here for view of Areca SAS (without disks attached): This image shows the card without disk units powered on, but I have also tested card with 6 RAIDS sets attached (all Apple HFS file sets) and they all come up ok. I am now having problem with crashes through after accessing the network for a while (via SmallTree Ethernet card). Not sure that the problems is but I get same problem with PC-Q35-2.9 and PC-Q35-2.11. Machine just cuts out instantly, with no failure message, so looks like something happening with KVM/QEMU, which is just pulling carpet from under the VM. I have also been working moving my physical machine to QEMU and so far I am still getting crash at boot at about 70% point on progress bar. I did comparison of the kexts on the crashing physical machine and the working clean hypervisor install, and removed the kexts that where different on the physical machines, but still got crash, but this it was further into the boot process. So still testing moving physical to virtual. Tools I have been using for this include: Carbon Copy Cloner to get physical disk image, VMWare Fusion to create vmdk image of physical disk and emu-img to convert VMWare image to qcow2 image. I have still not progressed DSDT setup on my MacOS VM's as setup up other disk images has kept me busy. EDIT: I have been testing the crash problem by running VM and doing tail -f on log. The crash when accessing external disk and network reports as: “PFLASH: Possible BUG - Write block confirm” This bug appears to have been reported some time ago (2013), but not recently. Has only one else seen this ? Cheers from Oz. Zebity Edited June 5, 2018 by zebity Add bug Link to comment Share on other sites More sharing options...
zebity Posted June 9, 2018 Author Share Posted June 9, 2018 (edited) Hi Mac Virtualizers. I have now been doing testing and updates with QEMU, OVMF, Clover and both Sierra & High Sierra, but am now blocked by QEMU: "PFLASH: Possible BUG - Write Block Confirm" error I am getting when I put stress on my VM. My objective was to try to get MacOS running with pretty much plain vanilla Ubuntu based installation as with so many Hacintosh out there this should not require a huge effort to get running from a user perspective. To this end I have a pretty vanilla set of installs: Lenovo X Series Server Ubuntu 18.04 Desktop (64 Bit) with all packages via "apt-get": QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.2) OVMF EDK II, UFI v2.70 (EDK II, ox00010000) ovmf/bionic,bionic,now 0~20180205.c0d9813c-2 While I started my testing with Clover Version 2.4k-r4458 (working), during testing moved to: -> 2.4k-r4497 (did not work, but I did not test it much) then went to: -> 2.4k-r4509 (worked and used for most of current testing) I have successfully got SmallTree 10GbE and Areca ARC-1883 working with via PCI Pass-through with standard MacOS drivers installed from running MacOS install. I also did in place upgrade from Sierra to High Sierra via QEMU emulation and this worked 100% ok. Here is my working drivers64UEFI directory for QEMU MacOS Sierra & High Sierra Spoiler apfs.efi AppleImageCodec-64.efi AppleKeyAggregator-64.efi AppleUITheme-64.efi DataHubDxe-64.efi FirmwareVolume-64.efi FSInject-64.efi HFSPlus.efi OsxFatBinaryDrv-64.efi PartitionDxe-64.efi SMCHelper-64.efi UsbKbDxe-64.efi UsbMouseDxe-64.efi So while everything looks ok, I keep getting the following crash error: PFLASH: Possible BUG - Write block confirm2018-06-05 07:00:59.406+0000: shutting down, reason=crashed NOTE: This was diagnosed by watching log on KVM host machine: "sudo tail -f /var/log/libvirt/qemu/MACHINE-ID.log" I can consistently reproduce the error by doing sftp transfer from guest to host to try to move data from one MacOS to Linux Host. It does not matter if I am using emulated E1000 network or PCI Pass-through SmallTree network. It also doesn't matter if you switch from server to client based transfer. It appears to related to throughput of transfer as I can do large transfers via internet (HTTP) ok (downloaded High Sierra 5GB install app from Apple App Store without a hitch). While these are large downloads the throughput is much lower than my large and sustained 30GB transfers form VM to Host. I have tested lots of variations of Clover versions, EFI drivers and SMBIOS machine variations and while I can get machine booting and working, I keep getting this error. To try to get this working I conducted the following variations: In all cases when I used any of: OsxAptioFixDrv-64.efi, OsxAptioFix2Drv-64.efi & OsxAptioFix3Drv-64.efi with both Sierra & High Sierra I always ended up with a CPU thread going to 100% and boot failing to complete. I am surprise that many of the current instructions/tests still recommend using these "fixes". I would have expected that QEMU would have hidden any hardware variation and so other testers would have seen the same results I am seeing. I tested with: OsxLowMemFixDrv-64.efi, this resulted in no difference in behaviour, and boot completed both with and without this "fix" installed. I tested with Apple HFSPlus.efi extract and VBoxHfs-64.efi and boot worked and there was no apparent difference in boot speed. I started my testing with MacPro6,1 (configured via "Clover Configurator") based SMBIOS and then moved to MacPro5,1 (always core dumped during boot) to MacMini6,2 (working ok). In both working cases I still get PFLASH error. As part of testing I have also changed the libvirt XML to remove reading osk key from file, which avoids permission problem with file and need to introduce the AppAmor work around I posted earlier. Here is my current libvirt XML file: Spoiler <domain type='kvm' id='28' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>MACHINE</name> <uuid>XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX</uuid> <memory unit='KiB'>25165824</memory> <currentMemory unit='KiB'>25165824</currentMemory> <vcpu placement='static'>8</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.11'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/home/USER/DIRECTORY/OVMF_VARS.fd</nvram> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Penryn</model> <vendor>GenuineIntel</vendor> <topology sockets='2' cores='4' threads='1'/> <feature policy='require' name='vme'/> <feature policy='require' name='x2apic'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='aes'/> <feature policy='require' name='xsave'/> <feature policy='require' name='avx'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='avx2'/> <feature policy='require' name='smep'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/kvm-spice</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/USER/DIRECTORY/MACHINE-hd3.qcow2'/> <backingStore/> <target dev='sdf' bus='sata'/> <alias name='sata0-0-5'/> <address type='drive' controller='0' bus='0' target='0' unit='5'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/USER/DIRECTORY/MACHINE-hd2.qcow2'/> <backingStore/> <target dev='sdh' bus='sata'/> <alias name='sata1-0-1'/> <address type='drive' controller='1' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/USER/DIRECTORY/clover-boot-24k-r4509.qcow2'/> <backingStore/> <target dev='sdi' bus='sata'/> <boot order='1'/> <alias name='sata1-0-2'/> <address type='drive' controller='1' bus='0' target='0' unit='2'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='sata' index='1'> <alias name='sata1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <model name='i82801b11-bridge'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <model name='pci-bridge'/> <target chassisNr='2'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x10'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x11'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x12'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x13'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x14'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0x15'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <serial type='pty'> <source path='/dev/pts/16'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/16'> <source path='/dev/pts/16'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='tablet' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='3'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <input type='keyboard' bus='usb'> <alias name='input3'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='spice' port='5912' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> <gl enable='no'/> </graphics> <video> <model type='vga' vram='65536' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> <address type='usb' bus='0' port='5'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir1'/> <address type='usb' bus='0' port='1'/> </redirdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'> <label>libvirt-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX</label> <imagelabel>libvirt-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+XXXXX:+XXX</label> <imagelabel>+XXXXX:+XXX</imagelabel> </seclabel> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='read the Inside OS X book by amit sigh to get code for this'/> </qemu:commandline> </domain> This is for machine configured with 3 PCI Pass-Through cards (2 x PCI for SmallTree/Intel X540-AT2 & 1 x PCI for Areca ARC-1883). You will note that I have, PC-Q35-2.11 as I tested with 2.11 & 2.9 (known good) and found that they both worked the same and both experienced the same "PFLASH" crash. For completeness here is current Clover config.plist file: Spoiler <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Boot</key> <dict> <key>Debug</key> <false/> <key>DefaultVolume</key> <string>clover</string> <key>Secure</key> <false/> <key>Timeout</key> <integer>3</integer> </dict> <key>GUI</key> <dict> <key>Scan</key> <dict> <key>Entries</key> <true/> <key>Tool</key> <true/> </dict> <key>ScreenResolution</key> <string>1024x768</string> <key>Theme</key> <string>embedded</string> </dict> <key>RtVariables</key> <dict> <key>BooterConfig</key> <string>0x28</string> <key>CsrActiveConfig</key> <string>0x3</string> </dict> <key>SMBIOS</key> <dict> <key>BiosReleaseDate</key> <string>02/02/2018</string> <key>BiosVendor</key> <string>Apple Inc.</string> <key>BiosVersion</key> <string>MM61.88Z.010D.B00.1802021217</string> <key>Board-ID</key> <string>Mac-XXXXXXXXXXXXXXXX</string> <key>BoardManufacturer</key> <string>Apple Inc.</string> <key>BoardSerialNumber</key> <string>XXXXXXXXXXXXXXXXXX</string> <key>BoardType</key> <integer>10</integer> <key>BoardVersion</key> <string>1.0</string> <key>ChassisAssetTag</key> <string>Mini-Aluminum</string> <key>ChassisManufacturer</key> <string>Apple Inc.</string> <key>ChassisType</key> <string>0x10</string> <key>Family</key> <string>Mac mini</string> <key>FirmwareFeatures</key> <string>0xE00DE137</string> <key>FirmwareFeaturesMask</key> <string>0xFF1FFF3F</string> <key>LocationInChassis</key> <string>Part Component</string> <key>Manufacturer</key> <string>Apple Inc.</string> <key>Mobile</key> <false/> <key>PlatformFeature</key> <string>0x01</string> <key>ProductName</key> <string>Macmini6,2</string> <key>SerialNumber</key> <string>XXXXXXXXXXXX</string> <key>SmUUID</key> <string>XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX</string> <key>Trust</key> <true/> <key>Version</key> <string>1.0</string> </dict> <key>SystemParameters</key> <dict> <key>InjectKexts</key> <string>Yes</string> <key>InjectSystemID</key> <true/> </dict> </dict> </plist> As noted, I moved from MacPro6,1 to MacMini6,2 as I thought maybe SMBIOS was a factor in exposing PFLASH defect. But again aside from finding that MacPro5,1 did not work, there is no difference in results once you get booting machine. Other setup items required if you want to use PCI Passthrough and QEMU with MacOS: Set up KVM iommu (Intel VT-D) support: Config File: /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=1 intel_iommu=on" Setup QEMU/KVM Options (just documenting locations and values, as I have not tested impact of these much) Config File: /etc/modprobe.d/qemu-system-x86.conf options kvm_intel nested=1 Config File: /sys/module/kvm/parameters/ignore_msrs value: Y Thoughts on ACPI/DSDT and QEMU: In some QEMU / MacOS threads/postings it recommends using CLOVER to configuration to load "q35-acpi-dsdt.aml". This file was a binary blob that was posted as part of QEMU source code. It appears that this file is now completely obsolete and it has been removed from the current QEMU Tree. Rather the Q35 emulator generates the ACPI tables dynamically as part of start up. So use of the old static ACPI table and any associated patches would appear to be more likely to break your QEMU configuration that fix it. As per Clover instructions you can press F4 during boot to dump the generated ACPI tables and use these dumps to look at the ACPI configurations. I have got Sierra and High Sierra running with pretty standard QEMU setup without need for any DSDT patching. Ubuntu 18.04, Netplan and Network Bridges: If you are trying to setup QEMU based MacOS servers with Ubuntu 18.04 then you will likely bumped into MacOS based VM & Netplan issues. In Ubuntu 18.04 you no longer configure the IP network settings via the /etc/network/interfaces file. Rather you have to use the new Netplan YAML based mechanism. To configure static IP addresses and bridges with Netplan you need to edit configuration files in "/etc/netplan/FILE.yaml" and then do an "sudo netplan apply". The YAML allows you to define the configuration of your machines networks, VLAN, LAG groups and Bridges in a single place. Typically with a simple configuration you for a set of server VM that are operating on a bridge on single physical interface you would declare the static IP addresses for the machines on the bridge and then within the VM guest you can simple use "ifconfig" to define the same static ip address on the interface of the virtual machine. This works with both Linux and FreeBSD VMs (I have not tested with Windows VMs), but with MacOS VM you get an "IP address already in use" conflict message when you configure a static IP address via MacOS Network Preferences. The reason is that the address has already been made visible on the bridge as part of bridge setup which the MacOS obviously probes against and it them stops the ifconfig from completing and the automatic /etc/resolv.conf (via linked file) does not occur. So your VM is stranded without network connection. To resolve this I simply removed the ip address definition from the netplan bridge configuration. See the following Netplan configuration example; Spoiler # Let NetworkManager manage all devices on this system network: version: 2 renderer: networkd ethernets: eno1: dhcp4: no dhcp6: no eno2: dhcp4: no addresses: [192.168.10.2/24] gateway4: 192.168.10.1 nameservers: search: - here.com - and.here.com addresses: [192.168.10.3] bridges: br30: interfaces: [eno1] dhcp4: no addresses: - 192.168.200.5/24 - 192.168.200.6/24 - 192.168.200.7/24 gateway4: 192.168.200.1 parameters: stp: false forward-delay: 0 # note: macOS server with static address is 192.168.200.8 - but it is not declared as this result in - IP address in use conflict from MacOS Guest In this case I have put static ip addresses on the bridge for 3 x Linux VM, but not static IP address for MacOS VM which has IP address 192.168.1.8 This is a workaround that appears to work. Though when you first use the interface you get packet loss, no doubt due to bridge having not yet learnt about the undeclared status ip address that is sitting on bridge. I suspect that once bridge forwards ARP requests, then VM responds and so connection is made through the bridge. This is apparent in the behaviour of bridged IP addresses, which are pingable before any VM has been brought up on the bridge, while for non declared status IP address, ping is only responded to once guest VM is up and its interface configured. A possible alternative approach would be to enable DHCP on the bridge and then return particular IP address, gateway, dns settings based on the requesting MAC address. I did not try this workaround as it required more Netplan expertise than I currently have. So if you are using Ubuntu 18.04 expect to have learn Netplan. NOTE: my 2 disk virtual machine (+ Clover Boot Disk) is no longer cleanly installed VM, but the re-creation of physical machines by using Carbon Copy Cloner to generate disk images and then restoring these onto VMWare vmdk images and then using "qemu-img convert" to convert to qcow2 disks. To get physical machines converted to QEMU machines, I first did kernel extensions clean up of the physical machine. This is needed as some of my physical Macs have accumulated updates over more than 5 years, as I have always updated machines by pulling or copying disks from old machines to new machine. Once you have a nice physical machine with only required extensions then moving to QEMU was quite straight forward. On the other hand I had a lot of trouble with machines with lots of historical extensions in them. To get new kext cache created you need to remove extensions from: /System/Library/Extensions & /Library/Extensions and then touch these two directories (/System/Library/Extensions & /Library/Extensions) and reboot to get rebuild happening. When doing reboot from virtualised machine after extension removal and touch, I always got boot problems. I suspect I will have to report the PFLASH problem as QEMU defect, unless someone can give to guidance on how to resolve or work around this. Thanks from Australia, Zebity. Edited June 12, 2018 by zebity Netplan update Link to comment Share on other sites More sharing options...
Recommended Posts