Graphics Acceleration in ESXi 5.1 and OS X Mountain Lion VM

[sgreszcz]

I have a couple of Mac Minis (core 2 duo) which I have been using for video projects and transcoding of my kid's DVDs and old analogue home movies into H.264. Even with queuing up transcoding, it can take quite a while to get things right so I have been experimenting with getting Mountain Lion running on my 8-core ESXi 5.1 lab box. The installation went smoothly, thanks to everyone in this forum and for those that created the Unocker!

 

When I have tried to install my video software I get error messages saying that I need video accelerated hardware to be present. I am running ESXi on an ASUS Z8NA-D6 server, which has basic onboard VGA video (aspeed ast2050) which I believe is not accelerated in any means.

 

Is it just a matter of finding a basic hardware accelerated video card that is supported by both ESXi and OS X? If so, does anyone have any experience in this and can recommend an inexpensive one that would do the job?

 

Thanks in advance!

[Donk]

VMware have no drivers which run QE/CI needed for video acceleration. The only possible solution is to use VT-d to pass through an OS X supported video card. You will have to check that the motherboard, BIOS and CPU support VT-d. Note this is different form VT-x.

 

Update: It looks like it does support VT-d so could be in luck. And this thread may help you find one.

 

http://www.insanelymac.com/forum/topic/219169-guide-how-to-install-snow-leo-on-asus-z8na-d6/

[sgreszcz]

Thanks For the reply Donk and for the excellent work on the unlocker. The installation went much smoother than expected.

 

I did some more research tonight and found out that luckily (as it is an older motherboard) that it supports VT-d.

 

I've ordered a NVIDIA GeForce GT640 which should be natively supported by mountain lion, so hopefully it will work in passthrough mode. Then I'll just have to figure out how to get sound working as I don't believe I have a sound card, USB or otherwise on the motherboard.

 

I'll report back with my findings and hopefully this goes easier than trying this the bare metal installation way.

 

 

[sebus]

Please do report back, as it looks like an interesting option

[sgreszcz]

OK, here is an update of the progress that I've made so far with installing Mountain Lion on my ESXi 5.1 box with GPU passthrough for accelerated graphics.

 

I spent a lot of time looking at these sites and searching with Google to find the most inexpensive and compatible out of box video card for Mountain Lion:

 

http://www.macbreake...hics-cards.html

######

 

I originally bought a nvidia GT 640, as it looked to be quite compatible with Mountain Lion, but in the end I found out that ESXi passthrough with nvidia cards did not work too well:

 

http://networkingbun...ts/vmdirectpath

 

So I returned the nvidia and spent time researching AMD Radeon cards here:

 

####

 

I found a Radeon HD6570 for about £40 and set up GPU passthrough in ESXi (I forgot to say that I did install Mountain Lion first without the Radeon card - I don't know if that would make any difference).

 

From your host, select the configuration tab, then "Advanced Settings", then "Edit", and select the ATI card.

 

 

Then you have to reboot your ESXi host for passthrough to take effect. You then can add the GPU to the OSX guest using the "Add PCI device" option. To allow the OSX guest to more memory, you need to adjust some settings on the host:

 

 

pciHole.start = "1200" and pciHole.end = "2200"

 

(Options/Advanced/General/Configuration Parameters) - in properties of VM.

 

 

I then had some errors when starting up the guest when I boosted the memory from 2GB to 16GB:

 

 

 

Failed to start the virtual machine.

Module MemSched power on failed.

An error occurred while parsing scheduler-specific configuration parameters.

Invalid memory setting: memory reservation (sched.mem.min) should be equal to memsize(xxxxx).

 

So to fix that I just clicked on the GPU PCI card that I added and clicked the option to "make reserved memory equal" for the OSX guest.

 

So after booting up, I was able to get all 16 virtual cores running, run Final Cut which didn't run before, and see the ATI card in system info:

 

 

I am using screen sharing from my Macbook Pro (also running Mountain Lion) to control the mouse and keyboard. I still haven't found a way to disable the VMware display, so I sort of get two of them, which I don't really want.

 

 

I also need to do some experimentation to see how well video editing and audio work, but at least now I can run programs that need video acceleration.

Edited April 8, 2013 by ErmaC
PLS read the forum rules. link removed

[Donk]

Brilliant write-up. Thanks for sharing the information with us.

[sebus]

Excellent, thanks for this, really clever way to do it

 

And what CPU do you run?

 

sebus

[sgreszcz]

 

And what CPU do you run?

 

 

2 x Intel Xeon E5620 Processor

[wxe911]

@ sgreszcz

First thanks for the detailed post and of course your research to get things to work.

 

I could not find any hint as to how you got sound to work?

 

Would you care to reveal your secret on that?

 

Regards, WXE911

[sgreszcz]

@ sgreszcz

First thanks for the detailed post and of course your research to get things to work.

 

I could not find any hint as to how you got sound to work?

 

Would you care to reveal your secret on that?

 

Regards, WXE911

 

Well, sound is a work in progress, and I still don't have it sorted out yet.

 

I installed Soundflower via a screen capture app which gave me some sound (albeit choppy for some reason - maybe as it was in software?) so I started to look for hardware solutions.

 

I tried to passthrough the USB hardware and use a USB dongle for sound. The USB dongle works natively on my Macbook, without needing drivers, but there is a bug in ESXi5.1 that allows you to passthrough the USB hardware, but after rebooting the host server to enable it, the USB device disappears.

 

[seems like VMware removed USB passthrough from 5.1 for some reason when it used to work in 5.0]

 

I then tried to add the USB sound dongle as a virtual USB device. I couldn't get that to work either - The guest OSX didn't detect it.

 

I then bought the M-Audio Audiophile 2496 which is a PCI card with older, but supposedly native drivers for OSX. I couldn't get the PCI card to passthrough. I could select it, but when I rebooted the ESXi host, I couldn't get Mountain Lion to boot.

 

I have now ordered a Maya 44 XTe PCI Express Audio Interface to see if I can passthrough that better (as I was able to passthrough the Radeon PCI Express card). It is a bit pricey, but has native Mountain Lion drivers supported by the manufacturer.

 

I'll report back when I receive the card and have time to test it. I think that this will be my last attempt. If it doesn't work, I might give up on using the ESXi/OSX server as a workstation and just use it for transcoding existing video with handbrake or other CPU/memory intensive work where I don't need sound.

[sebus]

At this point one could get Gigabyte motherboard that natively does ML (maybe not dual CPU & only 16Gb or RAM, but zero other problems)

 

sebus

[wxe911]

Thank you sgreszcz for your explanation. You have invested a lot of time, not to mention money, into this sound nightmare.

Please report back again.

 

@ sebus -- yes one could indeed build as suggested. In this thread it is all about vitualisation. Thanks anyway.

 

Cheers

[wxe911]

My experience:

ESXi 5.1 on Gigabyte Z68MA-D2H-B3 current EFI Firmware. 32GB RAM, 2TB HDD. Boot ESXi of USB stick.

 

working:

Passthrough Onboard Realtek ALC889 sound chip.

 

Windows 7:

1) Sound working (headphone plugged into rear green 3.5mm jack) displayed through console.

2) Sound working (speaker attached to Windows 7 workstation) displayed through Remote Desktop Connection.

Note: 2) also works if 1) no passthrough is done in the VM.

 

Setup for Windows was painless and it seems to work quite well.

 

 

Mac OS X Mountain Lion 10.8.2:

Mountain Lion does not see any device.

 

Read a lot here:

#####

#####

 

Downladed (MultiBeast5.2.1) and (Kex_Beast). Made a copy of AppleHDA kext. Installed Alc889 driver using (MultiBeast5.2.1). Then extracted the AppleHDA and HDAEnabler kext from S/L/E folder onto desktop.

Used (Kext_Beast) to install them again, just to be safe that permissions are done correctly.

 

Now Mountain Lion does recognise the onboard alc889 sound chip as Built-in device and HD audio from the HD4000 graphics inbuilt i2600 CPU.

 

It does produce sound coming out from the rear green jack, plugged into ESXi host, but the volume is faint and a lot of crackling noise. Effectively not usable.

Was unable to find a solution to make it sound better.

 

 

Installed the iRAPP server, but was not able to get any sound out of it when connected with the iRAPP Windows client. No solution found at all.

 

 

Just wanted to share my experience with you.

 

Update:

Tried (CoRD_0.5.7), free opensource RDC client for Mac. Can connect to iRAPP server installed on Mountain Lion, but no sound.

 

Connect with Windows 7 RDC to iRAPP server and finally got sound good sound.

Edited April 8, 2013 by ErmaC
Still by-pass the forum word filter and ignore the forum rules.

[sgreszcz]

 

Just wanted to share my experience with you.

 

 

Thanks for the input, I appreciate it. I have never messed much with kext files or anything, as the only hackintosh I have running is on an old Dell Mini9 netbook.

 

So here is some more of what I've tried:

 

I bought the Maya XTE PCIe audio card, but I could not get it to pass through to Mountain Lion and it caused me to lose connection from vSphere client and lock up the whole system forcing me to hard reboot. I tested this to see if it was only a problem with OSX, but it also caused the host to lock up when I tried passing the soundcard through to a Windows XP installation. I think it might be that the card is really a PCI card with a PCI/PCIe bridge and ESXi doesn't like that. I am returning the card...

 

I experimented with a couple of "virtual soundcards" for OSX, but couldn't really get them to work or they weren't supported in a virutal environment. I installed the latest soundflower virtual sound driver, and I did get really good sound when I tested capturing a youtube video using Snapz Pro X. This will work OK for that sort of thing. I can probably work remotely with splashtop2 which does a good job at remote streaming of video and audio. I haven't tried it from the virtualized Mountain Lion install yet.

 

I forgot to mention that I did manage to pass through the ASUS Mio 888 sound card, but it wasn't detected natively in Mountain Lion and I haven't dug much into how to get it working in ML with kexts (which I'm trying to avoid). I have no idea how to get the AppleHDA.kext to work with this soundcard. Maybe I'll try again...

 

I also tested a plantronics USB soundcard virtual device by passing it through to ML. It was detected natively by OSX and I got sound, however it was choppy and crackly. Now that I've managed to get ML running on ESXi5.0 u2 (which allows USB device passthrough - ESXi 5.1 does not) I will try again some time with one of the passed-through USB controllers from my motherboard and see if the performance is better.

 

I have also managed to run this server headless with graphics acceleration on by inserting a 100 ohm resistor into the analogue video port. This forces OSX to load the accelerated drivers without a monitor connected, otherwise you won't get the benefit of the GPU.

[sebus]

How to make VGA dummy

[sgreszcz]

How to make VGA dummy

 

Subus, thanks - I did do that directly on the VGA port on my Radeon card. The Radeon card I have has three connectors - VGA, HDMI and DVI-D. I didn't realise that the DVI is "D" - digital only - so it doesn't have any analogue signalling, and you can't use a DVI-I to VGA converter and connect to that with the 3 resistors. I was able to get it working with the VGA port, but you need to have a monitor, or the resistors, in place on the VGA port from first cold bootup of the ESXi host. You will not get any signal on the VGA if you plug it in later.

 

When the Radeon is on my motherboard, it seems to disable the onboard VGA and uses the Radeon to show BIOS and ESXi boot up sequence (on VGA or DVI, haven't tried HDMI). Then part of the way through the bootup the ESXi screen freezes until you start up Mountain Lion guest on the vSphere client, and then the video passthrough takes over and you get the OSX desktop on your monitor.

 

The performance is much better when the GPU is enabled, especially when using screen sharing.

 

For now I have sound and accelerated video on a headless setup. I will try the USB passthrough for mouse/sound/keyboard and I re-purchased the ASUS MIO 888 soundcard to see if I can get it working in passthrough with kext modifications. This would give me a good workstation configuration.

[qball4]

Any luck on the MIO 888 card? I am in the same boat with an ATI 6770 passed through perfectly but sound not working. I have tried (all with optical):

 

Maya U5 USB - on devices but no actual sound, works OOB on MBP 10.8.2

C-Media USB - on devices but no actual sound, works OOB on MBP 10.8.2

PCI Passthrough On-board ALC 889 - crackling/distortion, low volume

 

I thought perhaps the USB devices weren't getting enough bandwidth if sharing with all of the other devices I have plugged in so tried passing through pcie controllers:

Startech NEC 720200 USB 3.0 controller - esxi recognizes correctly, vm doesn't get past vmware screen, won't even go to bios setup, will however boot fine into Win7 vm... EDIT: Will boot to osx if no USB devices are plugged in, will not recognize any plugged in after boot

Frisco Logic 1009 USB 3.0 controller - esxi recognizes as "Unknown device" but will pass through, OSX does not recognize at all

[qball4]

I found a solution in case anyone else is trying the same:

 

I used an NEC chipset (PD720100 or PD720101) based iogear GIC251U PCI card in passthru then connected the AudioTrak Maya U5 and voila! it works!

 

Final VM setup:

esxi 5.1 patched to 1021289 (patch necessary for usb controller passthru support)

OS X 10.8.3 using Donk's unlocker

XFX 6770 1GB (HD-677X-ZAF4) in passthru - flashed to 5770, full QE/CI no kexts needed

iogear GIC251U PCI usb 2.0 card in passthru - works OOB

AudioTrak Maya U5 usb sound card - optical out 5.1, works OOB

 

tip: remember to re-select the usb card for passthru under configuration/advanced settings after every hypervisor reboot as there is still a bug in esxi that will remove it after two reboots

 

If anyone finds a way to get usb 3.0 working under esxi/OSX please share!

[wxe911]

@sgreszcz

 

The ASUS MIO 888 sound card is based on the Realtek ALC892

 

You may want to try this kext http://www.osx86.net...&id=3336&page=1

 

An easy way to deal/install kexts is with (####) found here ####

 

As always backup your original kext before installing modded ones.

Edited April 8, 2013 by ErmaC
We have word filter and forum rules.

[sgreszcz]

Thanks for the tips. I haven't had the time to continue working on the sound and the MIO888. I'll report back when I get some more time to mess with this.

[Wenle Z.]

Is it possible to run OSX 10.8 as a VM, but passthrough the iGPU HD4000 and onboard sound to the VM, and actually hook up a monitor to the ESXi and use the VM as a desktop?   

[MSoK]

Wenle z,

 

See my response to your post in the Workstation thread, see below:

 

http://www.insanelymac.com/forum/topic/267296-esxi-5-mac-os-x-unlocker/?p=1945917

[bookemdano]

Hi all, I'd like to add to this thread that I'm having some good success with virtualized OS X Mavericks on a Lenovo Thinkserver TS140 (Haswell) running on a Xeon E3-1225 v3 Quad core processor. 

 

First of all, the just-released ESXi 5.1 U2 appears to resolve the problem with USB controllers becoming de-selected from passthrough each reboot. I believe it still has some problems with USB 3.0 passthrough though, as I'm having some glitchy issues passing through various PCIe USB 3.0 cards to it. More on that later though, since this thread is really about graphics acceleration.

 

I too have a Mac Mini (2011) I've been using and its Core2Duo and Intel 3000 graphics are just not cutting it for iMovie/FCPX. I picked up this Lenovo for $320 on amazon which I consider quite the steal. The system can take non-ECC memory too so I got it loaded with 4x4GB of non-ECC UDIMMs (the system can take up to 32GB).

 

Donk's unlocker works wonderfully. Installing OS X was the easiest part. I've had some experience with VT-D/VMDirectPath before so I know it is really hit or miss. I took a chance on a Radeon 7750 card, which was the best I was going to be able to find considering I can't use Nvidia, didn't want to waste a second slot (I only have four on this case/mobo) and the Lenovo power supply is proprietary and only 280W without a video card connection. The 7750 gets all its power via the PCIe bus, takes up only one slot and was fairly cheap so I went with that. Even more importantly it's natively supported in Mavericks without any kext shenanigans--also an important consideration for me.

 

Passing that through was a cinch. In the mobo BIOS setup I could set the integrated Intel graphics as primary so that ESXi's console still uses it. Then the Radeon along with its HDMI audio device get passed through to OS X. I haven't actually tried the HDMI audio--have no idea if that works or not.

 

My bigger problem was USB. The Lenovo has 2-front and 4-back onboard SuperSpeed USB 3.0 ports... usually with ESXi you are presented with multiple controllers so you can keep one controller (housing the USB flash drive you're booting ESXi from) dedicated to ESXi and pass through the other. I was able to do this just fine... but for some reason I'm only presented with EHCI controllers to pass through, not the xHCI one(s) I need for USB 3.0 speeds. I haven't figured that one out yet--assume it's either a chipset or ESXi limitation (the onboard SATA controller also doesn't appear in the list of devices able to be passed through).

 

So I knew I needed to try a PCIe 1x USB 3.0 card. I already had a spare Renesas one. I got that working OK with the GenericxHCI kext, but for "fun" I wanted to try to get this VM running with only the built-in drivers. Now you can easily find the Fresco FL1100 and FL1009-based cards with native support in ML and Mavericks. I first tried the Orico 4-port model (http://www.amazon.com/Compatible-ORICO-PFU3-2P-Lightning-Controller/dp/B008V3TH48) but got kernel panics when I passed it thru. I still want to try and see if it works better under ESXi 5.5 but haven't gotten around to that yet.

 

For grins then I tried the 2-port Orico which uses FL1009. I am having much more success with that. Haven't run any speed tests but it installs great and everything I have tried connecting to it so far has worked (tried a USB 3.0 Optical drive enclosure, USB 3.0 external drive (bus powered) and a Transcend USB 3.0 memory card reader). Also, I've tried a Zalman USB Audio interface I've had kicking around forever. To my surprise and delight it works fine... Haven't tried listening to music or anything but the built-in OS X sounds play great with no crackling or distortion.

 

However, I'm having this issue now where ESXi is locking hard overnight. It started after I passed through the USB 3.0 card so I'm suspicious that is the reason but haven't done enough testing yet to narrow it down... could have been something I had plugged into the card too so tonight I will try it with nothing plugged in and see if it still hangs. If so then I will try ESXI 5.5 and see if that makes a difference.

 

For fun today I got out an old Dell SAS 6/ir RAID card (can be had very cheap on ebay), flashed it to the equivalent LSI 1068e card and got it passed through as well. Mavericks picked it up and it shows up as a SAS adapter. I've got a blu-ray writer and a spare SSD hooked up to it. So if in the end I can't get USB 3.0 to be stable then I'll probably just stick with USB 2.0 off the mobo and use the LSI card to connect drives via SATA.

 

Anyway, circling back to graphics acceleration. I haven't actually tried any tests to see if it's helping. I do know that Final Cut Pro X launched OK which I don't think it would do before the Radeon was passed through. I am confused though because I tried playing an avi using VLC on the display connected to the Radeon and I got a message from VLC "OpenGL acceleration is not supported on your Mac. Your Mac lacks Quartz Extreme acceleration, which is required for video output. It will still work, but much slower and with possibly unexpected results." 

 

The video played it wasn't terribly smooth. So I don't know if this is a VLC problem where it is still trying to use the VMWare video card (even though I had the VLC window on the monitor connected to the Radeon) or if that means that FCPX isn't going to be able to use the Radeon GPU for faster rendering either. I hope that's not the case.

 

Anyway, just thought I would share my experiences.

[bookemdano]

Following up, I found out that the new Maps app in Mavericks relies on QE/CI so I was able to use that to test things out. Turns out that with any kind of mirroring between the built-in VMWare display and the Radeon QE/CI is disabled (or at least the Maps app wouldn't draw anything). Nor would it do so if the VMWare monitor was the primary (with the menu bar and dock), even if I moved the app over to the Radeon display.

 

But if I set the Radeon as primary and kept the Maps app on it, QE/CI appeared to work as the Maps app functioned correctly.

 

Now, playing videos in Quicktime or VLC seemed to chew up the CPU more than I thought it would. But I was also trying this over screen sharing which may have been the culprit there. I will try hooking up a monitor directly to the Radeon and then try a video.

 

Edit: I also solved my nightly lockup issues by going back to ESXi 5.1 U2. That seems to be the version I'm having the most success with. I've also given up on the Orico USB 3.0 cards. I'm presently using a Renesys with the GenericUSBXHCI.kext and everything seems good.