ESXi 5 Mac OS X Unlocker

[Ken73]

Just wanted to stop in and say "THANKS!" to all involved in making this work! A day of reading this thread multiple times and Googling several tricks led me to a successful install of ML on ESXi 5.1!

 

Specs on my ESXi play box:

 

White box ESXi server: Supermicro X7DBN, twin Intel E5450 Xeon 3 gHz procs. 32gb RAM, 1tb of RAID 5 storage (including the hypervisor.) Been running this setup for a good 3+ years now as my test/play box at home. This box had originally started out life as an ESXi 3.x box and I've upgraded it over the years. Going from 5.0 to 5.1, I just downloaded the ISO and burned a copy; let the installer do the work for me. (And yes, the procs are E5450's, which from my understanding is some of the first Xeon's to have the VT-X bit, which I verified was on in BIOS.)

 

Figured I would share what I went through (and hopefully give a few of you a laugh.)

 

First, I found this thread - read all 18 pages over a few days in my "spare time" - well worth reading as you pick up lots of tricks through the way.

 

Second, got enough nerve up to download it and realize others had had success. Copied the contents (the esxi folder within the download) to my datastore. OK! Great! Now how the heck do I execute this? Read somewhere in here where I should probably do chmod +x at the command line.. how do I get to a command line on this thing? (Did I mention I've never used a command line on this machine?) Googling it finally leads me to turn on SSH under the Configuration>Security Profile. Got it - now to use PuTTY to SSH into the box. Where the heck is this stuff? I've used Linux before (my first stint was on 386's back in the day) so I had an idea of how to get around, finally found the /vmfs directory and made my way to where I had copied the files. Ran the "chmod +x *" trick just to be sure.

 

So then the dunce cap went on; I typed "install" (n00b mistake) and the ESXi installer came up. YIKES! This doesn't look right.. exit out of that! Now I try install.sh - that doesn't work either, can't find it. Can't find it? It's right there! Google some more.. forgot my old Linux.. okay, so it's "./install.sh" ..... finally! Patched just fine and dropped me back at a prompt. Reboot the host. Crossed my fingers but it booted up just fine and my auto-load VM's came up as normal.

 

Did I mention I have a Mac Mini? Great! I've got the install DVD here somewhere.. a quick search reveals 10.6.2 Snow Leopard on DVD! Let's give that a shot. Make a new VM, set the CD to the host device and let's go! I get a gray Apple screen and the twirly thing at the lower part of the screen is whizzing around. Looks good so far. Screen pops up with a language setting, I choose English. Progress bar appears and BAM! "Cannot install OSX on this computer." Really? I went through all that for nothing? Google some more.. turns out, the discs you get with your Mac are hardware-specific and looking for specific hardware. They won't work. Fine. Gotta be a way to get a generic DVD, right? Apple sells Snow Leopard for $20 on a DVD.. would really rather have the latest.

 

I haven't upgraded my Mac Mini to Lion or Mountain Lion yet - wonder if I can use those? Google some more - come to find out you can't get L or ML on DVD, so you have to download it. OK, so I do a little more reading about making a disc for it and turns out, you can download it right from the App store but instead of installing it right off (cancel the install after you've spent the $20 and 10 hours downloading it) copy the installESX.dmg file out of the installer, and put it in your datastore. Got it - hope like heck this is bootable!

 

So I whacked the other VM I made, let's make a new one; 1 proc 1 core (just as a test) - 2gb RAM, 40gb drive. Using installESD.dmg as the CD image. For some reason my ESXi 5.1 only shows up to Lion (10.7) so I chose that as the OS. Power on.. boots up to the CD... looks good! I get the menu for installing, restoring, disk util.. select install! Agree, agree, agree..... select disk to install to... but no disk shown? Wrong SCSI controller? Shutdown, try another one. Boot up, no go. Shutdown, try another one. Boot up, no go. Did this three times (yea I know, n00b err) .. still no go. Read through this thread again - someone else used the standard SCSI controller? WTH am I doing wrong? Maybe you have to partition it manually (?) before it sees the disk. Change back to default SCSI controller, boot up and choose the disk utility, create a partition, then try the installer.

 

VIOLA! Installed ML 10.8.2, seems to run fine other than when I click on "launchpad" to see what that was, it froze the machine. Installed VMWare Tools from the darwin.iso in the downloaded patch file. Now I'm having to remember all my tricks that I used to setup my Mini, but that's not a bad thing.

 

So there you have it, if a long-winded old country boy on a tractor can do it, you can too. Latest version of Mountain Lion (10.8.2.) on ESXi 5.1, fresh install, no problem. Thanks again to everyone on here that made it possible!

 

p.s. - Daft, YES.

[oli-j]

Richard post - have bookmarked it and will give it a go in the new year! Thanks!

[daft]

IM Protoge,

 

ahhhhh, thanks. you gave me the clue i needed. after making several attempts to install the Mac OS 10.8.2 image onto ESXi, i was finally clued in that i could install the OS via choosing the DMG image directly and not having to convert it via hdiutil. this was implied since ESXi installer would always prompt for an ISO image and nothing else. but by choosing the drop down menu to list "all files" i could choose the DMG image and it works.

 

so here's the breakdown for those that are interested and don't want to be confused as i by having to read into each one of these posts prior to mine.

 

steps i took:

 

- install ESXi 5.1, fresh install.

- enable the SSH and ESXi shell in 'security profile' (you can easily google this to figure out how)

- copy over the unlocker 1.1.0 into the datastore. (your default is ideal) make sure not to run all this within the root directory ('/') but in some other sub directory (such as '/tmp'), otherwise you will blow away your /bin directory like someone had done since they have no understanding of unix filesystems and what the general hierarchy is like.

- run './install.sh' within the esxi subfolder of the unlocker folder.

- reboot ESXi

- create new VM, I generally choose the default settings (as ESXi will recommend).

- before finishing the creation of the VM, you'll choose to edit settings near the end. modify CD/DVD drive and choose Datastore ISO file by clicking browse. choose the pull down menu and allow it to display 'all files' instead of just ISO Image. choose your DMG image.

- make sure you clicked on 'Connect at power on'

- click 'Ok' to close, and start the VM. it'll read the DMG image for install.

- make sure you erase your partition, and the installer will create a new MacOS partition for you, exit and install.

- it'll reboot and you're good to go.

 

simple.

[Ken73]

ahhhhh, thanks. you gave me the clue i needed. after making several attempts to install the Mac OS 10.8.2 image onto ESXi, i was finally clued in that i could install the OS via choosing the DMG image directly and not having to convert it via hdiutil. this was implied since ESXi installer would always prompt for an ISO image and nothing else. but by choosing the drop down menu to list "all files" i could choose the DMG image and it works.

 

Yep, I just went full steam ahead and chose "all files" and gave it a shot. I think it was MsoK that said you could just use it straight without any conversion, so that's what I did and it worked! I was originally going to try making a ML DVD, but everything I read said it was too big to fit on a single-layer disc, and I didn't have any dual-layer discs around so I just copied the DMG straight to the datastore. No ISO conversion necessary.

 

I should have a new license for Workstation 9 after the holidays, will give it a try on there as well.

[MSoK]

 

 

Yep, I just went full steam ahead and chose "all files" and gave it a shot. I think it was MsoK that said you could just use it straight without any conversion, so that's what I did and it worked! I was originally going to try making a ML DVD, but everything I read said it was too big to fit on a single-layer disc, and I didn't have any dual-layer discs around so I just copied the DMG straight to the datastore. No ISO conversion necessary.

 

I should have a new license for Workstation 9 after the holidays, will give it a try on there as well.

Ken and Daft,

Thanks for sharing your experiences installing ML on ESXi 5.1, welcome to the club, LoL! Just a heads up, the DMG you have of ML will not work with Workstation 9 or Player 5 mounting the .DMG, (ONLY Fusion 5 and ESXi 5.1 currently support mounting a .DMG), they still need to be in an ISO image format. So you can use hdiutil convert InstallESD.dmg -format UDTO -o InstallESD.iso on your MiniMac or your virtual instance of OS X, although this creates a .CDR file this can be renamed to .ISO, alternatively in Windows use MagicISO or ImgBurn to convert the .DMG file. The ISO image you have created will also work with ESXi 5.1 and unlike the DMG image will also work with ESXi 5.0 and 5.0u1.

[b419kid]

I found about the VMware Unlocker elsewhere when trying to run Mountain Lion in a vm. Worked perfectly. I wonder though: Do system updates work?

[MSoK]

I found about the VMware Unlocker elsewhere when trying to run Mountain Lion in a vm. Worked perfectly. I wonder though: Do system updates work?

What do you mean by system updates, if you mean OS X, i.e. 10.8.2 to 10.8.3, then yes, but better to download the combo update and install rather than doing a software update. If you mean VMware then you always need to re-apply the "Unlocker" following an update or patch.

[oli-j]

Followed all the above steps but I keep getting the message: "The CPU has been disabled by the guest operating system. Power off or reset the virtual machine."

 

I'm running ESXi 5.1 + unlocker on an HP Proliant Microserver. It's an AMD Turion II Neo N40L cpu.

 

Starting to think that this isn't going to be possible...

[Donk]

OS X does not run natively on AMD processors. There is work on this forum on a new kernel for AMD which may help. http://www.insanelymac.com/forum/topic/281450-thoughts-about-mountain-lion-with-amd-yes-amd/

[pip11]

The vmware.log files for the machines themselves weren't written to by the power on attempts, so they won't be much use, that snippet came from hostd.log...I'll send it your way shortly.

 

I also just saw this in the kernel logs:

 

2012-11-06T04:30:40.418Z cpu0:6003)VmkAccess: 637: hostd-vix-poll: access denied:: dom:appDom(2), obj:vmxSys(66), mode:syscall_allow(2)
2012-11-06T04:37:19.983Z cpu0:6003)VmkAccess: 637: hostd-vix-poll: access denied:: dom:appDom(2), obj:vmxSys(66), mode:syscall_allow(2)
2012-11-06T05:21:12.758Z cpu5:6003)VmkAccess: 637: hostd-vix-poll: access denied:: dom:appDom(2), obj:vmxSys(66), mode:syscall_allow(2)
2012-11-06T05:36:40.141Z cpu5:1037388)VmkAccess: 637: vmx: access denied:: dom:appDom(2), obj:vmxSys(66), mode:syscall_allow(2)
2012-11-06T05:36:46.002Z cpu3:1041489)VmkAccess: 637: vmx: access denied:: dom:appDom(2), obj:vmxSys(66), mode:syscall_allow(2)

 

Yes, it is a cluster -- though that host just got kicked out of it. We upgraded, re-applied unlocker, unapplied unlocker, and re-applied unlocker.

Thanks for the swift response!

 

I just updated my ESXi host to 5.1.0b (build 914609, released 12/20/2012) and I'm having the same problem. My host is a single host, no cluster or vCenter. With the unlocker applied, the system works fine for the first couple of VM power-ups, but then power-ups fail.

 

Once power-ups start failing, if I run /bin/vmx from the command line:

/var/log # /bin/vmx
Error during version check: Failed to get vmkernel version: 0xffffffff

 

And these lines appear in vmkernel.log every time I try to run /bin/vmx

2013-01-04T19:09:35.149Z cpu0:88884)VmkAccess: 637: vmx: access denied:: dom:appDom(2), obj:vmxSys(66), mode:syscall_allow(2)
2013-01-04T19:09:35.149Z cpu0:88884)VmkAccess: 922: VMkernel syscall invalid (16384)

 

Uninstalling the unlocker fixes the problem. I've tried installing/uninstalling the unlocker twice since updating to 5.1.0b.

[Ken73]

Got my Workstation 9 license; applied the patch, but I'm getting "Can't download the additional components needed to install OS X" .. I created the ISO on my virtual Mac with no (apparent) issues and it does boot up at least. Any ideas? What little I can garner from Google is it's a "bad disc" (image.)

[mattpayerle]

Anyone getting success booting workstation 9 with the DMG? Ken73, I'm not getting that error, when I boot it says unsuccessful boot, then tries to run the EFI Shell. I'll try making the ISO and trying again.

[Ken73]

OK, cured my own problem. It was apparently a bad "disc" - I recopied the InstallESD.dmg out of the install package and re-ran the hdiutil to make the ISO, but this time on my Mac Mini - and it worked. I've got it running under Workstation 9 just fine!

[Ken73]

Anyone getting success booting workstation 9 with the DMG? Ken73, I'm not getting that error, when I boot it says unsuccessful boot, then tries to run the EFI Shell. I'll try making the ISO and trying again.

 

You can't boot from the DMG, has to be converted to an ISO. You can do it easily within an already functioning Mac, or there are options to do it in Windows (I didn't have success doing it in Windows myself.)

[mattpayerle]

Yeah, I got it with the ISO

[orinoco253]

After struggling to get osx running on 4.1 I have upgraded to 5.1 applied the patch 1.10 from ssh ./install.sh rebooted mounted installesd.dmg when I start machine I get a general error the vm could not be started please can someone point me in the right direction 7 days of 4.1 and now day on 5.1 is not the best fro my sanity

[MSoK]

After struggling to get osx running on 4.1 I have upgraded to 5.1 applied the patch 1.10 from ssh ./install.sh rebooted mounted installesd.dmg when I start machine I get a general error the vm could not be started please can someone point me in the right direction 7 days of 4.1 and now day on 5.1 is not the best fro my sanity

OS X under ESXi 5.1 should work fine, so if you are having a problem, we will need a bit more detail, so what is your PC / Server spec (i.e. CPU etc.), which version of OS X are you wanting to install, is it a retail version (i.e. downloaded from the App Store), a txt copy of your .vmx config file. Finally has the "Unlocker" applied correctly, i.e. can you select OS X from the New Virtual Machine Wizard?

 

With more information hopefully we can help.

[orinoco253]

Thanks for reply MSOK, managed to get OSX 10.8.2 installed on ESXi 5.1 (799733) but having problems in that I am unable to start other VMS on the host I get a general error unable to start VM this is for all other vms

 

PC is a Gigabyte P67A U3 B3 32gb i7 2600k

[MSoK]

Thanks for reply MSOK, managed to get OSX 10.8.2 installed on ESXi 5.1 (799733) but having problems in that I am unable to start other VMS on the host I get a general error unable to start VM this is for all other vms

 

PC is a Gigabyte P67A U3 B3 32gb i7 2600k

Thanks for the additional information, so just to confirm your OS X 10.8.2 is up and running, but no other VM's work, i.e. Windows, Linux, etc. Was the OS X 10.8.2 a fresh install on your current hardware and ESXi 5.1 with "Unlocker" configuration? Are the non working VM's fresh installs on your current hardware / ESXi 5.1 or were they created previously. Have you tried a fresh install of a non working O/S type?

[orinoco253]

The VM's were all existing VM's, 10.8.2 was a fresh install, I will try a new install of a non working OS, the non working currently are a mixed bag, Ubuntu 12.04 Windows server 2008, Xp, Windows 7, Openindiana

[pip11]

Thanks for reply MSOK, managed to get OSX 10.8.2 installed on ESXi 5.1 (799733) but having problems in that I am unable to start other VMS on the host I get a general error unable to start VM this is for all other vms

 

PC is a Gigabyte P67A U3 B3 32gb i7 2600k

 

I believe I'm having the same problem on ESXi 5.1, there's more details in my last post. With the unlocker installed, after boot everything works correctly (I can start VMs running any OS), but after some amount of time/activity, VMs will not start. (VMs that are already running stay running however)

 

When I reboot ESXi, the cycle repeats: works fine for a time, then no VMs will start.

 

jimg also PM'd me and is having the same problem, he said that SSHing into ESXi and running '/sbin/services.sh restart' will temporarily allow VMs to start without having to reboot ESXi, but it's only temporary.

 

Donk, any ideas?

[sebus]

Thanks for reply MSOK, managed to get OSX 10.8.2 installed on ESXi 5.1 (799733) but having problems in that I am unable to start other VMS on the host I get a general error unable to start VM this is for all other vms

 

 

You are not running a cluster of ESXi with vCenter?

[orinoco253]

I am running a couple of identical ESXi servers with vcenter

[MSoK]

The VM's were all existing VM's, 10.8.2 was a fresh install, I will try a new install of a non working OS, the non working currently are a mixed bag, Ubuntu 12.04 Windows server 2008, Xp, Windows 7, Openindiana

OK, it is possible the new CPU is causing the problem for OS's built on different hardware, especially if a new build works fine. You could try "dumbing" down your new i7 2600K at the VM level by adding the following CPUID mask in to your .vmx configuration file.

 

cpuid.1.eax = "0000:0000:0000:0001:0000:0110:1010:0101"

 

see the original thread, link below:

 

http://www.insanelymac.com/forum/topic/232183-snow-leopard-on-vmware-and-disabled-cpu-error/#entry1551966

 

I am running a couple of identical ESXi servers with vcenter

Just as a reminder, vCenter does not support OS X or the "Unlocker", so to create and access OS X VM's go directly to your ESXi host in your vSphere client rather than via your vCenter Server.

[Donk]

Just as a reminder, vCenter does not support OS X or the "Unlocker", so to create and access OS X VM's go directly to your ESXi host in your vSphere client rather than via your vCenter Server.

 

I think vCenter has gotten smarter and is actively disconnecting ESXi hosts where a core file has been altered. As MSoK says the unlocker is only for standalone ESXi hosts, and I do no test it with vCenter as it has never worked with it.