Unsynchronised TSC after wake/reboot after wake (not Haswell-E)

[computergek80]

Hello everyone, 

 

I'm posting a new thread on behalf of myself and a few other members here (konrad11901, blackswords and Dmitry R). We've been experiencing an unstable system after wake.

 

My system is ab Asrock Z97M OC Formula with an i5-4670k, konrad11901 has an MSI B85-G43 Gaming with an i5-4460, blackswords has an MSI PC Mate Z97 with an i5-4690k and Dimitry R has an Asrock Z170M-ITX ac with an i3-6100. 

 

I recently got my Z97M OC Formula to replace a GA-Z87MX-D3H which suffered from corrupted memory after long sleep. I'm booting using Clover, with a patched DSDT, and an SSDT generated with PikerAlpha's ssdtPRGen.sh. I've attached the files of my configuration. 

 

Soon after wake, the log shows messages such as: 

2016-03-29 11:23:21,000 kernel[0]: Unsynchronized  TSC for cpu 1: 0x0000000193273184, delta 0x18e0a1aca
2016-03-29 11:23:21,000 kernel[0]: Unsynchronized  TSC for cpu 2: 0x0000000193428ced, delta 0x18e0a1aed
2016-03-29 11:23:21,000 kernel[0]: Unsynchronized  TSC for cpu 3: 0x00000001935de52f, delta 0x18e0a1aed

followed by process crashes, usually culminating in a launchd crash, causing a kernel panic and a reboot: 

*** Panic Report ***panic(cpu 1 caller 0xffffff800cb9aaa4): launchd exited (signal 8, exit status 0 )
…
Thread 3 crashed
…

We have noticed that the output of pmset -g log | grep -i failure points to various PCIe devices, most notably the graphics card. Here's my output at the moment: 

2016-03-29 13:11:50 +0200 Failure              Drivers Failure panic during wake due to PEG0(NVDA NVDA NVDA NVDA NVDA NVDA),IGPU():
2016-03-29 22:45:40 +0200 Failure              Drivers Failure panic during wake due to PEG0(NVDA NVDA NVDA NVDA NVDA NVDA),IGPU(): 

With VoodooTSCSync.kext, we've noticed launchd crashes less often, meaning waking from sleep works, apart from the other processes that do crash. 

 

My theory is that there's something wrong with the firmware of our boards, or an issue with the DSDT or with CPU power management. This underlying problem is causing a TSC sync issue, which causes problems with PCIe devices as well as processes. 

 

Does anyone know the cause? Are there more people with this issue? And can anyone explain the relationship between the TSC, the HPET and the RTC? 

 

Thanks in advance to anyone who can shed any light on this issue. 

 

Kind regards, 

 

Pieter

 

Edit: I forgot to mention that konrad11901 discovered that when using hibernation (hibernatemode=25) in stead of sleep there are no crashes, but of course, this isn't an ideal solution. 

Z97M OC Formula.zip

Edited March 29, 2016 by computergek80

[Dmitry R]

In my case I don't even see this output of pmset -g log | grep -i failure It's just empty. And changing hibernatemode from 0 to 25 doesn't help also. When it's 25 the system doesn't wake after Clover starts it. It just hangs up with a black screen until you press the reset button. Though I see the same Unsynchronized  TSC messages in my log file. VoodooTSCSync.kext doesn't remove those Unsynchronized TSC messages from a log.

[computergek80]

Ah, so there is a difference between the Haswell and Skylake systems. 

 

But yes, VoodooTSCSync.kext doesn't eliminate the log messages, it just tries to synchronise the cores when they occur. 

[computergek80]

Great news (for me at least)! I've managed to get rid of the kernel panics that were seemingly the result of the TSCSync issue. 

 

Ever since I got my new board, I've been using an Apple BCM94360CD with a PCIe 1x adapter. But the Asrock OC Formula wasn't detecting the PCIe device. But the Bluetooth was working using the USB header connection. 

 

I had found out that some motherboards get the wiring wrong for combined USB and PCIe devices and the stated solution would be to get an adapter with all four USB lines connected to the motherboard header. So, to get WiFi to work (I wasn't expecting our problem to be related), I was looking at a really expensive adapter that is only sold with the card, the wdxxfu version. And I was looking at the most expensive BCM943602CDP option as well. The purchasing agent somehow wouldn't place the order, so I had to wait for my money to get back to me. In the meantime I decided to just take the €12 risk and I bought this adapter from aliexpress.com. 

 

The cheap adapter arrived in record time, and it seems to have solved the problem. After noticing that there were no more crashes (not only launchd, but the other processes that would crash were no longer doing so), I changed some other things. I went back to letting Clover generate the P and C states, and I set a darkwake=9 bootflag. 

 

I hope this helps everyone else with this problem. I'd be interested to know wether you too have a PCIe device that is plugged in, but isn't detected by the motherboard. 

[konrad11901]

Congrats on finding the solution! Unluckily, I don't have any PCIe devices expect the graphics card So in my case it's not the problem.

[computergek80]

That's disappointing. Maybe it's one of the built-in PCIe devices in the board that is misbehaving, like the NIC or a USB controller. I suppose you could try disabling certain things in the BIOS.

 

I'm still getting TSCSync messages, and I haven't tried removing the Voodoo kext, so not everything is how I'd like it. But at least it's stable.

[thorton]

I had the same problem with my MSI P67 based motherboard. Found some useful information in this thread.

​

​I patched my bios to fix the issue.

[astroguy]

Hey folks, new guy here, new build, everything working lovely except same problem: when the screen goes black (display sleep or display off), computer likes to shut down that registers as a crash, and my error codes look much like the first post.  I'm also using an ASRock OC Formula board and Skylake 6700K CPU, I have an NVIDIA 960 in my primary PCIe slot.

 

I'm incredibly new to hackintoshing (can it be a verb?) and I'm interested in the "quick fix" of hibernatemode=25 ... um, how do I do that and where?

Okay, just did one thing that appears to have helped -- System Preferences > Energy Saver > Prevent computer from sleeping automatically when the display is off.  That may actually be a mid-term solution for me since this is going to be a computer that will likely always be on, but knowing about hibernate would be good ...

[maleorderbride]

I had the same problem with my MSI P67 based motherboard. Found some useful information in this thread.

​

​I patched my bios to fix the issue.

 

Thanks thoron! My ASUS X99 Deluxe and X99-A both have unsynchronized TSC after wake, but in my case at least VoodooTSCSync solves it 100% reliably.

 

I'll patch the BIOS though. Always in favor of one less kext

 

I have seem this off and on with different boards over the years, but in the X79 cases at least eventually more mature BIOS releases solved the problem themselves without any end-user patching. That said, I would doubt a Z97 board is going to get too many more BIOS updates, if any =/

 

 

Edit: I only see a single 0f 30 in the S3Restore moduel for my ASUS X99-A USB 3.1--does that seem right thorton?

[thorton]

Well, you can't be sure. 0F 30 is the opcode for writing to Model Specific Register. Whether it is writing to the Time Stamp Counter depends on the value in the ECX register - 0x10 for TSC. If you are familiar with assembly then you can verify this by looking at the disassembly. If not, upload your S3Restore, I don't mind taking a look at it.

[maleorderbride]

Well, you can't be sure. 0F 30 is the opcode for writing to Model Specific Register. Whether it is writing to the Time Stamp Counter depends on the value in the ECX register - 0x10 for TSC. If you are familiar with assembly then you can verify this by looking at the disassembly. If not, upload your S3Restore, I don't mind taking a look at it.

Thank you!

 

Here is a link to my BIOS as well as the extracted S3Restore: https://dl.dropboxusercontent.com/u/6316213/ASUS%20X99-A%20USB%203.1.zip

[thorton]

It seems that the offending code isn't in the S3Resume module. The one you found wasn't writing to TSC. I looked in several modules but couldn't find it. There might be some other issue at play here.

[maleorderbride]

It seems that the offending code isn't in the S3Resume module. The one you found wasn't writing to TSC. I looked in several modules but couldn't find it. There might be some other issue at play here.

 

OK, thanks for looking! I'll see what else I can track down. In the meantime, VoodooTSCSync does work for me.

[konrad11901]

Umm... can anyone give some specific instructions how to patch this S3Resume module? Or - if it's not a big problem - can anyone patch my motherboard's BIOS? Here's a link to it: 7816vC8.zip. Thanks in advance for any help!

[thorton]

I patched your BIOS. You might want to test it with the BIOS boot function if your motherboard has it. No reason it won't work but use at your own risk. Download - http://www.mediafire.com/download/3r6r322y82471cw/E7816IMS.C80_patched.zip

In case you want to patch it for your self
Open up the BIOS in UEFITool. Look for the S3Restore PEI module.
Expand it and select PE32 image section. Right click and extract body.

The next part is the tricky part, or not if you know some basic reverse engineering.
You can use any disassembler that can handle PE files, but I have Hopper Disassembler so I'll write about that. The trial is good enough for our purposes. Open up s3restore.bin in Hopper. It'll show us the disassembled code.
Now we need to find the instruction wrmsr.
wrmsr stands for Write to Model Specific Register. Which model specific register? That's specified by the value in the ECX register. For TSC aka Timestamp counter the value in the ECX register should be 0x10(or 16 when converted to decimal).

The reason for the Unsynchronised TSC error is that the BIOS writes to the TSC register of the first core after waking up from sleep, leaving other cores unsynchronised and OS X apparently does not like this. VoodooTSCSync works around this by writing the same value to TSC for all the cores. But it seems we can work around this by disabling the BIOS from writing to the TSC register.

So let's find the wrmsr instruction to disable.
Press CMD+F to Find. Enter wrmsr and make sure the Type is set to "String in Assembly". Now on the first match, we can see a few instructions above there's a mov ecx, 0xC0000080. This means the value in ECX is not 0x10. So we are not interested in this.
The next result, we see clearly that there's a mov ecx, 0x10 instruction about 7 instructions above. And the none of the proceeding instructions change the value of ECX. This is the one. We need to disable this.
Select the wrmsr instruction. In the status bar, you can see the file offset for this instruction in hex. In your case it's 0x29a(or 666 when converted to decimal). The specifics might be different for a different BIOS file.
Now we need to open up this file in a hex editor and change this instruction. The trial for hopper won't let us do this so I'm using Hex Fiend.
Open up s3restore.bin in a hex editor. Jump to offset 666. The bytes at the location would be 0F30, which is the opcode for wrmsr. To disable it, select and replace it by 9090. 90 stands for NOP(No Operation). Since wrmsr opcode is two bytes long we need two NOPs. Save the file and exit.

Now we just need to add this s3restore.bin back to the BIOS. Inside UEFITool, right click on the PE32 image section under S3Restore. Select replace body and replace it with the patched file. Save the image and that should be it.

[darthsian]

Hello thorton, i can't find S3Restore PEI module, in my BIOS theres only S3Resume PEI module... do you know, if it have same function as S3Restore module? Thanks.

This is link to my BIOS - ftp://europe.asrock.com/BIOS/1151/Z170%20Extreme4(3.20)ROM.zip

[thorton]

Yes, I think it serves the same purpose.

[thorton]

I patched your BIOS darthsian. Download - http://www.mediafire.com/download/k4qq4pkoa615635/Z17EX43.20.Patched.zip

Use at your own risk of course.

[konrad11901]

I patched your BIOS. You might want to test it with the BIOS boot function if your motherboard has it. No reason it won't work but use at your own risk. Download - http://www.mediafire.com/download/3r6r322y82471cw/E7816IMS.C80_patched.zip

 

In case you want to patch it for your self

Open up the BIOS in UEFITool. Look for the S3Restore PEI module.

Expand it and select PE32 image section. Right click and extract body.

 

The next part is the tricky part, or not if you know some basic reverse engineering.

You can use any disassembler that can handle PE files, but I have Hopper Disassembler so I'll write about that. The trial is good enough for our purposes. Open up s3restore.bin in Hopper. It'll show us the disassembled code.

Now we need to find the instruction wrmsr.

wrmsr stands for Write to Model Specific Register. Which model specific register? That's specified by the value in the ECX register. For TSC aka Timestamp counter the value in the ECX register should be 0x10(or 16 when converted to decimal).

 

The reason for the Unsynchronised TSC error is that the BIOS writes to the TSC register of the first core after waking up from sleep, leaving other cores unsynchronised and OS X apparently does not like this. VoodooTSCSync works around this by writing the same value to TSC for all the cores. But it seems we can work around this by disabling the BIOS from writing to the TSC register.

 

So let's find the wrmsr instruction to disable.

Press CMD+F to Find. Enter wrmsr and make sure the Type is set to "String in Assembly". Now on the first match, we can see a few instructions above there's a mov ecx, 0xC0000080. This means the value in ECX is not 0x10. So we are not interested in this.

The next result, we see clearly that there's a mov ecx, 0x10 instruction about 7 instructions above. And the none of the proceeding instructions change the value of ECX. This is the one. We need to disable this.

Select the wrmsr instruction. In the status bar, you can see the file offset for this instruction in hex. In your case it's 0x29a(or 666 when converted to decimal). The specifics might be different for a different BIOS file.

Now we need to open up this file in a hex editor and change this instruction. The trial for hopper won't let us do this so I'm using Hex Fiend.

Open up s3restore.bin in a hex editor. Jump to offset 666. The bytes at the location would be 0F30, which is the opcode for wrmsr. To disable it, select and replace it by 9090. 90 stands for NOP(No Operation). Since wrmsr opcode is two bytes long we need two NOPs. Save the file and exit.

 

Now we just need to add this s3restore.bin back to the BIOS. Inside UEFITool, right click on the PE32 image section under S3Restore. Select replace body and replace it with the patched file. Save the image and that should be it.

Wow, thank you very much for the instructions and patched BIOS! I'll edit this post when I'll test it.

 

EDIT: Wake from sleep now works without any problems! Thank you again thorton!

[darthsian]

I patched your BIOS darthsian. Download - http://www.mediafire.com/download/k4qq4pkoa615635/Z17EX43.20.Patched.zip

Use at your own risk of course.

Thank you very much, thorton. I edit BIOS following your instruction, so i can compare it with BIOS you patched if i do it right way. I'll test it yesterday and post here my results.

 

EDIT: So, it unfortunately does not work... if i use patched BIOS, computer won't POST. Black screen, fans on full load. It looks like Asrocks Z170 platform needs to patch it differently. Do you have any other ideas thorton, please?

[thorton]

It should POST by all means. I only changed two bytes. At worst it might not wake from sleep.

​

​I opened the edited BIOS with UEFITool and extracted out the patched s3resume module. But it did not match the patched module I'd inserted. The two files had many differences. So it seems UEFITool did something to it. I think this could be related to some incompatibility with UEFI tool and/or ASRock's uefi format.

​

Could try manually patching just those two bytes in the your UEFI file but that might mess up the signatures/hashes. My MSI motherboard doesn't care about that but I don't know if it will work on yours.

[darthsian]

It should POST by all means. I only changed two bytes. At worst it might not wake from sleep.

​

​I opened the edited BIOS with UEFITool and extracted out the patched s3resume module. But it did not match the patched module I'd inserted. The two files had many differences. So it seems UEFITool did something to it. I think this could be related to some incompatibility with UEFI tool and/or ASRock's uefi format.

​

Could try manually patching just those two bytes in the your UEFI file but that might mess up the signatures/hashes. My MSI motherboard doesn't care about that but I don't know if it will work on yours.

I manually patched UEFI file with Hex Fiend and wake from sleep works now! No more "Unsynchronised TSC ". Thanks again thorton 

[computergek80]

Hello everyone. 

 

I'm glad a few of you managed to solve your problems!

I thought my system was stable, but it doesn't seem to be after all. It still crashes randomly (usually not long after sleep), just much less often. But it's still annoying when it does. 

So I just examined my BIOS following your guide, thorton, but all I found was "mov ecx, 0xc0000080". No 0x10. There were a few other modules that seemed to have something to do with S3, but none of them contained a wrmsr call. 

What else can I try? Is my issue unrelated to all of yours? 

Kind regards, 

 

Pieter

 

Edit: Sorry, it's thorton's guide!

[darthsian]

Hello everyone. 

 

I'm glad a few of you managed to solve your problems!

I thought my system was stable, but it doesn't seem to be after all. It still crashes randomly (usually not long after sleep), just much less often. But it's still annoying when it does. 

So I just examined my BIOS following your guide, darthsian, but all I found was "mov ecx, 0xc0000080". No 0x10. There were a few other modules that seemed to have something to do with S3, but none of them contained a wrmsr call. 

What else can I try? Is my issue unrelated to all of yours? 

Kind regards, 

 

Pieter

It is not my guide, but thortons... i just following his lead 

I build 2 systems with Asrock Z97M OC Formula before, and im using VoodooTSCSync.kext on it and its stable. Without VoodooTSCSync.kext it restart right after waking from sleep.

But VoodooTSCSync.kext don't work on Asrock Z170 so i look for another solution. Found thortons guide to patch BIOS and its perfect solution.

I think on Asrock Z97 it will be possible too.

[computergek80]

Hi darthsian, 

 

Oh, I'm sorry, I got the user names mixed up.

That's weird. It's stable for you with VoodooTSCSync.kext? Or are the systems rebooting frequently? 

My problem is that I can't patch it, because the issue doesn't seem to be present. I must be dealing with a different problem. 

Kind regards, 

 

Pieter