TASK FORCE - [AMD] to missing SSE4.1 Instruction set - older CPUs on macOS Sierra (10.12.x)

[Shaneee]

@Spakk you've most likely tried it but have you tried using the Corecrypto from El Cap with that kernel?

[spakk]

@Spakk you've most likely tried it but have you tried using the Corecrypto from El Cap with that kernel?

No,  have not thought of it or tried. I will test it later. Thx Shane

[jsl]

Any one an idea??

Edit:

Just build a new one, is there someone who can test the kernel?

here the new

Thanks for this new testing kernel which got KP at 10.12.5 & 10.12.6 in FX-6300 hackintosh.

[jsl]

Any one an idea??

Edit:

Just build a new one, is there someone who can test the kernel?

here the new

It got KP at 10.12.5 too in Phenom II x6 1405T hackintosh.

[ITzTravelInTime]

i was thinking if you are using bridging to translate instructions in your test kernels, because the kernel must translate any sse4,1 command into instructions for older instructions sets, and also the kernel will check for the cpu having that instructions and matching the requirements (also cpuid and other stuff) so it's important to mod those checks and also would't be a bad idea to use the boot loader to disable all the system checks, and also a crazy idea: have you tried trowing some code from el capitan kernel into sierra kernel? as replacement of the new code that doesn't do the job on sierra

[Shaneee]

i was thinking if you are using bridging to translate instructions in your test kernels, because the kernel must translate any sse4,1 command into instructions for older instructions sets, and also the kernel will check for the cpu having that instructions and matching the requirements (also cpuid and other stuff) so it's important to mod those checks and also would't be a bad idea to use the boot loader to disable all the system checks, and also a crazy idea: have you tried trowing some code from el capitan kernel into sierra kernel? as replacement of the new code that doesn't do the job on sierra

 

Yeah tried some El Cap code in previous tests. The OPEMU has some support for SSE4.2 added by AnV but SSE4.1 is needed I think minimum. If you've got any ideas see my minimal source here, https://github.com/Shaneee92/SierraAMD-XNU-Minimal

 

Only the minimal edits to run on AMD CPU. Works well on my FX but as I said I think the OPEMU needs more added but not entirely sure how.

[spakk]

i was thinking if you are using bridging to translate instructions in your test kernels, because the kernel must translate any sse4,1 command into instructions for older instructions sets, and also the kernel will check for the cpu having that instructions and matching the requirements (also cpuid and other stuff) so it's important to mod those checks and also would't be a bad idea to use the boot loader to disable all the system checks, and also a crazy idea: have you tried trowing some code from el capitan kernel into sierra kernel? as replacement of the new code that doesn't do the job on sierra

 

Yeah tried some El Cap code in previous tests. The OPEMU has some support for SSE4.2 added by AnV but SSE4.1 is needed I think minimum. If you've got any ideas see my minimal source here, https://github.com/Shaneee92/SierraAMD-XNU-Minimal

 

Only the minimal edits to run on AMD CPU. Works well on my FX but as I said I think the OPEMU needs more added but not entirely sure how.

I am following this for some time, but I do not know how to do this :

 

 

 

https://software.intel.com/en-us/articles/intel-software-development-emulator

https://software.intel.com/en-us/articles/intel-software-development-emulator-release-notes

https://software.intel.com/en-us/isa-extensions

 

https://software.intel.com/en-us/articles/pre-release-license-agreement-for-intel-software-development-emulator-accept-end-user-license-agreement-and-download

 

( sde-external-8.9.0-2017-08-06-mac.tar.bz2 )

 

 

 

 

Very sad that Andy (AnV) is for over a year not online (I hope he is healthful) he could surely tell us with certainty whether that is possible or not

 

sde-external-8.9.0-2017-08-06-mac.tar.bz2.zip

[ITzTravelInTime]

but is there on the internet any open source sse4,1 emulator?

[spakk]

As far as I know, there is only the SSEPlus Project , look here: http://sseplus.sourceforge.net/
as well as the Udis86 project, look here: http://udis86.sourceforge.net/
on the one hand there are the modifications by Sinetek, Andy and Bronya
see the contents of the /osfmk/OPEMU folder, (modifying the Udis project)
and SSEPLus project, see the contents of the /osfmk/OPEMU folder, as well as the EXTERNAL_HEADERS folder

 

Note that the SSE4.1 emulation works on Phenom without SSE4.1 on windows os: https://www.google.de/search?q=sde-external+sse4.1+phenom+II+windows&ie=utf-8&oe=utf-8&client=firefox-b&gfe_rd=cr&dcr=0&ei=04GpWenlFKiV8QeFgLGoDQ

[ITzTravelInTime]

i have seen that files for sse4,1 are missing here

[ITzTravelInTime]

but how back, in time kernels with sse3 emulation implemented that? is it possible to use something similar to emulate sse4.1?

[ydeng]

As far as I know, there is only the SSEPlus Project , look here: http://sseplus.sourceforge.net/

as well as the Udis86 project, look here: http://udis86.sourceforge.net/

on the one hand there are the modifications by Sinetek, Andy and Bronya

see the contents of the /osfmk/OPEMU folder, (modifying the Udis project)

and SSEPLus project, see the contents of the /osfmk/OPEMU folder, as well as the EXTERNAL_HEADERS folder

 

Note that the SSE4.1 emulation works on Phenom without SSE4.1 on windows os: https://www.google.de/search?q=sde-external+sse4.1+phenom+II+windows&ie=utf-8&oe=utf-8&client=firefox-b&gfe_rd=cr&dcr=0&ei=04GpWenlFKiV8QeFgLGoDQ

None of the SSE4.1 instructions are implemented.  There are only 5 SSE4.2 instructions here https://github.com/Shaneee92/SierraAMD-XNU/blob/master/osfmk/OPEMU/ssse3.c#L119.  This is not going to work.  Each of these is emulated with a function which takes ssse3 as argument.

 

If you add error check here

https://github.com/Shaneee92/SierraAMD-XNU/blob/master/osfmk/i386/trap.c#L806

you should be able to see that emulation fais, returns nonzero.

 

Here are all the instructions that need to be implemented: https://software.intel.com/sites/landingpage/IntrinsicsGuide/#expand=1244&techs=SSE4_1 .

[spakk]

now hang at this point with last kernel

[ydeng]

no luck hes rite same as above ,just a few usb  messages sneaked in between the panic output 

 

You hit pextrq twice which is sse4.1.  These extract instructions are different because the destination is a regular register or memory.  The existing opemu code expect xmm registers as destination.  That's why fisttp emulation uses x87 instructions to modify memory location directly.  sse4.2 emulation writes to state64 to write registers.  Anyone knows x86 flags are restored when trap handler returns?  ptest instructions only changes the flags.  I could get the correct behavior by pushf/popf instuctions but if the flags get restored somewhere else.  These will have not effect.  Anyway to test opemu without working AMD Phenom system?  I have most of the emulation written.  Need to test against intrinsics first.  For pextr and ptest instructions, need to actually put this opemu.  Maybe I need to write a proper integration test.

now hang at this point with last kernel

It is futile without proper sse4.1 emulation.  SSE4.1 is turned on in Xcode by default.  User programs will have sse4.1 instructions.

You can verify this by the following command.

 

 

 

gcc -dM -E - < /dev/null | grep SSE

#define __SSE2_MATH__ 1

#define __SSE2__ 1

#define __SSE3__ 1

#define __SSE4_1__ 1

#define __SSE_MATH__ 1

#define __SSE__ 1

#define __SSSE3__ 1

[ydeng]

You hit pextrq twice which is sse4.1.  These extract instructions are different because the destination is a regular register or memory.  The existing opemu code expect xmm registers as destination.  That's why fisttp emulation uses x87 instructions to modify memory location directly.  sse4.2 emulation writes to state64 to write registers.  Anyone knows x86 flags are restored when trap handler returns?  ptest instructions only changes the flags.  I could get the correct behavior by pushf/popf instuctions but if the flags get restored somewhere else.  These will have not effect.  Anyway to test opemu without working AMD Phenom system?  I have most of the emulation written.  Need to test against intrinsics first.  For pextr and ptest instructions, need to actually put this opemu.  Maybe I need to write a proper integration test.

 

rflags are restored here https://github.com/Shaneee92/SierraAMD-XNU/blob/master/osfmk/x86_64/idt64.s#L539 from  x86_saved_state_t .

[spakk]

rflags are restored here https://github.com/Shaneee92/SierraAMD-XNU/blob/master/osfmk/x86_64/idt64.s#L539 from  x86_saved_state_t .

Thanks for the hint, I will build a new one and test whether it works

[ydeng]

Thanks for the hint, I will build a new one and test whether it works

This is only useful for committing emulation results of pextr and ptest.

I have coded all sse4.1 emulations.  Now I need to test them and integrate with opemu.

[spakk]

At the moment, I am stuck at this point.

[spakk]

Second pic.

the result in connection with corecrypto from 10.11.6

[ydeng]

Second pic.

the result in connection with corecrypto from 10.11.6

I took a look at corecrypto kext.  Lots of avx2 instructions.  Your processor supports them?

[spakk]

No,  have an AMD CPU  Phenom II X6 1100. This has only these CPU features: SSE4a, 3DNow !, SSE2, SSE, MMX, SSE3

 

Edit:

apologize ydeng, what are the exact features in the panic message, which shows the avx2 emulation or that the avx2 was emulated.

[ydeng]

 

No,  have an AMD CPU  Phenom II X6 1100. This has only these CPU features: SSE4a, 3DNow !, SSE2, SSE, MMX, SSE3

 

Edit:

apologize ydeng, what are the exact features in the panic message, which shows the avx2 emulation or that the avx2 was emulated.

 

I cannot tell why init got killed.

The corecryto one says the kext does not pass POST test, which means math error.

Apparently there are different versions of functions with different instruction sets.  avx2 functions should not be called if not available.

 

Have you tried to build the kernel with sse4.1 off?

 

Here's an implementation of corecrypto, it is worth a try.  https://github.com/samdmarshall/apple-corecrypto 

 

If opemu fails, it prints "OPEMU: instruction nmenic" , like the screen shots in AkimoA's post.

 

https://github.com/Shaneee92/SierraAMD-XNU/blob/master/osfmk/OPEMU/opemu.c#L104

[spakk]

here are my last two Test-kernels

1. with SSEPlus

2. with modified libudis and Sinetek OPEMU III

AMD-10.12.6_SSEPlus.zip

AMD-10.12.6-mod-Libudis.zip

[AXAXAXAXAXAX]

here are my last two Test-kernels

1. with SSEPlus

2. with modified libudis and Sinetek OPEMU III

 first kernel: 

second one:

[ydeng]

Intel has POPCNT as part of SSE4 spec.  But AMD has it in SSEa, starting from AMD 10h cpus.

This is not sse instruction.  I will not implement an emulation for this.