Skip to content
GitLab
Open
Issue created by Derek Bruening@derekbrueningContributor

drcpusim complains about CET instructions emitted by recent gcc

Pasting from https://github.com/DynamoRIO/dynamorio/issues/4953#issuecomment-922071343

For the drcpusim failures:

<Invalid Klamath instruction "nop" @ tool.cpuid+0x1190.  Aborting.>

It's not actually a nop: DR doesn't decode CET instructions yet (#4040):

bruening@ubuntu:~/dr/git/build_x86_dbg_tests$ objdump -d suite/tests/bin/tool.cpuid | grep -A 4 'main>:'
00001747 <main>:
    1747:	f3 0f 1e fb          	endbr32 
    174b:	8d 4c 24 04          	lea    0x4(%esp),%ecx

Compiling -march=pentium2 does not remove it which seems crazy. Compiling -fcf-protection=none does remove it from the .o.

There are some in the test tools library and drlibc too. If I build them with -fcf-protection=none then we just have instances in _start, __libc_csu_init, etc.:

bruening@ubuntu:~/dr/git/build_x86_dbg_tests$ objdump -d suite/tests/bin/tool.cpuid | grep -B 1 endbr
00001000 <.init>:
    1000:	f3 0f 1e fb          	endbr32 
--
00001190 <_start>:
    1190:	f3 0f 1e fb          	endbr32 
--
    126e:	66 90                	xchg   %ax,%ax
    1270:	f3 0f 1e fb          	endbr32 
--
    12bf:	90                   	nop
    12c0:	f3 0f 1e fb          	endbr32 
--
00002630 <__libc_csu_init>:
    2630:	f3 0f 1e fb          	endbr32 
--
000026a0 <__libc_csu_fini>:
    26a0:	f3 0f 1e fb          	endbr32 
--
000026ac <.fini>:
    26ac:	f3 0f 1e fb          	endbr32

Adding -mno-shstk (the other option at https://man7.org/linux/man-pages/man1/gcc.1.html that mentions CET) to all 3 targets: no difference.

Linker options:

  • -z cet-report=none
  • -no-shstk

Tried manually:

bruening@ubuntu:~/dr/git/build_x86_dbg_tests$ /usr/bin/cc -m32 -mno-shstk -fcf-protection=none -Wl,-z -Wl,cet-report=none  -Wl,--hash-style=both  -m32 -rdynamic  suite/tests/CMakeFiles/tool.cpuid.dir/__/__/clients/drcpusim/tests/cpuid.c.o  -o suite/tests/bin/tool.cpuid  /usr/lib32/libm.so  /usr/lib32/libdl.so  suite/tests/bin/libtools.a  lib32/libdrlibc.a
bruening@ubuntu:~/dr/git/build_x86_dbg_tests$ objdump -d suite/tests/bin/tool.cpuid | grep -B 1 endbr
00001000 <_init>:
    1000:	f3 0f 1e fb          	endbr32 
--
<...>

bruening@ubuntu:~/dr/git/build_x86_dbg_tests$ /usr/bin/cc -m32 -mno-shstk -fcf-protection=none  -Wl,--hash-style=both  -m32 -rdynamic  suite/tests/CMakeFiles/tool.cpuid.dir/__/__/clients/drcpusim/tests/cpuid.c.o  -o suite/tests/bin/tool.cpuid  /usr/lib32/libm.so  /usr/lib32/libdl.so  suite/tests/bin/libtools.a  lib32/libdrlibc.a -Wl,-no-shstk
/usr/bin/ld: cannot find -lgcc_s
/usr/bin/ld: cannot find -lgcc_s
collect2: error: ld returned 1 exit status
bruening@ubuntu:~/dr/git/build_x86_dbg_tests$ /usr/bin/cc -m32 -mno-shstk -fcf-protection=none -Wl,-no-shstk  -Wl,--hash-style=both  -m32 -rdynamic  suite/tests/CMakeFiles/tool.cpuid.dir/__/__/clients/drcpusim/tests/cpuid.c.o  -o suite/tests/bin/tool.cpuid  /usr/lib32/libm.so  /usr/lib32/libdl.so  suite/tests/bin/libtools.a  lib32/libdrlibc.a
/usr/bin/ld: attempted static link of dynamic object `/usr/lib32/libm.so'
collect2: error: ld returned 1 exit status

I can't even figure out why the Ubuntu20 VM gcc 9.3.0 has this enabled by default while my laptop 10.3.0 does not: gcc -v doesn't show a flag for the VM that's not there for my laptop that seems at all related to CET.

So stumped at how to even disable for a CET toolchain. The easiest solution is to remove these 2 tests are just too hard to run. But once we add #4040 I think many more drcpusim tests will fail when they realize these aren't multi-byte nops from SSE but are much later. Should we add an option to drcpusim to ignore these instructions? We'd need #4040 though.