Skylake: assertion fail in unix/signal.c: sc->fpstate->sw_reserved.extended_size <= XSTATE_FRAME_EXTRA

Created by: algr

On Intel Skylake model 78 with Ubuntu 16.04 LTS, many of the tests crash out with this assert. It turns out that extended_size = 1092 and XSTATE_FRAME_EXTRA = 896, i.e. there is an extra 196 bytes in the state.

This core has AVX2 but not (as far as I can tell) AVX-512. DR supports AVX2 and has already checked YMM_ENABLED() on this code path. But I don't know if the extra state is AVX-related. Even if it was AVX-512, that would only be an extra 32*16 = 128 bytes.

Some documentation explaining the rationale for what DR is doing here and how it arrives at XSTATE_FRAME_EXTRA would help enormously. It looks as if this code would have to be aware of all types of extra state including kinds not normally documented in the system's sigcontext.h.