`thread_exit_event` callback can be called twice for the same thread on Linux

Created by: petrochenkov

Not sure whether this can be classified as a bug in DynamoRIO, but it certainly represents some footgun for clients.

Consider the next setup - fork called in a process with multiple threads:

int main() { // Thread 1
    pthread_t t;
    pthread_create(&t, nullptr, do_work, nullptr); // Thread 2
    fork();
	
    // Child process:
    // Cleanup the now dead Thread 2 with `dynamo_other_thread_exit` in `dynamorio_fork_init`.
    // Calls the `thread_exit_event` callback for Thread 2.
	
    // Parent process:
    // Thread 2 eventually completes its work.
    // Calls the `thread_exit_event` callback for Thread 2.
}

Both times the thread_exit_event callback is called it sees the same value of drcontext and therefore the same thread ID dr_get_thread_id(drcontext) and same client-defined thread-local data. That's what I meant by "the same thread" in the title.

If the callback does some work outside of the process, e.g. dumps a per-thread trace/log into a file it really messes things up. The consecuences can range from corrupted file contents to SIGBUSes if the trace files are memory-mapped.

---

Possible solutions:

• Do not call thread_exit_event callbacks when tidying up dead threads with dynamorio_fork_init specifically or dynamo_other_thread_exit in general.
  • This is perhaps not the best solution because the callback still can do some useful work in this case, like free some memory or other process-local resources.
• Give client a way to determine whether the thread is actually exiting or is already dead and is being cleaned up by someone else.
• If such a way already exists, put it into documentation of dr_register_thread_exit_event.