Skip to content

Threads may access destructed global DbgCtl objects on shutdown #11819

@JosiahWI

Description

@JosiahWI

The issue was originally reported by TSan as a shutdown race. That report is easily reproducible, but I don't have a TSan build right now to do so. There were concerns it could be a false positive, but ASan happened to actually catch it in the wild on CI. This does not mean definitively that it isn't a false positive, but as TSan rarely has false positives (except in certain cases of which I believe this is not one) and both tools detected it, I think it's reasonable to presume this is a real race unless it can be proven otherwise. Here is the ASan report:

==6586==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000001178 at pc 0x0000004ab418 bp 0x7ff4f57e1cc0 sp 0x7ff4f57e1cb0
READ of size 1 at 0x604000001178 thread T1 ([ET_NET 0])
    #0 0x4ab417 in DbgCtl::on() const ../include/tsutil/DbgCtl.h:111
    #1 0x725570 in PollCont::do_poll(long) ../src/iocore/net/PollCont.cc:89
    #2 0x71d2bc in NetHandler::waitForActivity(long) ../src/iocore/net/NetHandler.cc:362
    #3 0xd4b8c2 in EThread::execute_regular() ../src/iocore/eventsystem/UnixEThread.cc:307
    #4 0xd4bddd in EThread::execute() ../src/iocore/eventsystem/UnixEThread.cc:358
    #5 0xd48a44 in spawn_thread_internal ../src/iocore/eventsystem/Thread.cc:75
    #6 0x7ff4f9db21c9 in start_thread (/lib64/libpthread.so.0+0x81c9)
    #7 0x7ff4f9a0d8d2 in __GI___clone (/lib64/libc.so.6+0x398d2)

This is likely very nontrivial to fix because of the difficulty of managing shutdown of global objects. Joining all threads before shutdown may be one possible fix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ASanAddress SanitizerTSan

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions