==3016==ERROR: AddressSanitizer: heap-use-after-free on address 0x620000000150 at pc 0x000100c3b071 bp 0x000006c54670 sp 0x000006c54668
READ of size 8 at 0x620000000150 thread T2
2020-01-24 15:32:43.152069+0900 atos[3105:14431995] examining /opt/ats-asf-9.0.x/*/traffic_server [3016]
#0 0x100c3b070 in ProxySession::connection_id() const ProxySession.cc:200
#1 0x1006cb241 in Http2Stream::destroy() Http2Stream.cc:769
#2 0x1006c2a25 in Http2Stream::terminate_if_possible() Http2Stream.cc:433
#3 0x1006bf8de in Http2Stream::main_event_handler(int, void*) Http2Stream.cc:169
#4 0x10002bca2 in Continuation::handleEvent(int, void*) I_Continuation.h:190
#5 0x100ea3de1 in EThread::process_event(Event*, int) UnixEThread.cc:136
#6 0x100ea4921 in EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*) UnixEThread.cc:175
#7 0x100ea564d in EThread::execute_regular() UnixEThread.cc:235
#8 0x100ea70f0 in EThread::execute() UnixEThread.cc:344
#9 0x100ea11e2 in spawn_thread_internal(void*) Thread.cc:92
#10 0x7fff663d3e64 in _pthread_start (libsystem_pthread.dylib:x86_64+0x5e64)
#11 0x7fff663cf83a in thread_start (libsystem_pthread.dylib:x86_64+0x183a)
0x620000000150 is located 208 bytes inside of 3616-byte region [0x620000000080,0x620000000ea0)
freed by thread T2 here:
#0 0x9d498d in wrap_free (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x6098d)
#1 0x111a9d in ats_memalign_free ink_memory.cc:138
#2 0x14bda8 in jearena::JemallocNodumpAllocator::deallocate(_InkFreeList*, void*) JeAllocator.cc:139
#3 0x118482 in malloc_free(_InkFreeList*, void*) ink_queue.cc:323
#4 0x114054 in ink_freelist_free ink_queue.cc:277
#5 0x100674212 in ClassAllocator<Http2ClientSession>::free(Http2ClientSession*) Allocator.h:145
#6 0x100651bbc in void thread_free<Http2ClientSession>(ClassAllocator<Http2ClientSession>&, Http2ClientSession*) I_ProxyAllocator.h:80
#7 0x100650cf8 in Http2ClientSession::free() Http2ClientSession.cc:162
#8 0x100c3ac51 in ProxySession::handle_api_return(int) ProxySession.cc:170
#9 0x100c3afbc in ProxySession::do_api_callout(TSHttpHookID) ProxySession.cc:148
#10 0x10064f801 in Http2ClientSession::destroy() Http2ClientSession.cc:78
#11 0x1006827bb in Http2ConnectionState::release_stream(Http2Stream*) Http2ConnectionState.cc:1377
#12 0x1006cb1ee in Http2Stream::destroy() Http2Stream.cc:767
#13 0x1006c2a25 in Http2Stream::terminate_if_possible() Http2Stream.cc:433
#14 0x1006bf8de in Http2Stream::main_event_handler(int, void*) Http2Stream.cc:169
#15 0x10002bca2 in Continuation::handleEvent(int, void*) I_Continuation.h:190
#16 0x100ea3de1 in EThread::process_event(Event*, int) UnixEThread.cc:136
#17 0x100ea4921 in EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*) UnixEThread.cc:175
#18 0x100ea564d in EThread::execute_regular() UnixEThread.cc:235
#19 0x100ea70f0 in EThread::execute() UnixEThread.cc:344
#20 0x100ea11e2 in spawn_thread_internal(void*) Thread.cc:92
#21 0x7fff663d3e64 in _pthread_start (libsystem_pthread.dylib:x86_64+0x5e64)
#22 0x7fff663cf83a in thread_start (libsystem_pthread.dylib:x86_64+0x183a)
previously allocated by thread T2 here:
#0 0x9d5147 in wrap_posix_memalign (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x61147)
#1 0x11183e in ats_memalign ink_memory.cc:102
#2 0x14bc85 in jearena::JemallocNodumpAllocator::allocate(_InkFreeList*) JeAllocator.cc:118
#3 0x118382 in malloc_new(_InkFreeList*) ink_queue.cc:264
#4 0x113e5a in ink_freelist_new ink_queue.cc:187
#5 0x1006e5fde in ClassAllocator<Http2ClientSession>::alloc() Allocator.h:131
#6 0x1006dce40 in Http2ClientSession* thread_alloc_init<Http2ClientSession>(ClassAllocator<Http2ClientSession>&, ProxyAllocator&) I_ProxyAllocator.h:73
#7 0x1006dc912 in Http2SessionAccept::accept(NetVConnection*, MIOBuffer*, IOBufferReader*) Http2SessionAccept.cc:56
#8 0x1006dcfd4 in Http2SessionAccept::mainEvent(int, void*) Http2SessionAccept.cc:72
#9 0x10002bca2 in Continuation::handleEvent(int, void*) I_Continuation.h:190
#10 0x100d42a23 in send_plugin_event(Continuation*, int, void*) SSLNextProtocolAccept.cc:33
#11 0x100d42544 in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) SSLNextProtocolAccept.cc:106
#12 0x10002bca2 in Continuation::handleEvent(int, void*) I_Continuation.h:190
#13 0x100df6d6a in read_signal_and_update(int, UnixNetVConnection*) UnixNetVConnection.cc:83
#14 0x100df6abd in read_signal_done(int, NetHandler*, UnixNetVConnection*) UnixNetVConnection.cc:144
#15 0x100df6a32 in UnixNetVConnection::readSignalDone(int, NetHandler*) UnixNetVConnection.cc:1000
#16 0x100d179f4 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) SSLNetVConnection.cc:637
#17 0x100dc5f3e in NetHandler::process_ready_list() UnixNet.cc:400
#18 0x100dc8391 in NetHandler::waitForActivity(long long) UnixNet.cc:535
#19 0x100ea5eb6 in EThread::execute_regular() UnixEThread.cc:283
#20 0x100ea70f0 in EThread::execute() UnixEThread.cc:344
#21 0x100ea11e2 in spawn_thread_internal(void*) Thread.cc:92
#22 0x7fff663d3e64 in _pthread_start (libsystem_pthread.dylib:x86_64+0x5e64)
#23 0x7fff663cf83a in thread_start (libsystem_pthread.dylib:x86_64+0x183a)
Thread T2 created by T0 here:
#0 0x9cc7cd in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x587cd)
#1 0x100ea0fa8 in ink_thread_create(_opaque_pthread_t**, void* (*)(void*), void*, int, unsigned long, void*) ink_thread.h:159
#2 0x100ea0d24 in Thread::start(char const*, void*, unsigned long, std::__1::function<void ()> const&) Thread.cc:109
#3 0x100eac629 in EventProcessor::spawn_event_threads(int, int, unsigned long) UnixEventProcessor.cc:392
#4 0x100ead8a8 in EventProcessor::start(int, unsigned long) UnixEventProcessor.cc:455
#5 0x100121e23 in main traffic_server.cc:1982
#6 0x7fff661cf7fc in start (libdyld.dylib:x86_64+0x1a7fc)
SUMMARY: AddressSanitizer: heap-use-after-free ProxySession.cc:200 in ProxySession::connection_id() const
Shadow bytes around the buggy address:
0x1c3fffffffd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1c3fffffffe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1c3ffffffff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1c4000000000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x1c4000000010: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x1c4000000020: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
0x1c4000000030: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x1c4000000040: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x1c4000000050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x1c4000000060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x1c4000000070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
I found another ASan crash related to HTTP/2, during the investigation of #6313 against 9.0.x branch.
This happens when clients abort in the middle of POST request.
Unfortunately, reverting dac1489 didn't work for this.