crash reloading parent.config on ATS 9

[randall]

We've seen this twice running ATS9. parent.config was updated (though I think the update was only a re-ordering of lines) and traffic_ctl config reload was issued. That resulted in the crash below:

#0  0x00007f9f22cafe6b in je_tcache_dalloc_small (tsd=<optimized out>, tcache=<optimized out>, ptr=<optimized out>, 
    binind=<optimized out>, slow_path=false) at include/jemalloc/internal/tcache.h:415
#1  je_arena_dalloc (tsdn=<optimized out>, ptr=<optimized out>, tcache=<optimized out>, slow_path=false)
    at include/jemalloc/internal/arena.h:1429
#2  je_idalloctm (is_metadata=false, slow_path=false, tsdn=<optimized out>, ptr=<optimized out>, 
    tcache=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1164
#3  je_iqalloc (tsd=<optimized out>, ptr=<optimized out>, tcache=<optimized out>, slow_path=<optimized out>)
    at include/jemalloc/internal/jemalloc_internal.h:1181
#4  ifree (tsd=0x7f9f13a8c700, ptr=<optimized out>, tcache=<optimized out>, slow_path=<optimized out>)
    at src/jemalloc.c:1871
#5  0x00007f9f24b0f9ad in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f1262dfd0) at /opt/ats/include/c++/v1/__tree:1830
#6  0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f12642a90) at /opt/ats/include/c++/v1/__tree:1831
#7  0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f12618df0) at /opt/ats/include/c++/v1/__tree:1831
#8  0x00007f9f24b0f9ad in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f129ef810) at /opt/ats/include/c++/v1/__tree:1830
#9  0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f12615760) at /opt/ats/include/c++/v1/__tree:1831
#10 0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f1260ecb0) at /opt/ats/include/c++/v1/__tree:1831
#11 0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f129ef870) at /opt/ats/include/c++/v1/__tree:1831
#12 0x00007f9f24b0f9ad in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f129f01a0) at /opt/ats/include/c++/v1/__tree:1830
#13 0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f129ef6c0) at /opt/ats/include/c++/v1/__tree:1831
#14 0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f129f0d40) at /opt/ats/include/c++/v1/__tree:1831
#15 0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f12612e20) at /opt/ats/include/c++/v1/__tree:1831
#16 0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f129ef1b0) at /opt/ats/include/c++/v1/__tree:1831
#17 0x00007f9f24b0f9b9 in std::__1::__tree<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__map_value_compare<unsigned long, std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::less<unsigned long>, true>, std::__1::allocator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*> > >::destroy (
    this=0x7f9f129ef0d0, __nd=0x7f9f129ef270) at /opt/ats/include/c++/v1/__tree:1831
#18 0x000055d58e34bd3c in ParentConsistentHash::~ParentConsistentHash (this=0x7f9f12b2fa80)
    at ParentConsistentHash.cc:60
#19 0x000055d58e34bdbe in ParentConsistentHash::~ParentConsistentHash (this=0x7f9f12b2fa80)
    at ParentConsistentHash.cc:58
#20 0x000055d58e330aeb in ParentRecord::~ParentRecord (this=0x7f9f1c18e700) at ParentSelection.cc:810
#21 0x000055d58e31b8e8 in ~BaseMatcher (this=<optimized out>) at ./ControlMatcher.h:169
#22 HostMatcher<ParentRecord, ParentResult>::~HostMatcher (this=<optimized out>) at ControlMatcher.cc:97
#23 0x000055d58e319353 in ControlMatcher<ParentRecord, ParentResult>::~ControlMatcher (this=0x7f9f21073400)
    at ControlMatcher.cc:709
#24 0x000055d58e32de8d in ParentConfigParams::~ParentConfigParams (this=0x7f9f125ec3f0) at ParentSelection.cc:93
#25 0x000055d58e32dece in ParentConfigParams::~ParentConfigParams (this=0x7f9f125ec3f0) at ParentSelection.cc:89
#26 0x000055d58e2aa539 in ConfigProcessor::release (this=0x55d58e79f640 <configProcessor>, id=3, info=0x7f9f125ec3f0)
    at ProxyConfig.cc:194
#27 0x000055d58e2aaf5c in ConfigInfoReleaser::handle_event (this=0x7f9ecb63c000) at ProxyConfig.cc:104
#28 0x000055d58e3aa129 in handleEvent (this=<optimized out>, event=<optimized out>, data=<optimized out>)
    at ./I_Continuation.h:190
#29 EThread::process_event (this=0x7f9f1d20d140, e=0x7f9f1fb74420, calling_code=2) at UnixEThread.cc:136
#30 0x000055d58e3aa930 in EThread::execute_regular (this=0x7f9f1d20d140) at UnixEThread.cc:249
#31 0x000055d58e3aad7c in EThread::execute (this=0x7f9f1d20d140) at UnixEThread.cc:338
#32 0x000055d58e3a975b in spawn_thread_internal (a=0x7f9f21054e70) at Thread.cc:92
#33 0x00007f9f22a8ee65 in start_thread () from /lib64/libpthread.so.0
#34 0x00007f9f21eac88d in clone () from /lib64/libc.so.6

[jrushford]

@randall Are you using "secondary_parents" in your parents.config?

[randall]

@jrushford In this particular case, no.

[jrushford]

@randall I've tried duplicating this while re-loading changes to parent.config. I have not been able to duplicate it so far but, I'm not using jemallac on my test machine. I'll try that next. Is there anything else that you can think of that I should try?

[randall]

@jrushford I'll try to run under valgrind/ASAN to see if I can get more details here.

[jrushford]

@randall ok, sounds good