quiche build: Regression tests seg fault

[bneradt]

Our quiche 10-Dev CI jobs are showing a segfault when the regression tests are run:

https://ci.trafficserver.apache.org/view/10-Dev/job/10-Dev/job/quiche/4/execution/node/77/log/

Note the following:

• 10-Dev does have the recent -R 3 regression test segfault fix merged in via the following 10-Dev merge: be0f694c
• This segfault happens with -R 1 as well as -R 3.

Stack Trace

[jenkins@ip-72-167-36-42 trafficserver]$ gdb --args /tmp/ats/bin/traffic_server -K -k -R 1

...

(gdb) run
Starting program: /tmp/ats/bin/traffic_server -K -k -R 1

...

Thread 1 "traffic_server" received signal SIGSEGV, Segmentation fault.
0x00007f3dc1de6c95 in __strlen_avx2 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.28-189.1.el8.x86_64 hwloc-libs-2.2.0-3.el8.x86_64 libblkid-2.32.1-35.el8.x86_64 libcap-2.48-2.el8.x86_64 libgcc-8.5.0-10.el8.x86_64 libselinux-2.9-5.el8.x86_64 libstdc++-8.5.0-10.el8.x86_64 libuuid-2.32.1-35.el8.x86_64 openssl-libs-1.1.1k-6.el8_5.x86_64 pcre-8.42-6.el8.x86_64 pcre2-10.32-2.el8.x86_64 systemd-libs-239-58.el8.x86_64 xz-libs-5.2.4-3.el8.1.x86_64 zlib-1.2.11-18.el8_5.x86_64

(gdb) bt
#0  0x00007f3dc1de6c95 in __strlen_avx2 () from /lib64/libc.so.6
#1  0x00007f3dc2a901b6 in core::ffi::c_str::CStr::from_ptr (ptr=0x0) at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ffi/c_str.rs:286
#2  quiche::ffi::quiche_config_load_cert_chain_from_pem_file (config=0x19a51c0, path=0x0) at quiche/src/ffi.rs:156
#3  0x0000000000c31b0b in QUICNetProcessor::start (this=0x109ef20 <quic_NetProcessor>, stacksize=1048576) at QUICNetProcessor_quiche.cc:84
#4  0x00000000008830d9 in main (argv=0x7ffe6ebdb068) at traffic_server/traffic_server.cc:2153

(gdb) f 1
#1  0x00007f3dc2a901b6 in core::ffi::c_str::CStr::from_ptr (ptr=0x0) at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ffi/c_str.rs:286
286     /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ffi/c_str.rs: No such file or directory.

(gdb) f 3
#3  0x0000000000c31b0b in QUICNetProcessor::start (this=0x109ef20 <quic_NetProcessor>, stacksize=1048576) at QUICNetProcessor_quiche.cc:84
84        quiche_config_load_cert_chain_from_pem_file(this->_quiche_config, context->userconfig->cert);

(gdb) p this->_quiche_config
$1 = (quiche_config *) 0x19a51c0

(gdb) p context->userconfig->cert
$2 = {<ats_scoped_resource<detail::SCOPED_MALLOC_TRAITS<char> >> = {_r = 0x0}, <No data fields>}

This is perhaps a nullptr dereference on context->userconfig->cert?

Reproduction Steps

docker run -it -u 1200:1200 --init --cap-add=SYS_PTRACE --network=host --name repro_quiche_regression_failure ci.trafficserver.apache.org/ats/rockylinux:8 /bin/bash

# Then, from within the docker container:
cd /var/tmp
git clone https://github.com/apache/trafficserver.git
cd trafficserver/
git checkout 10-Dev
source /opt/rh/gcc-toolset-11/enable
autoreconf -fiv
./configure --enable-experimental-plugins --enable-example-plugins --with-quiche=/opt/quiche --prefix=/tmp/ats --enable-werror --enable-debug
make -j4
make install
/tmp/ats/bin/traffic_server -K -k -R 3

Again, this crash happens with -R 1 as well as -R 3.

[brbzull0]

On my box(ubuntu 20.04), this also happenes if you just run the binary alone.