Skip to content

Segmentation fault while executing System.Net tests in linux x64 coreclr #83482

Closed

Description

@carlossanlop

Found in main: #83356
And also found in 7.0: #83467

There is also a crash in these tests, but affecting mono, and also happens in both main and 7.0. They could be related: #83481

The output of all the 3 executions is very similar:

Output (expand)
Console log: 'System.Net.Security.Unit.Tests' from job e1438f85-bd3c-4620-ae7b-ca686d95a426 workitem ee14f405-843d-442a-81cc-1dbd5bfc751f (redhat.7.amd64.open.svc) executed on machine a0009LP running Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-redhat-7.9-Maipo
+ ./RunTests.sh --runtime-path /mnt/work/AFB8097B/p
----- start Wed Mar 15 19:48:23 UTC 2023 =============== To repro directly: =====================================================
pushd .
/mnt/work/AFB8097B/p/dotnet exec --runtimeconfig System.Net.Security.Unit.Tests.runtimeconfig.json --depsfile System.Net.Security.Unit.Tests.deps.json xunit.console.dll System.Net.Security.Unit.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/mnt/work/AFB8097B/w/AEB109CE/e /mnt/work/AFB8097B/w/AEB109CE/e
  Discovering: System.Net.Security.Unit.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Security.Unit.Tests (found 74 of 76 test cases)
  Starting:    System.Net.Security.Unit.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 168: 24497 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Security.Unit.Tests.runtimeconfig.json --depsfile System.Net.Security.Unit.Tests.deps.json xunit.console.dll System.Net.Security.Unit.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/mnt/work/AFB8097B/w/AEB109CE/e
----- end Wed Mar 15 19:48:24 UTC 2023 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: unlimited
[    6.300543] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    6.300618] sd 3:0:1:0: Attached scsi generic sg1 type 0
[    6.300684] sr 2:0:0:0: Attached scsi generic sg2 type 5
[    6.390928] cryptd: max_cpu_qlen set to 1000
[    6.450462] XFS (sda2): Mounting V5 Filesystem
[    6.450493] XFS (dm-4): Mounting V5 Filesystem
[    6.450569] XFS (dm-0): Mounting V5 Filesystem
[    6.450600] XFS (dm-2): Mounting V5 Filesystem
[    6.450684] XFS (dm-3): Mounting V5 Filesystem
[    6.482819] XFS (dm-3): Ending clean mount
[    6.486027] XFS (sda2): Ending clean mount
[    6.616152] AVX2 version of gcm_enc/dec engaged.
[    6.616155] AES CTR mode by8 optimization enabled
[    6.619031] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[    6.619177] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni)
[    6.670151] XFS (dm-0): Ending clean mount
[    6.809153] XFS (dm-2): Ending clean mount
[    7.060075] XFS (dm-4): Ending clean mount
[    7.080417] systemd-journald[539]: Received request to flush runtime journal from PID 1
[    7.222850] type=1305 audit(1678908232.101:4): audit_pid=751 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[    8.830299] UDF-fs: INFO Mounting volume 'UDF Volume', timestamp 2023/03/16 00:00 (1000)
[    9.071384] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    9.071391] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    9.950462] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   10.410842] hv_utils: KVP IC version 4.0
[   11.666556]  sdb: sdb1
[   12.641607] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
[   18.061453] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   53.193928] hv_balloon: Max. dynamic memory size: 8192 MB
[   61.216830] SELinux: 2048 avtab hash slots, 112733 rules.
[   61.250698] SELinux: 2048 avtab hash slots, 112733 rules.
[   61.269664] SELinux:  8 users, 14 roles, 5046 types, 316 bools, 1 sens, 1024 cats
[   61.269668] SELinux:  130 classes, 112733 rules
[   61.272716] SELinux:  Converting 1337 SID table entries...
[   68.824122] SELinux: 2048 avtab hash slots, 112735 rules.
[   68.859169] SELinux: 2048 avtab hash slots, 112735 rules.
[   68.882094] SELinux:  8 users, 14 roles, 5046 types, 316 bools, 1 sens, 1024 cats
[   68.882098] SELinux:  130 classes, 112735 rules
[   68.885814] SELinux:  Converting 1337 SID table entries...
[   93.755199] SELinux: 2048 avtab hash slots, 112795 rules.
[   93.791316] SELinux: 2048 avtab hash slots, 112795 rules.
[   93.809913] SELinux:  8 users, 14 roles, 5047 types, 316 bools, 1 sens, 1024 cats
[   93.809916] SELinux:  130 classes, 112795 rules
[   93.812993] SELinux:  Converting 1338 SID table entries...
[   99.696803] type=1305 audit(1678908324.575:188): audit_pid=0 old=751 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   99.703269] type=1130 audit(1678908324.583:189): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   99.703284] type=1131 audit(1678908324.583:190): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   99.814557] type=1305 audit(1678908324.694:191): audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   99.814567] type=1305 audit(1678908324.694:192): audit_pid=9818 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[ 1479.203755] .NET Long Runni[24512]: segfault at 0 ip 00007f6c21d3f31f sp 00007f2b75feb410 error 6 in gssntlmssp.so[7f6c21d39000+1c000]
Waiting a few seconds for any dump to be written..
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dump..
Looking for files matching core.* ...
+ export _commandExitCode=139
+ _commandExitCode=139
+ /bin/python3 /mnt/work/AFB8097B/p/reporter/run.py https://dev.azure.com/dnceng-public/ public 3832409 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJhNWY3YjU2ZS1mY2M5LTQzYWQtODYwOC04N2M0NDcwOTIwOTMiLCJzaWQiOiI0NmY2NTNkOC03MDBlLTQxMTItYTA5Ny04YjVmNDMyYTQ1OWUiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzIwNTkxNiIsImpvYnJlZiI6IjQ2OTM5ZDM3LTBiMDYtNGI3YS1iYTg1LWM1ZDcxMWJkYTIyOTo3MWFmZTc2OC1lNDkzLTU5N2MtMjI4Yy0wMzc4MzQxOTNhZWIiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMjA1OTE2Iiwib3JjaGlkIjoiNDY5MzlkMzctMGIwNi00YjdhLWJhODUtYzVkNzExYmRhMjI5LmxpYnJhcmllc190ZXN0X3J1bl9yZWxlYXNlX2NvcmVjbHJfbGludXhfeDY0X2RlYnVnLl9fZGVmYXVsdCIsInJlcG9JZHMiOiIiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286NmZjYzkyZTUtNzNhNy00Zjg4LThkMTMtZDkwNDViNDVmYjI3IiwibmJmIjoxNjc4OTA3MzE0LCJleHAiOjE2Nzg5MTc1MTR9.Jw6444X6tGPly4azNQx08VtK4UBqtdp12wvNQix0XvW8k976BHgPX9pz3oj2GGV1zjDx8HGHMbUr-358uL1q7gGGdIdlzujkUoKaEttl2aCtdbWr6fiDdHR4aQWDgaIacp0Tb1AR61EG7NetmSlQoYPu-az6572Tjbsi17ULmbYm3Fxm_eWmhJ89ZO20jDNKu2ukWXJYPRjM3LRNgQ3bcucNQBkrZgKexmdmbBQ0MHsNv2b4RzzRQfBoZtyANz3JuExaC4Bh-ueBLei90NhYJYi9UBKpMYemRBJ9lr-OXtsWZBtK416V5IxmfHHu4C4WGRDDp6eBBC35aEGzsB9K5w
2023-03-15T19:48:34.809Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-03-15T19:48:34.809Z	INFO   	run.py	__init__(42)	read_results	Searching '/mnt/work/AFB8097B/w/AEB109CE/e' for test results files
2023-03-15T19:48:34.809Z	INFO   	run.py	__init__(42)	read_results	Searching '/mnt/work/AFB8097B/w/AEB109CE/uploads' for test results files
2023-03-15T19:48:34.809Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-03-15T19:48:34.810Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/mnt/work/AFB8097B/w/AEB109CE/e/__test_report.json'
2023-03-15T19:48:34.810Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1572 bytes
+ /bin/python3 /mnt/work/AFB8097B/p/gen-debug-dump-docs.py -buildid 205916 -workitem System.Net.Security.Unit.Tests -jobid e1438f85-bd3c-4620-ae7b-ca686d95a426 -outdir /mnt/work/AFB8097B/w/AEB109CE/uploads -templatedir /mnt/work/AFB8097B/p -dumpdir /home/helixbot/dotnetbuild/dumps -productver 7.0.5
read file: /mnt/work/AFB8097B/p/debug-dump-template.md
writing output file: /mnt/work/AFB8097B/w/AEB109CE/uploads/how-to-debug-dump.md
done writing debug dump information
+ exit 139
['System.Net.Security.Unit.Tests' END OF WORK ITEM LOG: Command exited with 139]

Activity

  1. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Mar 15, 2023
  2. changed the title [-][7.0] Segmentation fault while executing System.Net tests in linux x64 coreclr[/-] [+]Segmentation fault while executing System.Net tests in linux x64 coreclr[/+] on Mar 15, 2023
  3. changed the title [-]Segmentation fault while executing System.Net tests in linux x64 coreclr[/-] [+][7.0] Segmentation fault while executing System.Net tests in linux x64 coreclr[/+] on Mar 15, 2023
  4. changed the title [-][7.0] Segmentation fault while executing System.Net tests in linux x64 coreclr[/-] [+]Segmentation fault while executing System.Net tests in linux x64 coreclr[/+] on Mar 16, 2023
  5. wfurt commented on Mar 16, 2023

    @wfurt
    Member

    seems like dup of #83540. Probably broken by OS package update on RedHat.7 queue.

  6. ghost removed
    untriagedNew issue has not been triaged by the area owner
    on Mar 16, 2023
  7. added this to the 8.0.0 milestone on Mar 22, 2023
  8. ghost locked as resolved and limited conversation to collaborators on Apr 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions