System.Net.Requests.Tests, System.Net.Security.Tests failing with `Segmentation fault` on Debian 10

[ivanpovazan]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error legs:

• linux-x64 Release NativeAOT_Checked_Libs_SizeOpt
• linux-x64 Release NativeAOT_Checked_Libs_SpeedOpt

Tests failing:

• System.Net.Requests.Tests
• System.Net.Security.Tests

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "25 Segmentation fault      (core dumped) ./System.Net.Requests.Tests",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
299406	dotnet/runtime	Build and Package	Log	#87148
299407	dotnet/runtime	Build product	Log	#87148

Build	Definition	Test	Pull Request
299800	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299568	dotnet/runtime	JIT.opt.WorkItemExecution
299541	dotnet/runtime	JIT.opt.WorkItemExecution	#87261
299567	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299732	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299728	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299741	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299742	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299731	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299730	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299729	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299727	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299720	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299718	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299592	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86843
298109	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86578
299527	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86546
297387	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87189
299514	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299429	dotnet/runtime	JIT.opt.WorkItemExecution
299427	dotnet/runtime	JIT.opt.WorkItemExecution
299396	dotnet/runtime	JIT.opt.WorkItemExecution
298623	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86759
299397	dotnet/runtime	JIT.opt.WorkItemExecution
299365	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87204
299242	dotnet/runtime	JIT.opt.WorkItemExecution
298991	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87209
299244	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87099
299225	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87198
299206	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87065
299181	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87237
298055	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86652
299037	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
298966	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87233
298965	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87232
298677	dotnet/runtime	JIT.opt.WorkItemExecution
298811	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
298918	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#87230
298816	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
298414	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.WorkItemExecution	#87125
297910	dotnet/runtime	JIT.opt.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
35	43	43

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error legs:

• linux-x64 Release NativeAOT_Checked_Libs_SizeOpt
• linux-x64 Release NativeAOT_Checked_Libs_SpeedOpt

Tests failing:

• System.Net.Quic.Functional.Tests
• System.Net.Requests.Tests
• System.Net.Security.Tests

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Segmentation fault",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	ivanpovazan
Assignees:	-
Labels:	blocking-clean-ci, area-NativeAOT-coreclr, Known Build Error
Milestone:	-

[EgorBo]

I'll check whether it could be #87115

[ivanpovazan]

Updated the search term to use ErrorPattern instead, as the initial "ErrorMessage": "Segmentation fault" was too broad

[jkotas]

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

System.Net.Quic.Functional.Tests is known issue #87275.

System.Net.Requests.Tests is crashing is crashing in libcrypto.so.1.1 initialization on Debian 10. I do not think it has anything to do with the size/speed opt configs, we just happened to hit this problem in these configs first.

Unfortunately, it is not possible to get accurate stacktrace to debug this further since Debian does not publish symbols for security updates (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=894081).

We do not support Debian 10 for .NET 8 according to https://github.com/dotnet/core/blob/main/release-notes/8.0/supported-os.md#linux . We should "fix this" by deleting Debian 10 test runs for .NET 8.

#0  0x000000000008bb56 in ?? ()
#1  0x00007f8f8aea1cb0 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#2  0x00007f8f8ae30017 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#3  0x00007f8f88d18f48 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#4  0x00007fd02726e34a in call_init (l=0x563d7fe57920, argc=argc@entry=9, argv=argv@entry=0x7fffd8bbf948, env=env@entry=0x7fffd8bbf998) at dl-init.c:58
#5  0x00007fd02726e486 in call_init (env=0x7fffd8bbf998, argv=0x7fffd8bbf948, argc=9, l=<optimized out>) at dl-init.c:30
#6  _dl_init (main_map=main_map@entry=0x7f8f7800a270, argc=9, argv=0x7fffd8bbf948, env=0x7fffd8bbf998) at dl-init.c:119
#7  0x00007fd0272722e3 in dl_open_worker (a=a@entry=0x7f8f895a3d80) at dl-open.c:517
#8  0x00007fd026ffa8cf in __GI__dl_catch_exception (exception=exception@entry=0x7f8f895a3d60, operate=operate@entry=0x7fd027271f70 <dl_open_worker>, args=args@entry=0x7f8f895a3d80) at dl-error-skeleton.c:196
#9  0x00007fd027271bca in _dl_open (file=0x563d7d7efebb "libssl.so.1.1", mode=-2147483647, caller_dlopen=0x563d7dc62c5c, nsid=<optimized out>, argc=9, argv=0x7fffd8bbf948, env=0x7fffd8bbf998) at dl-open.c:599
#10 0x00007fd027253256 in dlopen_doit (a=a@entry=0x7f8f895a3fa0) at dlopen.c:66
#11 0x00007fd026ffa8cf in __GI__dl_catch_exception (exception=exception@entry=0x7f8f895a3f40, operate=operate@entry=0x7fd027253200 <dlopen_doit>, args=args@entry=0x7f8f895a3fa0) at dl-error-skeleton.c:196
#12 0x00007fd026ffa95f in __GI__dl_catch_error (objname=objname@entry=0x7f8f78009170, errstring=errstring@entry=0x7f8f78009178, mallocedp=mallocedp@entry=0x7f8f78009168,
    operate=operate@entry=0x7fd027253200 <dlopen_doit>, args=args@entry=0x7f8f895a3fa0) at dl-error-skeleton.c:215
#13 0x00007fd027253995 in _dlerror_run (operate=operate@entry=0x7fd027253200 <dlopen_doit>, args=args@entry=0x7f8f895a3fa0) at dlerror.c:170
#14 0x00007fd0272532e6 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#15 0x0000563d7dc62c5c in ?? ()
#16 0x00007fd027095997 in __pthread_once_slow (once_control=0x563d7e618b68, init_routine=0x563d7dc62b80) at pthread_once.c:116
#17 0x0000563d7dc62d99 in InitializeOpenSSLShim ()
#18 0x0000563d7dc5bd8f in ?? ()
#19 0x00007fd027095997 in __pthread_once_slow (once_control=0x563d7e618b2c, init_routine=0x563d7dc5bd80) at pthread_once.c:116
#20 0x0000563d7dc5bd77 in CryptoNative_EnsureOpenSslInitialized ()
#21 0x0000563d7df5a4df in TestUtilities_Interop_CryptoInitializer__EnsureOpenSslInitialized ()
#22 0x0000563d7df5a469 in TestUtilities_Interop_CryptoInitializer___cctor ()
#23 0x0000563d7dee317e in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun ()
#24 0x0000563d7dee309d in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnNonGCStaticBase ()
#25 0x0000563d7df5a3a9 in TestUtilities_Interop_OpenSsl___cctor ()
#26 0x0000563d7dee317e in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun ()
#27 0x0000563d7dee309d in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnNonGCStaticBase ()
#28 0x0000563d7df5a3fe in TestUtilities_Interop_OpenSsl__OpenSslVersionNumber ()
#29 0x0000563d7df5487c in TestUtilities_System_PlatformDetection__GetOpenSslVersion ()
#30 0x0000563d7df54499 in TestUtilities_System_PlatformDetection__get_OpenSslVersion ()

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error legs:

• linux-x64 Release NativeAOT_Checked_Libs_SizeOpt
• linux-x64 Release NativeAOT_Checked_Libs_SpeedOpt

Tests failing:

• System.Net.Quic.Functional.Tests
• System.Net.Requests.Tests
• System.Net.Security.Tests

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "RunTests\\.sh.+Segmentation fault",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
299406	dotnet/runtime	Build and Package	Log	#87148
299407	dotnet/runtime	Build product	Log	#87148

Build	Definition	Test	Pull Request
299800	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299568	dotnet/runtime	JIT.opt.WorkItemExecution
299541	dotnet/runtime	JIT.opt.WorkItemExecution	#87261
299567	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299732	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299728	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299741	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299742	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299731	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299730	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299729	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299727	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299720	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299718	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299592	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86843
298109	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86578
299527	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86546
297387	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87189
299514	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
299429	dotnet/runtime	JIT.opt.WorkItemExecution
299427	dotnet/runtime	JIT.opt.WorkItemExecution
299396	dotnet/runtime	JIT.opt.WorkItemExecution
298623	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86759
299397	dotnet/runtime	JIT.opt.WorkItemExecution
299365	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87204
299242	dotnet/runtime	JIT.opt.WorkItemExecution
298991	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87209
299244	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87099
299225	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87198
299206	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87065
299181	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87237
298055	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#86652
299037	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
298966	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87233
298965	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	#87232
298677	dotnet/runtime	JIT.opt.WorkItemExecution
298811	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
298918	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#87230
298816	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
298414	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.WorkItemExecution	#87125
297910	dotnet/runtime	JIT.opt.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
41	43	43

Author:	ivanpovazan
Assignees:	-
Labels:	area-System.Net, blocking-clean-ci, untriaged, area-NativeAOT-coreclr, Known Build Error
Milestone:	-