CreateDefaultBuilder_SecretsDoesReload test failed in CI

[stephentoub]

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-48684-merge-1dda5503a115420687/Microsoft.Extensions.Hosting.Unit.Tests/console.b5b90d65.log?sv=2019-07-07&se=2021-03-16T05%3A35%3A14Z&sr=c&sp=rl&sig=ytXzgYX%2F6x0X9rBYl8sm%2B1JpNoEfEPjQFQhlMtH69tU%3D

  Discovering: Microsoft.Extensions.Hosting.Unit.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  Microsoft.Extensions.Hosting.Unit.Tests (found 99 test cases)
  Starting:    Microsoft.Extensions.Hosting.Unit.Tests (parallel test collections = on, max threads = 2)
�[40m�[32minfo�[39m�[22m�[49m: Microsoft.Extensions.Hosting.Tests.HostTests[0]
      Request starting
    Microsoft.Extensions.Hosting.Tests.HostTests.CreateDefaultBuilder_SecretsDoesReload [FAIL]
      Assert.Equal() Failure
                                    ↓ (pos 20)
      Expected: Hello ASP.NET Core: f577b280d3814d6c935970a1aadbb8e6
      Actual:   Hello ASP.NET Core: 407a83ef08c04e1ca1e07690cb30dc0a
                                    ↑ (pos 20)
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Hosting/tests/UnitTests/HostTests.cs(264,0): at Microsoft.Extensions.Hosting.Tests.HostTests.CreateDefaultBuilder_SecretsDoesReload()
        --- End of stack trace from previous location ---
  Finished:    Microsoft.Extensions.Hosting.Unit.Tests

Tagging subscribers to this area: @eerhardt, @maryamariyan
See info in area-owners.md if you want to be subscribed.

Issue Details

---

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-48684-merge-1dda5503a115420687/Microsoft.Extensions.Hosting.Unit.Tests/console.b5b90d65.log?sv=2019-07-07&se=2021-03-16T05%3A35%3A14Z&sr=c&sp=rl&sig=ytXzgYX%2F6x0X9rBYl8sm%2B1JpNoEfEPjQFQhlMtH69tU%3D

  Discovering: Microsoft.Extensions.Hosting.Unit.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  Microsoft.Extensions.Hosting.Unit.Tests (found 99 test cases)
  Starting:    Microsoft.Extensions.Hosting.Unit.Tests (parallel test collections = on, max threads = 2)
�[40m�[32minfo�[39m�[22m�[49m: Microsoft.Extensions.Hosting.Tests.HostTests[0]
      Request starting
    Microsoft.Extensions.Hosting.Tests.HostTests.CreateDefaultBuilder_SecretsDoesReload [FAIL]
      Assert.Equal() Failure
                                    ↓ (pos 20)
      Expected: Hello ASP.NET Core: f577b280d3814d6c935970a1aadbb8e6
      Actual:   Hello ASP.NET Core: 407a83ef08c04e1ca1e07690cb30dc0a
                                    ↑ (pos 20)
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Hosting/tests/UnitTests/HostTests.cs(264,0): at Microsoft.Extensions.Hosting.Tests.HostTests.CreateDefaultBuilder_SecretsDoesReload()
        --- End of stack trace from previous location ---
  Finished:    Microsoft.Extensions.Hosting.Unit.Tests

Author:	stephentoub
Assignees:	-
Labels:	area-Extensions-Hosting
Milestone:	-

[eerhardt]

Looking at the test log, it looks like that test took 1 minute to run:

Looking at the test, it looks like it is waiting for 1 minute:

runtime/src/libraries/Microsoft.Extensions.Hosting/tests/UnitTests/HostTests.cs

Lines 211 to 218 in 5a3d44e

	config.GetReloadToken().RegisterChangeCallback(o =>
	{
	configReloadedCancelTokenSource.Cancel();
	}, null);
	// Wait for up to 1 minute, if config reloads at any time, cancel the wait.
	await Task.WhenAny(Task.Delay(TimeSpan.FromMinutes(1), configReloadedCancelToken)); // Task.WhenAny ignores the task throwing on cancellation.
	Assert.NotEqual(dynamicConfigMessage1, dynamicConfigMessage2); // Messages are different.
	Assert.Equal(dynamicConfigMessage2, config["Hello"]); // Config DID reload from disk

So this leads me to believe that the change callback was never invoked, which is why we are getting stale data in the config.

[stephentoub]

Failed again here:
https://dev.azure.com/dnceng/public/_build/results?buildId=1061100&view=ms.vss-test-web.build-test-results-tab&runId=32699320&paneView=attachments&resultId=181769
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-50357-merge-ede7a005545e42cf8b/Microsoft.Extensions.Hosting.Unit.Tests/console.39af16ec.log?sv=2019-07-07&se=2021-04-18T11%3A42%3A53Z&sr=c&sp=rl&sig=k%2BL7uF4zgHywaVYUegb2Pi%2F%2F7AIejrTog53v%2FftznF0%3D

  Discovering: Microsoft.Extensions.Hosting.Unit.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  Microsoft.Extensions.Hosting.Unit.Tests (found 100 test cases)
  Starting:    Microsoft.Extensions.Hosting.Unit.Tests (parallel test collections = on, max threads = 2)
    Microsoft.Extensions.Hosting.Tests.HostTests.CreateDefaultBuilder_SecretsDoesReload [FAIL]
      Assert.Equal() Failure
                                    ↓ (pos 20)
      Expected: Hello ASP.NET Core: 1466b2f4da5249bdaef71d3aeee50e47
      Actual:   Hello ASP.NET Core: 8ab58f24c0c34494a4a4969709dc8a26
                                    ↑ (pos 20)
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Hosting/tests/UnitTests/HostTests.cs(264,0): at Microsoft.Extensions.Hosting.Tests.HostTests.CreateDefaultBuilder_SecretsDoesReload()
        --- End of stack trace from previous location ---
�[40m�[32minfo�[39m�[22m�[49m: Microsoft.Extensions.Hosting.Tests.HostTests[0]
      Request starting
  Finished:    Microsoft.Extensions.Hosting.Unit.Tests

[maryamariyan]

The test was added in #43807

[maryamariyan]

Removed blocking-clean-ci as the ActiveIssue attribute is now added to the test.

Keeping issue open to track re-enabling it properly