Deadlock when calling System.Environment.Exit(int exitCode) during Startup.Configure(..)

In some situations my web application requires to immediately quit during start-up. When doing so via System.Environment.Exit(int exitCode), the dotnet process does not exit.

Minimal set-up to reproduce:

public static void Main(string[] args)
{
    var host = new WebHostBuilder()
        .UseKestrel()
        .UseStartup<Startup>()
        .Build();

    host.Run();
}

public override void Configure(IApplicationBuilder app)
{
	System.Environment.Exit(1);
}

It seems that in this scenario Set() will no longer be called on the instance of ManualResetEventSlim passed to AttachCtrlcSigtermShutdown(..) resulting in the Shutdown() method to block indefinitely.

https://github.com/aspnet/Hosting/blob/97c9510a95e676324ed8fa03d8b6dcf3b1c19ccf/src/Microsoft.AspNetCore.Hosting/WebHostExtensions.cs#L134-L162

One possible solution could be applying the WebHostOptions.ShutdownTimeout when waiting for the manual reset event.

System Info:

dotnet core: 2.1
Microsoft.AspNetCore.Hosting: 2.1.1

Are you using 2.0 or 2.1?

Have you considered throwing an exception from Startup.Configure rather than calling exit?

Thanks for the quick reply!

I am using 2.1.

I considered throwing exceptions from Startup.Configure, but I am using a JSON-based log-format and would like to keep the stacktrace in one log-entry (rather than plain-text, line-separated).

@Tratcher

@Tratcher a bug?

Yes, with a workaround.

I am encountering the same behaviour. When throwing an exception instead it results in the same behaviour. As strack trace is written and I am stuck at

Application is shutting down...

I am running 2.1 on Linux (Fedora).

I encounter the same behavior in 2.2. I had relied on Environment.Exit in 2.0 on Linux/Windows without problem, but now in 2.2 it hangs on both Windows and Linux. Attached is a debugger which I believe shows the deadlock in the console logger.

I see the same issue on Windows. One workaround appears to be simply P/Invoking the native C exit function in a class somewhere as follows:

private const string NativeCLibName = "ucrtbase.dll"; // assumes Windows; on Mac it will be "libSystem.dylib", on Linux probably "libc.so"

private const string ExitFunctionNativeName = "exit";

[DllImport(NativeCLibName, EntryPoint = ExitFunctionNativeName)]
public static extern void NativeExit(int exitCode);

Then wherever you would have called Environment.Exit, just call NativeExit instead.

If we can reproduce this with the generic host then we can move this to dotnet/runtime. It might require changes there anyways.

This isn't something we want to fix in the webhost, but we should fix this in the generic host.

Here's the repro with the generic host:

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var host = Host.CreateDefaultBuilder()
        .ConfigureServices(services =>
        {
            services.AddHostedService<Worker>();
        })
        .Build();

host.Run();

class Worker : IHostedService
{
    public Task StartAsync(CancellationToken cancellationToken)
    {
        Environment.Exit(1);
        return Task.CompletedTask;
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        return Task.CompletedTask;
    }
}

Results in hanging at this:

@janvorli

@janvorli All of hosting weird designs are around trying to let the program continue execution before sigterm tears the application down (blocking on a wait handle in the event and waiting for the application code to signal it when it's done). Is there anyway we can stop doing this? Any cleaner way the runtime can support this natively without us doing these tricks?

@davidfowl

@davidfowl I am not sure if I fully understand what tricks you are talking about, so let me answer a bit broadly. We've been trying to get rid of all attempts to synchronize with process exiting / cleaning stuff up in the runtime during the past years, as they've often lead to intermittent, sometimes very rare and hard to debug deadlocks during process exit. Or process shutdown delays. On Unix, after the main thread exits, the secondary threads can still be running code (including managed code) for some time until the OS shuts down the process.

As for SIGTERM handling in the runtime, we cannot execute the termination code in the SIGTERM handler directly, as only so called async safe functions can be called from it. The reason is that the signal can be delivered to any thread of the process any time. So for example, if the thread that the signal handler is called on was inside of malloc code and the handler then attempted to allocate memory too, it could lead to deadlock or memory corruption depending on where in the malloc code the thread was running when it was interrupted for the signal. That's why we run it on a special thread and the signal handler just notifies that thread.

Right. What we're trying to do is let the main thread unwind. Many systems use SIGTERM as a way to start a graceful shutdown. To make this work we need to start the shutdown process and wait for some timeframe until it's done. To coordinate that, we set up a bunch of events to allow the main thread to exit before continuing the process exit handler.

Ideally we could have the runtime coordinate this so that we can remove our code (similar to how Console.CancelKeyPress allows cancelling shutdown). Here's a narrowed down piece of code:

using System;
using System.Threading;

namespace ConsoleApp35
{
    class Program
    {
        static void Main(string[] args)
        {
            var waitForProcessShutdownStart = new ManualResetEventSlim();
            var waitForMainExit = new ManualResetEventSlim();

            AppDomain.CurrentDomain.ProcessExit += (sender, e) =>
            {
                // Tell everyone that shutdown has started
                waitForProcessShutdownStart.Set();

                Console.WriteLine("Waiting for main");
                // Don't unwind until main exists
                waitForMainExit.Wait();
            };

            Console.CancelKeyPress += (sender, e) =>
            {
                e.Cancel = true;

                // Tell everyone that shutdown has started
                waitForProcessShutdownStart.Set();
            };

            Console.WriteLine("Waiting for shutdown");
            // Wait for shutdown to start
            waitForProcessShutdownStart.Wait();

            // This is where we do graceful shutdown

            // This will cause the hang because it invokes the process exit handler directly
            // Environment.Exit(1);

            // Now we're done with main, tell the shutdown handler
            waitForMainExit.Set();
        }
    }
}

@maryamariyan

To makes matters more complex, Environment.Exit, synchronously triggering the OnProcessExit callback. In the above scenario this causes a deadlock. The most pragmatic thing we could do here is to timeout the wait so it isn't a permanent deadlock...

cc @maryamariyan

Closing this in favor of #50527

Navigation Menu

Repository navigation

Deadlock when calling System.Environment.Exit(int exitCode) during Startup.Configure(..) #50397

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Deadlock when calling System.Environment.Exit(int exitCode) during Startup.Configure(..) #50397

Description

Activity

Tratcher commented on Jul 6, 2018

armingerten commented on Jul 6, 2018

muratg commented on Aug 28, 2018

Tratcher commented on Aug 29, 2018

b0wter commented on Dec 21, 2018

alex-sherman commented on Aug 29, 2019

ddgcodes commented on May 10, 2020

davidfowl commented on Mar 28, 2021

davidfowl commented on Mar 30, 2021

davidfowl commented on Mar 30, 2021

davidfowl commented on Mar 30, 2021

janvorli commented on Mar 30, 2021

davidfowl commented on Mar 31, 2021

davidfowl commented on Mar 31, 2021

davidfowl commented on Mar 31, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions