Repository navigation
CoreCLR's CNTRL_SHUTDOWN_EVENT handler prevents graceful exit of services during system shutdown #83093
Description
Description
I am not sure if this belongs here, but this is a new weird behaviour I have been fighting with lately.
I have a service, where the 'Main_Worker' is a 'IHostedService' and a 'BackgroundService'. I have overriden the 'StopAsync' function and the only thing it really does is write to the Windows Event Log a message and save a txt file with some data.
While I was working with this structure in .NET 6, it all worked preatty well. Then I upgraded to .Net 7, and it seems that now 'StopAsync' is not triggering when I turn of the PC by a normal Power -> Shutdown or if I use 'Process.Start("cmd", @"/c shutdown -s -t 0");'
When in .NET 6, both ways it worked without problems.
Reproduction Steps
public async override Task<Task> StopAsync(CancellationToken cancellationToken) { Log.Text("Stopping Worker_Main service..."); logger.LogWarning("Test Service is Stopping..."); await Log.SaveCurrentLog(); return base.StopAsync(cancellationToken); }
That is my StopAsync function, it used to work like this with no problems. But now, even if you take away the await line, not even the logger is appearing.
Expected behavior
At least I would expect to see the logged message in the Windows Event Logger when the PC is turned on again.
Actual behavior
No trace of the message being logged or the file being written. The service probably closes, yet I have no idea if it is in a forced mode or if it is actually being closed in a gracefull way.
Regression?
No response
Known Workarounds
I do not have workarounds at the moment.
Configuration
Have tested in Windows 10 & 11
Both are x64
Other information
No response
Activity
gkapellmann commentedon Mar 8, 2023
As a matter of fact, just to give a bit more of context, here is a test I have been doing:
lifetime.ApplicationStarted.Register(() => logger.LogInformation("ApplicationStarted Done...")); lifetime.ApplicationStopping.Register(() => logger.LogInformation("ApplicationStopping Done...")); lifetime.ApplicationStopped.Register(() => logger.LogInformation("ApplicationStopped Done..."));ApplicationStarted appears in the event viewer perfectly well. But the other two are not. This when restaring or shutdown.
But if I stop gracefully the service manually, both messageds appear fine.
buyaa-n commentedon Mar 13, 2023
Looks similar to #62579 and #81274, @gkapellmann could you try the workaround mentioned and let us know if that helps your issue? The workaround is for unblocking you for short term, if you could let us know how it goes, it would be a useful data for to the team to find appropriate fix.
ericstj commentedon Mar 15, 2023
This one is actually different than the error 1067 issue. In that case the service process was exiting before completing ServiceBase's OnStop method which is what notified SCM that the service cleanly stopped. In that case it would run all ApplicationStopping and ApplicationStopped.
In this case the machine is shutting down. I was looking at a similar issue internally and found that the problem was actually due to #41101. During a shutdown a service will first receive a CNTRL_SHUTDOWN_EVENT. The default is to do nothing for a service process per https://learn.microsoft.com/en-us/windows/console/handlerroutine#remarks, but that was changed for .NET processes in #41101.
I was able to workaround this by adding ConsoleControlHandler that cancels other handlers when seeing CNTRL_SHUTDOWN_EVENT - stopping the runtime's handler from shutting down the process and letting the service exit normally. In .NET 6.0 you can do this with a one-liner:
Just make sure that registration stays alive for the entire duration of the service - I put it in the Main() routine.
We should look into adding something like this to ServiceBase -- or having the runtime try to avoid handling CNTRL_SHUTDOWN_EVENT for services.
31 remaining items
janvorli commentedon Apr 10, 2024
The primary issue is that a service should not call ExitProcess when receiving the CTRL_SHUTDOWN_EVENT, as described in https://learn.microsoft.com/en-us/windows/console/handlerroutine#remarks:
The problem got introduced when .NET started to register the control handler in #55333.
To fix that, we should change the behavior of our handler when the .NET app is running as a service. In fact, I've discussed that with Windows developers in the past and they said that the MSDN doc is slightly misleading and this is true for system session, not just services.
janvorli commentedon Apr 10, 2024
I haven't looked into the details of our implementation of the code related to the handler, maybe it would be sufficient to return FALSE from the HandlerRoutine if we detect we are running in the system session at this point:
runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/PosixSignalRegistration.Windows.cs
Lines 83 to 85 in ac07ea6
ericstj commentedon Apr 10, 2024
@janvorli has it right
The problem is the process is getting two different shutdown signals. Services should only be shutdown through SCM - and that's how it works in .NETFramework and earlier .NET versions. It broke with #55333. The Windows default control handler identifies a service process and avoids sending it CTRL_SHUTDOWN_EVENT - the CoreCLR should do the same since it's behaving like the OS and registering a handler for all application types. To behave most like the OS here we should try to identify if the process is a service process - we can't count on it using System.ServiceProcess - some .NET processes opt to PInvoke directly to SCM. If possible we should replicate the OS's detection mechanism.
jkotas commentedon Apr 10, 2024
#50527 and #55333 introduced a low-level signal handling API. I do not think that the introduction of these APIs alone broke anything. The break was introduced as side-effect of ConsoleLifeTime using these APIs in #56057.
It does not make sense for this low-level signal handling API to be opinionated and handle app-model specific concerns, like skipping the handler registration for Windows Services. Windows Services are not the only ones with this type of issue. For example, the Win32 GUI apps should be only shutdown through WM_QUIT message.
I think that the problem is caused by ConsoleLifeTime being used by Windows Services. Windows Services are not console apps, so it does not make sense for them to use Console app model. (And similarly, Win32 GUI apps are not console apps and so they should not use ConsoleLifeTime. Dtto for all other appmodels - WinUI, MAUI, Avalonia, ... .)
janvorli commentedon Apr 10, 2024
It is true that the #55333 on its own without any SIGTERM registered handler doesn't cause the issue and the problem is that the
HandlePosixSignalsetscontext.Cancel=truewhich was introduced by the #56057. So, it seems the fix should be rather in theHandlePosixSignal- to not to set context.Cancel for processes running under system session.janvorli commentedon Apr 10, 2024
Or I guess just not using the ConsoleLifetime. That's what you were trying to say, @jkotas, right?
jkotas commentedon Apr 10, 2024
Right. ConsoleLifetime implements console appmodel that is not compatible with Windows services.
Note that we have SystemdLifetime that handles same concern for Systemd processes ("Unix services"):
runtime/src/libraries/Microsoft.Extensions.Hosting.Systemd/src/SystemdLifetime.netcoreapp.cs
Lines 15 to 18 in d88c7ba
ericstj commentedon May 5, 2025
Folks aren't using
ConsoleLifetimefor windows services, they are usingWindowsServiceLifetimewhich doesn't use Posix registration - but instead uses SCM for these events. I don't think there's anything wrong the user code or the hosting code is doing here. They are expecting for the process to continue running until they can complete the SCM request to shutdown. That's not happening because CoreCLR registered a console handler and the OS's default handler (which avoids shutting down services) is no longer used. It's not running because of some console lifetime registration - the coreCLR registers this for all applications. There is no way for an app to tell the coreCLR not do this, (other than the hack I shared which inserts another handler and avoids calling the coreCLR).runtime/src/coreclr/src/vm/ceemain.cpp
Line 639 in 29bd3c4
runtime/src/coreclr/src/vm/ceemain.cpp
Lines 404 to 408 in 29bd3c4
jkotas commentedon May 5, 2025
Makes sense. It is .NET Core -> .NET Framework regression introduced by dotnet/coreclr#17265.
dotnet/coreclr#17265 was an attempt to fix the console lifetime issues without introducing proper signal handling APIs. It did not work well, and we ended up introducing proper signal handling APIs anyway.
Should we revert dotnet/coreclr#17265 and go back to .NET Framework behavior so that we are not baking app-model specific concerns into runtime?
jkotas commentedon May 16, 2025
Fixed by #115494