Repository navigation
Windows service infrequently results in error 1067 on stop operation #62579
Description
Description
.NET 6 published application sometimes crashes on service stop operation. I created a web application from the template and only added the Microsoft.Extensions.Hosting.WindowsServices (version 6.0.0) package and called UseWindowsService() to make sure the application listens for service controller requests.
Here is the code:
using Microsoft.Extensions.Hosting.WindowsServices;
var options = new WebApplicationOptions
{
Args = args,
ContentRootPath = WindowsServiceHelpers.IsWindowsService() ? AppContext.BaseDirectory : default
};
var builder = WebApplication.CreateBuilder(options);
// Add services to the container.
builder.Services.AddRazorPages();
builder.Host.UseWindowsService();
var app = builder.Build();
// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
app.UseExceptionHandler("/Error");
}
app.UseStaticFiles();
app.UseRouting();
app.UseAuthorization();
app.MapRazorPages();
app.Run();Reproduction Steps
Repository with code, script and repro steps: https://github.com/zryska/net6-repro
- Create a new Web application using a template
- Make the changes to add UseWindowsService() call as shown above
- Publish the application as SFD or FDD for win-x64 runtime
- Create a new service using
sc createcommand - Start the service and verify it's running
- Stop the service
- Start the service
- Repeat steps 4 and 5 until the issue is reproduced
It can be done through the services UI or using a script. On my system it usually happens 1 time out of 10 graceful shutdowns - but it isn't deterministic.
When done through services UI the following will be displayed:
Even log => Windows Logs -> System will contain the following message (when done through script or UI):
The .NET6 test service service terminated unexpectedly. It has done this 26 time(s).
Expected behavior
Service stops gracefully every time.
Actual behavior
Non-graceful shutdowns are happening.
Regression?
No response
Known Workarounds
Explicitly assigning value to ServiceName fixes the problem.
builder.Host.UseWindowsService(o => o.ServiceName = "AnyServiceName");
Configuration
.NET version: .NET 6.0.100
OS version: Windows 10 pro build 19044.1348
Architecture: x64
Other information
No response
Activity
eerhardt commentedon Dec 9, 2021
Any chance you can get more information here to narrow down where the problem is? Maybe try getting a crash dump.
https://github.com/dotnet/diagnostics/blob/main/documentation/dotnet-dump-instructions.md
Or adding a
try-catchsomewhere in your project to try to catch the exception.Can you attach a Debugger before stopping the service? If you do, does it still crash?
zryska commentedon Dec 9, 2021
I wasn't able to get a crash dump generated for some reason. Also when I included
try-catchblock around the main method it didn't catch anything. The application seems to be crashing after it leaves the main method. When I run the same code on Linux just withsystemdpackage instead ofWindowsServicesthe same code works just fine.eerhardt commentedon Dec 9, 2021
Your best bet here is to get more information - what is the stacktrace / message / etc that is causing the process to crash.
zryska commentedon Dec 9, 2021
The issue is not surfacing when I have debugger attached. Would you be able to give it a try on your end by following the repro steps or is there something else I could provide in terms of repro to make it easier?
eerhardt commentedon Dec 9, 2021
Was dotnet-dump not working for you? What was the reason?
zryska commentedon Dec 9, 2021
My understanding was that
dotnet-dumpallows only to generate a dump at time of executing the collect command. I was using SysInternalsProcDumpbut it didn't create any crash dump even when service manager reported crash.This is procdump output from event when service controller reported ungraceful shutdown:
eerhardt commentedon Dec 9, 2021
There's no dump files in that folder?
Can you try https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps? Or possibly https://docs.microsoft.com/en-us/visualstudio/debugger/debug-using-the-just-in-time-debugger?view=vs-2022#BKMK_Enabling?
zryska commentedon Dec 9, 2021
No. No dumps were created. I should also mention that the main method apparently exists gracefully since in case I add a line of code to create a file before exiting main the file gets created even when the service control manager reports that the application crashed. I was trying the registry approach in the past when I was suspecting that something is wrong in our code application without results but I can give it a try again with the empty WebApplication project.
davidfowl commentedon Dec 10, 2021
Look at the event log to se if there's more information.
zryska commentedon Dec 10, 2021
Hi David, unfortunately the event log doesn't contain anything actionable. The alleged crash event is logged in the system log and there is nothing in the Application log around that time.
Here is the raw event (removed computer name) in case there is something I missed:
zryska commentedon Dec 10, 2021
I tried to replicate it on another machine with lower HW specs:
CPU: 4 cores
RAM: 8 GB
OS: Windows 10 Enterprise build: 19042.1348
It still happened but much less frequently. I let the script run for a while and out of 1000 restarts there were just two "crash" events reported in the event log.
For comparison my development machine HW specs:
CPU: 16 cores
RAM: 32 GB
OS: Windows 10 Pro build: 19044.1348
I moved the repro code into a separate repository if it helps: https://github.com/zryska/net6-repro
OwnageIsMagic commentedon Dec 12, 2021
@zryska use
procdump64.exe -mp -t -e 1 -x . "WebApplication1.exe"27 remaining items
ericstj commentedon Mar 24, 2023
We should prioritize the fix for this and consider backporting to .NET 7.0 and 6.0. The workaround to wait some time before exiting doesn't sufficiently solve the problem, because it doesn't hold up the progress of
runtime/src/libraries/Microsoft.Extensions.Hosting.Abstractions/src/HostingAbstractionsHostExtensions.cs
Line 75 in 967250c
Which will dispose the service and make it impossible for the service state to be updated.
@buyaa-n as we discussed, we can try the fix of signaling an in Run when ServiceBase.Run(..) returns, then waiting on this in StopAsync. This will ensure that the service has safely exited.
ericstj commentedon Apr 6, 2023
Keeping open for servicing.
MD-V commentedon Apr 21, 2023
@ericstj would be awesome if this would be backported to .NET 6 and 7. We are currently experiencing this exact problem with one of our applications.
ericstj commentedon Apr 21, 2023
Yes, that’s the plan. Most likely the June release.
If you have the ability to test daily bits you can try the net8.0 package to confirm the fix works for you. That can be restored from
runtime/NuGet.config
Line 22 in f107b4b
If you are able to try and confirm the fix that helps us ensure we got things right. Thanks!
FlorianAlbert commentedon Apr 27, 2023
@ericstj I tried the fix on behalf of @MD-V and so far everything seems to work properly.