Better Dangling snapshots detection

docs/dangling-files.md

@@ -12,13 +12,42 @@ A dangling snapshot file are when a `.verified.` file exist with no correspondin
 
 ## How dangling snapshot checks works
 
- * When each test is executed, any snapshots produced are recorded.
- * After all tests are executed, all recorded snapshots are checked against the snapshots that exist on disk
+ * When each test is executed, two things are recorded: the snapshots it produced, and the name it produces them under, up to the end of the type and method. The name is recorded even when the test produces no snapshot at all, so an inline test, or one that threw, still counts as existing.
+ * After all tests are executed, the snapshots on disk are checked against both.
  * An exception is thrown if any files:
    * exist on disk and do not have a corresponding recorded test
    * have casing that does not match the test case
 
 
+## Which run a snapshot belongs to
+
+A snapshot file the current run did not produce is not necessarily redundant. It can belong to another target framework, another OS, or another architecture, none of which are running. The two recorded halves answer different parts of that question.
+
+A file whose name starts with no recorded test name belongs to no run at all. The set of tests in an assembly does not vary by framework, OS or architecture, so a snapshot no test claims is redundant however much uniqueness its name carries. `SomeTests.Deleted.DotNet9_0.verified.txt` is reported as readily as `SomeTests.Deleted.verified.txt`.
+
+A file whose name does start with a recorded test name is a variant of a live test, and what follows that name decides it. Everything between the test name and the `.verified.` marker is split into segments and each is compared, whole, against the values [`Namer`](https://github.com/VerifyTests/Verify/blob/main/src/Verify/Naming/Namer.cs) produces for this run. A segment naming a different OS or architecture belongs to a run on another machine and is left alone. A segment naming a different target framework is covered below. Anything else is not uniqueness, so the file is reported.
+
+Matching whole segments is what separates a uniqueness segment from a name that merely starts the same way: `SomeTests.NetworkClient.verified.txt` is a test called `NetworkClient`, not a `Net` uniqueness segment.
+
+
+## Multi targeted projects
+
+A multi targeted project runs its tests once per target framework, so most of the snapshots on disk during any one run belong to a framework that is not currently running. A `UniqueForRuntime` or `UniqueForTargetFramework` snapshot of a deleted test is indistinguishable, by name, from one that another framework still owns.
+
+To tell them apart, each run records what it tracked to a manifest in the intermediate (obj) directory. The manifests are named after the target framework and share one directory across all frameworks of the project, so a run can read what the other runs tracked. Once every target framework has a manifest, the union of them is the complete set of snapshot files the project owns, and a target framework segment no longer excuses a file: nothing produces it, so it is dangling.
+
+Both recorded halves go in the manifest. The test names matter across frameworks as much as the files do: a test behind an `#if NET48` exists only in the net48 run, and without its name in the union every other run would report its snapshots.
+
+In a multi targeted run this means the check is at its most accurate on the last framework to run: the earlier runs cannot yet account for the frameworks still to come, and leave the target framework segments alone. Snapshots of deleted tests are reported by every run, since no manifest is needed to know that no test claims them.
+
+The manifests are scoped to the build configuration, and are ignored if they predate the assembly running the check, so a stale manifest cannot mask a dangling file. They live in obj and are removed by a clean.
+
+Two axes cannot be settled this way:
+
+ * `UniqueForOSPlatform` and `UniqueForArchitecture`, since the runs that produce those files are on other machines with their own intermediate directories. A segment naming an OS or architecture other than the current one is always left alone.
+ * `UniqueForAssemblyConfiguration`, which has no enumerable set of values and so is not recognised as uniqueness at all. A snapshot only another configuration produces is reported.
+
+
 ## Experimental
 
 `DanglingSnapshots` is an experimental feature (marked with `[Experimental("VerifyDanglingSnapshots")]`) and is subject to change in minor version.
@@ -80,6 +109,8 @@ public static class SetUp
 <sup><a href='/src/DanglingSnapshotsNUnitUsage/DanglingSnapshots.cs#L1-L9' title='Snippet source file'>snippet source</a> | <a href='#snippet-DanglingSnapshotsNUnitUsage/DanglingSnapshots.cs' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
+The NUnit runner prints a teardown failure but does not count it, so the run is summarised as passed. To keep a dangling snapshot from going unnoticed, the NUnit integration sets the process exit code to 1 when the check fails, and writes the reason after the runner's summary.
+
 
 ### XUnitV3
 
@@ -118,3 +149,82 @@ public class Tests
 ```
 <sup><a href='/src/DanglingSnapshotsXunitV3Usage/Tests.cs#L1-L8' title='Snippet source file'>snippet source</a> | <a href='#snippet-XunitV3DanglingCollection' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
+
+
+### TUnit
+
+Use the `[After(TestSession)]` feature:
+
+<!-- snippet: DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs -->
+<a id='snippet-DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs'></a>
+```cs
+#pragma warning disable VerifyDanglingSnapshots
+
+public static class Cleanup
+{
+    [After(TestSession)]
+    public static void Run() =>
+        DanglingSnapshots.Run();
+}
+```
+<sup><a href='/src/DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs#L1-L8' title='Snippet source file'>snippet source</a> | <a href='#snippet-DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs' title='Start of snippet'>anchor</a></sup>
+<!-- endSnippet -->
+
+
+### Expecto
+
+Expecto test projects are console applications, so the check goes in the entry point, after the run:
+
+<!-- snippet: DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs -->
+<a id='snippet-DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs'></a>
+```cs
+#pragma warning disable VerifyDanglingSnapshots
+
+var result = Runner.RunTestsInAssemblyWithCLIArgs([], args);
+
+DanglingSnapshots.Run();
+
+return result;
+```
+<sup><a href='/src/DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs#L1-L7' title='Snippet source file'>snippet source</a> | <a href='#snippet-DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs' title='Start of snippet'>anchor</a></sup>
+<!-- endSnippet -->
+
+
+### Fixie
+
+Fixie already requires an `IExecution` implementation for Verify. Add the check to the end of `Run`:
+
+<!-- snippet: DanglingSnapshotsFixieUsage/DanglingSnapshots.cs -->
+<a id='snippet-DanglingSnapshotsFixieUsage/DanglingSnapshots.cs'></a>
+```cs
+#pragma warning disable VerifyDanglingSnapshots
+
+public class TestProject :
+    ITestProject,
+    IExecution
+{
+    public void Configure(TestConfiguration configuration, TestEnvironment environment)
+    {
+        VerifierSettings.AssignTargetAssembly(environment.Assembly);
+        configuration.Conventions.Add<DefaultDiscovery, TestProject>();
+    }
+
+    public async Task Run(TestSuite testSuite)
+    {
+        foreach (var testClass in testSuite.TestClasses)
+        {
+            foreach (var test in testClass.Tests)
+            {
+                using (ExecutionState.Set(testClass, test, null))
+                {
+                    await test.Run();
+                }
+            }
+        }
+
+        DanglingSnapshots.Run();
+    }
+}
+```
+<sup><a href='/src/DanglingSnapshotsFixieUsage/DanglingSnapshots.cs#L1-L28' title='Snippet source file'>snippet source</a> | <a href='#snippet-DanglingSnapshotsFixieUsage/DanglingSnapshots.cs' title='Start of snippet'>anchor</a></sup>
+<!-- endSnippet -->

docs/mdsource/dangling-files.source.md

@@ -5,13 +5,42 @@ A dangling snapshot file are when a `.verified.` file exist with no correspondin
 
 ## How dangling snapshot checks works
 
- * When each test is executed, any snapshots produced are recorded.
- * After all tests are executed, all recorded snapshots are checked against the snapshots that exist on disk
+ * When each test is executed, two things are recorded: the snapshots it produced, and the name it produces them under, up to the end of the type and method. The name is recorded even when the test produces no snapshot at all, so an inline test, or one that threw, still counts as existing.
+ * After all tests are executed, the snapshots on disk are checked against both.
  * An exception is thrown if any files:
    * exist on disk and do not have a corresponding recorded test
    * have casing that does not match the test case
 
 
+## Which run a snapshot belongs to
+
+A snapshot file the current run did not produce is not necessarily redundant. It can belong to another target framework, another OS, or another architecture, none of which are running. The two recorded halves answer different parts of that question.
+
+A file whose name starts with no recorded test name belongs to no run at all. The set of tests in an assembly does not vary by framework, OS or architecture, so a snapshot no test claims is redundant however much uniqueness its name carries. `SomeTests.Deleted.DotNet9_0.verified.txt` is reported as readily as `SomeTests.Deleted.verified.txt`.
+
+A file whose name does start with a recorded test name is a variant of a live test, and what follows that name decides it. Everything between the test name and the `.verified.` marker is split into segments and each is compared, whole, against the values [`Namer`](https://github.com/VerifyTests/Verify/blob/main/src/Verify/Naming/Namer.cs) produces for this run. A segment naming a different OS or architecture belongs to a run on another machine and is left alone. A segment naming a different target framework is covered below. Anything else is not uniqueness, so the file is reported.
+
+Matching whole segments is what separates a uniqueness segment from a name that merely starts the same way: `SomeTests.NetworkClient.verified.txt` is a test called `NetworkClient`, not a `Net` uniqueness segment.
+
+
+## Multi targeted projects
+
+A multi targeted project runs its tests once per target framework, so most of the snapshots on disk during any one run belong to a framework that is not currently running. A `UniqueForRuntime` or `UniqueForTargetFramework` snapshot of a deleted test is indistinguishable, by name, from one that another framework still owns.
+
+To tell them apart, each run records what it tracked to a manifest in the intermediate (obj) directory. The manifests are named after the target framework and share one directory across all frameworks of the project, so a run can read what the other runs tracked. Once every target framework has a manifest, the union of them is the complete set of snapshot files the project owns, and a target framework segment no longer excuses a file: nothing produces it, so it is dangling.
+
+Both recorded halves go in the manifest. The test names matter across frameworks as much as the files do: a test behind an `#if NET48` exists only in the net48 run, and without its name in the union every other run would report its snapshots.
+
+In a multi targeted run this means the check is at its most accurate on the last framework to run: the earlier runs cannot yet account for the frameworks still to come, and leave the target framework segments alone. Snapshots of deleted tests are reported by every run, since no manifest is needed to know that no test claims them.
+
+The manifests are scoped to the build configuration, and are ignored if they predate the assembly running the check, so a stale manifest cannot mask a dangling file. They live in obj and are removed by a clean.
+
+Two axes cannot be settled this way:
+
+ * `UniqueForOSPlatform` and `UniqueForArchitecture`, since the runs that produce those files are on other machines with their own intermediate directories. A segment naming an OS or architecture other than the current one is always left alone.
+ * `UniqueForAssemblyConfiguration`, which has no enumerable set of values and so is not recognised as uniqueness at all. A snapshot only another configuration produces is reported.
+
+
 ## Experimental
 
 `DanglingSnapshots` is an experimental feature (marked with `[Experimental("VerifyDanglingSnapshots")]`) and is subject to change in minor version.
@@ -46,6 +75,8 @@ use the `[OneTimeTearDown]` feature:
 
 snippet: DanglingSnapshotsNUnitUsage/DanglingSnapshots.cs
 
+The NUnit runner prints a teardown failure but does not count it, so the run is summarised as passed. To keep a dangling snapshot from going unnoticed, the NUnit integration sets the process exit code to 1 when the check fails, and writes the reason after the runner's summary.
+
 
 ### XUnitV3
 
@@ -56,3 +87,24 @@ snippet: DanglingSnapshotsXUnitV3Usage/DanglingSnapshots.cs
 Apply that collection to all tests:
 
 snippet: XunitV3DanglingCollection
+
+
+### TUnit
+
+Use the `[After(TestSession)]` feature:
+
+snippet: DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs
+
+
+### Expecto
+
+Expecto test projects are console applications, so the check goes in the entry point, after the run:
+
+snippet: DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs
+
+
+### Fixie
+
+Fixie already requires an `IExecution` implementation for Verify. Add the check to the end of `Run`:
+
+snippet: DanglingSnapshotsFixieUsage/DanglingSnapshots.cs

src/DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs

@@ -0,0 +1,7 @@
+#pragma warning disable VerifyDanglingSnapshots
+
+var result = Runner.RunTestsInAssemblyWithCLIArgs([], args);
+
+DanglingSnapshots.Run();
+
+return result;

src/DanglingSnapshotsExpectoUsage/DanglingSnapshotsExpectoUsage.csproj

@@ -0,0 +1,23 @@
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <TargetFramework>net11.0</TargetFramework>
+    <GenerateAssemblyInfo>true</GenerateAssemblyInfo>
+    <IsPackageProject>false</IsPackageProject>
+    <!-- The other usage projects get this from their test SDK, which marks them as test projects.
+         An Expecto project is a plain console application, so nothing sets it, and the defaults
+         would otherwise try to pack this sample as a NuGet package. -->
+    <IsPackable>false</IsPackable>
+    <DisableImplicitNamespaceImports>true</DisableImplicitNamespaceImports>
+    <OutputType>Exe</OutputType>
+    <SignAssembly>False</SignAssembly>
+  </PropertyGroup>
+  <ItemGroup>
+    <PackageReference Include="Expecto" />
+    <PackageReference Include="FSharp.Core" />
+    <PackageReference Include="ProjectDefaults" PrivateAssets="all" />
+    <ProjectReference Include="..\Verify.Expecto\Verify.Expecto.csproj" />
+    <ProjectReference Include="..\Verify\Verify.csproj" />
+  </ItemGroup>
+  <Import Project="$(ProjectDir)..\Verify\buildTransitive\Verify.props" />
+  <Import Project="$(ProjectDir)..\Verify.Expecto\buildTransitive\Verify.Expecto.props" />
+</Project>

src/DanglingSnapshotsExpectoUsage/Tests.cs

@@ -0,0 +1,18 @@
+using Expecto;
+
+public class Tests
+{
+    [Tests]
+    public static Test Simple = Runner.TestCase(
+        nameof(Simple),
+        () => Verify(
+            name: nameof(Simple),
+            target: "Foo"));
+
+    [Tests]
+    public static Test Second = Runner.TestCase(
+        nameof(Second),
+        () => Verify(
+            name: nameof(Second),
+            target: "Foo"));
+}

src/DanglingSnapshotsFixieUsage/DanglingSnapshots.cs

@@ -0,0 +1,28 @@
+#pragma warning disable VerifyDanglingSnapshots
+
+public class TestProject :
+    ITestProject,
+    IExecution
+{
+    public void Configure(TestConfiguration configuration, TestEnvironment environment)
+    {
+        VerifierSettings.AssignTargetAssembly(environment.Assembly);
+        configuration.Conventions.Add<DefaultDiscovery, TestProject>();
+    }
+
+    public async Task Run(TestSuite testSuite)
+    {
+        foreach (var testClass in testSuite.TestClasses)
+        {
+            foreach (var test in testClass.Tests)
+            {
+                using (ExecutionState.Set(testClass, test, null))
+                {
+                    await test.Run();
+                }
+            }
+        }
+
+        DanglingSnapshots.Run();
+    }
+}

src/DanglingSnapshotsFixieUsage/DanglingSnapshotsFixieUsage.csproj

@@ -0,0 +1,17 @@
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <TargetFramework>net11.0</TargetFramework>
+    <GenerateAssemblyInfo>true</GenerateAssemblyInfo>
+    <IsPackageProject>false</IsPackageProject>
+    <DisableImplicitNamespaceImports>true</DisableImplicitNamespaceImports>
+    <SignAssembly>false</SignAssembly>
+  </PropertyGroup>
+  <ItemGroup>
+    <PackageReference Include="Fixie.TestAdapter" />
+    <PackageReference Include="ProjectDefaults" PrivateAssets="all" />
+    <ProjectReference Include="..\Verify.Fixie\Verify.Fixie.csproj" />
+    <ProjectReference Include="..\Verify\Verify.csproj" />
+  </ItemGroup>
+  <Import Project="$(ProjectDir)..\Verify\buildTransitive\Verify.props" />
+  <Import Project="$(ProjectDir)..\Verify.Fixie\buildTransitive\Verify.Fixie.props" />
+</Project>

src/DanglingSnapshotsFixieUsage/Tests.cs

@@ -0,0 +1,8 @@
+public class Tests
+{
+    public Task Simple() =>
+        Verify("Foo");
+
+    public Task Second() =>
+        Verify("Foo");
+}

src/DanglingSnapshotsMSTestUsage/AssemblyInfo.cs

@@ -1,5 +1,5 @@
 [assembly: Parallelize]
 // Without this every verification fails with "TestContext is null". Applied to the assembly rather
 // than the test class so that building this project also covers the generator skipping the static
-// [TestClass] that hosts the [AssemblyCleanup] below.
+// [TestClass] in DanglingSnapshots.cs, which an assembly wide attribute also reaches.
 [assembly: UsesVerify]

src/DanglingSnapshotsMSTestUsage/Tests.cs

@@ -6,6 +6,6 @@ public Task Simple() =>
         Verify("Foo");
 
     [TestMethod]
-    public Task IncorrectCase() =>
+    public Task Second() =>
         Verify("Foo");
 }

src/DanglingSnapshotsNUnitUsage/Tests.cs

@@ -6,6 +6,6 @@ public Task Simple() =>
         Verify("Foo");
 
     [Test]
-    public Task IncorrectCase() =>
+    public Task Second() =>
         Verify("Foo");
 }

src/DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs

@@ -0,0 +1,8 @@
+#pragma warning disable VerifyDanglingSnapshots
+
+public static class Cleanup
+{
+    [After(TestSession)]
+    public static void Run() =>
+        DanglingSnapshots.Run();
+}

src/DanglingSnapshotsTUnitUsage/DanglingSnapshotsTUnitUsage.csproj

@@ -0,0 +1,17 @@
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <TargetFramework>net11.0</TargetFramework>
+    <GenerateAssemblyInfo>true</GenerateAssemblyInfo>
+    <IsPackageProject>false</IsPackageProject>
+    <DisableImplicitNamespaceImports>true</DisableImplicitNamespaceImports>
+    <NoWarn>$(NoWarn);CA1822</NoWarn>
+  </PropertyGroup>
+  <ItemGroup>
+    <PackageReference Include="TUnit" />
+    <PackageReference Include="ProjectDefaults" PrivateAssets="all" />
+    <ProjectReference Include="..\Verify.TUnit\Verify.TUnit.csproj" />
+    <ProjectReference Include="..\Verify\Verify.csproj" />
+  </ItemGroup>
+  <Import Project="$(ProjectDir)..\Verify\buildTransitive\Verify.props" />
+  <Import Project="$(ProjectDir)..\Verify.TUnit\buildTransitive\Verify.TUnit.props" />
+</Project>

src/DanglingSnapshotsTUnitUsage/Tests.Second.verified.txt

@@ -0,0 +1 @@
+Foo

src/DanglingSnapshotsTUnitUsage/Tests.Simple.verified.txt

@@ -0,0 +1 @@
+Foo

src/DanglingSnapshotsTUnitUsage/Tests.cs

@@ -0,0 +1,10 @@
+public class Tests
+{
+    [Test]
+    public Task Simple() =>
+        Verify("Foo");
+
+    [Test]
+    public Task Second() =>
+        Verify("Foo");
+}

src/DanglingSnapshotsXunitV3Usage/Tests.Second.verified.txt

@@ -0,0 +1 @@
+Foo