diff --git a/docs/dangling-files.md b/docs/dangling-files.md index 2efb8bf5fb..fd69333475 100644 --- a/docs/dangling-files.md +++ b/docs/dangling-files.md @@ -12,13 +12,42 @@ A dangling snapshot file are when a `.verified.` file exist with no correspondin ## How dangling snapshot checks works - * When each test is executed, any snapshots produced are recorded. - * After all tests are executed, all recorded snapshots are checked against the snapshots that exist on disk + * When each test is executed, two things are recorded: the snapshots it produced, and the name it produces them under, up to the end of the type and method. The name is recorded even when the test produces no snapshot at all, so an inline test, or one that threw, still counts as existing. + * After all tests are executed, the snapshots on disk are checked against both. * An exception is thrown if any files: * exist on disk and do not have a corresponding recorded test * have casing that does not match the test case +## Which run a snapshot belongs to + +A snapshot file the current run did not produce is not necessarily redundant. It can belong to another target framework, another OS, or another architecture, none of which are running. The two recorded halves answer different parts of that question. + +A file whose name starts with no recorded test name belongs to no run at all. The set of tests in an assembly does not vary by framework, OS or architecture, so a snapshot no test claims is redundant however much uniqueness its name carries. `SomeTests.Deleted.DotNet9_0.verified.txt` is reported as readily as `SomeTests.Deleted.verified.txt`. + +A file whose name does start with a recorded test name is a variant of a live test, and what follows that name decides it. Everything between the test name and the `.verified.` marker is split into segments and each is compared, whole, against the values [`Namer`](https://github.com/VerifyTests/Verify/blob/main/src/Verify/Naming/Namer.cs) produces for this run. A segment naming a different OS or architecture belongs to a run on another machine and is left alone. A segment naming a different target framework is covered below. Anything else is not uniqueness, so the file is reported. + +Matching whole segments is what separates a uniqueness segment from a name that merely starts the same way: `SomeTests.NetworkClient.verified.txt` is a test called `NetworkClient`, not a `Net` uniqueness segment. + + +## Multi targeted projects + +A multi targeted project runs its tests once per target framework, so most of the snapshots on disk during any one run belong to a framework that is not currently running. A `UniqueForRuntime` or `UniqueForTargetFramework` snapshot of a deleted test is indistinguishable, by name, from one that another framework still owns. + +To tell them apart, each run records what it tracked to a manifest in the intermediate (obj) directory. The manifests are named after the target framework and share one directory across all frameworks of the project, so a run can read what the other runs tracked. Once every target framework has a manifest, the union of them is the complete set of snapshot files the project owns, and a target framework segment no longer excuses a file: nothing produces it, so it is dangling. + +Both recorded halves go in the manifest. The test names matter across frameworks as much as the files do: a test behind an `#if NET48` exists only in the net48 run, and without its name in the union every other run would report its snapshots. + +In a multi targeted run this means the check is at its most accurate on the last framework to run: the earlier runs cannot yet account for the frameworks still to come, and leave the target framework segments alone. Snapshots of deleted tests are reported by every run, since no manifest is needed to know that no test claims them. + +The manifests are scoped to the build configuration, and are ignored if they predate the assembly running the check, so a stale manifest cannot mask a dangling file. They live in obj and are removed by a clean. + +Two axes cannot be settled this way: + + * `UniqueForOSPlatform` and `UniqueForArchitecture`, since the runs that produce those files are on other machines with their own intermediate directories. A segment naming an OS or architecture other than the current one is always left alone. + * `UniqueForAssemblyConfiguration`, which has no enumerable set of values and so is not recognised as uniqueness at all. A snapshot only another configuration produces is reported. + + ## Experimental `DanglingSnapshots` is an experimental feature (marked with `[Experimental("VerifyDanglingSnapshots")]`) and is subject to change in minor version. @@ -80,6 +109,8 @@ public static class SetUp snippet source | anchor +The NUnit runner prints a teardown failure but does not count it, so the run is summarised as passed. To keep a dangling snapshot from going unnoticed, the NUnit integration sets the process exit code to 1 when the check fails, and writes the reason after the runner's summary. + ### XUnitV3 @@ -118,3 +149,82 @@ public class Tests ``` snippet source | anchor + + +### TUnit + +Use the `[After(TestSession)]` feature: + + + +```cs +#pragma warning disable VerifyDanglingSnapshots + +public static class Cleanup +{ + [After(TestSession)] + public static void Run() => + DanglingSnapshots.Run(); +} +``` +snippet source | anchor + + + +### Expecto + +Expecto test projects are console applications, so the check goes in the entry point, after the run: + + + +```cs +#pragma warning disable VerifyDanglingSnapshots + +var result = Runner.RunTestsInAssemblyWithCLIArgs([], args); + +DanglingSnapshots.Run(); + +return result; +``` +snippet source | anchor + + + +### Fixie + +Fixie already requires an `IExecution` implementation for Verify. Add the check to the end of `Run`: + + + +```cs +#pragma warning disable VerifyDanglingSnapshots + +public class TestProject : + ITestProject, + IExecution +{ + public void Configure(TestConfiguration configuration, TestEnvironment environment) + { + VerifierSettings.AssignTargetAssembly(environment.Assembly); + configuration.Conventions.Add(); + } + + public async Task Run(TestSuite testSuite) + { + foreach (var testClass in testSuite.TestClasses) + { + foreach (var test in testClass.Tests) + { + using (ExecutionState.Set(testClass, test, null)) + { + await test.Run(); + } + } + } + + DanglingSnapshots.Run(); + } +} +``` +snippet source | anchor + diff --git a/docs/mdsource/dangling-files.source.md b/docs/mdsource/dangling-files.source.md index cdcd7c89d1..4915036d8e 100644 --- a/docs/mdsource/dangling-files.source.md +++ b/docs/mdsource/dangling-files.source.md @@ -5,13 +5,42 @@ A dangling snapshot file are when a `.verified.` file exist with no correspondin ## How dangling snapshot checks works - * When each test is executed, any snapshots produced are recorded. - * After all tests are executed, all recorded snapshots are checked against the snapshots that exist on disk + * When each test is executed, two things are recorded: the snapshots it produced, and the name it produces them under, up to the end of the type and method. The name is recorded even when the test produces no snapshot at all, so an inline test, or one that threw, still counts as existing. + * After all tests are executed, the snapshots on disk are checked against both. * An exception is thrown if any files: * exist on disk and do not have a corresponding recorded test * have casing that does not match the test case +## Which run a snapshot belongs to + +A snapshot file the current run did not produce is not necessarily redundant. It can belong to another target framework, another OS, or another architecture, none of which are running. The two recorded halves answer different parts of that question. + +A file whose name starts with no recorded test name belongs to no run at all. The set of tests in an assembly does not vary by framework, OS or architecture, so a snapshot no test claims is redundant however much uniqueness its name carries. `SomeTests.Deleted.DotNet9_0.verified.txt` is reported as readily as `SomeTests.Deleted.verified.txt`. + +A file whose name does start with a recorded test name is a variant of a live test, and what follows that name decides it. Everything between the test name and the `.verified.` marker is split into segments and each is compared, whole, against the values [`Namer`](https://github.com/VerifyTests/Verify/blob/main/src/Verify/Naming/Namer.cs) produces for this run. A segment naming a different OS or architecture belongs to a run on another machine and is left alone. A segment naming a different target framework is covered below. Anything else is not uniqueness, so the file is reported. + +Matching whole segments is what separates a uniqueness segment from a name that merely starts the same way: `SomeTests.NetworkClient.verified.txt` is a test called `NetworkClient`, not a `Net` uniqueness segment. + + +## Multi targeted projects + +A multi targeted project runs its tests once per target framework, so most of the snapshots on disk during any one run belong to a framework that is not currently running. A `UniqueForRuntime` or `UniqueForTargetFramework` snapshot of a deleted test is indistinguishable, by name, from one that another framework still owns. + +To tell them apart, each run records what it tracked to a manifest in the intermediate (obj) directory. The manifests are named after the target framework and share one directory across all frameworks of the project, so a run can read what the other runs tracked. Once every target framework has a manifest, the union of them is the complete set of snapshot files the project owns, and a target framework segment no longer excuses a file: nothing produces it, so it is dangling. + +Both recorded halves go in the manifest. The test names matter across frameworks as much as the files do: a test behind an `#if NET48` exists only in the net48 run, and without its name in the union every other run would report its snapshots. + +In a multi targeted run this means the check is at its most accurate on the last framework to run: the earlier runs cannot yet account for the frameworks still to come, and leave the target framework segments alone. Snapshots of deleted tests are reported by every run, since no manifest is needed to know that no test claims them. + +The manifests are scoped to the build configuration, and are ignored if they predate the assembly running the check, so a stale manifest cannot mask a dangling file. They live in obj and are removed by a clean. + +Two axes cannot be settled this way: + + * `UniqueForOSPlatform` and `UniqueForArchitecture`, since the runs that produce those files are on other machines with their own intermediate directories. A segment naming an OS or architecture other than the current one is always left alone. + * `UniqueForAssemblyConfiguration`, which has no enumerable set of values and so is not recognised as uniqueness at all. A snapshot only another configuration produces is reported. + + ## Experimental `DanglingSnapshots` is an experimental feature (marked with `[Experimental("VerifyDanglingSnapshots")]`) and is subject to change in minor version. @@ -46,6 +75,8 @@ use the `[OneTimeTearDown]` feature: snippet: DanglingSnapshotsNUnitUsage/DanglingSnapshots.cs +The NUnit runner prints a teardown failure but does not count it, so the run is summarised as passed. To keep a dangling snapshot from going unnoticed, the NUnit integration sets the process exit code to 1 when the check fails, and writes the reason after the runner's summary. + ### XUnitV3 @@ -56,3 +87,24 @@ snippet: DanglingSnapshotsXUnitV3Usage/DanglingSnapshots.cs Apply that collection to all tests: snippet: XunitV3DanglingCollection + + +### TUnit + +Use the `[After(TestSession)]` feature: + +snippet: DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs + + +### Expecto + +Expecto test projects are console applications, so the check goes in the entry point, after the run: + +snippet: DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs + + +### Fixie + +Fixie already requires an `IExecution` implementation for Verify. Add the check to the end of `Run`: + +snippet: DanglingSnapshotsFixieUsage/DanglingSnapshots.cs diff --git a/src/DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs b/src/DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs new file mode 100644 index 0000000000..e95d0a12d7 --- /dev/null +++ b/src/DanglingSnapshotsExpectoUsage/DanglingSnapshots.cs @@ -0,0 +1,7 @@ +#pragma warning disable VerifyDanglingSnapshots + +var result = Runner.RunTestsInAssemblyWithCLIArgs([], args); + +DanglingSnapshots.Run(); + +return result; diff --git a/src/DanglingSnapshotsExpectoUsage/DanglingSnapshotsExpectoUsage.csproj b/src/DanglingSnapshotsExpectoUsage/DanglingSnapshotsExpectoUsage.csproj new file mode 100644 index 0000000000..92ebfb1620 --- /dev/null +++ b/src/DanglingSnapshotsExpectoUsage/DanglingSnapshotsExpectoUsage.csproj @@ -0,0 +1,23 @@ + + + net11.0 + true + false + + false + true + Exe + False + + + + + + + + + + + diff --git a/src/DanglingSnapshotsMSTestUsage/Tests.Dangling.verified.txt b/src/DanglingSnapshotsExpectoUsage/Tests.Second.verified.txt similarity index 100% rename from src/DanglingSnapshotsMSTestUsage/Tests.Dangling.verified.txt rename to src/DanglingSnapshotsExpectoUsage/Tests.Second.verified.txt diff --git a/src/DanglingSnapshotsMSTestUsage/Tests.Incorrectcase.verified.txt b/src/DanglingSnapshotsExpectoUsage/Tests.Simple.verified.txt similarity index 100% rename from src/DanglingSnapshotsMSTestUsage/Tests.Incorrectcase.verified.txt rename to src/DanglingSnapshotsExpectoUsage/Tests.Simple.verified.txt diff --git a/src/DanglingSnapshotsExpectoUsage/Tests.cs b/src/DanglingSnapshotsExpectoUsage/Tests.cs new file mode 100644 index 0000000000..87b79b7bcc --- /dev/null +++ b/src/DanglingSnapshotsExpectoUsage/Tests.cs @@ -0,0 +1,18 @@ +using Expecto; + +public class Tests +{ + [Tests] + public static Test Simple = Runner.TestCase( + nameof(Simple), + () => Verify( + name: nameof(Simple), + target: "Foo")); + + [Tests] + public static Test Second = Runner.TestCase( + nameof(Second), + () => Verify( + name: nameof(Second), + target: "Foo")); +} diff --git a/src/DanglingSnapshotsFixieUsage/DanglingSnapshots.cs b/src/DanglingSnapshotsFixieUsage/DanglingSnapshots.cs new file mode 100644 index 0000000000..649e7ab1cf --- /dev/null +++ b/src/DanglingSnapshotsFixieUsage/DanglingSnapshots.cs @@ -0,0 +1,28 @@ +#pragma warning disable VerifyDanglingSnapshots + +public class TestProject : + ITestProject, + IExecution +{ + public void Configure(TestConfiguration configuration, TestEnvironment environment) + { + VerifierSettings.AssignTargetAssembly(environment.Assembly); + configuration.Conventions.Add(); + } + + public async Task Run(TestSuite testSuite) + { + foreach (var testClass in testSuite.TestClasses) + { + foreach (var test in testClass.Tests) + { + using (ExecutionState.Set(testClass, test, null)) + { + await test.Run(); + } + } + } + + DanglingSnapshots.Run(); + } +} diff --git a/src/DanglingSnapshotsFixieUsage/DanglingSnapshotsFixieUsage.csproj b/src/DanglingSnapshotsFixieUsage/DanglingSnapshotsFixieUsage.csproj new file mode 100644 index 0000000000..b1fc09dd7a --- /dev/null +++ b/src/DanglingSnapshotsFixieUsage/DanglingSnapshotsFixieUsage.csproj @@ -0,0 +1,17 @@ + + + net11.0 + true + false + true + false + + + + + + + + + + diff --git a/src/DanglingSnapshotsNUnitUsage/Tests.Dangling.verified.txt b/src/DanglingSnapshotsFixieUsage/Tests.Second.verified.txt similarity index 100% rename from src/DanglingSnapshotsNUnitUsage/Tests.Dangling.verified.txt rename to src/DanglingSnapshotsFixieUsage/Tests.Second.verified.txt diff --git a/src/DanglingSnapshotsNUnitUsage/Tests.Incorrectcase.verified.txt b/src/DanglingSnapshotsFixieUsage/Tests.Simple.verified.txt similarity index 100% rename from src/DanglingSnapshotsNUnitUsage/Tests.Incorrectcase.verified.txt rename to src/DanglingSnapshotsFixieUsage/Tests.Simple.verified.txt diff --git a/src/DanglingSnapshotsFixieUsage/Tests.cs b/src/DanglingSnapshotsFixieUsage/Tests.cs new file mode 100644 index 0000000000..24b18e90dc --- /dev/null +++ b/src/DanglingSnapshotsFixieUsage/Tests.cs @@ -0,0 +1,8 @@ +public class Tests +{ + public Task Simple() => + Verify("Foo"); + + public Task Second() => + Verify("Foo"); +} diff --git a/src/DanglingSnapshotsMSTestUsage/AssemblyInfo.cs b/src/DanglingSnapshotsMSTestUsage/AssemblyInfo.cs index f247aba7e8..68ac8f25ac 100644 --- a/src/DanglingSnapshotsMSTestUsage/AssemblyInfo.cs +++ b/src/DanglingSnapshotsMSTestUsage/AssemblyInfo.cs @@ -1,5 +1,5 @@ [assembly: Parallelize] // Without this every verification fails with "TestContext is null". Applied to the assembly rather // than the test class so that building this project also covers the generator skipping the static -// [TestClass] that hosts the [AssemblyCleanup] below. +// [TestClass] in DanglingSnapshots.cs, which an assembly wide attribute also reaches. [assembly: UsesVerify] diff --git a/src/DanglingSnapshotsXunitV3Usage/Tests.Dangling.verified.txt b/src/DanglingSnapshotsMSTestUsage/Tests.Second.verified.txt similarity index 100% rename from src/DanglingSnapshotsXunitV3Usage/Tests.Dangling.verified.txt rename to src/DanglingSnapshotsMSTestUsage/Tests.Second.verified.txt diff --git a/src/DanglingSnapshotsMSTestUsage/Tests.cs b/src/DanglingSnapshotsMSTestUsage/Tests.cs index 17e5762e77..6524fb29bf 100644 --- a/src/DanglingSnapshotsMSTestUsage/Tests.cs +++ b/src/DanglingSnapshotsMSTestUsage/Tests.cs @@ -6,6 +6,6 @@ public Task Simple() => Verify("Foo"); [TestMethod] - public Task IncorrectCase() => + public Task Second() => Verify("Foo"); } \ No newline at end of file diff --git a/src/DanglingSnapshotsXunitV3Usage/Tests.Incorrectcase.verified.txt b/src/DanglingSnapshotsNUnitUsage/Tests.Second.verified.txt similarity index 100% rename from src/DanglingSnapshotsXunitV3Usage/Tests.Incorrectcase.verified.txt rename to src/DanglingSnapshotsNUnitUsage/Tests.Second.verified.txt diff --git a/src/DanglingSnapshotsNUnitUsage/Tests.cs b/src/DanglingSnapshotsNUnitUsage/Tests.cs index ab6c1d612f..f74ab098fc 100644 --- a/src/DanglingSnapshotsNUnitUsage/Tests.cs +++ b/src/DanglingSnapshotsNUnitUsage/Tests.cs @@ -6,6 +6,6 @@ public Task Simple() => Verify("Foo"); [Test] - public Task IncorrectCase() => + public Task Second() => Verify("Foo"); } \ No newline at end of file diff --git a/src/DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs b/src/DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs new file mode 100644 index 0000000000..1ee42328a6 --- /dev/null +++ b/src/DanglingSnapshotsTUnitUsage/DanglingSnapshots.cs @@ -0,0 +1,8 @@ +#pragma warning disable VerifyDanglingSnapshots + +public static class Cleanup +{ + [After(TestSession)] + public static void Run() => + DanglingSnapshots.Run(); +} diff --git a/src/DanglingSnapshotsTUnitUsage/DanglingSnapshotsTUnitUsage.csproj b/src/DanglingSnapshotsTUnitUsage/DanglingSnapshotsTUnitUsage.csproj new file mode 100644 index 0000000000..828f4ad537 --- /dev/null +++ b/src/DanglingSnapshotsTUnitUsage/DanglingSnapshotsTUnitUsage.csproj @@ -0,0 +1,17 @@ + + + net11.0 + true + false + true + $(NoWarn);CA1822 + + + + + + + + + + diff --git a/src/DanglingSnapshotsTUnitUsage/Tests.Second.verified.txt b/src/DanglingSnapshotsTUnitUsage/Tests.Second.verified.txt new file mode 100644 index 0000000000..fdf74cdc4b --- /dev/null +++ b/src/DanglingSnapshotsTUnitUsage/Tests.Second.verified.txt @@ -0,0 +1 @@ +Foo \ No newline at end of file diff --git a/src/DanglingSnapshotsTUnitUsage/Tests.Simple.verified.txt b/src/DanglingSnapshotsTUnitUsage/Tests.Simple.verified.txt new file mode 100644 index 0000000000..fdf74cdc4b --- /dev/null +++ b/src/DanglingSnapshotsTUnitUsage/Tests.Simple.verified.txt @@ -0,0 +1 @@ +Foo \ No newline at end of file diff --git a/src/DanglingSnapshotsTUnitUsage/Tests.cs b/src/DanglingSnapshotsTUnitUsage/Tests.cs new file mode 100644 index 0000000000..fb2911ac01 --- /dev/null +++ b/src/DanglingSnapshotsTUnitUsage/Tests.cs @@ -0,0 +1,10 @@ +public class Tests +{ + [Test] + public Task Simple() => + Verify("Foo"); + + [Test] + public Task Second() => + Verify("Foo"); +} diff --git a/src/DanglingSnapshotsXunitV3Usage/Tests.Second.verified.txt b/src/DanglingSnapshotsXunitV3Usage/Tests.Second.verified.txt new file mode 100644 index 0000000000..fdf74cdc4b --- /dev/null +++ b/src/DanglingSnapshotsXunitV3Usage/Tests.Second.verified.txt @@ -0,0 +1 @@ +Foo \ No newline at end of file diff --git a/src/DanglingSnapshotsXunitV3Usage/Tests.cs b/src/DanglingSnapshotsXunitV3Usage/Tests.cs index c588b8ba06..d9dbb6c046 100644 --- a/src/DanglingSnapshotsXunitV3Usage/Tests.cs +++ b/src/DanglingSnapshotsXunitV3Usage/Tests.cs @@ -8,7 +8,7 @@ public Task Simple() => #endregion [Fact] - public Task IncorrectCase() => + public Task Second() => Verify("Foo"); } diff --git a/src/Verify.NUnit/DanglingSnapshots.cs b/src/Verify.NUnit/DanglingSnapshots.cs index dde2e3f066..335e39b778 100644 --- a/src/Verify.NUnit/DanglingSnapshots.cs +++ b/src/Verify.NUnit/DanglingSnapshots.cs @@ -3,6 +3,44 @@ [Experimental("VerifyDanglingSnapshots")] public static class DanglingSnapshots { - public static void Run() => - DanglingSnapshotsCheck.Run(); -} \ No newline at end of file + public static void Run() + { + try + { + DanglingSnapshotsCheck.Run(); + } + catch (Exception exception) + { + FailProcessOnExit(exception); + throw; + } + } + + /// + /// NUnit's Microsoft.Testing.Platform runner prints a teardown failure but does not count it: the + /// run is summarised as passed and the process exits zero, so a dangling snapshot would not fail + /// a build. That holds for [SetUpFixture] and [TestFixture] alike, and there is no + /// teardown hook whose failure the runner does surface, so the exit code is set here instead. + /// + /// Set from ProcessExit, because the runner's own result replaces anything assigned while it is + /// still running. Registered only once the check has already failed, and only over a zero, so a + /// run that failed for its own reasons keeps the code it earned. + /// + static void FailProcessOnExit(Exception exception) => + AppDomain.CurrentDomain.ProcessExit += (_, _) => + { + if (Environment.ExitCode != 0) + { + return; + } + + Environment.ExitCode = 1; + // Written from the handler so it lands after the runner's summary, which is where a + // passing summary sitting next to a failing exit code needs explaining. + Console.Error.WriteLine( + $""" + Verify has failed the dangling snapshot check. The NUnit runner does not count a teardown failure, so it reported the run as passed; the exit code has been set to 1. + {exception.Message} + """); + }; +} diff --git a/src/Verify.Tests/DanglingManifestTests.OrphanAndRetiredFrameworkSurviveTheUnion.verified.txt b/src/Verify.Tests/DanglingManifestTests.OrphanAndRetiredFrameworkSurviveTheUnion.verified.txt new file mode 100644 index 0000000000..d2ddea70b4 --- /dev/null +++ b/src/Verify.Tests/DanglingManifestTests.OrphanAndRetiredFrameworkSurviveTheUnion.verified.txt @@ -0,0 +1,11 @@ +{ + Type: Exception, + Message: +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Tests.Deleted.DotNet8_0.verified.txt + * Tests.Alive.Mono3_1.verified.txt + +} \ No newline at end of file diff --git a/src/Verify.Tests/DanglingManifestTests.WithoutCoverageOnlyTheOrphanIsReported.verified.txt b/src/Verify.Tests/DanglingManifestTests.WithoutCoverageOnlyTheOrphanIsReported.verified.txt new file mode 100644 index 0000000000..92e34d56ed --- /dev/null +++ b/src/Verify.Tests/DanglingManifestTests.WithoutCoverageOnlyTheOrphanIsReported.verified.txt @@ -0,0 +1,10 @@ +{ + Type: Exception, + Message: +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Tests.Deleted.DotNet8_0.verified.txt + +} \ No newline at end of file diff --git a/src/Verify.Tests/DanglingManifestTests.cs b/src/Verify.Tests/DanglingManifestTests.cs new file mode 100644 index 0000000000..7e20e1e118 --- /dev/null +++ b/src/Verify.Tests/DanglingManifestTests.cs @@ -0,0 +1,305 @@ +#pragma warning disable VerifyDanglingSnapshots +public class DanglingManifestTests +{ + static readonly DateTime noStaleFiltering = DateTime.MinValue; + + [Fact] + public void SingleFrameworkIsCoveredByItsOwnManifest() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net9.0", ["one.verified.txt", "two.verified.txt"], ["one", "two"]); + + var merged = DanglingManifest.Read(directory, [], "net9.0", noStaleFiltering); + + Assert.True(merged.FrameworksCovered); + Assert.Equal(["one.verified.txt", "two.verified.txt"], merged.Files.Order()); + Assert.Equal(["one", "two"], merged.Prefixes.Order()); + } + + /// + /// The point of the manifests: the run of one framework sees what the runs of the others + /// tracked, so their files are not mistaken for dangling ones. + /// + [Fact] + public void EveryFrameworkPresentIsCovered() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net8.0", ["Alive.DotNet8_0.verified.txt"], ["Alive"]); + DanglingManifest.Write(directory, "net9.0", ["Alive.DotNet9_0.verified.txt"], ["Alive"]); + + var merged = DanglingManifest.Read(directory, ["net8.0", "net9.0"], "net9.0", noStaleFiltering); + + Assert.True(merged.FrameworksCovered); + Assert.Equal(["Alive.DotNet8_0.verified.txt", "Alive.DotNet9_0.verified.txt"], merged.Files.Order()); + Assert.Equal(["Alive"], merged.Prefixes); + } + + /// + /// A test behind a #if exists only in the runs of the frameworks that compile it. Without + /// its prefix in the union, every other framework's run would see its snapshots as belonging to + /// no test at all, and report them. + /// + [Fact] + public void FrameworkSpecificTestIsKnownToTheOtherFrameworks() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net48", ["OnlyOnNet48.verified.txt"], ["OnlyOnNet48"]); + DanglingManifest.Write(directory, "net9.0", ["Shared.verified.txt"], ["Shared"]); + + var merged = DanglingManifest.Read(directory, ["net48", "net9.0"], "net9.0", noStaleFiltering); + + Assert.True(merged.FrameworksCovered); + Assert.Contains("OnlyOnNet48", merged.Prefixes); + Assert.Contains("OnlyOnNet48.verified.txt", merged.Files); + } + + /// + /// The first framework to run has only its own manifest, so it cannot account for the files of + /// the frameworks still to run and has to leave the framework axis to the name. + /// + [Fact] + public void MissingFrameworkIsNotCovered() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net9.0", ["Alive.DotNet9_0.verified.txt"], ["Alive"]); + + var merged = DanglingManifest.Read(directory, ["net8.0", "net9.0"], "net9.0", noStaleFiltering); + + Assert.False(merged.FrameworksCovered); + Assert.Equal(["Alive.DotNet9_0.verified.txt"], merged.Files); + } + + /// + /// A manifest naming a framework the project no longer targets is read for its contents, but + /// says nothing about coverage. + /// + [Fact] + public void RetiredFrameworkDoesNotSatisfyCoverage() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net7.0", ["Alive.DotNet7_0.verified.txt"], ["Alive"]); + DanglingManifest.Write(directory, "net9.0", ["Alive.DotNet9_0.verified.txt"], ["Alive"]); + + var merged = DanglingManifest.Read(directory, ["net8.0", "net9.0"], "net9.0", noStaleFiltering); + + Assert.False(merged.FrameworksCovered); + } + + /// + /// A manifest from before the last build may name tests and files that have since been renamed + /// or deleted, and counting those as tracked would hide exactly the danglers the check is for. + /// + [Fact] + public void StaleManifestIsIgnored() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net8.0", ["Stale.DotNet8_0.verified.txt"], ["Stale"]); + DanglingManifest.Write(directory, "net9.0", ["Alive.DotNet9_0.verified.txt"], ["Alive"]); + + // Pinned rather than left to the clock: the two manifests are written milliseconds apart, + // and file timestamp granularity is coarser than that on some file systems. + var buildTime = DateTime.UtcNow; + File.SetLastWriteTimeUtc( + DanglingManifest.PathFor(directory, "net8.0"), + buildTime.AddHours(-1)); + File.SetLastWriteTimeUtc( + DanglingManifest.PathFor(directory, "net9.0"), + buildTime.AddHours(1)); + + var merged = DanglingManifest.Read(directory, ["net8.0", "net9.0"], "net9.0", buildTime); + + Assert.False(merged.FrameworksCovered); + Assert.Equal(["Alive.DotNet9_0.verified.txt"], merged.Files); + Assert.Equal(["Alive"], merged.Prefixes); + } + + /// + /// A re run of one framework replaces its manifest rather than adding to it, so a file or test + /// that run no longer tracks stops being tracked. + /// + [Fact] + public void ReRunReplacesManifest() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net9.0", ["Before.verified.txt"], ["Before"]); + DanglingManifest.Write(directory, "net9.0", ["After.verified.txt"], ["After"]); + + var merged = DanglingManifest.Read(directory, [], "net9.0", noStaleFiltering); + + Assert.Equal(["After.verified.txt"], merged.Files); + Assert.Equal(["After"], merged.Prefixes); + } + + /// + /// Several verifications can resolve to one verified path, and every case of a parameterised + /// test records the same prefix, so both bags hold duplicates. + /// + [Fact] + public void DuplicatesAreWrittenOnce() + { + using var directory = new TempDirectory(); + DanglingManifest.Write( + directory, + "net9.0", + ["Alive.verified.txt", "Alive.verified.txt"], + ["Alive", "Alive", "Alive"]); + + var lines = File.ReadAllLines(DanglingManifest.PathFor(directory, "net9.0")); + + Assert.Equal(["[files]", "Alive.verified.txt", "[prefixes]", "Alive"], lines); + } + + [Fact] + public void NoDirectoryIsNotCovered() + { + using var directory = new TempDirectory(); + var missing = directory.BuildPath("absent"); + + var merged = DanglingManifest.Read(missing, ["net9.0"], "net9.0", noStaleFiltering); + + Assert.False(merged.FrameworksCovered); + Assert.Empty(merged.Files); + Assert.Empty(merged.Prefixes); + } + + [Fact] + public void EmptyManifestIsStillCoverage() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net9.0", [], []); + + var merged = DanglingManifest.Read(directory, ["net9.0"], "net9.0", noStaleFiltering); + + Assert.True(merged.FrameworksCovered); + Assert.Empty(merged.Files); + Assert.Empty(merged.Prefixes); + } + + /// + /// A test that produced no file at all - inline, excluded targets, or one that threw - records a + /// prefix and nothing else. That is the case the prefix half exists for. + /// + [Fact] + public void PrefixWithoutFiles() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net9.0", [], ["Inline"]); + + var merged = DanglingManifest.Read(directory, [], "net9.0", noStaleFiltering); + + Assert.Empty(merged.Files); + Assert.Equal(["Inline"], merged.Prefixes); + } + + /// + /// A verified path can be anything the file system accepts, so the manifest has to survive + /// spaces and the characters Verify puts in a name for parameters, targets and indexes. + /// + [Fact] + public void AwkwardPathsRoundTrip() + { + using var directory = new TempDirectory(); + string[] files = + [ + @"D:\a b\Tests.Method_param=a b.DotNet9_0#00.verified.txt", + "/home/a b/Tests.Method#name.verified.txt", + @"D:\a\Tests.Method_param=Ünïcödé.verified.txt" + ]; + string[] prefixes = + [ + @"D:\a b\Tests.Method", + "/home/a b/Tests.Method", + @"D:\a\[files]Tests.Method" + ]; + DanglingManifest.Write(directory, "net9.0", files, prefixes); + + var merged = DanglingManifest.Read(directory, [], "net9.0", noStaleFiltering); + + Assert.Equal(files.Order(), merged.Files.Order()); + Assert.Equal(prefixes.Order(), merged.Prefixes.Order()); + } + + /// + /// Written through a temporary file and moved into place. A partially written manifest read by + /// another framework's run would drop paths and report live snapshots as dangling. + /// + [Fact] + public void WriteLeavesNoTemporaryFiles() + { + using var directory = new TempDirectory(); + DanglingManifest.Write(directory, "net9.0", ["Alive.verified.txt"], ["Alive"]); + + var files = Directory.GetFiles(directory) + .Select(Path.GetFileName) + .Order(); + + Assert.Equal(["net9.0.txt"], files); + } + + /// + /// End to end: two frameworks, one snapshot each, one left behind by a test that was deleted, + /// and one left behind by a framework that was dropped. The deleted test's file is reported + /// either way, since no prefix owns it. The retired framework's file needs the complete union: + /// its test is still there, so only the absence of a run that claims it says anything. + /// + [Fact] + public Task OrphanAndRetiredFrameworkSurviveTheUnion() + { + using var directory = new TempDirectory(); + var root = directory.Path; + var aliveOnNet8 = Path.Combine(root, "Tests.Alive.DotNet8_0.verified.txt"); + var aliveOnNet9 = Path.Combine(root, "Tests.Alive.DotNet9_0.verified.txt"); + var deletedTest = Path.Combine(root, "Tests.Deleted.DotNet8_0.verified.txt"); + // A runtime no run of this project produces, so the verdict is the same on every target + // framework and the snapshot can be shared between them. + var retiredFramework = Path.Combine(root, "Tests.Alive.Mono3_1.verified.txt"); + var alivePrefix = Path.Combine(root, "Tests.Alive"); + + DanglingManifest.Write(directory, "net8.0", [aliveOnNet8], [alivePrefix]); + DanglingManifest.Write(directory, "net9.0", [aliveOnNet9], [alivePrefix]); + + var merged = DanglingManifest.Read(directory, ["net8.0", "net9.0"], "net9.0", noStaleFiltering); + + Assert.True(merged.FrameworksCovered); + return Throws( + () => DanglingSnapshotsCheck.CheckFiles( + [aliveOnNet8, aliveOnNet9, deletedTest, retiredFramework], + merged.Files, + merged.Prefixes, + root, + merged.FrameworksCovered)) + .IgnoreStackTrace(); + } + + /// + /// The same inputs without full coverage. The deleted test's snapshot is still reported, because + /// no prefix owns it whatever framework it names. The retired framework's one is not: its test + /// is alive, and without every manifest there is no way to know no run produces it. + /// + [Fact] + public Task WithoutCoverageOnlyTheOrphanIsReported() + { + using var directory = new TempDirectory(); + var root = directory.Path; + var aliveOnNet9 = Path.Combine(root, "Tests.Alive.DotNet9_0.verified.txt"); + var deletedTest = Path.Combine(root, "Tests.Deleted.DotNet8_0.verified.txt"); + // A runtime no run of this project produces, so the verdict is the same on every target + // framework and the snapshot can be shared between them. + var retiredFramework = Path.Combine(root, "Tests.Alive.Mono3_1.verified.txt"); + var alivePrefix = Path.Combine(root, "Tests.Alive"); + + DanglingManifest.Write(directory, "net9.0", [aliveOnNet9], [alivePrefix]); + + var merged = DanglingManifest.Read(directory, ["net8.0", "net9.0"], "net9.0", noStaleFiltering); + + Assert.False(merged.FrameworksCovered); + return Throws( + () => DanglingSnapshotsCheck.CheckFiles( + [aliveOnNet9, deletedTest, retiredFramework], + merged.Files, + merged.Prefixes, + root, + merged.FrameworksCovered)) + .IgnoreStackTrace(); + } +} diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.DirectoryCaseMismatch.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.DirectoryCaseMismatch.verified.txt new file mode 100644 index 0000000000..cc0604d2d7 --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.DirectoryCaseMismatch.verified.txt @@ -0,0 +1,15 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have been tracked with incorrect case: + + * Nested/Alive.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have been tracked with incorrect case: + + * Nested/Alive.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.DirectoryWithTrailingSeparator.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.DirectoryWithTrailingSeparator.verified.txt new file mode 100644 index 0000000000..1367861742 --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.DirectoryWithTrailingSeparator.verified.txt @@ -0,0 +1,15 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.DuplicateTrackedEntries.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.DuplicateTrackedEntries.verified.txt new file mode 100644 index 0000000000..cd471dbf3d --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.DuplicateTrackedEntries.verified.txt @@ -0,0 +1,4 @@ +== Frameworks not covered == +Nothing reported. +== Frameworks covered == +Nothing reported. \ No newline at end of file diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.LongestPrefixWins.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.LongestPrefixWins.verified.txt new file mode 100644 index 0000000000..20c95507ab --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.LongestPrefixWins.verified.txt @@ -0,0 +1,16 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Tests.Method.Nested.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Tests.Method.Nested.verified.txt + * Tests.Method.Nested.Mono3_1.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.NoPrefixesRecorded.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.NoPrefixesRecorded.verified.txt new file mode 100644 index 0000000000..cd471dbf3d --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.NoPrefixesRecorded.verified.txt @@ -0,0 +1,4 @@ +== Frameworks not covered == +Nothing reported. +== Frameworks covered == +Nothing reported. \ No newline at end of file diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.NothingOnDisk.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.NothingOnDisk.verified.txt new file mode 100644 index 0000000000..cd471dbf3d --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.NothingOnDisk.verified.txt @@ -0,0 +1,4 @@ +== Frameworks not covered == +Nothing reported. +== Frameworks covered == +Nothing reported. \ No newline at end of file diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedNamesContainingNet.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedNamesContainingNet.verified.txt new file mode 100644 index 0000000000..4b0529a1da --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedNamesContainingNet.verified.txt @@ -0,0 +1,21 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Tests.NetworkClient.verified.txt + * Tests.NetCoreShim.verified.txt + * Tests.Nettle.verified.txt + * HttpTests.NetworkFailure.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Tests.NetworkClient.verified.txt + * Tests.NetCoreShim.verified.txt + * Tests.Nettle.verified.txt + * HttpTests.NetworkFailure.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedUnderDirectoryContainingUniquenessToken.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedUnderDirectoryContainingUniquenessToken.verified.txt new file mode 100644 index 0000000000..b9769bd09f --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedUnderDirectoryContainingUniquenessToken.verified.txt @@ -0,0 +1,19 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Shared.NetCore/Deleted.verified.txt + * Snapshots.Windows.Only/Deleted.verified.txt + * Plain/Deleted.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Shared.NetCore/Deleted.verified.txt + * Snapshots.Windows.Only/Deleted.verified.txt + * Plain/Deleted.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedWithIndexesAndParameters.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedWithIndexesAndParameters.verified.txt new file mode 100644 index 0000000000..3615ed54b9 --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedWithIndexesAndParameters.verified.txt @@ -0,0 +1,23 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted#00.verified.txt + * Deleted#name.verified.txt + * Deleted_param=value.verified.txt + * Deleted_param=value.DotNet9_0#00.verified.txt + * Deleted.DotNet9_0#00.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted#00.verified.txt + * Deleted#name.verified.txt + * Deleted_param=value.verified.txt + * Deleted_param=value.DotNet9_0#00.verified.txt + * Deleted.DotNet9_0#00.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedWithLowerCaseUniquenessSegments.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedWithLowerCaseUniquenessSegments.verified.txt new file mode 100644 index 0000000000..f00e1a207c --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedWithLowerCaseUniquenessSegments.verified.txt @@ -0,0 +1,21 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.windows.verified.txt + * Deleted.linux.verified.txt + * Deleted.osx.verified.txt + * Deleted.dotnet9_0.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.windows.verified.txt + * Deleted.linux.verified.txt + * Deleted.osx.verified.txt + * Deleted.dotnet9_0.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedWithUniqueness.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedWithUniqueness.verified.txt new file mode 100644 index 0000000000..45e4b1aa05 --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.OrphanedWithUniqueness.verified.txt @@ -0,0 +1,45 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.verified.txt + * Deleted.DotNet.verified.txt + * Deleted.DotNet9_0.verified.txt + * Deleted.Net.verified.txt + * Deleted.Net4_8.verified.txt + * Deleted.Mono.verified.txt + * Deleted.Mono6_12.verified.txt + * Deleted.Windows.verified.txt + * Deleted.Linux.verified.txt + * Deleted.OSX.verified.txt + * Deleted.Android.verified.txt + * Deleted.IOS.verified.txt + * Deleted.x64.verified.txt + * Deleted.arm64.verified.txt + * Deleted.Debug.verified.txt + * Deleted.Release.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.verified.txt + * Deleted.DotNet.verified.txt + * Deleted.DotNet9_0.verified.txt + * Deleted.Net.verified.txt + * Deleted.Net4_8.verified.txt + * Deleted.Mono.verified.txt + * Deleted.Mono6_12.verified.txt + * Deleted.Windows.verified.txt + * Deleted.Linux.verified.txt + * Deleted.OSX.verified.txt + * Deleted.Android.verified.txt + * Deleted.IOS.verified.txt + * Deleted.x64.verified.txt + * Deleted.arm64.verified.txt + * Deleted.Debug.verified.txt + * Deleted.Release.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.PrefixIsNotASubstringMatch.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.PrefixIsNotASubstringMatch.verified.txt new file mode 100644 index 0000000000..f6a23380be --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.PrefixIsNotASubstringMatch.verified.txt @@ -0,0 +1,17 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * AliveToo.verified.txt + * AliveToo.Mono3_1.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * AliveToo.verified.txt + * AliveToo.Mono3_1.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.PrefixMatchIgnoresCase.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.PrefixMatchIgnoresCase.verified.txt new file mode 100644 index 0000000000..de4984a47c --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.PrefixMatchIgnoresCase.verified.txt @@ -0,0 +1,9 @@ +== Frameworks not covered == +Nothing reported. +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Nested/Alive.Mono4_2.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.RetiredFrameworkSnapshots.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.RetiredFrameworkSnapshots.verified.txt new file mode 100644 index 0000000000..656b7263cb --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.RetiredFrameworkSnapshots.verified.txt @@ -0,0 +1,9 @@ +== Frameworks not covered == +Nothing reported. +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Alive.Mono4_2.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.StaleParameterOnLiveTest.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.StaleParameterOnLiveTest.verified.txt new file mode 100644 index 0000000000..ddf144af46 --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.StaleParameterOnLiveTest.verified.txt @@ -0,0 +1,15 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Alive_param=removed.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Alive_param=removed.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.TrackedFileMissingFromDisk.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.TrackedFileMissingFromDisk.verified.txt new file mode 100644 index 0000000000..cd471dbf3d --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.TrackedFileMissingFromDisk.verified.txt @@ -0,0 +1,4 @@ +== Frameworks not covered == +Nothing reported. +== Frameworks covered == +Nothing reported. \ No newline at end of file diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.UniqueDirectoryPrefixes.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.UniqueDirectoryPrefixes.verified.txt new file mode 100644 index 0000000000..9896216ff2 --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.UniqueDirectoryPrefixes.verified.txt @@ -0,0 +1,15 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.Mono3_1/target.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.Mono3_1/target.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.UniquenessFromAnotherRun.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.UniquenessFromAnotherRun.verified.txt new file mode 100644 index 0000000000..3425df79bc --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.UniquenessFromAnotherRun.verified.txt @@ -0,0 +1,15 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Alive.Debug.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Alive.Debug.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.UnnormalizedTrackedPath.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.UnnormalizedTrackedPath.verified.txt new file mode 100644 index 0000000000..acde8d9292 --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.UnnormalizedTrackedPath.verified.txt @@ -0,0 +1,15 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Snapshots/Alive.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Snapshots/Alive.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.UntrackedAndIncorrectCase.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.UntrackedAndIncorrectCase.verified.txt new file mode 100644 index 0000000000..ab35b9092e --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.UntrackedAndIncorrectCase.verified.txt @@ -0,0 +1,25 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.verified.txt + + +The following files have been tracked with incorrect case: + + * AliveWithCase.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Deleted.verified.txt + + +The following files have been tracked with incorrect case: + + * AliveWithCase.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.UseFileNamePrefix.verified.txt b/src/Verify.Tests/DanglingSnapshotsCheckTests.UseFileNamePrefix.verified.txt new file mode 100644 index 0000000000..d9731af288 --- /dev/null +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.UseFileNamePrefix.verified.txt @@ -0,0 +1,16 @@ +== Frameworks not covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Unpinned.Mono3_1.verified.txt + +== Frameworks covered == + +Verify has detected the following issues with snapshot files: + +The following files have not been tracked: + + * Pinned.Mono3_1.verified.txt + * Unpinned.Mono3_1.verified.txt diff --git a/src/Verify.Tests/DanglingSnapshotsCheckTests.cs b/src/Verify.Tests/DanglingSnapshotsCheckTests.cs index b98fd95d74..c96f4795e9 100644 --- a/src/Verify.Tests/DanglingSnapshotsCheckTests.cs +++ b/src/Verify.Tests/DanglingSnapshotsCheckTests.cs @@ -1,6 +1,19 @@ -#pragma warning disable VerifyDanglingSnapshots +#pragma warning disable VerifyDanglingSnapshots public class DanglingSnapshotsCheckTests { + const string root = "path/to"; + + // Runtimes and architectures no run of this project produces, so the classifier reaches the same + // verdict on every target framework and machine and the snapshots can be shared between them. + // A test that hard coded, say, DotNet8_0 would report it as belonging to another run everywhere + // except the net8.0 run, which owns it. + const string foreignRuntime = "Mono3_1"; + const string otherForeignRuntime = "Mono4_2"; + const string foreignArchitecture = "s390x"; + + static readonly string[] alive = ["path/to/Alive.verified.txt"]; + static readonly string[] alivePrefix = ["path/to/Alive"]; + [Fact] public Task Untracked() { @@ -13,7 +26,7 @@ public Task Untracked() "path/to/tracked.verified.txt" }; - return Throws(() => DanglingSnapshotsCheck.CheckFiles(filesOnDisk, trackedFiles, "path/to")) + return Throws(() => DanglingSnapshotsCheck.CheckFiles(filesOnDisk, trackedFiles, ["path/to/tracked"], root, false)) .IgnoreStackTrace(); } @@ -29,7 +42,7 @@ public Task IncorrectCase() "path/to/tracked.verified.txt" }; - return Throws(() => DanglingSnapshotsCheck.CheckFiles(filesOnDisk, trackedFiles, "path/to")) + return Throws(() => DanglingSnapshotsCheck.CheckFiles(filesOnDisk, trackedFiles, ["path/to/tracked"], root, false)) .IgnoreStackTrace(); } @@ -39,6 +52,448 @@ public void AllTracked() var filesOnDisk = new List { "path/to/tracked.verified.txt" }; var trackedFiles = new ConcurrentBag { "path/to/tracked.verified.txt" }; - DanglingSnapshotsCheck.CheckFiles(filesOnDisk, trackedFiles, "path/to"); + DanglingSnapshotsCheck.CheckFiles(filesOnDisk, trackedFiles, ["path/to/tracked"], root, false); + } + + /// + /// Every file here is a snapshot for a test that no longer exists: no recorded prefix owns a + /// name starting where theirs do. The set of tests does not vary by framework, OS or + /// architecture, so there is no run left for the uniqueness in the name to belong to, and all of + /// them are reported whether or not the manifests cover the frameworks. + /// + [Fact] + public Task OrphanedWithUniqueness() => + Verify( + Report( + [ + "path/to/Deleted.verified.txt", + "path/to/Deleted.DotNet.verified.txt", + "path/to/Deleted.DotNet9_0.verified.txt", + "path/to/Deleted.Net.verified.txt", + "path/to/Deleted.Net4_8.verified.txt", + "path/to/Deleted.Mono.verified.txt", + "path/to/Deleted.Mono6_12.verified.txt", + "path/to/Deleted.Windows.verified.txt", + "path/to/Deleted.Linux.verified.txt", + "path/to/Deleted.OSX.verified.txt", + "path/to/Deleted.Android.verified.txt", + "path/to/Deleted.IOS.verified.txt", + "path/to/Deleted.x64.verified.txt", + "path/to/Deleted.arm64.verified.txt", + "path/to/Deleted.Debug.verified.txt", + "path/to/Deleted.Release.verified.txt" + ], + alive, + alivePrefix)); + + /// + /// The axes no manifest can settle. The architecture one is skipped whether or not the + /// frameworks are covered, since the run that owns it is on another machine. The configuration + /// one has no enumerable value set, so it is not recognised as uniqueness at all and is + /// reported. + /// + [Fact] + public Task UniquenessFromAnotherRun() => + Verify( + Report( + [ + $"path/to/Alive.{foreignArchitecture}.verified.txt", + "path/to/Alive.Debug.verified.txt" + ], + alive, + alivePrefix)); + + /// + /// A project that dropped a target framework keeps the snapshots that framework owned. The test + /// is still there, so only a complete union can say the file is stale: by name it is + /// indistinguishable from the file of a framework that simply is not running. + /// + [Fact] + public Task RetiredFrameworkSnapshots() => + Verify( + Report( + [ + $"path/to/Alive.{foreignRuntime}.verified.txt", + $"path/to/Alive.{otherForeignRuntime}.verified.txt" + ], + [$"path/to/Alive.{foreignRuntime}.verified.txt"], + alivePrefix)); + + /// + /// Names that merely start a word with Net. A whole segment match cannot confuse them + /// with a uniqueness segment, and none of them is owned by a live test. + /// + [Fact] + public Task OrphanedNamesContainingNet() => + Verify( + Report( + [ + "path/to/Tests.NetworkClient.verified.txt", + "path/to/Tests.NetCoreShim.verified.txt", + "path/to/Tests.Nettle.verified.txt", + "path/to/HttpTests.NetworkFailure.verified.txt" + ], + alive, + alivePrefix)); + + /// + /// A directory whose name contains a uniqueness token holds snapshots like any other. Only the + /// segments after a recorded prefix are uniqueness, so the directory name cannot hide them. + /// + [Fact] + public Task OrphanedUnderDirectoryContainingUniquenessToken() => + Verify( + Report( + [ + "path/to/Shared.NetCore/Deleted.verified.txt", + "path/to/Snapshots.Windows.Only/Deleted.verified.txt", + "path/to/Plain/Deleted.verified.txt" + ], + alive, + alivePrefix)); + + [Fact] + public Task OrphanedWithLowerCaseUniquenessSegments() => + Verify( + Report( + [ + "path/to/Deleted.windows.verified.txt", + "path/to/Deleted.linux.verified.txt", + "path/to/Deleted.osx.verified.txt", + "path/to/Deleted.dotnet9_0.verified.txt" + ], + alive, + alivePrefix)); + + /// + /// A verified name is {Type}.{Method}{_parameters}{.uniqueness}#{index}. Everything after + /// the type and method varies per run, so none of it identifies the test. + /// + [Fact] + public Task OrphanedWithIndexesAndParameters() => + Verify( + Report( + [ + "path/to/Deleted#00.verified.txt", + "path/to/Deleted#name.verified.txt", + "path/to/Deleted_param=value.verified.txt", + "path/to/Deleted_param=value.DotNet9_0#00.verified.txt", + "path/to/Deleted.DotNet9_0#00.verified.txt" + ], + alive, + alivePrefix)); + + /// + /// A test that is still there but whose parameter set changed leaves a snapshot behind. The + /// prefix is live and no uniqueness segment excuses the file, so it is reported. + /// + [Fact] + public Task StaleParameterOnLiveTest() => + Verify( + Report( + [ + "path/to/Alive_param=current.verified.txt", + "path/to/Alive_param=removed.verified.txt" + ], + ["path/to/Alive_param=current.verified.txt"], + alivePrefix)); + + /// + /// A prefix ends at the type and method, so a test whose name merely starts with another's does + /// not lend it its identity. + /// + [Fact] + public Task PrefixIsNotASubstringMatch() => + Verify( + Report( + [ + "path/to/Alive.verified.txt", + "path/to/AliveToo.verified.txt", + $"path/to/AliveToo.{foreignRuntime}.verified.txt" + ], + alive, + alivePrefix)); + + /// + /// Two tests where one name extends the other. The longer prefix has to win, or the shorter + /// one's uniqueness rules would be applied to the longer one's files. + /// + [Fact] + public Task LongestPrefixWins() => + Verify( + Report( + [ + "path/to/Tests.Method.verified.txt", + "path/to/Tests.Method.Nested.verified.txt", + $"path/to/Tests.Method.Nested.{foreignRuntime}.verified.txt" + ], + ["path/to/Tests.Method.verified.txt"], + ["path/to/Tests.Method", "path/to/Tests.Method.Nested"])); + + /// + /// The directory half of a prefix comes from the compiler's caller file path, while the scan + /// starts at the project directory MSBuild recorded. The two disagree on case often enough that + /// matching them ordinally would call every test unknown. + /// + [Fact] + public Task PrefixMatchIgnoresCase() => + Verify( + Report( + [$"path/to/Nested/Alive.{otherForeignRuntime}.verified.txt"], + [$"path/to/Nested/Alive.{foreignRuntime}.verified.txt"], + ["path/to/nested/Alive"])); + + /// + /// The unique directory conventions put the parameters and uniqueness in a directory name, with + /// the snapshots below it, so a prefix is followed by a separator rather than by a dot. + /// + [Fact] + public Task UniqueDirectoryPrefixes() => + Verify( + Report( + [ + "path/to/Alive/target.verified.txt", + $"path/to/Alive.{foreignRuntime}/target.verified.txt", + $"path/to/Deleted.{foreignRuntime}/target.verified.txt" + ], + [ + "path/to/Alive/target.verified.txt", + $"path/to/Alive.{foreignRuntime}/target.verified.txt" + ], + alivePrefix)); + + /// + /// UseFileName pins the verified name, so the recorded prefix is that name rather than + /// the type and method. + /// + [Fact] + public Task UseFileNamePrefix() => + Verify( + Report( + [ + "path/to/Pinned.verified.txt", + $"path/to/Pinned.{foreignRuntime}.verified.txt", + $"path/to/Unpinned.{foreignRuntime}.verified.txt" + ], + ["path/to/Pinned.verified.txt"], + ["path/to/Pinned"])); + + /// + /// Nothing recorded the tests, so there is no identity to reason from and every file would look + /// unowned. Only reachable through a caller that tracks files alone, or a manifest written + /// before prefixes were recorded. + /// + [Fact] + public Task NoPrefixesRecorded() => + Verify( + Report( + [ + "path/to/Deleted.verified.txt", + $"path/to/Deleted.{foreignRuntime}.verified.txt" + ], + alive, + [])); + + /// + /// ResolveDirectory combines the source file directory with a relative + /// UseDirectory through Path.Combine, which does not normalise, so the tracked + /// path keeps its ... The scan returns the normalised path, and neither the file nor the + /// prefix matches it. + /// + [Fact] + public Task UnnormalizedTrackedPath() => + Verify( + Report( + ["path/to/Snapshots/Alive.verified.txt"], + ["path/to/Tests/../Snapshots/Alive.verified.txt"], + ["path/to/Tests/../Snapshots/Alive"])); + + /// + /// Only the case of the file name is meaningful. The prefix matches either way, so the file + /// reaches the casing check rather than being reported as an unknown test. + /// + [Fact] + public Task DirectoryCaseMismatch() => + Verify( + Report( + ["path/to/Nested/Alive.verified.txt"], + ["path/to/nested/Alive.verified.txt"], + ["path/to/nested/Alive"])); + + /// + /// The project directory recorded by MSBuild ends with a separator, so the relative suffix has + /// to survive both forms. + /// + [Fact] + public Task DirectoryWithTrailingSeparator() => + Verify( + ReportIn( + "path/to/", + ["path/to/Deleted.verified.txt"], + alive, + alivePrefix)); + + [Fact] + public Task UntrackedAndIncorrectCase() => + Verify( + Report( + [ + "path/to/Deleted.verified.txt", + "path/to/AliveWithCase.verified.txt" + ], + ["path/to/alivewithcase.verified.txt"], + ["path/to/alivewithcase"])); + + /// + /// A test can track a verified path that has no file yet: the first run of a new test, or a run + /// where every target was new. That is not a dangling file. + /// + [Fact] + public Task TrackedFileMissingFromDisk() => + Verify(Report([], ["path/to/NotYetAccepted.verified.txt"], ["path/to/NotYetAccepted"])); + + /// + /// Several verifications can resolve to one verified path, through UseFileName or a + /// DerivePathInfo that collapses them, so the bag can hold duplicates. + /// + [Fact] + public Task DuplicateTrackedEntries() => + Verify( + Report( + ["path/to/Alive.verified.txt"], + ["path/to/Alive.verified.txt", "path/to/Alive.verified.txt"], + ["path/to/Alive", "path/to/Alive"])); + + [Fact] + public Task NothingOnDisk() => + Verify(Report([], alive, alivePrefix)); + + /// + /// UseUniqueDirectory in split mode writes to + /// {prefix}.verified\{target}.{extension}, so .verified. is in the directory name + /// rather than the file name and the scan's glob never matches. A deleted test leaves its whole + /// snapshot directory behind. + /// + [Fact] + public void FindSplitModeUniqueDirectory() + { + using var directory = new TempDirectory(); + WriteNested(directory, "Tests.Split.verified", "target.txt"); + WriteNested(directory, "Tests.Split.verified", "target#00.txt"); + WriteNested(directory, "Tests.Split.verified", "nested", "target.txt"); + + Assert.Empty(DanglingSnapshotsCheck.FindSnapshotFiles(directory)); + } + + /// + /// The non split unique directory convention keeps .verified. in the file name, so those + /// files are found. + /// + [Fact] + public void FindNonSplitUniqueDirectory() + { + using var directory = new TempDirectory(); + WriteNested(directory, "Tests.Unique", "target.verified.txt"); + WriteNested(directory, "Tests.Unique", "target#00.verified.txt"); + + Assert.Equal(2, DanglingSnapshotsCheck.FindSnapshotFiles(directory).Count()); + } + + /// + /// The scan walks every directory below the project, build output included. Harmless while + /// nothing copies snapshots there, and a report of files that are not snapshot sources for + /// every project that sets CopyToOutputDirectory on them. + /// + [Fact] + public void FindIncludesBinAndObj() + { + using var directory = new TempDirectory(); + WriteNested(directory, "bin", "Release", "net9.0", "Tests.Copied.verified.txt"); + WriteNested(directory, "obj", "Release", "net9.0", "Tests.Copied.verified.txt"); + + Assert.Equal(2, DanglingSnapshotsCheck.FindSnapshotFiles(directory).Count()); + } + + [Fact] + public void FindExcludesReceived() + { + using var directory = new TempDirectory(); + WriteNested(directory, "Tests.Alive.received.txt"); + WriteNested(directory, "Tests.Alive.verified.txt"); + + var found = DanglingSnapshotsCheck.FindSnapshotFiles(directory) + .ToList(); + + Assert.Single(found); + Assert.EndsWith("Tests.Alive.verified.txt", found[0]); + } + + /// + /// The Win32 glob *.verified.* also matches a name ending in .verified with no + /// extension, which is not a file Verify writes. + /// + [Fact] + public void FindIncludesExtensionlessVerified() + { + using var directory = new TempDirectory(); + WriteNested(directory, "Tests.Stray.verified"); + + Assert.Single(DanglingSnapshotsCheck.FindSnapshotFiles(directory)); + } + + [Fact] + public void FindIncludesNestedDirectories() + { + using var directory = new TempDirectory(); + WriteNested(directory, "Tests.Alive.verified.txt"); + WriteNested(directory, "Snapshots", "Tests.Alive.verified.txt"); + WriteNested(directory, "Snapshots", "Deep", "Tests.Alive.verified.txt"); + + Assert.Equal(3, DanglingSnapshotsCheck.FindSnapshotFiles(directory).Count()); + } + + static string WriteNested(TempDirectory directory, params string[] segments) + { + var path = directory.BuildPath(segments); + Directory.CreateDirectory(Path.GetDirectoryName(path)!); + File.WriteAllText(path, "content"); + return path; + } + + static string Report(IEnumerable filesOnDisk, IReadOnlyCollection tracked, IReadOnlyCollection prefixes) => + ReportIn(root, filesOnDisk, tracked, prefixes); + + /// + /// Reports both ways round. Without a manifest from every target framework the check leaves the + /// runtime and framework axis to the name; with one the union decides it instead, and the + /// difference between the two halves is what the manifests bought. + /// + static string ReportIn(string directory, IEnumerable filesOnDisk, IReadOnlyCollection tracked, IReadOnlyCollection prefixes) + { + var files = filesOnDisk.ToList(); + var builder = new StringBuilder(); + builder.AppendLine("== Frameworks not covered =="); + builder.AppendLine(Check(files, tracked, prefixes, directory, false)); + builder.AppendLine("== Frameworks covered =="); + builder.Append(Check(files, tracked, prefixes, directory, true)); + return builder.ToString(); + } + + static string Check( + IReadOnlyCollection filesOnDisk, + IReadOnlyCollection tracked, + IReadOnlyCollection prefixes, + string directory, + bool frameworksCovered) + { + try + { + DanglingSnapshotsCheck.CheckFiles(filesOnDisk, tracked, prefixes, directory, frameworksCovered); + return "Nothing reported."; + } + catch (Exception exception) + { + return exception.Message; + } } -} \ No newline at end of file +} diff --git a/src/Verify.Tests/UniquenessSegmentsTests.cs b/src/Verify.Tests/UniquenessSegmentsTests.cs new file mode 100644 index 0000000000..663494b675 --- /dev/null +++ b/src/Verify.Tests/UniquenessSegmentsTests.cs @@ -0,0 +1,186 @@ +#pragma warning disable VerifyDanglingSnapshots + +/// +/// Written against the values Namer reports for the run executing them, rather than against +/// literals, so the same assertions hold on every target framework, OS and architecture. +/// +public class UniquenessSegmentsTests +{ + // A runtime name that is never the one this run produces. + static readonly string foreignRuntime = Namer.Runtime == "Mono" ? "DotNet" : "Mono"; + + static readonly string foreignRuntimeAndVersion = $"{foreignRuntime}3_1"; + + static readonly string foreignPlatform = Namer.OperatingSystemPlatform == "Linux" ? "Windows" : "Linux"; + + static readonly string foreignArchitecture = Namer.Architecture == "arm64" ? "x64" : "arm64"; + + [Fact] + public void NoUniquenessIsNotAnotherRun() + { + Assert.False(BelongsToAnotherRun(".verified.txt")); + Assert.False(BelongsToAnotherRun("_param=value.verified.txt")); + Assert.False(BelongsToAnotherRun("#00.verified.txt")); + Assert.False(BelongsToAnotherRun("#name.verified.txt")); + } + + [Fact] + public void ThisRuntimeIsNotAnotherRun() + { + Assert.False(BelongsToAnotherRun($".{Namer.Runtime}.verified.txt")); + Assert.False(BelongsToAnotherRun($".{Namer.RuntimeAndVersion}.verified.txt")); + } + + [Fact] + public void AnotherRuntimeIsAnotherRun() + { + Assert.True(BelongsToAnotherRun($".{foreignRuntime}.verified.txt")); + Assert.True(BelongsToAnotherRun($".{foreignRuntimeAndVersion}.verified.txt")); + } + + /// + /// A version on the same runtime as this run, but not this run's version. The version has to be + /// compared, not just the runtime name, or every framework of a multi targeted project would + /// look like this one. + /// + [Fact] + public void AnotherVersionOfThisRuntimeIsAnotherRun() + { + var other = $"{Namer.Runtime}99_9"; + + Assert.NotEqual(Namer.RuntimeAndVersion, other); + Assert.True(BelongsToAnotherRun($".{other}.verified.txt")); + } + + /// + /// The manifests settle the runtime and framework axis, so once they cover every framework the + /// segments naming it say nothing. + /// + [Fact] + public void RuntimeIsIgnoredOnceFrameworksAreCovered() => + Assert.False(BelongsToAnotherRun($".{foreignRuntimeAndVersion}.verified.txt", frameworksCovered: true)); + + [Fact] + public void ThisPlatformIsNotAnotherRun() => + Assert.False(BelongsToAnotherRun($".{Namer.OperatingSystemPlatform}.verified.txt")); + + /// + /// No manifest can settle the OS axis: those runs are on other machines, with their own + /// intermediate directories. So a foreign platform stays foreign even when covered. + /// + [Fact] + public void AnotherPlatformIsAnotherRunEvenWhenCovered() + { + Assert.True(BelongsToAnotherRun($".{foreignPlatform}.verified.txt")); + Assert.True(BelongsToAnotherRun($".{foreignPlatform}.verified.txt", frameworksCovered: true)); + } + + [Fact] + public void ThisArchitectureIsNotAnotherRun() => + Assert.False(BelongsToAnotherRun($".{Namer.Architecture}.verified.txt")); + + [Fact] + public void AnotherArchitectureIsAnotherRunEvenWhenCovered() + { + Assert.True(BelongsToAnotherRun($".{foreignArchitecture}.verified.txt")); + Assert.True(BelongsToAnotherRun($".{foreignArchitecture}.verified.txt", frameworksCovered: true)); + } + + /// + /// The configuration axis has no enumerable value set, so a segment naming one is not treated as + /// uniqueness at all. A snapshot only another configuration produces is reported. + /// + [Fact] + public void ConfigurationIsNotRecognised() + { + Assert.True(!BelongsToAnotherRun(".Debug.verified.txt")); + Assert.True(!BelongsToAnotherRun(".Release.verified.txt")); + } + + /// + /// A whole segment match, so a name that merely starts a word with a uniqueness token is left + /// alone. These are what the old substring list could not tell apart. + /// + [Fact] + public void NamesStartingWithAUniquenessTokenAreNotUniqueness() + { + Assert.False(BelongsToAnotherRun(".NetworkClient.verified.txt")); + Assert.False(BelongsToAnotherRun(".NetCoreShim.verified.txt")); + Assert.False(BelongsToAnotherRun(".Nettle.verified.txt")); + Assert.False(BelongsToAnotherRun(".Windowsill.verified.txt")); + Assert.False(BelongsToAnotherRun(".Linuxish.verified.txt")); + } + + [Fact] + public void CaseIsPartOfTheMatch() + { + Assert.False(BelongsToAnotherRun(".windows.verified.txt")); + Assert.False(BelongsToAnotherRun($".{foreignRuntime.ToLowerInvariant()}.verified.txt")); + } + + /// + /// The first segment is the parameter text, which is never uniqueness. A parameter value that + /// happens to read like a runtime must not be mistaken for one. + /// + [Fact] + public void ParametersAreNotUniqueness() => + Assert.False(BelongsToAnotherRun($"_param={foreignRuntimeAndVersion}.verified.txt")); + + [Fact] + public void UniquenessAfterParameters() => + Assert.True(BelongsToAnotherRun($"_param=value.{foreignRuntimeAndVersion}.verified.txt")); + + [Fact] + public void UniquenessBeforeAnIndex() + { + Assert.True(BelongsToAnotherRun($".{foreignRuntimeAndVersion}#00.verified.txt")); + Assert.True(BelongsToAnotherRun($".{foreignRuntimeAndVersion}#name.verified.txt")); + } + + [Fact] + public void SeveralUniquenessSegments() + { + Assert.True(BelongsToAnotherRun($".{foreignRuntimeAndVersion}.{Namer.OperatingSystemPlatform}.verified.txt")); + Assert.True(BelongsToAnotherRun($".{Namer.RuntimeAndVersion}.{foreignPlatform}.verified.txt")); + Assert.False(BelongsToAnotherRun($".{Namer.RuntimeAndVersion}.{Namer.OperatingSystemPlatform}.verified.txt")); + } + + /// + /// The unique directory conventions put the uniqueness in a directory name. Uniqueness ends + /// where that directory does, so the file name below it is never read as uniqueness. + /// + [Fact] + public void UniqueDirectory() + { + Assert.True(BelongsToAnotherRun($".{foreignRuntimeAndVersion}{Path.DirectorySeparatorChar}target.verified.txt")); + Assert.False(BelongsToAnotherRun($"{Path.DirectorySeparatorChar}{foreignRuntimeAndVersion}.verified.txt")); + } + + /// + /// Split mode names the directory {prefix}{uniqueness}.verified, so uniqueness ends at + /// the verified marker even though it is a directory rather than a file. + /// + [Fact] + public void SplitModeUniqueDirectory() + { + Assert.True(BelongsToAnotherRun($".{foreignRuntimeAndVersion}.verified{Path.DirectorySeparatorChar}target.txt")); + Assert.False(BelongsToAnotherRun($".verified{Path.DirectorySeparatorChar}target.txt")); + } + + /// + /// Only {name}{major}_{minor} is a versioned segment. Anything else keeps its underscores + /// and fails the name lookup whole. + /// + [Fact] + public void OnlyMajorMinorIsAVersion() + { + Assert.False(BelongsToAnotherRun($".{foreignRuntime}_.verified.txt")); + Assert.False(BelongsToAnotherRun($".{foreignRuntime}_x.verified.txt")); + Assert.False(BelongsToAnotherRun($".{foreignRuntime}3_.verified.txt")); + Assert.False(BelongsToAnotherRun($".{foreignRuntime}_1.verified.txt")); + Assert.True(BelongsToAnotherRun($".{foreignRuntime}10_11.verified.txt")); + } + + static bool BelongsToAnotherRun(string tail, bool frameworksCovered = false) => + UniquenessSegments.BelongsToAnotherRun(tail, frameworksCovered); +} diff --git a/src/Verify/ConventionCheck/DanglingManifest.cs b/src/Verify/ConventionCheck/DanglingManifest.cs new file mode 100644 index 0000000000..0674a63290 --- /dev/null +++ b/src/Verify/ConventionCheck/DanglingManifest.cs @@ -0,0 +1,153 @@ +namespace VerifyTests; + +/// +/// Records what one target framework's test run tracked, so that the runs of the other target +/// frameworks can read it back. +/// +/// Without this a run only knows what it did itself. A multi targeted project runs once per +/// framework, so most of the snapshots on disk belong to a framework that is not running, and the +/// only way to tell those from a snapshot whose test was deleted was to guess from the file name. +/// Unioning the manifests replaces the guess with the answer for that axis. +/// +/// Both halves are recorded. The files say which snapshots exist; the prefixes say which tests do, +/// which matters just as much across frameworks: a test behind a #if NET48 exists only in +/// the net48 run, and without its prefix every other run would call its snapshots unowned. +/// +/// Manifests go in the intermediate (obj) directory, below BaseIntermediateOutputPath, which unlike +/// IntermediateOutputPath is shared by every target framework of the project. They are scoped by +/// configuration, since a Debug run says nothing about the snapshots a Release run owns, and named +/// after the target framework, so a re run overwrites rather than accumulates. +/// +/// Staleness is bounded two ways: obj is wiped by a clean, and a manifest older than the running +/// assembly is from before the last build and is ignored. A run that cannot account for every target +/// framework says so, and the caller leaves the framework axis to the name instead. +/// +static class DanglingManifest +{ + const string extension = ".txt"; + const string filesHeader = "[files]"; + const string prefixesHeader = "[prefixes]"; + + internal readonly struct Merged(IReadOnlyCollection files, IReadOnlyCollection prefixes, bool frameworksCovered) + { + public IReadOnlyCollection Files { get; } = files; + public IReadOnlyCollection Prefixes { get; } = prefixes; + public bool FrameworksCovered { get; } = frameworksCovered; + } + + public static string PathFor(string directory, string targetFramework) => + Path.Combine(directory, $"{targetFramework}{extension}"); + + /// + /// Written through a temporary file and moved into place, so a run of another target framework + /// reading concurrently sees either the previous manifest or this one, never half of one. + /// + public static void Write( + string directory, + string targetFramework, + IEnumerable files, + IEnumerable prefixes) + { + Directory.CreateDirectory(directory); + var path = PathFor(directory, targetFramework); + var temp = $"{path}.{Guid.NewGuid():N}.tmp"; + try + { + // The headers cannot collide with a recorded path: both halves are absolute, so neither + // can be a line that is exactly "[files]" or "[prefixes]". + List lines = [filesHeader, ..files.Distinct(), prefixesHeader, ..prefixes.Distinct()]; + File.WriteAllLines(temp, lines); + File.Move(temp, path, true); + } + catch + { + IoHelpers.DeleteFile(temp); + throw; + } + } + + /// + /// Everything recorded by the manifests in , and whether those + /// manifests account for every framework in . + /// + /// + /// Manifests last written before this are from before the last build and are ignored. The + /// running assembly's timestamp is the caller's value: a build server builds every framework + /// before running any of them, so a manifest from this build is always newer. + /// + public static Merged Read( + string directory, + IReadOnlyList targetFrameworks, + string targetFramework, + DateTime staleBefore) + { + HashSet files = []; + HashSet prefixes = []; + HashSet frameworksRead = new(StringComparer.OrdinalIgnoreCase); + + if (Directory.Exists(directory)) + { + foreach (var file in Directory.EnumerateFiles(directory, $"*{extension}")) + { + if (File.GetLastWriteTimeUtc(file) < staleBefore) + { + continue; + } + + ReadInto(file, files, prefixes); + frameworksRead.Add(Path.GetFileNameWithoutExtension(file)); + } + } + + return new(files, prefixes, IsCovered(targetFrameworks, targetFramework, frameworksRead)); + } + + static void ReadInto(string file, HashSet files, HashSet prefixes) + { + var target = files; + foreach (var line in File.ReadLines(file)) + { + if (line.Length == 0) + { + continue; + } + + if (line == filesHeader) + { + target = files; + continue; + } + + if (line == prefixesHeader) + { + target = prefixes; + continue; + } + + target.Add(line); + } + } + + /// + /// A single targeted project has no TargetFrameworks value, so its own manifest is the whole + /// picture. A multi targeted project needs one manifest per framework: anything less and the + /// files of the frameworks still to run are indistinguishable from dangling ones. + /// + static bool IsCovered(IReadOnlyList targetFrameworks, string targetFramework, HashSet frameworksRead) + { + if (targetFrameworks.Count == 0) + { + return frameworksRead.Contains(targetFramework); + } + + foreach (var framework in targetFrameworks) + { + if (!frameworksRead.Contains(framework)) + { + return false; + } + } + + return true; + } +} diff --git a/src/Verify/ConventionCheck/DanglingSnapshotsCheck.cs b/src/Verify/ConventionCheck/DanglingSnapshotsCheck.cs index d2d1d2bba9..c88ef73c98 100644 --- a/src/Verify/ConventionCheck/DanglingSnapshotsCheck.cs +++ b/src/Verify/ConventionCheck/DanglingSnapshotsCheck.cs @@ -7,9 +7,17 @@ namespace VerifyTests; public static class DanglingSnapshotsCheck { static ConcurrentBag? trackedVerifiedFiles; + static ConcurrentBag? trackedPrefixes; internal static void TrackVerifiedFile(string path) => trackedVerifiedFiles?.Add(path); + /// + /// The verified path up to the end of the type and method, before parameters, uniqueness and + /// index. Recorded once per verification, from the constructor, so a test is known to exist + /// even when it produced no file: it was inline, it threw, or every target was excluded. + /// + internal static void TrackPrefix(string prefix) => trackedPrefixes?.Add(prefix); + public static void Run() { if (!BuildServerDetector.Detected) @@ -17,12 +25,89 @@ public static void Run() return; } - var directory = AttributeReader.GetProjectDirectory(VerifierSettings.Assembly); - var files = Directory.EnumerateFiles(directory, "*.verified.*", SearchOption.AllDirectories); - CheckFiles(files, trackedVerifiedFiles!, directory); + if (VerifierSettings.AssemblyOrNull is not { } assembly) + { + // No verification ran, so nothing is tracked and every snapshot on disk would be + // reported. A run that verified nothing is something the test framework reports; adding + // a second failure here only buries it. + return; + } + + var directory = AttributeReader.GetProjectDirectory(assembly); + var merged = MergeWithOtherFrameworks(assembly); + CheckFiles(FindSnapshotFiles(directory), merged.Files, merged.Prefixes, directory, merged.FrameworksCovered); + } + + internal static IEnumerable FindSnapshotFiles(string directory) => + Directory.EnumerateFiles(directory, "*.verified.*", SearchOption.AllDirectories); + + /// + /// Publishes what this run tracked, then reads back what the runs of the other target frameworks + /// tracked. When every framework is accounted for, the union is the complete set of verified + /// files the project owns, and a file outside it is dangling whatever its name says. + /// + static DanglingManifest.Merged MergeWithOtherFrameworks(Assembly assembly) + { + var files = trackedVerifiedFiles!; + var prefixes = trackedPrefixes!; + var directory = VerifierSettings.DanglingDir; + var targetFramework = VerifierSettings.TargetFramework; + if (directory is null || + targetFramework is null) + { + // The project does not consume Verify's build props, so there is nowhere to put a + // manifest and no framework list to check it against. + return new(files, prefixes, false); + } + + try + { + DanglingManifest.Write(directory, targetFramework, files, prefixes); + return DanglingManifest.Read( + directory, + VerifierSettings.TargetFrameworks, + targetFramework, + AssemblyWriteTime(assembly)); + } + catch (Exception exception) + when (exception is IOException or UnauthorizedAccessException) + { + // The manifest is an optimization: without it the check still runs, it just cannot + // settle the framework axis. Failing to read or write one must not fail the test run. + return new(files, prefixes, false); + } + } + + /// + /// A manifest older than the assembly running it is from before the last build, so whatever it + /// records may since have been renamed or deleted. Unknown timestamp means no filtering, which + /// is what a single file or in memory assembly gets. + /// + static DateTime AssemblyWriteTime(Assembly assembly) + { + var location = assembly.Location; + if (location.Length == 0) + { + return DateTime.MinValue; + } + + try + { + return File.GetLastWriteTimeUtc(location); + } + catch (Exception exception) + when (exception is IOException or UnauthorizedAccessException) + { + return DateTime.MinValue; + } } - internal static void CheckFiles(IEnumerable filesOnDisk, ConcurrentBag trackedFiles, string directory) + internal static void CheckFiles( + IEnumerable filesOnDisk, + IReadOnlyCollection trackedFiles, + IReadOnlyCollection prefixes, + string directory, + bool frameworksCovered) { static void AppendItems(StringBuilder builder, List list, string title) { @@ -46,6 +131,11 @@ static void AppendItems(StringBuilder builder, List list, string title) // making the check O(files * tracked). var tracked = new HashSet(trackedFiles); var trackedIgnoreCase = new HashSet(trackedFiles, StringComparer.OrdinalIgnoreCase); + // A prefix identifies a test, and case is not part of that identity. The casing check below + // is about the snapshot's own name; the directory half of a prefix comes from whatever + // string each side built its path from, and the two disagree often enough that an ordinal + // comparison would call every test unknown. + var knownPrefixes = new HashSet(prefixes, StringComparer.OrdinalIgnoreCase); List untracked = []; List incorrectCase = []; @@ -56,13 +146,13 @@ static void AppendItems(StringBuilder builder, List list, string title) continue; } - var suffix = file[directory.Length..]; - suffix = suffix.TrimStart(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar); - if (IfFileUnique(suffix)) + if (!IsDangling(file, knownPrefixes, frameworksCovered)) { continue; } + var suffix = file[directory.Length..]; + suffix = suffix.TrimStart(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar); if (trackedIgnoreCase.Contains(file)) { incorrectCase.Add(suffix); @@ -90,13 +180,64 @@ static void AppendItems(StringBuilder builder, List list, string title) throw new(builder.ToString()); } - static bool IfFileUnique(string file) => - file.Contains(".Net") || - file.Contains(".DotNet") || - file.Contains(".Mono.") || - file.Contains(".OSX.") || - file.Contains(".Windows.") || - file.Contains(".Linux."); + /// + /// A file the run did not write is dangling unless some other run owns it. + /// + /// When no test of this assembly owns a name starting where this one does, no run of the project + /// owns it either: the set of tests does not vary by framework, OS or architecture, so there is + /// nothing left for a name to excuse. That is the whole point of tracking prefixes, and it is + /// what makes a deleted test's snapshot visible however much uniqueness its name carries. + /// + /// When a test does own the name, the file is either a variant this run does not produce, which + /// the uniqueness segments say, or a leftover, which they do not. + /// + static bool IsDangling(string file, HashSet prefixes, bool frameworksCovered) + { + if (prefixes.Count == 0) + { + // Nothing recorded the tests, so there is no identity to reason from. Only reachable + // through a caller that tracks files alone, or a manifest written before prefixes were + // recorded. Treating every file as unowned would report all of them. + return false; + } + + if (!TryMatchPrefix(file, prefixes, out var tailIndex)) + { + return true; + } + + return !UniquenessSegments.BelongsToAnotherRun(file[tailIndex..], frameworksCovered); + } + + /// + /// The longest recorded prefix this file continues from. A verified name continues from the type + /// and method with parameters (_), uniqueness or the verified marker (.), a target + /// name or index (#), or, for the unique directory conventions, a directory separator. + /// Probing every such position finds the prefix without having to know which of them the name + /// uses, and searching from the end takes the longest match. + /// + static bool TryMatchPrefix(string file, HashSet prefixes, out int tailIndex) + { + for (var index = file.Length - 1; index > 0; index--) + { + var character = file[index]; + if (character is not ('.' or '_' or '#') && + character != Path.DirectorySeparatorChar && + character != Path.AltDirectorySeparatorChar) + { + continue; + } + + if (prefixes.Contains(file[..index])) + { + tailIndex = index; + return true; + } + } + + tailIndex = 0; + return false; + } [ModuleInitializer] internal static void Init() @@ -104,6 +245,7 @@ internal static void Init() if (BuildServerDetector.Detected) { trackedVerifiedFiles = []; + trackedPrefixes = []; } } -} \ No newline at end of file +} diff --git a/src/Verify/ConventionCheck/UniquenessSegments.cs b/src/Verify/ConventionCheck/UniquenessSegments.cs new file mode 100644 index 0000000000..f9e93b214c --- /dev/null +++ b/src/Verify/ConventionCheck/UniquenessSegments.cs @@ -0,0 +1,217 @@ +namespace VerifyTests; + +/// +/// Decides whether what follows a verified prefix names a run other than this one. +/// +/// A verified name is +/// {Type}.{Method}{_parameters}{.uniqueness}{#target}{#index}.verified.{extension}, so the +/// uniqueness segments sit between the parameters and the target name, the index, or the verified +/// marker. Under the unique directory conventions the same segments are in a directory name, with +/// the snapshot files below it, so the segments also end at a directory separator. +/// +/// Segments are matched whole, against the values can actually produce. A +/// substring search cannot tell Tests.NetworkClient from a Net uniqueness segment, and +/// cannot tell a DotNet8_0 written by another target framework from the DotNet9_0 this +/// run writes itself. +/// +static class UniquenessSegments +{ + static HashSet platforms = new(StringComparer.Ordinal) + { + "Windows", + "Linux", + "OSX", + "Android", + "IOS" + }; + + // Namer.Architecture is RuntimeInformation.ProcessArchitecture lowercased. Listed rather than + // reflected off the Architecture enum, which grew over time: reflecting it would make a name + // recognised or not depending on the runtime reading it, so a snapshot an s390x agent produced + // would go unrecognised by a .NET Framework run, which only knows the first four. + static HashSet architectures = new(StringComparer.Ordinal) + { + "x86", + "x64", + "arm", + "arm64", + "wasm", + "s390x", + "loongarch64", + "armv6", + "ppc64le", + "riscv64" + }; + + // Every value Namer.GetRuntimeAndVersion and Namer.GetSimpleFrameworkName can return, before + // the major and minor version is appended. + static HashSet frameworks = new(StringComparer.Ordinal) + { + "Net", + "DotNet", + "Mono" + }; + + /// The verified path from the end of the matched prefix. + /// + /// Whether a manifest is in hand for every target framework. When one is, the union of them has + /// already settled the runtime and framework axis and the segments naming it are not consulted. + /// + public static bool BelongsToAnotherRun(string tail, bool frameworksCovered) + { + var end = EndOfUniqueness(tail); + var start = 0; + // The first segment is the parameter text, which is not uniqueness. Under the file + // convention the tail opens with the separator, making that segment empty. + var isParameters = true; + while (start < end) + { + var next = tail.IndexOf('.', start, end - start); + if (next == -1) + { + next = end; + } + + if (!isParameters && + IsFromAnotherRun(tail[start..next], frameworksCovered)) + { + return true; + } + + isParameters = false; + start = next + 1; + } + + return false; + } + + /// + /// Uniqueness runs out at the target name or index, at the verified marker, or, for the unique + /// directory conventions, where the directory holding the snapshots ends. + /// + static int EndOfUniqueness(string tail) + { + var end = tail.Length; + for (var index = 0; index < tail.Length; index++) + { + var character = tail[index]; + if (character == '#' || + character == Path.DirectorySeparatorChar || + character == Path.AltDirectorySeparatorChar) + { + return index; + } + } + + var marker = tail.IndexOf(".verified", StringComparison.Ordinal); + if (marker != -1 && + marker < end) + { + return marker; + } + + return end; + } + + static bool IsFromAnotherRun(string segment, bool frameworksCovered) + { + if (segment.Length == 0) + { + return false; + } + + if (platforms.Contains(segment)) + { + return !MatchesCurrentPlatform(segment); + } + + if (architectures.Contains(segment)) + { + return !string.Equals(segment, Namer.Architecture, StringComparison.Ordinal); + } + + if (frameworksCovered) + { + return false; + } + + return IsFramework(segment) && + !MatchesCurrentFramework(segment); + } + + /// + /// Namer.OperatingSystemPlatform throws rather than name a platform it does not know. Nothing + /// this run produces can be said to match in that case, so the file is left alone. + /// + static bool MatchesCurrentPlatform(string segment) + { + try + { + return string.Equals(segment, Namer.OperatingSystemPlatform, StringComparison.Ordinal); + } + catch + { + return false; + } + } + + static bool MatchesCurrentFramework(string segment) + { + if (string.Equals(segment, Namer.Runtime, StringComparison.Ordinal) || + string.Equals(segment, Namer.RuntimeAndVersion, StringComparison.Ordinal)) + { + return true; + } + + // The target framework names need a TargetFrameworkAttribute, and throw without one. + try + { + return string.Equals(segment, Namer.TargetFrameworkName, StringComparison.Ordinal) || + string.Equals(segment, Namer.TargetFrameworkNameAndVersion, StringComparison.Ordinal); + } + catch + { + return false; + } + } + + static bool IsFramework(string segment) => + frameworks.Contains(WithoutVersion(segment)); + + /// + /// Strips the {major}_{minor} a versioned uniqueness segment ends with, so that + /// DotNet9_0 is recognised as the same axis as DotNet. Anything that is not + /// exactly that shape is returned whole, and fails the name lookup on its own. + /// + static string WithoutVersion(string segment) + { + var underscore = segment.LastIndexOf('_'); + if (underscore < 1 || + underscore == segment.Length - 1) + { + return segment; + } + + for (var index = underscore + 1; index < segment.Length; index++) + { + if (!char.IsDigit(segment[index])) + { + return segment; + } + } + + var nameEnd = underscore; + while (nameEnd > 0 && + char.IsDigit(segment[nameEnd - 1])) + { + nameEnd--; + } + + if (nameEnd == underscore) + { + return segment; + } + + return segment[..nameEnd]; + } +} diff --git a/src/Verify/DerivePaths/AttributeReader.cs b/src/Verify/DerivePaths/AttributeReader.cs index d89f5fe1e3..e68f034b9b 100644 --- a/src/Verify/DerivePaths/AttributeReader.cs +++ b/src/Verify/DerivePaths/AttributeReader.cs @@ -14,6 +14,19 @@ public static bool TryGetTargetFrameworks([NotNullWhen(true)] out string? target public static bool TryGetTargetFrameworks(Assembly assembly, [NotNullWhen(true)] out string? targetFrameworks) => TryGetValue(assembly, "Verify.TargetFrameworks", out targetFrameworks); + /// + /// The single target framework this assembly was built for, as written in the project file. + /// + public static bool TryGetTargetFramework(Assembly assembly, [NotNullWhen(true)] out string? targetFramework) => + TryGetValue(assembly, "Verify.TargetFramework", out targetFramework); + + /// + /// The directory holding the dangling snapshot manifests, shared by every target framework of + /// the project and scoped to the build configuration. + /// + internal static bool TryGetDanglingDirectory(Assembly assembly, [NotNullWhen(true)] out string? danglingDirectory) => + TryGetValue(assembly, "Verify.DanglingDirectory", out danglingDirectory); + public static string GetProjectDirectory() => GetProjectDirectory(Assembly.GetCallingAssembly()); diff --git a/src/Verify/Verifier/InnerVerifier.cs b/src/Verify/Verifier/InnerVerifier.cs index 898d834760..9fe06664b8 100644 --- a/src/Verify/Verifier/InnerVerifier.cs +++ b/src/Verify/Verifier/InnerVerifier.cs @@ -1,3 +1,4 @@ +#pragma warning disable VerifyDanglingSnapshots namespace VerifyTests; public partial class InnerVerifier : @@ -102,6 +103,12 @@ public InnerVerifier( Directory.CreateDirectory(directory); + // Recorded here rather than alongside the files, so that a test counts as existing even when + // it produces none: an inline snapshot, a verification that throws, or one whose targets are + // all excluded. Everything the naming appends after this point - parameters, uniqueness, + // target name, index - varies per run and per case, so none of it identifies the test. + DanglingSnapshotsCheck.TrackPrefix(Path.Combine(directory, settings.FileName ?? typeAndMethod)); + if (settings.UniqueDirectory) { if (settings.inline != null) @@ -211,6 +218,7 @@ public InnerVerifier(string directory, string name, VerifySettings? settings = n Directory.CreateDirectory(directory); var prefix = Path.Combine(directory, name); + DanglingSnapshotsCheck.TrackPrefix(prefix); ValidatePrefix(this.settings, prefix); verifiedFiles = MatchingFileFinder.FindVerified(name, directory); diff --git a/src/Verify/VerifierSettings_TargetAssembly.cs b/src/Verify/VerifierSettings_TargetAssembly.cs index b1135aa90e..e1042fe0c3 100644 --- a/src/Verify/VerifierSettings_TargetAssembly.cs +++ b/src/Verify/VerifierSettings_TargetAssembly.cs @@ -16,6 +16,24 @@ public static partial class VerifierSettings internal static bool TargetsMultipleFramework { get; private set; } = true; + /// + /// Every target framework of the project, as written in the project file. Empty when the project + /// does not consume Verify's build props, or targets a single framework. + /// + internal static IReadOnlyList TargetFrameworks { get; private set; } = []; + + /// + /// The single target framework this assembly was built for. Null when the project does not + /// consume Verify's build props. + /// + internal static string? TargetFramework { get; private set; } + + /// + /// The directory holding the dangling snapshot manifests. Null when the project does not consume + /// Verify's build props, in which case no manifests are written or read. + /// + internal static string? DanglingDir { get; private set; } + [Experimental("VerifierSettingsTestAssembly")] public static Assembly Assembly { @@ -30,6 +48,12 @@ public static Assembly Assembly } } + /// + /// Null until the first verification runs. Unlike , does not throw, for + /// callers whose whole job is to do nothing when no verification ran. + /// + internal static Assembly? AssemblyOrNull => assembly; + static Lock locker = new(); public static void AssignTargetAssembly(Assembly assembly) @@ -56,8 +80,18 @@ public static void AssignTargetAssembly(Assembly assembly) if (AttributeReader.TryGetTargetFrameworks(assembly, out var targetFrameworks)) { TargetsMultipleFramework = targetFrameworks.Contains(';'); + TargetFrameworks = targetFrameworks + .Split(';', StringSplitOptions.RemoveEmptyEntries) + .Select(_ => _.Trim()) + .Where(_ => _.Length > 0) + .ToList(); } + AttributeReader.TryGetTargetFramework(assembly, out var targetFramework); + TargetFramework = targetFramework; + AttributeReader.TryGetDanglingDirectory(assembly, out var danglingDir); + DanglingDir = danglingDir; + DirectoryReplacements.UseAssembly(solutionDir, ProjectDir); VerifierSettings.assembly = assembly; } diff --git a/src/Verify/buildTransitive/Verify.props b/src/Verify/buildTransitive/Verify.props index 2ac2d64015..02ff56cccb 100644 --- a/src/Verify/buildTransitive/Verify.props +++ b/src/Verify/buildTransitive/Verify.props @@ -108,12 +108,25 @@ $([System.IO.Path]::Combine('$(MSBuildProjectDirectory)', '$(IntermediateOutputPath)')) + + $([System.IO.Path]::Combine('$(MSBuildProjectDirectory)', '$(BaseIntermediateOutputPath)', 'VerifyDangling', '$(Configuration)')) <_Parameter1>Verify.TargetFrameworks <_Parameter2>$(TargetFrameworks) + + <_Parameter1>Verify.TargetFramework + <_Parameter2>$(TargetFramework) + + + <_Parameter1>Verify.DanglingDirectory + <_Parameter2>$(VerifyDanglingDirectory) + <_Parameter1>Verify.IntermediateDirectory <_Parameter2>$(VerifyIntermediateDirectory) diff --git a/src/VerifyDangling.slnx b/src/VerifyDangling.slnx index e96e7d8ca1..4fe7e58b21 100644 --- a/src/VerifyDangling.slnx +++ b/src/VerifyDangling.slnx @@ -11,8 +11,11 @@ + + + diff --git a/src/appveyor.yml b/src/appveyor.yml index 61df4ad56e..06f9f2aa87 100644 --- a/src/appveyor.yml +++ b/src/appveyor.yml @@ -58,11 +58,15 @@ build_script: # Run it via the fixie.console tool (restored from src/.config/dotnet-tools.json, which is only discoverable from src). # Use pushd/popd so the `cd` into src does not persist into subsequent AppVeyor script lines. - cmd: pushd src && dotnet fixie Verify.Fixie.Tests --configuration Release --no-build && popd +# The dangling snapshot check only runs on a build server, so these projects are the only place it +# is exercised end to end: adapter wiring, module initializer, teardown hook, manifest and scan. - dotnet build src/VerifyDangling.slnx --configuration Release --verbosity minimal -#- dotnet test %CD%/src/DanglingSnapshotsMSTestUsage --configuration Release --no-build --no-restore -#- dotnet test %CD%/src/DanglingSnapshotsNUnitUsage --configuration Release --no-build --no-restore -#- dotnet test %CD%/src/DanglingSnapshotsXunitUsage --configuration Release --no-build --no-restore -#- dotnet test %CD%/src/DanglingSnapshotsXunitV3Usage --configuration Release --no-build --no-restore +- dotnet test %CD%/src/DanglingSnapshotsMSTestUsage --configuration Release --no-build --no-restore --verbosity minimal +- dotnet test %CD%/src/DanglingSnapshotsNUnitUsage --configuration Release --no-build --no-restore --verbosity minimal +- dotnet test %CD%/src/DanglingSnapshotsXunitV3Usage --configuration Release --no-build --no-restore --verbosity minimal +- dotnet run --project src/DanglingSnapshotsExpectoUsage --configuration Release --no-build --no-restore --verbosity minimal +- dotnet run --project src/DanglingSnapshotsTUnitUsage/DanglingSnapshotsTUnitUsage.csproj --configuration Release --no-build --no-restore --verbosity minimal +- cmd: pushd src && dotnet fixie DanglingSnapshotsFixieUsage --configuration Release --no-build && popd #- pwsh: |