Skip to content

Runtime dynamic tests (AddDynamicTest) discard the Task of async test methods, hiding failures #7249

Open

Description

@thomhurst

What happens

A dynamic test added at run time with TestContext.AddDynamicTest whose method returns Task passes even when the method throws after its first await. For example, a failing await Assert.That(...) is never reported.

TestRegistry.CreateRuntimeTestInvoker (src/TUnit.Engine/Services/TestRegistry.cs, around line 338) compiles DynamicTest<T>.TestMethod. That is an Expression<Action<T>>, so the compiled delegate is an Action<T>. invokeMethod.Invoke(...) returns null, and the invokeResult is Task task branch never runs, so the method's Task is dropped.

The [DynamicTestBuilder] path (ReflectionTestDataCollector.CreateDynamicTestInvoker) is not affected: it extracts the MethodInfo and awaits the returned task.

How to reproduce

public class Consumer
{
    public async Task Fails()
    {
        await Task.Yield();
        throw new Exception("should fail");
    }
}

public class Builder
{
    [Test]
    public async Task Adds()
    {
        await TestContext.Current!.AddDynamicTest(new DynamicTest<Consumer>
        {
            TestMethod = c => c.Fails(),
            Attributes = []
        });
    }
}

The dynamic Fails test is reported as passed. I found this while writing the regression test for #7204 (PR #7248): an async Task assertion on a disposed fixture passed until I made the method synchronous.

Expected

The dynamic test fails. One option: when the lambda body is a method call that returns Task or ValueTask, invoke it through its MethodInfo (as CreateDynamicTestInvoker does), or compile the body into a Func<T, object?> so the result can be awaited.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions