Skip to content

Merge main into live #45092

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 2, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/ai/conceptual/evaluation-libraries.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The evaluation libraries, which are built on top of the [Microsoft.Extensions.AI

## Test integration

The libraries are designed to integrate smoothly with existing .NET apps, allowing you to leverage existing testing infrastructures and familiar syntax to evaluate intelligent apps. You can use any test framework (for example, [MSTest](../../core/testing/index.md#mstest), [xUnit](../../core/testing/index.md#xunit), or [NUnit](../../core/testing/index.md#nunit)) and testing workflow (for example, [Test Explorer](/visualstudio/test/run-unit-tests-with-test-explorer), [dotnet test](../../core/tools/dotnet-test.md), or a CI/CD pipeline). The library also provides easy ways to do online evaluations of your application by publishing evaluation scores to telemetry and monitoring dashboards.
The libraries are designed to integrate smoothly with existing .NET apps, allowing you to leverage existing testing infrastructures and familiar syntax to evaluate intelligent apps. You can use any test framework (for example, [MSTest](../../core/testing/index.md#mstest), [xUnit](../../core/testing/index.md#xunitnet), or [NUnit](../../core/testing/index.md#nunit)) and testing workflow (for example, [Test Explorer](/visualstudio/test/run-unit-tests-with-test-explorer), [dotnet test](../../core/tools/dotnet-test.md), or a CI/CD pipeline). The library also provides easy ways to do online evaluations of your application by publishing evaluation scores to telemetry and monitoring dashboards.

## Comprehensive evaluation metrics

Expand Down
51 changes: 34 additions & 17 deletions docs/core/testing/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,47 +35,64 @@ Keep in mind there are [best practices](unit-testing-best-practices.md) for writ

## Testing tools

.NET is a multi-language development platform, and you can write various test types for [C#](../../csharp/index.yml), [F#](../../fsharp/index.yml), and [Visual Basic](../../visual-basic/index.yml). For each of these languages, you can choose between several test frameworks.
When running tests in .NET, there are two components involved: the test platform and the test framework.

### xUnit
### Test platforms

[xUnit](https://xunit.net) is a free, open-source, community-focused unit testing tool for .NET. The original inventor of NUnit v2 wrote xUnit.net. xUnit.net is the latest technology for unit testing .NET apps. It also works with ReSharper, CodeRush, and TestDriven.NET. xUnit.net is a project of the [.NET Foundation](https://dotnetfoundation.org) and operates under its code of conduct.
The test platform is the engine that runs the tests and acts as a communication channel with IDEs. For example, Visual Studio can send a discovery request to the test platform so that it can display the available tests in Test Explorer. The test platform responds back to the IDE with the tests it found. Similar communication happens for test execution.

VSTest has been used for many years in .NET and was the only test platform in the ecosystem. Early in 2024, the first stable version of a new test platform, called [Microsoft.Testing.Platform (MTP)](./unit-testing-platform-intro.md), was released.

### Test frameworks

The test framework is built on top of the test platform. It defines the set of attributes and APIs that are available to you, as a test author. It's usually powered by a test adapter, which acts as a communication layer between the test framework and the test platform. The popular test frameworks are MSTest, NUnit, TUnit, and xUnit.net.

#### MSTest

[MSTest](https://github.com/microsoft/testfx) is the Microsoft test framework for all .NET languages. It's extensible and works with .NET CLI, Visual Studio, Visual Studio Code, and Rider. It supports both VSTest and Microsoft.Testing.Platform.

For more information, see the following resources:

- [Unit testing with C#](unit-testing-with-dotnet-test.md)
- [Unit testing with F#](unit-testing-fsharp-with-dotnet-test.md)
- [Unit testing with Visual Basic](unit-testing-visual-basic-with-dotnet-test.md)
- [Microsoft.Testing.Platform support in MSTest (MSTest runner)](unit-testing-mstest-runner-intro.md)
- [Unit testing with C#](unit-testing-with-mstest.md)
- [Unit testing with F#](unit-testing-fsharp-with-mstest.md)
- [Unit testing with Visual Basic](unit-testing-visual-basic-with-mstest.md)

### NUnit
#### NUnit

[NUnit](https://nunit.org) is a unit-testing framework for all .NET languages. Initially, NUnit was ported from JUnit, and the current production release has been rewritten with many new features and support for a wide range of .NET platforms. It's a project of the [.NET Foundation](https://dotnetfoundation.org).
[NUnit](https://nunit.org) is a unit-testing framework for all .NET languages. Initially, NUnit was ported from JUnit, and the current production release has been rewritten with many new features and support for a wide range of .NET platforms. It's a project of the [.NET Foundation](https://dotnetfoundation.org). It supports both VSTest and Microsoft.Testing.Platform.

For more information, see the following resources:

- [Microsoft.Testing.Platform support in NUnit (NUnit runner)](unit-testing-nunit-runner-intro.md)
- [Unit testing with C#](unit-testing-with-nunit.md)
- [Unit testing with F#](unit-testing-fsharp-with-nunit.md)
- [Unit testing with Visual Basic](unit-testing-visual-basic-with-nunit.md)

### MSTest
#### TUnit

[MSTest](https://github.com/microsoft/testfx) is the Microsoft test framework for all .NET languages. It's extensible and works with both .NET CLI and Visual Studio. For more information, see the following resources:
[TUnit](https://thomhurst.github.io/TUnit/) is entirely built on top of Microsoft.Testing.Platform and doesn't support VSTest. For more information, refer to TUnit documentation.

- [Unit testing with C#](unit-testing-with-mstest.md)
- [Unit testing with F#](unit-testing-fsharp-with-mstest.md)
- [Unit testing with Visual Basic](unit-testing-visual-basic-with-mstest.md)
#### xUnit.net

[xUnit.net](https://xunit.net) is a free, open-source, community-focused unit testing tool for .NET. The original inventor of NUnit v2 wrote xUnit.net. xUnit.net is the latest technology for unit testing .NET apps. It also works with ReSharper, CodeRush, and TestDriven.NET. xUnit.net is a project of the [.NET Foundation](https://dotnetfoundation.org) and operates under its code of conduct. It supports both VSTest and Microsoft.Testing.Platform

#### MSTest runner
For more information, see the following resources:

- [Microsoft.Testing.Platform support in xUnit.net v3](https://xunit.net/docs/getting-started/v3/microsoft-testing-platform)
- [Unit testing with C#](unit-testing-with-dotnet-test.md)
- [Unit testing with F#](unit-testing-fsharp-with-dotnet-test.md)
- [Unit testing with Visual Basic](unit-testing-visual-basic-with-dotnet-test.md)

The MSTest runner is a lightweight and portable alternative to [VSTest](https://github.com/microsoft/vstest) for running tests in continuous integration (CI) pipelines, and in Visual Studio Test Explorer. For more information, see [MSTest runner overview](unit-testing-mstest-runner-intro.md).
## Running tests

### .NET CLI

You can run a solutions unit test from the [.NET CLI](../tools/index.md) with the [dotnet test](../tools/dotnet-test.md) command. The .NET CLI exposes most of the functionality that [Integrated Development Environments (IDEs)](#ide) make available through user interfaces. The .NET CLI is cross-platform and available to use as part of continuous integration and delivery pipelines. The .NET CLI is used with scripted processes to automate common tasks.
You can run unit tests from all test projects in a solution using the [.NET CLI](../tools/index.md) with the [dotnet test](../tools/dotnet-test.md) command. The .NET CLI exposes most of the functionality that [Integrated Development Environments (IDEs)](#ide) make available through user interfaces. The .NET CLI is cross-platform and available to use as part of continuous integration and delivery pipelines. The .NET CLI is used with scripted processes to automate common tasks.

### IDE

Whether you're using Visual Studio or Visual Studio Code, there are graphical user interfaces for testing functionality. There are more features available to IDEs than the CLI, for example, [Live Unit Testing](/visualstudio/test/live-unit-testing). For more information, see [Including and excluding tests with Visual Studio](/visualstudio/test/live-unit-testing#include-and-exclude-test-projects-and-test-methods).
Whether you're using Visual Studio, Visual Studio Code, or Rider, there are graphical user interfaces for testing functionality. There are more features available to IDEs than the CLI, for example, [Live Unit Testing](/visualstudio/test/live-unit-testing). For more information, see [Including and excluding tests with Visual Studio](/visualstudio/test/live-unit-testing#include-and-exclude-test-projects-and-test-methods).

## See also

Expand Down
30 changes: 19 additions & 11 deletions docs/core/testing/unit-testing-mstest-runner-intro.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
---
title: MSTest runner overview
title: Microsoft.Testing.Platform support in MSTest (MSTest runner)
description: Learn about the MSTest runner, a lightweight way to run tests without depending on the .NET SDK.
author: nohwnd
ms.author: jajares
ms.date: 12/15/2023
---

# MSTest runner overview
# Microsoft.Testing.Platform support in MSTest (MSTest runner)

The MSTest runner is a lightweight and portable alternative to [VSTest](https://github.com/microsoft/vstest) for running tests in all contexts (for example, continuous integration (CI) pipelines, CLI, Visual Studio Test Explorer, and VS Code Test Explorer). The MSTest runner is embedded directly in your MSTest test projects, and there are no other app dependencies, such as `vstest.console` or `dotnet test`, needed to run your tests.
MSTest supports running tests with both VSTest and [Microsoft.Testing.Platform (MTP)](./unit-testing-platform-intro.md). The support for MTP is powered by the MSTest runner, which can run tests in all contexts (for example, continuous integration (CI) pipelines, CLI, Visual Studio Test Explorer, and VS Code Text Explorer). The MSTest runner is embedded directly in your MSTest test projects, and there are no other app dependencies, such as `vstest.console` or `dotnet test`, needed to run your tests. However, you can still run your tests using `dotnet test`.

The MSTest runner is open source, and builds on a [`Microsoft.Testing.Platform`](./unit-testing-platform-intro.md) library. You can find `Microsoft.Testing.Platform` code in [microsoft/testfx](https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform) GitHub repository. The MSTest runner comes bundled with `MSTest in 3.2.0-preview.23623.1` or newer.
The MSTest runner is open source and builds on the [`Microsoft.Testing.Platform`](./unit-testing-platform-intro.md) library. You can find `Microsoft.Testing.Platform` code in the [microsoft/testfx](https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform) GitHub repository. The MSTest runner comes bundled with `MSTest in 3.2.0` or newer.

## Enable MSTest runner in an MSTest project

Expand All @@ -19,7 +19,7 @@ It's recommended to use [MSTest SDK](./unit-testing-mstest-sdk.md) as it greatly
When you use `MSTest SDK`, by default you're opted in to using MSTest runner.

```xml
<Project Sdk="MSTest.Sdk/3.3.1">
<Project Sdk="MSTest.Sdk/3.8.2">

<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
Expand All @@ -30,7 +30,7 @@ When you use `MSTest SDK`, by default you're opted in to using MSTest runner.
</Project>
```

Alternatively, you can enable MSTest runner by adding the `EnableMSTestRunner` property and setting `OutputType` to `Exe` in your project file. You also need to ensure that you're using `MSTest 3.2.0-preview.23623.1` or newer.
Alternatively, you can enable MSTest runner by adding the `EnableMSTestRunner` property and setting `OutputType` to `Exe` in your project file. You also need to ensure that you're using `MSTest 3.2.0` or newer. We strongly recommend you update to the latest MSTest version available.

Consider the following example project file:

Expand All @@ -40,14 +40,19 @@ Consider the following example project file:
<PropertyGroup>
<!-- Enable the MSTest runner, this is an opt-in feature -->
<EnableMSTestRunner>true</EnableMSTestRunner>
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>

<!--
Displays error on console in addition to the log file. Note that this feature comes with a performance impact.
For more information, visit https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test#show-failure-per-test
-->
<TestingPlatformShowTestsFailure>true</TestingPlatformShowTestsFailure>

<OutputType>Exe</OutputType>

<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>

<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
</PropertyGroup>

<ItemGroup>
Expand All @@ -58,8 +63,11 @@ Consider the following example project file:
MSTest.TestAdapter
MSTest.TestFramework
MSTest.Analyzers
Starting with 3.8, it also includes:
Microsoft.Testing.Extensions.TrxReport
Microsoft.Testing.Extensions.CodeCoverage
-->
<PackageReference Include="MSTest" Version="3.2.0" />
<PackageReference Include="MSTest" Version="3.8.0" />

<!--
Coverlet collector isn't compatible with MSTest runner, you can
Expand All @@ -75,7 +83,7 @@ Consider the following example project file:
```

> [!TIP]
> It's advised to set the `EnableMSTestRunner` property in *Directory.Build.props* file instead of *csproj* file to ensure all test projects in your solution are using the MSTest runner.
> To ensure all test projects in your solution use the MSTest runner, set the `EnableMSTestRunner` and `TestingPlatformDotnetTestSupport` properties in *Directory.Build.props* file instead of individual project files.

## Configurations and filters

Expand Down
30 changes: 19 additions & 11 deletions docs/core/testing/unit-testing-nunit-runner-intro.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,23 @@
---
title: NUnit runner overview
title: Microsoft.Testing.Platform support in NUnit (NUnit runner)
description: Learn about the NUnit runner, a lightweight way to run tests without depending on the .NET SDK.
author: Evangelink
ms.author: amauryleve
ms.date: 05/21/2024
---

# NUnit runner overview
# Microsoft.Testing.Platform support in NUnit (NUnit runner)

The NUnit runner is a lightweight and portable alternative to [VSTest](https://github.com/microsoft/vstest) for running tests in all contexts (for example, continuous integration (CI) pipelines, CLI, Visual Studio Test Explorer, and VS Code Text Explorer). The NUnit runner is embedded directly in your NUnit test projects, and there are no other app dependencies, such as `vstest.console` or `dotnet test`, needed to run your tests.
NUnit supports running tests with both VSTest and [Microsoft.Testing.Platform (MTP)](./unit-testing-platform-intro.md). The support for MTP is powered by the NUnit runner, which can run tests in all contexts (for example, continuous integration (CI) pipelines, CLI, Visual Studio Test Explorer, and VS Code Text Explorer). The NUnit runner is embedded directly in your NUnit test projects, and there are no other app dependencies, such as `vstest.console` or `dotnet test`, needed to run your tests. However, you can still run your tests using `dotnet test`.

The NUnit runner is open source, and builds on a [`Microsoft.Testing.Platform`](./unit-testing-platform-intro.md) library. You can find `Microsoft.Testing.Platform` code in [microsoft/testfx](https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform) GitHub repository. The NUnit runner comes bundled with `NUnit 5.0.0-beta.2` or newer.
The NUnit runner is open source, and builds on top of [`Microsoft.Testing.Platform`](./unit-testing-platform-intro.md). You can find `Microsoft.Testing.Platform` code in [microsoft/testfx](https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform) GitHub repository. The NUnit runner is supported in NUnit3TestAdapter version 5.0 or greater. For more information, see [NUnit and Microsoft.Testing.Platform](https://docs.nunit.org/articles/vs-test-adapter/NUnit-And-Microsoft-Test-Platform.html)

## Enable NUnit runner in a NUnit project

You can enable NUnit runner by adding the `EnableNUnitRunner` property and setting `OutputType` to `Exe` in your project file. You also need to ensure that you're using `NUnit 5.0.0-beta.2` or newer.
You can enable NUnit runner by adding the `EnableNUnitRunner` property and setting `OutputType` to `Exe` in your project file. You also need to ensure that you're using `NUnit3TestAdapter` version 5.0 or newer.

> [!TIP]
> To ensure all test projects in your solution use the NUnit runner, set the `EnableNUnitRunner` and `TestingPlatformDotnetTestSupport` properties in *Directory.Build.props* file instead of individual project files.

Consider the following example project file:

Expand All @@ -24,24 +27,29 @@ Consider the following example project file:
<PropertyGroup>
<!-- Enable the NUnit runner, this is an opt-in feature -->
<EnableNUnitRunner>true</EnableNUnitRunner>
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>

<!--
Displays error on console in addition to the log file. Note that this feature comes with a performance impact.
For more information, visit https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test#show-failure-per-test
-->
<TestingPlatformShowTestsFailure>true</TestingPlatformShowTestsFailure>

<OutputType>Exe</OutputType>

<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>

<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
<PackageReference Include="NUnit" Version="4.1.0" />
<PackageReference Include="NUnit.Analyzers" Version="4.2.0">
<PackageReference Include="NUnit" Version="4.3.2" />
<PackageReference Include="NUnit.Analyzers" Version="4.6.0">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="NUnit3TestAdapter" Version="5.0.0-beta.2" />
<PackageReference Include="NUnit3TestAdapter" Version="5.0.0" />

<!--
Coverlet collector isn't compatible with NUnit runner, you can
Expand Down
Loading
Loading