diff --git a/docs/ai/conceptual/evaluation-libraries.md b/docs/ai/conceptual/evaluation-libraries.md index e07e9eca8c355..6101eb1d61d69 100644 --- a/docs/ai/conceptual/evaluation-libraries.md +++ b/docs/ai/conceptual/evaluation-libraries.md @@ -18,7 +18,7 @@ The evaluation libraries, which are built on top of the [Microsoft.Extensions.AI ## Test integration -The libraries are designed to integrate smoothly with existing .NET apps, allowing you to leverage existing testing infrastructures and familiar syntax to evaluate intelligent apps. You can use any test framework (for example, [MSTest](../../core/testing/index.md#mstest), [xUnit](../../core/testing/index.md#xunit), or [NUnit](../../core/testing/index.md#nunit)) and testing workflow (for example, [Test Explorer](/visualstudio/test/run-unit-tests-with-test-explorer), [dotnet test](../../core/tools/dotnet-test.md), or a CI/CD pipeline). The library also provides easy ways to do online evaluations of your application by publishing evaluation scores to telemetry and monitoring dashboards. +The libraries are designed to integrate smoothly with existing .NET apps, allowing you to leverage existing testing infrastructures and familiar syntax to evaluate intelligent apps. You can use any test framework (for example, [MSTest](../../core/testing/index.md#mstest), [xUnit](../../core/testing/index.md#xunitnet), or [NUnit](../../core/testing/index.md#nunit)) and testing workflow (for example, [Test Explorer](/visualstudio/test/run-unit-tests-with-test-explorer), [dotnet test](../../core/tools/dotnet-test.md), or a CI/CD pipeline). The library also provides easy ways to do online evaluations of your application by publishing evaluation scores to telemetry and monitoring dashboards. ## Comprehensive evaluation metrics diff --git a/docs/core/testing/index.md b/docs/core/testing/index.md index 221dc0ca560af..e3f1b6babdc96 100644 --- a/docs/core/testing/index.md +++ b/docs/core/testing/index.md @@ -35,47 +35,64 @@ Keep in mind there are [best practices](unit-testing-best-practices.md) for writ ## Testing tools -.NET is a multi-language development platform, and you can write various test types for [C#](../../csharp/index.yml), [F#](../../fsharp/index.yml), and [Visual Basic](../../visual-basic/index.yml). For each of these languages, you can choose between several test frameworks. +When running tests in .NET, there are two components involved: the test platform and the test framework. -### xUnit +### Test platforms -[xUnit](https://xunit.net) is a free, open-source, community-focused unit testing tool for .NET. The original inventor of NUnit v2 wrote xUnit.net. xUnit.net is the latest technology for unit testing .NET apps. It also works with ReSharper, CodeRush, and TestDriven.NET. xUnit.net is a project of the [.NET Foundation](https://dotnetfoundation.org) and operates under its code of conduct. +The test platform is the engine that runs the tests and acts as a communication channel with IDEs. For example, Visual Studio can send a discovery request to the test platform so that it can display the available tests in Test Explorer. The test platform responds back to the IDE with the tests it found. Similar communication happens for test execution. + +VSTest has been used for many years in .NET and was the only test platform in the ecosystem. Early in 2024, the first stable version of a new test platform, called [Microsoft.Testing.Platform (MTP)](./unit-testing-platform-intro.md), was released. + +### Test frameworks + +The test framework is built on top of the test platform. It defines the set of attributes and APIs that are available to you, as a test author. It's usually powered by a test adapter, which acts as a communication layer between the test framework and the test platform. The popular test frameworks are MSTest, NUnit, TUnit, and xUnit.net. + +#### MSTest + +[MSTest](https://github.com/microsoft/testfx) is the Microsoft test framework for all .NET languages. It's extensible and works with .NET CLI, Visual Studio, Visual Studio Code, and Rider. It supports both VSTest and Microsoft.Testing.Platform. For more information, see the following resources: -- [Unit testing with C#](unit-testing-with-dotnet-test.md) -- [Unit testing with F#](unit-testing-fsharp-with-dotnet-test.md) -- [Unit testing with Visual Basic](unit-testing-visual-basic-with-dotnet-test.md) +- [Microsoft.Testing.Platform support in MSTest (MSTest runner)](unit-testing-mstest-runner-intro.md) +- [Unit testing with C#](unit-testing-with-mstest.md) +- [Unit testing with F#](unit-testing-fsharp-with-mstest.md) +- [Unit testing with Visual Basic](unit-testing-visual-basic-with-mstest.md) -### NUnit +#### NUnit -[NUnit](https://nunit.org) is a unit-testing framework for all .NET languages. Initially, NUnit was ported from JUnit, and the current production release has been rewritten with many new features and support for a wide range of .NET platforms. It's a project of the [.NET Foundation](https://dotnetfoundation.org). +[NUnit](https://nunit.org) is a unit-testing framework for all .NET languages. Initially, NUnit was ported from JUnit, and the current production release has been rewritten with many new features and support for a wide range of .NET platforms. It's a project of the [.NET Foundation](https://dotnetfoundation.org). It supports both VSTest and Microsoft.Testing.Platform. For more information, see the following resources: +- [Microsoft.Testing.Platform support in NUnit (NUnit runner)](unit-testing-nunit-runner-intro.md) - [Unit testing with C#](unit-testing-with-nunit.md) - [Unit testing with F#](unit-testing-fsharp-with-nunit.md) - [Unit testing with Visual Basic](unit-testing-visual-basic-with-nunit.md) -### MSTest +#### TUnit -[MSTest](https://github.com/microsoft/testfx) is the Microsoft test framework for all .NET languages. It's extensible and works with both .NET CLI and Visual Studio. For more information, see the following resources: +[TUnit](https://thomhurst.github.io/TUnit/) is entirely built on top of Microsoft.Testing.Platform and doesn't support VSTest. For more information, refer to TUnit documentation. -- [Unit testing with C#](unit-testing-with-mstest.md) -- [Unit testing with F#](unit-testing-fsharp-with-mstest.md) -- [Unit testing with Visual Basic](unit-testing-visual-basic-with-mstest.md) +#### xUnit.net + +[xUnit.net](https://xunit.net) is a free, open-source, community-focused unit testing tool for .NET. The original inventor of NUnit v2 wrote xUnit.net. xUnit.net is the latest technology for unit testing .NET apps. It also works with ReSharper, CodeRush, and TestDriven.NET. xUnit.net is a project of the [.NET Foundation](https://dotnetfoundation.org) and operates under its code of conduct. It supports both VSTest and Microsoft.Testing.Platform -#### MSTest runner +For more information, see the following resources: + +- [Microsoft.Testing.Platform support in xUnit.net v3](https://xunit.net/docs/getting-started/v3/microsoft-testing-platform) +- [Unit testing with C#](unit-testing-with-dotnet-test.md) +- [Unit testing with F#](unit-testing-fsharp-with-dotnet-test.md) +- [Unit testing with Visual Basic](unit-testing-visual-basic-with-dotnet-test.md) -The MSTest runner is a lightweight and portable alternative to [VSTest](https://github.com/microsoft/vstest) for running tests in continuous integration (CI) pipelines, and in Visual Studio Test Explorer. For more information, see [MSTest runner overview](unit-testing-mstest-runner-intro.md). +## Running tests ### .NET CLI -You can run a solutions unit test from the [.NET CLI](../tools/index.md) with the [dotnet test](../tools/dotnet-test.md) command. The .NET CLI exposes most of the functionality that [Integrated Development Environments (IDEs)](#ide) make available through user interfaces. The .NET CLI is cross-platform and available to use as part of continuous integration and delivery pipelines. The .NET CLI is used with scripted processes to automate common tasks. +You can run unit tests from all test projects in a solution using the [.NET CLI](../tools/index.md) with the [dotnet test](../tools/dotnet-test.md) command. The .NET CLI exposes most of the functionality that [Integrated Development Environments (IDEs)](#ide) make available through user interfaces. The .NET CLI is cross-platform and available to use as part of continuous integration and delivery pipelines. The .NET CLI is used with scripted processes to automate common tasks. ### IDE -Whether you're using Visual Studio or Visual Studio Code, there are graphical user interfaces for testing functionality. There are more features available to IDEs than the CLI, for example, [Live Unit Testing](/visualstudio/test/live-unit-testing). For more information, see [Including and excluding tests with Visual Studio](/visualstudio/test/live-unit-testing#include-and-exclude-test-projects-and-test-methods). +Whether you're using Visual Studio, Visual Studio Code, or Rider, there are graphical user interfaces for testing functionality. There are more features available to IDEs than the CLI, for example, [Live Unit Testing](/visualstudio/test/live-unit-testing). For more information, see [Including and excluding tests with Visual Studio](/visualstudio/test/live-unit-testing#include-and-exclude-test-projects-and-test-methods). ## See also diff --git a/docs/core/testing/unit-testing-mstest-runner-intro.md b/docs/core/testing/unit-testing-mstest-runner-intro.md index ee3ec54f447ec..747de12784ebb 100644 --- a/docs/core/testing/unit-testing-mstest-runner-intro.md +++ b/docs/core/testing/unit-testing-mstest-runner-intro.md @@ -1,16 +1,16 @@ --- -title: MSTest runner overview +title: Microsoft.Testing.Platform support in MSTest (MSTest runner) description: Learn about the MSTest runner, a lightweight way to run tests without depending on the .NET SDK. author: nohwnd ms.author: jajares ms.date: 12/15/2023 --- -# MSTest runner overview +# Microsoft.Testing.Platform support in MSTest (MSTest runner) -The MSTest runner is a lightweight and portable alternative to [VSTest](https://github.com/microsoft/vstest) for running tests in all contexts (for example, continuous integration (CI) pipelines, CLI, Visual Studio Test Explorer, and VS Code Test Explorer). The MSTest runner is embedded directly in your MSTest test projects, and there are no other app dependencies, such as `vstest.console` or `dotnet test`, needed to run your tests. +MSTest supports running tests with both VSTest and [Microsoft.Testing.Platform (MTP)](./unit-testing-platform-intro.md). The support for MTP is powered by the MSTest runner, which can run tests in all contexts (for example, continuous integration (CI) pipelines, CLI, Visual Studio Test Explorer, and VS Code Text Explorer). The MSTest runner is embedded directly in your MSTest test projects, and there are no other app dependencies, such as `vstest.console` or `dotnet test`, needed to run your tests. However, you can still run your tests using `dotnet test`. -The MSTest runner is open source, and builds on a [`Microsoft.Testing.Platform`](./unit-testing-platform-intro.md) library. You can find `Microsoft.Testing.Platform` code in [microsoft/testfx](https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform) GitHub repository. The MSTest runner comes bundled with `MSTest in 3.2.0-preview.23623.1` or newer. +The MSTest runner is open source and builds on the [`Microsoft.Testing.Platform`](./unit-testing-platform-intro.md) library. You can find `Microsoft.Testing.Platform` code in the [microsoft/testfx](https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform) GitHub repository. The MSTest runner comes bundled with `MSTest in 3.2.0` or newer. ## Enable MSTest runner in an MSTest project @@ -19,7 +19,7 @@ It's recommended to use [MSTest SDK](./unit-testing-mstest-sdk.md) as it greatly When you use `MSTest SDK`, by default you're opted in to using MSTest runner. ```xml - + net8.0 @@ -30,7 +30,7 @@ When you use `MSTest SDK`, by default you're opted in to using MSTest runner. ``` -Alternatively, you can enable MSTest runner by adding the `EnableMSTestRunner` property and setting `OutputType` to `Exe` in your project file. You also need to ensure that you're using `MSTest 3.2.0-preview.23623.1` or newer. +Alternatively, you can enable MSTest runner by adding the `EnableMSTestRunner` property and setting `OutputType` to `Exe` in your project file. You also need to ensure that you're using `MSTest 3.2.0` or newer. We strongly recommend you update to the latest MSTest version available. Consider the following example project file: @@ -40,14 +40,19 @@ Consider the following example project file: true + true + + + true + Exe net8.0 enable enable - - false - true @@ -58,8 +63,11 @@ Consider the following example project file: MSTest.TestAdapter MSTest.TestFramework MSTest.Analyzers + Starting with 3.8, it also includes: + Microsoft.Testing.Extensions.TrxReport + Microsoft.Testing.Extensions.CodeCoverage --> - + true + true + + + true + Exe net8.0 enable enable - - false - true - - + + runtime; build; native; contentfiles; analyzers; buildtransitive all - +