Publishing test results with failed tests does not fail the build

[strtdusty]

I am using xunit test through DNX. For each project we generate a test result xml file and use the "publish test results" task to publish the results to the build. The results get published correctly and we can see that some tests failed. When tests fail, however, the build is not marked as failed. I could fail the build when running xunit, but the results would not get published. Is there a way for the build to look at the published test results and fail if there are test failures?

[chrispat]

You can mark the Publish Test Results task as Always Run and it will run even for a failed build.

[richardszalay]

@chrisrpatterson Your response is unrelated to @strtdusty's issue. The issue is that if a test fail, the "build" is still marked as successful. The successful build contains a failed test, sure, but it will go unnoticed so it's almost irrelevant.

[liamnichols]

+1 for this. I want to use the VSTS build system for our mobile projects but back it via Jenkins jobs as the Xcode support is somewhat lacking for my needs.

My problem is that I have to mark the Jenkins job as unstable as I can only download artefacts (my junit results) from the "last successful build". This then means that my build appears to have succeeded if the user doesn't actually click into the build to investigate.

[Yaash19]

@strtdusty

how did you resolve this ? i am having similar issue.

[strtdusty]

I used @chrisrpatterson 's suggestion. Our builds end up being marked as failed though which is correct.

[asinitson]

Why this is marked as closed? Although problem is resolved for the original request, the issue Publishing test results with failed tests does not fail the build is still not resolved as @liamnichols mentioned. Here are the steps to reproduce:

1. Create build definition with only one step: "Publish Test Results"
2. Run the build

Expected result:
If some of the tests are failed, then build should also fail.

Actual result:
Build always passes, irrespective of the results of the test.

Solution proposed above only works if you run the tests as a previous build step. But it is still impossible to fail the build purely depending on the published results.

Why is this important? If you have your build running outside of TFS bubble and just "publish" the results of the tests, then you still want to know a status of the build.

[pakdev]

@chrisrpatterson Can you please re-open this issue given @asinitson's rationale?

[alejandro5042]

There should at least be a checkbox for selecting whether the published test results can fail the build.

[kngrace]

I'm running two different test suites and I am able to fail the build with the test task itself for only one of them but not for the other. I'd like to be able to rely on the Publish Test Results task to mark a build failure if any of the tests it has published have failed.

[pakdev]

This is also a problem for Javascript tests/linting. My jshint/jscs failures are reported in the Tests tab, but the "npm test" and "Publish Test Results" tasks pass.

[asinitson]

Hey, everyone!

Since original bug reporter has his problem solved and this bug report is closed, I guess it makes sense to add another bug report. @chrisrpatterson: Is anybody looking at the comments for this closed bug report? Are you considering re-opening it?

I will wait a day or two for the reply and then create another report, so we at least would have some kind of statement: "won't fix" or "planned in some future release".

[ericsciple]

User voice is probably a better place for this feedback.

I'll re-open and assign to the appropriate team to see the feedback. This repo isn't used to track feature requests, so I would expect the issue to get closed again.