'Run tests|Debug tests' Pester code lens does not show when Describe block is interpolated string

[bergmeister]

I sometimes include test details in the describe block that are sometimes only known at runtime, however, the Pester code lens does not show any more once I use an interpolated string in the Describe block.
Given the following Pester file (foo.tests.ps1):

$version = $PSVersionTable.PSVersion.Major
Describe "foo on PSVersion $version" {
	It "bar $version" {

	}
}

The Run tests|Debug tests code lens does not show, I see that under the hood, the extension creates parameters to Invoke-Pester that look like -PesterOption @{IncludeVSCodeMarker=$true} -TestName 'foo', in this case, the extension would either need to evaluate the interpolated string or just get rid of the -TestName parameter.

[rjmholt]

Evaluating the string will either be tricky or dangerous, so I think if we can pass this off to Pester we should.

[rkeithhill]

What if we created the command line using double quotes e.g. -PesterOption @{IncludeVSCodeMarker=$true} -TestName "foo on PSVersion $version"? Or maybe, if we can pull the string from the Describe block, it would be better to use that string quoting. That way if you used single quotes to prevent interpolation, we'd honor that.

[bergmeister]

Interestingly the single quote is by design, which this issue documents as it used to be double quotes but was not working for interpolated strings in special scenarios where ToString() was resulting in something that Pester could not understand, maybe this is also an issue with Pester...
Maybe the better solution is to just run Invoke-Pester without the -TestName (i.e. on the whole file) if the Describe string contains a dollar sign

[rkeithhill]

@SeeminglyScience and I had a discussion on the Slack where I think we came to the conclusion there is no good way to support interpolated strings. Many times the variables used in the TestName interpolated string are defined in the test file. However, we must provide a string to Invoke-Pester -TestName that gets executed in the PS integrated console. So we have a catch 22, the variable doesn't get defined until the test executes, but we can't successfully execute the test by name because the variable is not defined yet.

I think the best we can do here is change the code lens for the Describe block to offer to run all tests in the file. This would probably be better than silently not offering anything.

[bergmeister]

Yes, I agree to just omit the TestName parameter if and only if the interpolated string contains an expression to be evaluated, I guess just checking that it contains a dollar sign should be enough to detect that.
I'd like to give it a try to do that myself please since I recently figured out how to build and install the extension myself and I'd like to get some more practice with typescript

[SeeminglyScience]

I'm still very uneasy about the idea of some of the code lenses running all tests and some running only the nearest describe block. I'd much rather see a descriptive error message explaining that the test name should be static.

It's probably not a huge deal for just describes, but ideally we'd support a code lens for every it block. I believe @nohwnd mentioned filtering by it block names is potentially coming in the next major release.

[nohwnd]

Yes, there is v5-alpha1 since yesterday that can discover tests and filter them by It, and only run appropriate setups and teardowns, so it should be ideal for running tests selectively.

It already can filter by path, you can see me do someting very similar with re-running failed tests here.

At the moment those published functions do not take the result object + paths (to keep the api smaller and simpler), instead test discovery is done every time from scratch.

But internally the limitation is not there, if I add functions that clean a dirty discover/run result, it could easily take a cached discovery object and run with different filters. Which is useful not only for this scenario, but for example if you want to observe filesystem, discover tests in a single file, join that discovery result to your cached results and re-run all previously failed tests in all files. Or when you have a huge test suite where discovery takes signifcant amount of time, and multiple configurations to run it with.

[rjmholt]

I'm still very uneasy about the idea of some of the code lenses running all tests and some running only the nearest describe block. I'd much rather see a descriptive error message explaining that the test name should be static.

I 100% agree. I think executing the CodeLens should always do the same thing or fail informatively.

The truth is that you can embed arbitrary expressions in PowerShell strings. But executing a Pester CodeLens, you're going to execute arbitrary code anyway. So maybe we should just get PowerShell to evaluate the string when we think it's not static? Except the big problem there is that executing a single test in a vacuum will quite likely not have the variable populated. Which is a scenario I don't think we should, or can, aim to support -- there's nothing easy about trying to reconstruct arbitrary execution states so that variables exist for interpolation.

So personally I would be happy with any of:

• We reject non-static strings, have a nice informative message saying this, and take the philosophy that parameterised tests don't make sense being plucked out of a file and run individually with the CodeLens (the term "Lens" implies you've lost outside context to me as well).
• We have a heuristic to identify non-static strings and try to evaluate them in our currently PowerShell context when we see them to instantiate the Pester CodeLens name. It might give a different string to running the whole test script and people will expect it to work when it doesn't, but we just would take the line that we meet them halfway and can't take it any further because we can't pick apart the mixing of imperative and declarative paradigms. It will work if the user executes the necessary header lines above it, and I think that's in the spirit of the Pester CodeLens -- pick and choose how you execute your tests.

[SeeminglyScience]

So maybe we should just get PowerShell to evaluate the string when we think it's not static? Except the big problem there is that executing a single test in a vacuum will quite likely not have the variable populated.

Yeah, every example I've seen so far has had the variable defined at the top of the test file. More specifically, it's not even going to be defined at the point we run Invoke-Pester.

For example:

# Invoke-MyCommand.Tests.ps1
$CommandName = $MyInvocation.MyCommand.Name.Replace('Tests.ps1', '')

Describe "$CommandName Integration Tests" {
    # ...
}

If we get PowerShell to evaluate:

Invoke-Pester Invoke-MyCommand.Tests.ps1 -TestName "$CommandName Integration Tests"

Then TestName will evaluate to Integration Tests. The only way I could see that being supportable is if the TestName could take a ScriptBlock as an argument (or if the discovery object scenario @nohwnd described would support that)