Skip to content

[Experimental] Capturing values in exit tests #1040

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Apr 15, 2025

Conversation

grynspan
Copy link
Contributor

@grynspan grynspan commented Mar 27, 2025

This PR implements an experimental form of state capture in exit tests. If you specify a capture list on the test's body closure and explicitly write the type of each captured value, and each value conforms to Sendable, Codable, and (implicitly) Copyable, we'll encode them and send them "over the wire" to the child process:

let a = Int.random(in: 100 ..< 200)
await #expect(exitsWith: .failure) { [a = a as Int] in
  assert(a > 500)
}

This PR is incomplete. Among other details:

  • Need to properly transmit the data, not stuff it in an environment variable
  • Need to implement diagnostics correctly
  • Need to figure out if ExitTest.CapturedValue and __Expression.Value have any synergy. (They do, but it's beyond the scope of this initial/experimental PR.)

We are ultimately constrained by the language here as we don't have real type information for the captured values, nor can we infer captures by inspecting the syntax of the exit test body (hence the need for an explicit capture list with types.) If we had something like decltype() we could apply during macro expansion, you wouldn't need to write x = x as T and could just write x. The macro would use decltype() to produce a thunk function of the form:

@Sendable func __compiler_generated_name__(: decltype(a),: decltype(b),: decltype(c)) async throws {
  let (a, b, c) = (,,)
  try await { /* ... */ }()
}

Checklist:

  • Code and documentation should follow the style of the Style Guide.
  • If public symbols are renamed or modified, DocC references should be updated.

@grynspan grynspan added enhancement New feature or request public-api Affects public API exit-tests ☠️ Work related to exit tests parameterized-testing Related to parameterized testing functionality macros 🔭 Related to Swift macros such as @Test or #expect labels Mar 27, 2025
@grynspan grynspan added this to the Swift 6.x milestone Mar 27, 2025
@grynspan grynspan self-assigned this Mar 27, 2025
@grynspan
Copy link
Contributor Author

@swift-ci test

4 similar comments
@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan grynspan force-pushed the jgrynspan/exit-test-value-capture branch from c2a2534 to 74101e0 Compare March 29, 2025 17:34
@grynspan
Copy link
Contributor Author

@swift-ci test

3 similar comments
@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan grynspan force-pushed the jgrynspan/exit-test-value-capture branch from ec3ff3d to 514ed01 Compare March 31, 2025 19:10
@grynspan
Copy link
Contributor Author

@swift-ci test

1 similar comment
@grynspan
Copy link
Contributor Author

grynspan commented Apr 1, 2025

@swift-ci test

@grynspan grynspan changed the title Capturing values in exit tests [WIP, DNM] Capturing values in exit tests Apr 1, 2025
@grynspan grynspan force-pushed the jgrynspan/exit-test-value-capture branch from 9f06c93 to 6db82cf Compare April 1, 2025 18:06
@grynspan grynspan changed the title [WIP, DNM] Capturing values in exit tests [Experimental] Capturing values in exit tests Apr 6, 2025
This PR implements an experimental form of state capture in exit tests. If you
specify a capture list on the test's body closure and explicitly write the type
of each captured value, _and_ each value conforms to `Sendable`, `Codable`, and
(implicitly) `Copyable`, we'll encode them and send them "over the wire" to the
child process:

```swift
let a = Int.random(in: 100 ..< 200)
await #expect(exitsWith: .failure) { [a = a as Int] in
  assert(a > 500)
}
```

This PR is incomplete. Among other details:

- [] Need to properly transmit the data, not stuff it in an environment variable
- [] Need to capture source location information correctly for error handling
     during value decoding
- [] Need to implement diagnostics correctly

We are ultimately constrained by the language here as we don't have real type
information for the captured values, nor can we infer captures by inspecting the
syntax of the exit test body (hence the need for an explicit capture list with
types.) If we had something like `decltype()` we could apply during macro
expansion, you wouldn't need to write `x = x as T` and could just write `x`.
@grynspan grynspan force-pushed the jgrynspan/exit-test-value-capture branch from 6db82cf to 6f87c49 Compare April 6, 2025 21:41
@grynspan
Copy link
Contributor Author

grynspan commented Apr 6, 2025

@swift-ci test

@grynspan grynspan marked this pull request as ready for review April 6, 2025 21:44
@grynspan
Copy link
Contributor Author

grynspan commented Apr 9, 2025

@swift-ci test

@grynspan
Copy link
Contributor Author

grynspan commented Apr 9, 2025

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan grynspan requested a review from briancroom April 14, 2025 22:18
@grynspan
Copy link
Contributor Author

@swift-ci test

Comment on lines 775 to +778
childEnvironment["SWT_EXPERIMENTAL_BACKCHANNEL"] = backChannelEnvironmentVariable
}
if let capturedValuesEnvironmentVariable = _makeEnvironmentVariable(for: capturedValuesReadEnd) {
childEnvironment["SWT_EXPERIMENTAL_CAPTURED_VALUES"] = capturedValuesEnvironmentVariable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd find it nice if the two env vars used a similar naming convention as each other

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, sorry for not elaborating…

My understanding of these two env vars is that they essentially contain a file handle to a pipe, and there's now two such pipes: one allowing communication from the parent to exit test child, and one allowing communication from the exit test child to the parent.

My suggestion was to choose names for these two env vars that more closely resemble each other, reflecting that they are "opposites", and convey the direction of communication (since they are each one-way). For example, you might choose:

  • SWT_EXPERIMENTAL_PARENT_TO_EXIT_TEST_FILE_HANDLE, and
  • SWT_EXPERIMENTAL_EXIT_TEST_TO_PARENT_FILE_HANDLE

You can tweak the wording of course, but I like that this example has symmetry between the names, you can tell at a glance which one is which direction, and they are more future-proof in the sense that each pipe could evolve to include more kinds of data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the moment, their names represent what they are and what they do. In the future, if either/both become host to more functionality, we can rename them. There's no ABI stability concern here as both endpoints are the same version of Swift Testing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood that we're free to evolve them at any time and that's great. Still, I find them more self-describing if they conveyed their direction in the names. For example I don't know whether the "back channel" is going from parent to exit test or vice-versa without looking it up. Both pipes could be considered "back channels".

Can we incorporate this feedback if/when we elevate the feature to public, since that will involve removing "experimental" from the names?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's save this discussion for a separate PR.

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

Copy link
Contributor

@stmontgomery stmontgomery left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be helpful to rename the env vars as I suggested in a comment still, mainly to make their roles more self-describing. On the whole, this looks great though!

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan grynspan merged commit 4e4885d into main Apr 15, 2025
3 checks passed
@grynspan grynspan deleted the jgrynspan/exit-test-value-capture branch April 15, 2025 17:33
grynspan added a commit that referenced this pull request Apr 15, 2025
@stmontgomery stmontgomery modified the milestones: Swift 6.x, Swift 6.2 Apr 24, 2025
@grynspan grynspan added the exit-test-capture-lists 🥍 Work related to exit test capture lists label Jun 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request exit-test-capture-lists 🥍 Work related to exit test capture lists exit-tests ☠️ Work related to exit tests macros 🔭 Related to Swift macros such as @Test or #expect parameterized-testing Related to parameterized testing functionality public-api Affects public API
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants