Rework Configuration-Time Test Category Filtering Mechanism #192
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm proposing to change the mechanism for filtering the test categories (= subfolders inside
tests/
containing aCMakeLists.txt
file) during configuration time.Whereas the previous approach (
SYCL_CTS_TEST_FILTER
) used to specify all test categories that are to be included in the compilation, I'm proposing to do the opposite, i.e., list all categories that are to be excluded. To underline this change, I've renamed the CMake config parameter toSYCL_CTS_EXCLUDE_TEST_CATEGORIES
.Switching from inclusion to exclusion has the benefit that (when a filter is used) newly added tests are being compiled by default, instead of having to be added to a list manually. If we end up compiling tests during CI (which I'm still very much planning on doing), we will have to limit the set of tests compiled by each implementation somehow (as no implementation can currently compile all tests). Using an exclusion filter will therefore highlight potential problems for different SYCL implementations early.
I've added filters for ComputeCpp, DPCPP and hipSYCL, reflecting their current state of CTS support. It would be great if someone could confirm these on their setup (either use them and check that everything else compiles, or ideally re-generate the filters -- see below).
Having these filters within the repository for each implementation also serves as a nice way of tracking the current state of CTS support, where an empty list indicates that all tests compile successfully. It also shows which test categories we may want to focus on most. For example, currently none of the implementations seem to be able to compile the
accessor
tests.As tests and implementations improve over time, these filters should be regenerated and updated within the repository. To streamline this process, I'm introducing a new utility script called
generate_exclude_filter.py
that makes this rather simple. Basically, the script attempts to compile all targets for a given SYCL implementation, spitting out the failing ones, which can then be used as a filter.I've removed the ability to specify a filter when generating a conformance report using the
run_conformance_tests.py
script. Frankly I never quite understood why this feature was there (and mandatory!) in the first place. We can re-introduce it should the need arise.Lastly, I've removed the
conformance_filters
directory containing thecore.csv
andcodeplay.csv
filters.