xdist is not executing tests if their parametrizations were collected in a different order #596
Description
Originally reported by: BitBucket: liori, GitHub: liori
Related to #594, but slightly different.
I generated my parametrizations by iterating over a set of strings. However, this iteration depends on the specific values the strings hash to, and these values might be different for every process. Therefore, whereas I was always generating exactly the same parametrizations, they were in different order.
A simple test case that shows the problem:
#!python
import pytest
my_names = {'john', 'kate', 'alfred', 'paul', 'mary'}
@pytest.mark.parametrize('name', list(my_names), ids=list(my_names))
def test_is_name_short(name):
assert len(name) < 7
Run with PYTHONHASHSEED=random py.test -n 4 to make sure you trigger randomized hashing for strings.
I think that a simple sort in report_collection_diff before running unified_diff might help with this problem.