CI check each minimum dependency is enforced in *.yaml and environment.yml files

[MarcoGorelli]

xref #50205 (comment)

If any dependency has a minimum version specified in

pandas/pyproject.toml

Lines 57 to 119 in 16b9c98

	[project.optional-dependencies]
	test = ['hypothesis>=5.5.3', 'pytest>=6.0', 'pytest-xdist>=1.31', 'pytest-asyncio>=0.17.0']
	performance = ['bottleneck>=1.3.2', 'numba>=0.53.1', 'numexpr>=2.7.1']
	timezone = ['tzdata>=2022.1']
	computation = ['scipy>=1.7.1', 'xarray>=0.21.0']
	fss = ['fsspec>=2021.07.0']
	aws = ['s3fs>=2021.08.0']
	gcp = ['gcsfs>=2021.07.0', 'pandas-gbq>=0.15.0']
	excel = ['odfpy>=1.4.1', 'openpyxl>=3.0.7', 'pyxlsb>=1.0.8', 'xlrd>=2.0.1', 'xlsxwriter>=1.4.3']
	parquet = ['pyarrow>=6.0.0']
	feather = ['pyarrow>=6.0.0']
	hdf5 = [# blosc only available on conda (https://github.com/Blosc/python-blosc/issues/297)
	#'blosc>=1.20.1',
	'tables>=3.6.1']
	spss = ['pyreadstat>=1.1.2']
	postgresql = ['SQLAlchemy>=1.4.16', 'psycopg2>=2.8.6']
	mysql = ['SQLAlchemy>=1.4.16', 'pymysql>=1.0.2']
	sql-other = ['SQLAlchemy>=1.4.16']
	html = ['beautifulsoup4>=4.9.3', 'html5lib>=1.1', 'lxml>=4.6.3']
	xml = ['lxml>=4.6.3']
	plot = ['matplotlib>=3.6.1']
	output_formatting = ['jinja2>=3.0.0', 'tabulate>=0.8.9']
	clipboard = ['PyQt5>=5.15.1', 'qtpy>=2.2.0']
	compression = ['brotlipy>=0.7.0', 'python-snappy>=0.6.0', 'zstandard>=0.15.2']
	all = ['beautifulsoup4>=4.9.3',
	# blosc only available on conda (https://github.com/Blosc/python-blosc/issues/297)
	#'blosc>=1.21.0',
	'bottleneck>=1.3.2',
	'brotlipy>=0.7.0',
	'fastparquet>=0.6.3',
	'fsspec>=2021.07.0',
	'gcsfs>=2021.07.0',
	'html5lib>=1.1',
	'hypothesis>=6.13.0',
	'jinja2>=3.0.0',
	'lxml>=4.6.3',
	'matplotlib>=3.6.1',
	'numba>=0.53.1',
	'numexpr>=2.7.3',
	'odfpy>=1.4.1',
	'openpyxl>=3.0.7',
	'pandas-gbq>=0.15.0',
	'psycopg2>=2.8.6',
	'pyarrow>=6.0.0',
	'pymysql>=1.0.2',
	'PyQt5>=5.15.1',
	'pyreadstat>=1.1.2',
	'pytest>=6.0',
	'pytest-xdist>=1.31',
	'pytest-asyncio>=0.17.0',
	'python-snappy>=0.6.0',
	'pyxlsb>=1.0.8',
	'qtpy>=2.2.0',
	'scipy>=1.7.1',
	's3fs>=2021.08.0',
	'SQLAlchemy>=1.4.16',
	'tables>=3.6.1',
	'tabulate>=0.8.9',
	'tzdata>=2022.1',
	'xarray>=0.21.0',
	'xlrd>=2.0.1',
	'xlsxwriter>=1.4.3',
	'zstandard>=0.15.2']

then if that dependency appears in environment.yml or any of the *.yaml files, then it should either be pinned to that minimum version, or be marked as >= that minimum version

Let's add a script to automate checking this!

https://github.com/pandas-dev/pandas/blob/main/scripts/validate_min_versions_in_sync.py is kind of related

---

Feel free to tag me for review if you take this on

[mroeschke]

Notes:

1. ci/deps/actions-*-minimum_versions.yaml should be pinned as = instead of >=
2. IMO validate_min_versions_in_sync.py should be updated to ensure the dependency files are also in sync

Personally not sure if this is a good first issue as there are a lot of moving parts but happy to be wrong

[seanjedi]

Hmm, I believe I understand the ask.
So essentially get all the dependencies from pyproject.toml optional-dependencies, and compare them against the dependencies within environment.yml, and if there are any dependencies that are newer within pyproject.toml tan environment.yaml, then update environment.yaml to either = or >= correct?
So if that is the case, then:

1. how do we determine when we want to have it = or >=? Would that be depdendent on how pyproject.toml has it?
2. what if environment.yml somehow has a newer depdency, then should be ignore it, or should we downgrade it to the version that pyproject has it at?

Also can I take this?

[MarcoGorelli]

I'd say:

1. if it already has = then keep as-is, else, add >=
2. not sure I understand the question, can you show an example?

And yes, sure!

[seanjedi]

For 2 let's say hypothetically we have a situation where
In environment.yml we have:

- numba=0.53.1

and in pyproject we have:

numba>=0.52

Should I check if the dependencies in pyproject are indeed greater than the ones in environment, or if they differ then set the dependencies to pyproject
This might be all moot, but I wanted to doublecheck since in my head the way I was thinking of comparing the two is if pyproject has a dependency greater than environment, then update environment, otherwise stay the same, or do we want the two be equal?
I think the case we have, is that we want it to be equal, so above even though environment has a great version, we should have it pinned to be numba>=0.52

[seanjedi]

take

[MarcoGorelli]

If there's already a pin, then as long as the version is greater than the minimum version, you can leave it as it is

If there's no pin, then just put >= and the minimum version

[seanjedi]

So should I create a new file for this, or extend validate_min_versions_in_sync.py to be able to update these dependencies as well?

[MarcoGorelli]

it's probably fine to extend that file if possible

[seanjedi]

Created a branch here to work on the issue: seanjedi-50207

[seanjedi]

@MarcoGorelli I have a question regarding which YAML files to check
Right now I am checking this list of YAML file (these are all the .yml files I can find in both the root and ci/deps, not all of these have dependencies, that is correct right? )

Also looking at how we were checking the dependencies earlier, it seems that we were only concerned with the optional dependencies, should I only be checking optional dependencies, or if I find any dependency that is both within pyproject.toml and any yaml file, then I should pin it?

[MarcoGorelli]

you only need to look at environment.yml and ci/deps/*.yaml, the other yaml files are irrelevant

I find any dependency that is both within pyproject.toml and any yaml file, then I should pin it

yup (unless it's pinned already)

[seanjedi]

Update (Since I have not updated in a long while): I am still working on this issue, but progress has slowed down a lot since the last commit I made in my branch.
This began with the Holiday's taking my time away so that I can spend it with family.
Now it is due to graduate school, and a few requirements for graduation that appeared that I have to appeal against now due to administration.
I bring this up so that if someone sees this, I want to ensure that this issue is not dead and that if anyone else wants to work on this issue, I would be open to that (especially collaboratively), and please let me know so that I can let you know what I have worked on so that we don't end up doing the same work.

Otherwise, I will continue working on this issue when I have the chance.