Discussion: Migrating to setup.py to poetry

[chayim]

As redis-py 4.0.0 now targets python versions > 3.6 we can discuss new ways of packaging. Poetry makes dependnecy management for both development and the library itself simpler - reducing the need to manually track the dev_requirements.txt. It easily integrates with tox via the tox-poetry plugin, and makes publishing packages to Pypi easier that twine.

I'm openly declaring that my bias is to move from setup.py to poetry, including some of the pain points that comes with it. I'd love to hear what the community thinks. Please tag others so that they can weigh in if they so desire.

[chayim]

@WisdomPill @2014BDuck @barshaul @davidylee Starting here, since we've recently collaborated on PRs.

[WisdomPill]

I personally do not use poetry, for personal and work related projects I use pipenv, but I have heard that there is a lot of excitement about poetry and want to further explore it to see if it fits my needs.

So for me it's okay, I can always learn from these migrations.

[jiekun]

Personally speaking, I don't use poetry (not yet) so I may not be able to provide useful suggestions. Obviously, it's for developers, so an upgrade/migration is good to go if it could provide enough guide/document.

We have a few lines describing the installing things on README. So I would prefer some description about new things we are going to use on README too.

Lastly, maybe we could have someone who has experience with poetry to share his feeling. Maybe in this issue or somewhere else. :)

[chayim]

I mean I have a tonne of experience with poetry, but I wanted to not bias. Since you asked, here's what I like (in short):

• The pyproject.toml is PEP (508, 516, 517) compliant. This means that we can specify building the package, it's dependencies, pypi fields, etc in one place.
  • The outcome of this is that one can still pip install directly from the repo, and pypi
• Separation of project dependencies, optional dependencies, and developer dependencies
• Dependency resolution that can be python specific (though only one poetry.lock)
• Parallel dependency fetching
• Eliminates the pip freeze need / issue - where we track non-project dependencies, due to the nature of pip.
• Manages virtualenvs, and can also not (I choose not personally, but that's a local setting)
• Built-in package publishing

[barshaul]

I've never worked with poetry before, but it sounds interesting. Will this migration have any downsides?

[chayim]

The only issue is that it can be more difficult to work on multiple versions of python in parallel - given the poetry.lock file, should versions need to be semantically different per python. However, redis-py has no actual dependencies (just dev deps, and an optional hiredis), so IMHO there isn't a realistic downside.

@AvitalFineRedis I realize I didn't include you - what do you think?

[ddevault]

As a distribution maintainer: please no. Poetry is great for workstations and awful for distributions.

[chayim]

@ddevault Can you help me understand how - I'm all good with not, but I'd love to know how it's bad for distributions, since I've never experienced this. Thank you in advance!

[ddevault]

setup.py has already been through the gamut of distribution support patches and has already been refined to support each possible need. Python modules have been packaged by a huge variety of distros with a huge diversity of needs and configurations - Alpine, Debian, NixOS, Guix, FreeBSD, OpenBSD, NetBSD, Illumos, even Plan 9 - all have different needs, and all already know how to deal with setup.py, and setup.py already knows how to deal with all of them. Poetry is one of many attempts by the Docker generation to build the next iteration of virtualenvs, entirely disregarding the idea that a distro might even be involved and encouraging huge, stale dependency trees and cowboy deployments instead. It has zero accomodations for distro needs and has not had nearly enough time to mature to support as many use-cases as setup.py already does.

[chayim]

@ddevault I actually came here all prepared to argue with you - well disagree, but empathising with your position. My poetry use case definitely isn't for venv management (I view it as a dep manager, nothing else). It is purely for dependency management and separation, and using pyproject.toml Realistically, I see us moving there in the future since pyproject.toml is the ongoing future of python and there's less to maintain. Than being the case I'm not arguing.

Yesterday I was bit by the pain in python typing. Today I went too far down the rabbit hole. I spent enough time fighting that one.

poetry overall is a great tool to setup venv which I use daily for python projects.
A few considerations I discovered while using are below:

1. can have slow dependency resolution if unbounded
   Poetry is extremely slow when resolving the dependencies python-poetry/poetry#2094 (comment)
   using verbose logging helps to troubleshoot:
   poetry install -vvv | tee ./poetry_install.log
   and setting fixed dependency/python version constraints (similar to how Airflow does it) also helps to limit the iterative checks

2. if you would like to use a dynamic/editable library, the syntax is a bit different from pipenv
   inside the 'pyproject.toml', add:
   customlib= {path = "./customlib", develop=true}

3. poetry updates are available about 12-24 hours after dependency release
   One example I encountered was 'boto3' was updated by Amazon, poetry did not find/recognize the latest version until later.
   After running poetry update command the next day, was a non-issue.

[chayim]

1. can have slow dependency resolution if unbounded
2. poetry updates are available about 12-24 hours after dependency release

@averille-cloud-dev I've definitely experienced both 1 and 3 on your list. Though, given how small our dependency list is, I don't really see 1 as an issue. Given our use case, I'd argue the same for 3.

2. if you would like to use a dynamic/editable library, the syntax is a bit different from pipenv

Since this is poetry and not pipenv I see this as unrelated.

Notably, were we building something that uses pyproject.toml, we'd need to embrace something similar, or use PyPA Build.