Support Pipfile

[gsemet]

I prefer using Pipfile over requirements.txt. Please support this file format to set up the virtual environment.

[RichardLitt]

Thanks @stibbons. Contributions are always welcome, of course, if you'd like to submit a PR for this.

[agjohnson]

I started playing with support for pipenv, it looks like it might be a drop in replacement for calls to pip install. We'll probably want to offer the option through our readthedocs.yml file, so this will take:

• Conditional call to pipenv in doc_builder/python_environments.py
• readthedocs.yml support in our readthedocs-build repo
• readthedocs.yml support in this repo

[gsemet]

yes, it is a straigh replacement of pip (actually, a wrapper), you do pipenv install thispackage and it installs the package in the virtualenv (using pew) and modify Pipfile accordingly. For ReadTheDocs, I suggest you do pipenv install --dev and ensure the lib is installed in the environment. I usually use -e . to inject my lib under developement into the virtualenv, but I am not sure if everybody does it

[tuukkamustonen]

You should actually do pipenv install --dev --ignore-pipfile. Without --ignore-pipfile, pipenv will just seek to upgrade the packages in Pipfile and not really do deterministic build. See https://github.com/kennethreitz/pipenv/issues/954#issuecomment-338777638.

[gsemet]

indeed

[agjohnson]

I'm already using pipenv for other projects, so have some familiarity. I believe for our case, we actually want --system as well, to utilize our existing virtualenvs. We have special storage for this, so that venvs persist across builds.

@tuukkamustonen this confuses me a little, but I've only recently started with pipenv and pipfile. What does ignoring the pipfile mean for projects that don't check in their pipfile.lock? It doesn't seem checking in pipfile.lock is a requirement.

I suppose the most correct is check for pipfile.lock, if so, --ignore-pipfile, otherwise don't?

Some of these options will likely be added/allowed override through our readthedocs.yml, --dev is likely one of those.

In the case of a requirements.txt, we should be able to just use pipenv install instead of pip install

[gsemet]

Both side is ok. When you freeze the requirements.txt (ex using pip-tools), it act like when you use the lock file.
When you only describe the version by range or without version, it act like Pipfile.
The lock file is specific to the machine, it does not have the markers (ex « python_version < 3 »), but both should work.
I would use ‘pipenv install —dev —system’, it is like ´pip install -r requirements.txt -r requirements-dev.txt’

[tuukkamustonen]

@agjohnson I am new to pipenv as well, but as I understand it, if you run just pipenv install it will NOT follow your Pipfile.lock, but instead look into Pipfile, and upgrade any packages that you haven't locked into exact version, and then update Pipfile.lock. So it's like pip-compile and pip-sync (from pip-tools package) combined. I think it's a bit confusing.

I suppose the most correct is check for pipfile.lock, if so, --ignore-pipfile, otherwise don't?

Yeah, if we are after deterministic builds, then the lock-file needs to be followed and --ignore-pipfile given. I don't know how it would make sense otherwise. And then there is pipenv update, that uninstalls/installs packages like pip-sync but it doesn't have --ignore-pipfile option... hmm.

The lock file is specific to the machine, it does not have the markers (ex « python_version < 3 »), but both should work.

This is a good reminder. Neither pip-tools or pipenv (which uses pip-tools underneath) support "universal" lock files. So if you create the lockfile under python 3.6 on Windows and then run it on 3.4 on Linux, things may break, because lockfile expects 3.6/Windows environment. See https://github.com/kennethreitz/pipenv/issues/857 and jazzband/pip-tools#563.

What does ignoring the pipfile mean for projects that don't check in their pipfile.lock

Those projects won't get deterministic builds. It's ok, but should be documented. One of the main benefits with pipenv is to get deterministic builds (improvement over simple requirements.txt), but as you say, it's not a requirement. User should just acknowledge that.

In the case of a requirements.txt, we should be able to just use pipenv install instead of pip install

I don't think pipenv reads requirements.txt-style files. In case there is no Pipfile why not just resort to old behavior?

[agjohnson]

This is a good reminder. Neither pip-tools or pipenv (which uses pip-tools underneath) support "universal" lock files. So if you create the lockfile under python 3.6 on Windows and then run it on 3.4 on Linux, things may break, because lockfile expects 3.6/Windows environment. See kennethreitz/pipenv#857 and jazzband/pip-tools#563.

Ah yes, so it seems that ignoring the Pipfile and relying on Pipfile.lock should be an option, but is off by default. Without universal lock files, we can't guarantee that the environment is the same OS or even python version as our build images.

I don't think pipenv reads requirements.txt-style files. In case there is no Pipfile why not just resort to old behavior?

It looks like if a Pipfile is missing and a requirements.txt is available, pipenv install will automatically translate this to a Pipfile. Likewise, if the file is named foo.txt, pipenv install -r foo.txt will pick up the arbitrarily named file. These are the two modes we need to support on RTD.

[gsemet]

It does not change from requirements.txt, it may or may not be reproductible, depending on the work of the programmer... I would not bother, simply use the --dev, because most of the time ReadTheDoc is triggered at the same time than travis.

Later when universal lock file will be settled this can be adapted to Rtd, but this is not as "important" than in the travis build (because the artefact is the doc, not the actual wheel), and we dont have this ability in the travis.

Or, simpler, you let the developer write its install line (pip install -r requirements.txt, or pipenv install --dev), it is the more versatile solution.

[sobolevn]

What's the status of this issue? Maybe there is something I can help with?
Thanks!