Gitpod break the behavior of pre-commit of Python

[jefer94]

Bug description

[INFO] Restored changes from /home/gitpod/.cache/pre-commit/patch1626779319-12649.
An unexpected error has occurred: CalledProcessError: command: ('/home/gitpod/.cache/pre-commit/reponkw94t64/py_env-python3.9/bin/python', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
    
Check the log at /home/gitpod/.cache/pre-commit/pre-commit.log

version information

pre-commit version: 2.13.0
sys.version:
    3.9.6 (default, Jul  3 2021, 16:40:50) 
    [GCC 9.3.0]
sys.executable: /workspace/apiv2/.venv/bin/python
os.name: posix
sys.platform: linux

### error information

An unexpected error has occurred: CalledProcessError: command: ('/home/gitpod/.cache/pre-commit/reponkw94t64/py_env-python3.9/bin/python', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
    
Traceback (most recent call last):
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/error_handler.py", line 65, in error_handler
    yield
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/main.py", line 385, in main
    return run(args.config, store, args)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/commands/run.py", line 408, in run
    install_hook_envs(to_install, store)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/repository.py", line 224, in install_hook_envs
    _hook_install(hook)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/repository.py", line 82, in _hook_install
    lang.install_environment(
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/languages/python.py", line 204, in install_environment
    helpers.run_setup_cmd(prefix, install_cmd)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/languages/helpers.py", line 52, in run_setup_cmd
    cmd_output_b(*cmd, cwd=prefix.prefix_dir)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/util.py", line 154, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/home/gitpod/.cache/pre-commit/reponkw94t64/py_env-python3.9/bin/python', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

Steps to reproduce

Select a workspace with Django, install pipenv, install pre-commit, do anything that use pre-commit

Expected behavior

That pre-commit run in Gitpod without modify the .bashrc also is common break the docker image if we trying to mount other .bashrc

Example repository

No response

Anything else?

Here is my solution pre-commit/pre-commit#1985

[csweichel]

Sounds like we should not be setting PIP_USER=1 in workspace-full. Would that solve the issue?

[jefer94]

Really yes, the team of pre-commit perform a pip install ... to install the hooks dependencies from .pre-commit-config.yaml that normally work because Python by default not set PIP_USER=1 but in Gitpod this environment is set, this configuration should still generating problems with more packages that should work locally but don't in Gitpod and this for any newbie is weird

[billsioros]

Hello everybody!

I'm facing a similar (maybe even the exact same, but I'm not quite sure as I'm pretty new to Gitpod 😄) issue. Given the following .gitpod.Dockerfile:

FROM gitpod/workspace-full

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

ENV PATH="${HOME}/.poetry/bin:${PATH}"
ENV PIP_USER="no"

RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3 - \
    && poetry config virtualenvs.create "true" \
    && poetry config virtualenvs.in-project "true"

PIP_USER remains set to yes. I have tried various values just in case (false, 0, no), as well as not using Environment replacement, with no results whatsoever.

[jefer94]

Should be great that .bashrc was generated getting the environment

@billsioros this was my solution

FROM gitpod/workspace-postgres

RUN sudo apt-get update \
    && sudo apt-get update \
    && sudo apt-get install -y redis-server \
    && sudo apt-get clean \
    && sudo rm -rf /var/cache/apt/* /var/lib/apt/lists/* /tmp/*

RUN pyenv update && pyenv install 3.9.6 && pyenv global 3.9.6
RUN pip install pipenv

# remove PIP_USER environment
USER gitpod
RUN if ! grep -q "export PIP_USER=no" "$HOME/.bashrc"; then printf '%s\n' "export PIP_USER=no" >> "$HOME/.bashrc"; fi

[billsioros]

Hi @jefer94, thank you very much! Sadly my issue has not been resolved. Shouldn't PIP_USER (and possibly other environment variables set by the Dockerfile ) be respected, in any case?

Edit: I'm not quite sure, but could it be due to spawning a python virtual environment and afterwards installing the dependencies?

[jefer94]

@billsioros no necesaryly, you have the PIP_USER=yes because the gitpod image has your own .bashrc file, if you run gp open ~/.bashrc you should see PIP_USER is being setting, and this file is static

[billsioros]

Hello again @jefer94. PIP_USER was being set to no, but after instnatiating a new virtual environment, attempting to install the dependencies resulted in nothing. I was able to resolve my issue by not instantiating a new virtual environment.

If I'm correct, Gitpod does not proceed to execute commands specified by tasks if a new shell is instantiated. But should this be considered an issue? I'm not sure about any security complications it might introduce, but I think something like this should be at least documented (I'm sorry if it is already documented, I search for information on the matter but couldn't come up with anything 😢)

[jefer94]

In Gitpod I think is not necesary use virtualenv, but for you resolve you issue you need to set the PIP_USER before Gitpod open the custom docker image and for me it's safe, venv just prevent have conflicts dependencies, in you local machine is not recommended install each dependency of each project in the system or user packages

[csweichel]

This is an issue with workspace-full itself. I've raised a PR in workspace-images to set PIP_USER=no

[jankeromnes]

This is an issue with workspace-full itself.

Isn't this rather an issue with the gitpod-layer?

gitpod/components/image-builder/workspace-image-layer/gitpod-layer/debian/gitpod/.bashrc-prepend

Lines 39 to 44 in af51739

	export PIPENV_VENV_IN_PROJECT=true
	export PIP_USER=yes
	export PYTHONUSERBASE=/workspace/.pip-modules
	export PATH=$PYTHONUSERBASE/bin:$PATH
	unset PIP_TARGET
	unset PYTHONPATH

[da-moon]

@billsioros I got that problem fixed in a hacky way by setting env vars in files under /etc/profile.d/ and having /etc/bash.bashrc to source all files under /etc/profile.d/ and some sed statements to update local ~/.bashrc and ~/profile file.

# ────────────────────────────────────────────────────────────────────────────────
USER root
ENV EDITOR="nvim"
ENV VISUAL="nvim"
ENV PATH="${PATH}:/usr/local/bin"
WORKDIR "/etc/profile.d"
RUN \
  [ -d "${HOME}/.bashrc.d" ] && while read i; do ln -sf "$i" "/etc/profile.d/$(basename $i).sh"; done < <(find ${HOME}/.bashrc.d -type f ) || true ; \
  curl \
  -fsSl https://raw.githubusercontent.com/3hhh/fzfuncs/master/bashrc_fzf \
  -o /etc/profile.d/bashrc_fzf.sh \
  && ( \
  echo "export EDITOR=\"${EDITOR}\"" ; \
  echo "export VISUAL=\"${VISUAL}\"" ; \
  echo 'export DOCKER_BUILDKIT="0"' ; \
  echo 'export BUILDKIT_PROGRESS="auto"' ; \
  echo 'export COMPOSE_DOCKER_CLI_BUILD="0"' ; \
  echo 'export GEM_HOME=/workspace/.rvm' ; \
  echo 'export GEM_PATH=$GEM_HOME:$GEM_PATH' ; \
  echo 'export GOROOT="${HOME}/go" ;' ; \
  echo 'export GOPATH="/workspace/go-packages" ;' ; \
  echo 'export PYTHONUSERBASE=/workspace/.pip-modules ;' ; \
  echo 'export PATH="${PATH}:/usr/local/bin"' ; \
  echo '[ -d ${GEM_HOME} ] && export PATH="${GEM_HOME}/bin:$PATH" ;' ; \
  echo '[ -d ${GOROOT} ] && export PATH="${GO_ROOT}/bin:$PATH" ;' ; \
  echo '[ -d ${GOPATH} ] && export PATH="${GOPATH}/bin:$PATH" ;' ; \
  echo '[ -d ${HOME}/.local/bin ] && export PATH="${HOME}/.local/bin:${PATH}"' ; \
  echo '[ -d ${HOME}/.cargo/bin ] && export PATH="${HOME}/.cargo/bin:${PATH}";';  \
  echo '[ -d ${PYTHONUSERBASE} ] && export PATH="${PYTHONUSERBASE}/bin:$PATH" ;' ; \
  echo '[ -d ${HOME}/.pyenv/bin ] && export PATH="${HOME}/.pyenv/bin:${PATH}";' ; \
  echo '[ -d ${HOME}/.pyenv/shims ] && export PATH="${HOME}/.pyenv/shims:${PATH}" ;' ; \
  echo '[ -d ${HOME}/bin ] && export PATH="${HOME}/bin":${PATH}' ; \
  echo 'npm --version >/dev/null 2>&1 && export PATH="$(npm -g bin):${PATH}" ;' ; \
  echo 'yarn --version >/dev/null 2>&1 && export PATH="$(yarn global bin):${PATH}" ;' ; \
  echo 'gp --help >/dev/null 2>&1 && (eval $(gp env | sed -e "s/\(=[[:blank:]]*\)\(.*\)/\1\"\2\" ;/" -e "s/^/export /g") ; source <(gp completion) ;)' ; \
  echo 'starship --version >/dev/null 2>&1 && eval "$(starship init bash)" ;' ; \
  echo 'just --version >/dev/null 2>&1 && eval "$(just --completions bash)" ;' ; \
  echo 'rustup --version >/dev/null 2>&1 && eval "$(rustup completions bash rustup)" && eval "$(rustup completions bash cargo)";' ; \
  echo 'curl --version >/dev/null 2>&1 && source <(curl -sSL git.io/forgit) ;' ; \
  ) | tee 00-env.sh > /dev/null \
  && ( \
  echo "alias apt='sudo apt'" ; \
  echo "alias g='git'" ; \
  echo 'exa --version >/dev/null 2>&1 && alias la="exa -A";alias ll="exa -alhF";alias ls="exa";' ; \
  echo 'bat --version >/dev/null 2>&1 && alias cat="bat -pp";' ; \
  echo 'just --version >/dev/null 2>&1 && alias j="just";' ; \
  echo "alias nvim='nvim --headless -c \"call dein#recache_runtimepath()\" -c \"qall\" && nvim'" ; \
  ) | tee 01-alias.sh > /dev/null \
  && ( \
  echo '(rg -h >/dev/null 2>&1 && bat -h >/dev/null 2>&1 && sk -h >/dev/null 2>&1 ) && function fp { rg --files $1 | sk --preview="bat {} --color=always --style=grid,numbers" ; }' ; \
  echo '(rg -h >/dev/null 2>&1 && sk -h >/dev/null 2>&1 ) && function fs { sk -i -c "rg {} --color=always $1" --ansi ; }' ; \
  ) | tee 02-func.sh > /dev/null \
  && [ -r /etc/bash.bashrc ] && (\
    echo 'while read i; do source "$i"; done < <(find /etc/profile.d/ -name "*.sh") ;' ; \
  ) | tee -a /etc/bash.bashrc
# ────────────────────────────────────────────────────────────────────────────────
USER gitpod
WORKDIR "${HOME}"
RUN \
  [ -r "${HOME}/.bashrc.d/60-python" ] \
  && sed -i -e 's/pyenv/${HOME}\/.pyenv\/bin\/pyenv/g'  "${HOME}/.bashrc.d/60-python" \
  && sed -i \
    -e '/\$HOME\/\.bashrc\.d/d' \
    -e '/PIP/d' \
    -e '/PYTHONUSERBASE/d' \
    -e '/gp completion/d' \
    -e '/\.cargo\/env/d' \
    -e '/rvm/d' \
    -e '/GEM/d' \
    "${HOME}/.bashrc" \
  && ( \
  echo "[ -r '\${HOME}/.bashrc' ] && sed -i \\" ; \
  echo "-e '/\\\$HOME\/\.bashrc\.d/d' \\" ; \
  echo "-e '/cargo/d' \\" ; \
  echo "-e '/PIP/d' \\" ; \
  echo "-e '/PYTHONUSERBASE/d' \\" ; \
  echo "-e '/gp completion/d' \\" ; \
  echo "-e '/rvm/d' \\" ; \
  echo "-e '/GEM/d' \\" ; \
  echo "'\${HOME}/.bashrc' \\" ; \
  echo '&& . ${HOME}/.bashrc' ; \
  ) | tee "${HOME}/.profile" > /dev/null ;
# ────────────────────────────────────────────────────────────────────────────────

there are more steps involved so I would recommend you to study my Gitpod docker image here.

To test and see if your repo and local config works with my changes, you can open my gitpod-workspace repo in Gitpod, clone your repo under workspace/ and see if everything works for you.