使用 pip -r requirements.txt 安装 python deps 失败,但是逐个安装却成功。为什么?

使用 pip -r requirements.txt 安装 python deps 失败,但是逐个安装却成功。为什么?

我有以下requirements.txt文件:

appdirs==1.4.3
decorator==4.3.0
numpy==1.15.4
pybind11==2.2.4
pyopencl==2018.2.2
pytools==2018.5.2
six==1.12.0

运行pip install -r requirements.txt失败,显示:

Collecting appdirs==1.4.3 (from -r reqs.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Collecting decorator==4.3.0 (from -r reqs.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
Collecting numpy==1.15.4 (from -r reqs.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/86/04/bd774106ae0ae1ada68c67efe89f1a16b2aa373cc2db15d974002a9f136d/numpy-1.15.4-cp35-cp35m-manylinux1_x86_64.whl (13.8MB)
    100% |████████████████████████████████| 13.8MB 1.2MB/s 
Collecting pybind11==2.2.4 (from -r reqs.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/f2/7c/e71995e59e108799800cb0fce6c4b4927914d7eada0723dd20bae3b51786/pybind11-2.2.4-py2.py3-none-any.whl (145kB)
    100% |████████████████████████████████| 153kB 1.2MB/s 
Collecting pyopencl==2018.2.2 (from -r reqs.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/bc/58/3ab1246e94986f1b6953e76d7ea7e69d2dbfef7b3f3874eded48524a024f/pyopencl-2018.2.2.tar.gz (341kB)
    100% |████████████████████████████████| 348kB 1.1MB/s 
    Complete output from command python setup.py egg_info:
    ---------------------------------------------------------------------------
    Pybind11 is not installed.
    ---------------------------------------------------------------------------
    Very likely, the build process after this message will fail.

    Simply press Ctrl+C and type
    python -m pip install pybind11
    to fix this. If you don't, the build will continue
    in a few seconds.

    [1] https://pybind11.readthedocs.io/en/stable/
    ---------------------------------------------------------------------------
    Continuing in 1 seconds...    
    ---------------------------------------------------------------------------
    Mako is not installed.
    ---------------------------------------------------------------------------
    That is not a problem, as most of PyOpenCL will be just fine
    without it. Some higher-level parts of pyopencl (such as
    pyopencl.reduction) will not function without the templating engine
    Mako [1] being installed. If you would like this functionality to
    work, you might want to install Mako after you finish
    installing PyOpenCL.

    Simply type
    python -m pip install mako
    either now or after the installation completes to fix this.

    [1] http://www.makotemplates.org/
    ---------------------------------------------------------------------------
    Hit Ctrl-C now if you'd like to think about the situation.
    ---------------------------------------------------------------------------
    Continuing in 1 seconds...   
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-d25rz2_5/pyopencl/setup.py", line 353, in <module>
        main()
      File "/tmp/pip-install-d25rz2_5/pyopencl/setup.py", line 320, in main
        language='c++',
      File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 41, in __init__
        self._include_dirs = self.include_dirs
      File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 55, in get_include_dirs
        return self._include_dirs + self.get_additional_include_dirs()
      File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 52, in get_additional_include_dirs
        return [self.get_numpy_incpath()]
      File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 47, in get_numpy_incpath
        file, pathname, descr = find_module("numpy")
      File "/home/app/.venv/lib/python3.5/imp.py", line 296, in find_module
        raise ImportError(_ERR_MSG.format(name), name=name)
    ImportError: No module named 'numpy'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-d25rz2_5/pyopencl/

运行时:

pip install numpy==1.15.4
pip install pybind11==2.2.4
pip install pyopencl==2018.2.2

效果非常好。知道为什么吗?提前谢谢。

- - - - - 编辑: - - - - -

我也尝试过但pip-tools没有成功...在这种情况下我创建了一个文件requirements.in

numpy
pybind11
pyopencl

并运行pip-compile,但失败了:

Traceback (most recent call last):
  File "/home/app/.venv/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/scripts/compile.py", line 196, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 101, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 198, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 284, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 217, in get_dependencies
    self._dependencies_cache[ireq] = self.resolve_reqs(download_dir, ireq, wheel_cache)
  File "/home/app/.venv/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 183, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/operations/prepare.py", line 126, in prep_for_dist
    self.req.run_egg_info()
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 473, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/utils/misc.py", line 705, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmp0ykgmwk4build/pyopencl/

答案1

运行pip install -r requirements.txt失败... 而运行:

pip install numpy==1.15.4
pip install pybind11==2.2.4
pip install pyopencl==2018.2.2

效果非常好。知道为什么吗?

这是因为这两个命令并不相同。pip install支持多个参数(一次安装多个包),因此相当于的命令pip install -r requirements.txt宁愿是

$ pip install numpy==1.15.4 pybind11==2.2.4 pyopencl==2018.2.2  # etc

并且这个会失败。原因是pyopencl不仅包含 Python 代码,还包含 C 代码(所谓的“C 扩展”),需要使用 C 编译器进行编译才能安装。对于编译,在开始构建扩展时必须已经安装numpy和,但它们仅在那时安排安装。因此,无法构建,因此无法准备安装,最终整个命令失败,什么都没有安装。pybind11pyopenclpippyopencl

不幸的是,这是一个相当常见的问题。有两种方法可以避免它:

  1. 基本上,这与你已经发现的方法相同:在单独的pip install命令中安装不同的软件包组。将pip install -r requirements.txt命令拆分为两个:创建requirements-pre.txt(名称无关紧要)。移动numpypybind11requirements.txtrequirements-pre.txt。现在使用安装软件包

    $ pip install -r requirements-pre.txt && pip install -r requirements.txt
    
  2. 使用pipenv和 Pipfiles 而不是pip“需求”文本文件,因为pipenv它知道如何一次性重新安装失败的依赖项。软件包保留在 Pipfile 中

    [[source]]
    url = "https://pypi.python.org/simple"
    verify_ssl = true
    
    [packages]
    numpy = "==1.15.4"
    pybind11 = "==2.2.4"
    pyopencl = "==2018.2.2"
    

可以用一个pipenv install命令来安装。

并运行pip-compile,但失败了:

这与 的错误相同pip install,只是原始回溯被 隐藏了pip-compile

相关内容