问题描述
我正在尝试在 Google Cloud 上设置一个 centos-7 GPU(Nvidia Tesla K80)实例,以执行 CUDA 工作。
不幸的是,我似乎无法正确安装/配置驱动程序。
事实上,当尝试与nvidia-smi
(NVIDIA 系统管理界面)交互时会发生以下情况:
# nvidia-smi -pm 1
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
使用更新方法的相同操作nvidia-persistenced
:
# nvidia-persistenced
nvidia-persistenced failed to initialize. Check syslog for more details.
+ 我在 syslog 中收到以下错误(使用journalctl
命令):
Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 0 has read and write permissions for those files.
确实,不存在任何 nvidia 设备:
# ll /dev/nvidia*
ls: cannot access /dev/nvidia*: No such file or directory
但是,这里有一个证明 GPU 已正确连接到实例的证明:
# lshw -numeric -C display
*-display UNCLAIMED
description: 3D controller
product: GK210GL [Tesla K80] [10DE:102D]
vendor: NVIDIA Corporation [10DE]
physical id: 4
bus info: pci@0000:00:04.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: msi pm cap_list
configuration: latency=0
resources: iomemory:40-3f iomemory:80-7f memory:fc000000-fcffffff memory:400000000-7ffffffff memory:800000000-801ffffff ioport:c000(size=128)
我遵循的安装过程
创建 centos-7 实例,如下所示Google Cloud 文档的此部分:
gcloud compute instances create test-gpu-drivers \
--machine-type n1-standard-2 \
--boot-disk-size 250GB \
--accelerator type=nvidia-tesla-k80,count=1 \
--image-family centos-7 --image-project centos-cloud \
--maintenance-policy TERMINATE
然后,我遵循的驱动程序和 CUDA 安装过程受到以下启发:Google Cloud 文档,但改为使用最新版本:
gcloud compute ssh test-gpu-drivers
sudo su
yum -y update
# Reboot for kernel update to be taken into account
reboot
gcloud compute ssh test-gpu-drivers
sudo su
# Install nvidia drivers repository, found here: https://www.nvidia.com/Download/index.aspx?lang=en-us
curl -J -O http://us.download.nvidia.com/tesla/410.72/nvidia-diag-driver-local-repo-rhel7-410.72-1.0-1.x86_64.rpm
yum -y install ./nvidia-diag-driver-local-repo-rhel7-410.72-1.0-1.x86_64.rpm
# Install CUDA repository, found here: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=CentOS&target_version=7&target_type=rpmlocal
curl -J -O https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-repo-rhel7-10.0.130-1.x86_64.rpm
yum -y install ./cuda-repo-rhel7-10.0.130-1.x86_64.rpm
# Install CUDA & drivers & dependencies
yum clean all
yum -y install cuda
nvidia-smi -pm 1
reboot
gcloud compute ssh test-gpu-drivers
sudo su
nvidia-smi -pm 1
完整日志这里。
(我也尝试了精确的 GCE 驱动程序安装脚本,没有升级版本,但也没有成功)
环境
发行版本
[root@test-gpu-drivers myuser]# cat /etc/*-release | head -n 1 CentOS Linux release 7.6.1810 (Core)
内核发布
[root@test-gpu-drivers myuser]# uname -r 3.10.0-957.1.3.el7.x86_64
我可以让它在 Ubuntu 上运行!
为了分析这个问题,我决定在 Ubuntu 18.04 (LTS) 上尝试做同样的事情。这次,我没有遇到任何问题。
实例创建:
gcloud compute instances create gpu-ubuntu-1804 \
--machine-type n1-standard-2 \
--boot-disk-size 250GB \
--accelerator type=nvidia-tesla-k80,count=1 \
--image-family ubuntu-1804-lts --image-project ubuntu-os-cloud \
--maintenance-policy TERMINATE
安装过程:
gcloud compute ssh gpu-ubuntu-1804
sudo su
apt update
apt -y upgrade
reboot
gcloud compute ssh gpu-ubuntu-1804
sudo su
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
apt -y install ./cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
rm cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
apt-get update
apt-get -y install cuda
nvidia-smi -pm 1
提供完整安装日志这里。
测试:
# nvidia-smi -pm 1
Enabled persistence mode for GPU 00000000:00:04.0.
All done.
# ll /dev/nvidia*
crw-rw-rw- 1 root root 241, 0 Dec 4 14:01 /dev/nvidia-uvm
crw-rw-rw- 1 root root 195, 0 Dec 4 14:01 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Dec 4 14:01 /dev/nvidiactl
我注意到的一件事是,在 Ubuntu 上安装包nvidia-dkms
会触发一些东西,而在 centos 上我没有看到这些东西:
Setting up nvidia-dkms-410 (410.79-0ubuntu1) ...
update-initramfs: deferring update (trigger activated)
A modprobe blacklist file has been created at /etc/modprobe.d to prevent Nouveau
from loading. This can be reverted by deleting the following file:
/etc/modprobe.d/nvidia-graphics-drivers.conf
A new initrd image has also been created. To revert, please regenerate your
initrd by running the following command after deleting the modprobe.d file:
`/usr/sbin/initramfs -u`
*****************************************************************************
*** Reboot your computer and verify that the NVIDIA graphics driver can ***
*** be loaded. ***
*****************************************************************************
Loading new nvidia-410.79 DKMS files...
Building for 4.15.0-1025-gcp
Building for architecture x86_64
Building initial module for 4.15.0-1025-gcp
Generating a 2048 bit RSA private key
.............................................................................................................+++
..........+++
writing new private key to '/var/lib/shim-signed/mok/MOK.priv'
-----
EFI variables are not supported on this system
/sys/firmware/efi/efivars not found, aborting.
Done.
nvidia:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.15.0-1025-gcp/updates/dkms/
nvidia-modeset.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.15.0-1025-gcp/updates/dkms/
nvidia-drm.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.15.0-1025-gcp/updates/dkms/
nvidia-uvm.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.15.0-1025-gcp/updates/dkms/
depmod...
DKMS: install completed.
环境
发行版本
root@gpu-ubuntu-1804:/home/elouan_keryell-even# cat /etc/*-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS" NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.1 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic
内核发布
root@gpu-ubuntu-1804:/home/elouan_keryell-even# uname -r 4.15.0-1025-gcp
问题
有人知道我在 Centos 7 上安装 NVIDIA 驱动程序出了什么问题吗?
答案1
有两个问题:
- CentOS 默认使用 nouveau 开源驱动,该驱动与 nvidia 不兼容,必须停用
- 来自 nvidia repo 的驱动程序似乎不起作用,因为需要 nvidia dkms 模块。
要做到这一点 :
安装一些必需的包
yum install kernel-devel epel-release dkms
编辑
/etc/default/grub
并nouveau.modeset=0
添加GRUB_CMDLINE_LINUX
重新生成 grub 配置以应用更改:
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
重新启动以使更改生效。
然后直接安装此驱动程序:http://fr.download.nvidia.com/tesla/410.79/NVIDIA-Linux-x86_64-410.79.run
此后,nvidia-smi 应该可以正常工作
答案2
这个问题已经报告给谷歌,正在处理中这里