我想查看我的显卡的 GPU 使用情况。我在 Kubuntu 14.04 LTS x64 上使用，配有两张 Nvidia GTX 690 显卡。运行时nvidia-smi -q我得到：
francky@francky-Aurora-R4:~$ nvidia-smi -q
==============NVSMI LOG==============
Timestamp : Sun Nov 22 08:37:22 2015
Driver Version : 352.39
Attached GPUs : 2
GPU 0000:04:00.0
Product Name : GeForce GTX 690
Product Brand : GeForce
Display Mode : N/A
Display Active : N/A
Persistence Mode : Disabled
Accounting Mode : N/A
Accounting Mode Buffer Size : N/A
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-e770f6fe-e8c6-466b-89e1-bc39d2a280bb
Minor Number : 0
VBIOS Version : 80.04.4C.00.0E
MultiGPU Board : N/A
Board ID : N/A
Inforom Version
Image Version : N/A
OEM Object : N/A
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
PCI
Bus : 0x04
Device : 0x00
Domain : 0x0000
Device Id : 0x118810DE
Bus Id : 0000:04:00.0
Sub System Id : 0x095B10DE
GPU Link Info
PCIe Generation
Max : N/A
Current : N/A
Link Width
Max : N/A
Current : N/A
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 43 %
Performance State : P0
Clocks Throttle Reasons : N/A
FB Memory Usage
Total : 2047 MiB
Used : 237 MiB
Free : 1810 MiB
BAR1 Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
Gpu : N/A
Memory : N/A
Encoder : N/A
Decoder : N/A
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 59 C
GPU Shutdown Temp : N/A
GPU Slowdown Temp : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Default Power Limit : N/A
Enforced Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Max Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : N/A
GPU 0000:05:00.0
Product Name : GeForce GTX 690
Product Brand : GeForce
Display Mode : N/A
Display Active : N/A
Persistence Mode : Disabled
Accounting Mode : N/A
Accounting Mode Buffer Size : N/A
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-7a0d0c06-7285-a503-c130-32de1ce1d24d
Minor Number : 1
VBIOS Version : 80.04.4C.00.0F
MultiGPU Board : N/A
Board ID : N/A
Inforom Version
Image Version : N/A
OEM Object : N/A
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
PCI
Bus : 0x05
Device : 0x00
Domain : 0x0000
Device Id : 0x118810DE
Bus Id : 0000:05:00.0
Sub System Id : 0x095B10DE
GPU Link Info
PCIe Generation
Max : N/A
Current : N/A
Link Width
Max : N/A
Current : N/A
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 30 %
Performance State : P8
Clocks Throttle Reasons : N/A
FB Memory Usage
Total : 2047 MiB
Used : 244 MiB
Free : 1803 MiB
BAR1 Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
Gpu : N/A
Memory : N/A
Encoder : N/A
Decoder : N/A
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 33 C
GPU Shutdown Temp : N/A
GPU Slowdown Temp : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Default Power Limit : N/A
Enforced Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Max Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : N/A
为什么 GPU 利用率 N/A，如何修复或规避该问题？我只想要 GPU 利用率，并且我可以使用其他工具来实现这一点。
正常情况下，GPU利用率显示如下：
nvidia-smi -q
==============NVSMI LOG==============
Timestamp : Fri Dec 11 11:40:25 2015
Driver Version : 352.39
Attached GPUs : 3
GPU 0000:01:00.0
Product Name : GeForce GTX TITAN X
Product Brand : GeForce
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0420115017035
GPU UUID : GPU-4ef51df0-f0bb-0da5-9418-ef1060190073
Minor Number : 0
VBIOS Version : 84.00.1F.00.90
MultiGPU Board : No
Board ID : 0x100
Inforom Version
Image Version : G001.0000.01.03
OEM Object : 1.1
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
PCI
Bus : 0x01
Device : 0x00
Domain : 0x0000
Device Id : 0x17C210DE
Bus Id : 0000:01:00.0
Sub System Id : 0x29923842
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 16x
Current : 8x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : 388000 KB/s
Rx Throughput : 80000 KB/s
Fan Speed : 46 %
Performance State : P2
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Unknown : Active
FB Memory Usage
Total : 12287 MiB
Used : 7490 MiB
Free : 4797 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 4 MiB
Free : 252 MiB
Compute Mode : Default
Utilization
Gpu : 71 %
Memory : 37 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 81 C
GPU Shutdown Temp : 97 C
GPU Slowdown Temp : 92 C
Power Readings
Power Management : Supported
Power Draw : 139.90 W
Power Limit : 250.00 W
Default Power Limit : 250.00 W
Enforced Power Limit : 250.00 W
Min Power Limit : 150.00 W
Max Power Limit : 275.00 W
Clocks
Graphics : 1341 MHz
SM : 1341 MHz
Memory : 3304 MHz
Applications Clocks
Graphics : 1126 MHz
Memory : 3505 MHz
Default Applications Clocks
Graphics : 1126 MHz
Memory : 3505 MHz
Max Clocks
Graphics : 1518 MHz
SM : 1518 MHz
Memory : 3505 MHz
Clock Policy
Auto Boost : On
Auto Boost Default : On
Processes
Process ID : 30463
Type : C
Name : /home/francky/torch/install/bin/luajit
Used GPU Memory : 7465 MiB
答案1
自驱动程序版本 270.26 起，Nvidia 不再支持 GeForce 卡的分析功能。目前只有 Quadro 卡支持 GPU 使用情况监控。
答案2
您使用的是四年前的版本nvidia-smi。新版本支持我的 GTX 970M，并且毫无疑问会支持您同一系列的 GTX 960：
$ nvidia-smi
Tue Apr 30 17:07:43 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 970M Off | 00000000:01:00.0 On | N/A |
| N/A 59C P0 25W / N/A | 1287MiB / 3016MiB | 37% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1300 G /usr/lib/xorg/Xorg 1050MiB |
| 0 2482 G compiz 233MiB |
+-----------------------------------------------------------------------------+
输出片段nvidia-smi -q：
$ nvidia-smi -q
==============NVSMI LOG==============
Timestamp : Tue Apr 30 17:08:30 2019
Driver Version : 384.130
Attached GPUs : 1
GPU 00000000:01:00.0
Product Name : GeForce GTX 970M
Product Brand : GeForce
Display Mode : Enabled
Display Active : Enabled
Persistence Mode : Disabled
Accounting Mode : Disabled
(.... SNIP ....)
Utilization
Gpu : 36 %
Memory : 11 %
Encoder : 0 %
Decoder : 0 %
(.... SNIP ....)
正如您所看到的，GPU 曾经说过N/A现在也这么说36 %（就我的情况和时间点而言）。