调用 NumPy 对象时 RAM 内存不足

调用 NumPy 对象时 RAM 内存不足

我在 Inspiron 15 3000 系列上安装了 Linux 16.04 和 Windows 10 双启动,调用 NumPy 对象时似乎内存不足。我尝试下载博士问答,一种用于开放域问答的阅读理解系统,由 Facebook 研究小组开发。它包括调用提供的训练模型和数据进行维基百科问答。我尝试运行来interactive.py尝试它,但在向提示符提出我的第一个问题后,它就内存不足了。

(wiki_env) mike@mike-thinks:~/Programing/WikiQueri/DrQA$ python3.6 scripts/pipeline/interactive.py 
07/19/2018 01:49:54 PM: [ Running on CPU only. ]
07/19/2018 01:49:54 PM: [ Initializing pipeline... ]
07/19/2018 01:49:54 PM: [ Initializing document ranker... ]
07/19/2018 01:49:54 PM: [ Loading /home/mike/Programing/WikiQueri/DrQA/data/wikipedia/docs-tfidf-ngram=2-hash=16777216-tokenizer=simple.npz ]
Traceback (most recent call last):
  File "scripts/pipeline/interactive.py", line 69, in <module>
    reader_model=args.reader_model,
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/pipeline/drqa.py", line 109, in __init__
    self.ranker = ranker_class(**ranker_opts)
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/retriever/tfidf_doc_ranker.py", line 37, in __init__
    matrix, metadata = utils.load_sparse_csr(tfidf_path)
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/retriever/utils.py", line 34, in load_sparse_csr
    matrix = sp.csr_matrix((loader['data'], loader['indices'],
  File "/home/mike/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 235, in __getitem__
    pickle_kwargs=self.pickle_kwargs)
  File "/home/mike/.local/lib/python3.6/site-packages/numpy/lib/format.py", line 674, in read_array
    array = numpy.ndarray(count, dtype=dtype)
MemoryError
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "scripts/pipeline/interactive.py", line 69, in <module>
    reader_model=args.reader_model,
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/pipeline/drqa.py", line 109, in __init__
    self.ranker = ranker_class(**ranker_opts)
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/retriever/tfidf_doc_ranker.py", line 37, in __init__
    matrix, metadata = utils.load_sparse_csr(tfidf_path)
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/retriever/utils.py", line 34, in load_sparse_csr
    matrix = sp.csr_matrix((loader['data'], loader['indices'],
  File "/home/mike/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 235, in __getitem__
    pickle_kwargs=self.pickle_kwargs)
  File "/home/mike/.local/lib/python3.6/site-packages/numpy/lib/format.py", line 674, in read_array
    array = numpy.ndarray(count, dtype=dtype)
MemoryError

所以我了解到这意味着我的内存用完了,我该怎么做才能应对这个错误?是因为模型太大吗?

以下是我的内存使用情况的输出free -m

              total        used        free      shared  buff/cache   available
Mem:           7888        5515        1013         320        1359        1718
Swap:             0           0           0

mike@mike-thinks:~/Programming/WikiQueri/DrQA$ cat /proc/meminfo
MemTotal:        8078272 kB
MemFree:          291604 kB
MemAvailable:    2326220 kB
Buffers:          262392 kB
Cached:          2212332 kB
SwapCached:            0 kB
Active:          3691168 kB
Inactive:        1485432 kB
Active(anon):    2727772 kB
Inactive(anon):   274408 kB
Active(file):     963396 kB
Inactive(file):  1211024 kB
Unevictable:     2238236 kB
Mlocked:         2238252 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               104 kB
Writeback:             0 kB
AnonPages:       4940072 kB
Mapped:           508460 kB
Shmem:            275728 kB
Slab:             243928 kB
SReclaimable:     158948 kB
SUnreclaim:        84980 kB
KernelStack:       13024 kB
PageTables:        49456 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4039136 kB
Committed_AS:   11909484 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      161648 kB
DirectMap2M:     7088128 kB
DirectMap1G:     2097152 kB

我需要大约 16 GB 的 RAM,我不知道如何读取这些表格。你知道我是否可以做到吗?

相关内容