☰

In Landscape

tag-icon • tag-icon server memory landscape
In Landscape

I'm trying to debug an OOM situation in an Ubuntu 12.04 server, and looking at the Memory graphs in Landscape, I noticed that there wasn't any serious memory usage spike. Then I looked at the output of the free command and I wasn't quite sure how both memory usage results relate to each other.

Here's landscape's output on the server:

$ landscape-sysinfo 
  System load:  0.0               Processes:           93
  Usage of /:   5.6% of 19.48GB   Users logged in:     1
  Memory usage: 26%               IP address for eth0: -
  Swap usage:   2%

Then I ran the free command and I get:

$ free -m
             total       used       free     shared    buffers     cached
Mem:           486        381        105          0          4        165
-/+ buffers/cache:        212        274
Swap:          255          7        248

I can understand the 2% swap usage, but where does the 26% memory usage come from?

答案1

In Landscape

landscape-sysinfo actually gets its data directly from /proc/meminfo:

dpb@helo:~$ cat /proc/meminfo |egrep 'MemTotal:|Active:'
MemTotal:       12286760 kB
Active:          3794832 kB
dpb@helo:~$

The calculation of "Memory Usage" in this case would be:

((MemTotal - Active) / MemTotal) * 100

You can see these calculations in:

/usr/share/pyshared/landscape/sysinfo/memory.py
/usr/share/pyshared/landscape/lib/sysstats.py

gets its data directly from /proc/meminfo:

In free

The free utility also gets its data directly from /proc/meminfo:

Mem

  • total: MemTotal
  • used: MemTotal - MemFree
  • free: MemFree
  • buffers: Buffers
  • cached: Cached

Buffers/cache

  • used: MemTotal - MemFree - Buffers - Cached
  • free: MemFree + Buffers + Cached

Swap

  • total: SwapTotal
  • used: SwapTotal - SwapFree
  • free: SwapFree

Total

  • total: MemTotal + SwapTotal
  • used: MemTotal - MemFree + SwapTotal - SwapFree
  • free: MemFree + SwapFree

Corrected cached -- lzap

答案2

Those graphs don't reflect every single memory allocation/freeing event, but samples from /proc/meminfo (exactly as dpb described) at given intervals. A slightly speculative explanation for why it's not showing in the graph would be that it simply occurred between to points when memory usage was sampled.

I suspect that what has happened here is that some process acquired lots of memory in a hurry and OOM killer disposed of it before a sample could be made. That would be a fairly extreme circumstance, and also one that would mean the whole machine was running slowly as it was swapping heavily. This loading on the machine would reduce the likelihood of the system having available time to sample memory usage during that window and report it back to the Landscape server.

答案3

To add to dpb's correct answer, we need to define the difference between memory types. Going by redhat's documentation on /proc/meminfo:

The main definitions to use:

  • Active memory is memory has been used recently and shouldn't be replaced by the OS when new pages are brought in.
  • MemTotal is how much usable memory your system has (physical memory minus a tiny amount of memory used for the kernel code).
  • MemFree is memory currently unused memory. Memory used as disk cache is not free.
  • MemUsed is memory being used = MemTotal - MemFree.
  • Cached is memory used as a cache for files on disk - if you recently read a file from disk maybe you'll look at it again in the near future so let's keep it in memory when its available (as essentially free and could dramatically speed up your computer as reads from disk take several milliseconds while reads from memory take microseconds or less).
  • Buffers is a buffer for the IO devices located -- e.g., when you change values of a file, it first gets saved to the buffers (in your main memory) and then the changes get written to the disk as writing to the disk is slow. This buffered memory is short lived and tiny on modern kernels. (Granted the distinction between cached/buffers is less clear on modern linux kernel see [1]).
  • MemUsed - Buffers/Cached (first number in second line of free) is how much memory is used by your processes. This is how much memory is in use ignoring the disk cache (which your system is free to over write)..

You could easily have more active memory than MemUsed - Buffers/Cached. This scenario would occur when you are frequently accessing parts of disk that have been cached in memory. Similarly, you could have processes that have allocated memory but haven't used that allocated memory recently -- maybe its ideal to move this allocated memory to swap if you need more memory.

相关内容