UNIX Memory

Memory/Swap Information

Linux/UNIX will use free RAM for things like disk caching. This artificially elevates the “used” memory figure and reduces the “free” memory figure. The RAM used as disk cache is flagged as both “used” and “available” because it can be very easily reclaimed at any time.

free

free -m

free -mt

free -gt

Show figures in KB

Show figures in MB

Show figures in MB with Totals

Show figures in GB with Totals

Example output from free...              total        used        free      shared  buff/cache   availableMem:       61580608    42263196     1706948     3940240    17610464    13240884Swap:      16777212      174308    16602904
This output from free shows that the system has a total of 61,580,608K of memory, of which 42,263,196K is considered used. This used figure includes 17,610,464K of OS buffers/caches and 3,940,240K of shared memory. The available memory is calculated based on some additional information (available from /proc/zoneinfo and /proc/meminfo). 

See the awk script below for a reproduction of the internal available memory calculation (adapted from Stephen Kitt via unix.stackexchange)... zi=/proc/zoneinfomi=/proc/meminfoawk -v low=$(grep low ${zi}|awk '{k+=$2}END{print k}') \ '{a[$1]=$2} END{ print a["MemFree:"]+a["Active(file):"]+a["Inactive(file):"]+a["SReclaimable:"]-(12*low); }' ${mi}
Keeping available memory at around 20% of total memory is a good rule of thumb (but remember every rule of thumb has its exceptions). You can calculate your current percentage using...

awk '{a[$1]=$2} END{ print a["MemAvailable:"]/a["MemTotal:"]*100; }' /proc/meminfo

prtconf -m

Example output from prtconf -mMemory Size: 32768 MB

svmon -G

               size       inuse        free         pin     virtual   mmodememory      2490368     1661688      828680      524156      601823     Dedpg space    2490368        3796
               work        pers        clnt       otherpin          439364           0        3656       81136in use       601823           0     1059865
PageSize   PoolSize       inuse        pgsp         pin     virtuals    4 KB         -     1216136        3796      134348      156271m   64 KB         -       27847           0       24363       27847L   16 MB         -           0           0           0           0S   16 GB         -           0           0           0           0

svmon -O summary=basic,unit=auto

One of the key things to note in this output is that "free" memory is only 49.6M yet "available" memory is 3.06G. This is because, in normal operation, AIX will attempt to use all free memory as a filesystem cache. As long as you don't see much use of paging space then use the "available" figure when working out how much memory you have to spare for other things (like Oracle Database memory tunables).
Unit: auto--------------------------------------------------------------------------------------               size       inuse        free         pin     virtual  available   mmodememory        32.0G       32.0G       49.6M       6.04G       28.5G      3.06G     Dedpg space      25.6G       2.26G
               work        pers        clnt       otherpin           5.09G          0K       14.3M     962.50Min use        27.6G          0K       4.31G

vmstat

vmstat 5 5

vmstat -a

vmstat -Itw 5 5


5 iterations, 5 seconds apart

Show inactive and active memory (instead of buff and cache)

(AIX) I/O oriented view ,with timestamp, wide mode

Example output from vmstat -Itw 5 5System configuration: lcpu=16 mem=32768MB ent=2.00
   kthr            memory                         page                       faults                 cpu             time----------- --------------------- ------------------------------------ ------------------- ----------------------- --------  r   b   p        avm        fre    fi    fo    pi    po    fr     sr    in     sy     cs us sy id wa    pc    ec hr mi se 27  13   0    8138927       5875 69120 10313     0     0 78175 155309 15391  56136 118802 45 47  3  5  3.90 195.0 13:12:35 32  11   0    8128587       5929 62101  5547     0     0 63686 159470 14366 184458 105148 47 46  3  4  3.90 194.8 13:12:40 30   7   0    8110765       5545 48682  2918     0     0 45613 105090 13471 330653  93227 54 36  5  5  3.88 193.9 13:12:45 18   7   0    8107437       6394 52499  3051     0     0 52707 109837 14381 391281  99667 55 35  5  5  3.93 196.4 13:12:50 43  14   0    8106457       5310 92296  2260     0     0 92307 207717 12866 367300  91384 48 42  4  6  3.92 196.1 13:12:55
System Configuration
  • lcpu - logical processors
  • mem - amount of memory
  • ent - entitled capacity

Note: Entitled capacity relates to AIX LPAR functionality
kthrKernel Thread states
  • r - average runnable threads over the sampling interval
  • b - average kernel threads placed in VMM wait queue (awaiting resourcem awaiting I/O) over the sampling interval
  • p - average threads waiting for I/O messages from raw devices

SMT

In our example (on AIX) we can see (using smtctl) that 16 lcpu equates to 4 physical cpu with 4 SMT threads each
memory
  • avm - active virtual pages shown as 4K chunks. Divide by 256 to convert to MB. 
  • fre - size of the free list.
If we divide mem by avm (in MB) we can see that, on this system, virtual memory accounts for over 96% of total memory.
The free list often remains small because a large proportion of real memory will be used as a filesystem cache.
page
  • fi - file page-ins per second
  • fo - file page-outs per second
  • pi - number of pages paged in from paging space over the sampling interval
  • po - number of pages paged out to paging space over the sampling interval
  • fr - pages freed (page replacement)
  • sr - pages scanned by page-replacement algorithm
faults
  • in - device interrupts
  • sy - system calls
  • cs - kernel thread context switches
CPU
  • us - % user time
  • sy - system time
  • id - idle time
  • wa - wait time (idle time during which the system had outstanding disk/NFS I/O requests)
  • pc - processors consumed
  • ec - entitled capacity
If the current processor consumption exceeds the entitled capacity, the percentage becomes relative to the physical processor consumed (pc)

More information...

cat /proc/meminfo

dmidecode --type 17

lshw -short -C memory

Pagesize...

getconf PAGESIZE

Memory Nodes/Zones/Fragmentation...

cat /proc/buddyinfo

Information about storage used for swap...

cat /proc/swaps

cat /etc/fstab

Processes by Memory Usage

dstat -d --top-mem

Processes by % Memory Usage...

for file in /proc/*/status

do

  awk '/VmSize|Name|Uid/{printf $2 " " $3}END{ print ""}' $file

done | grep 101 | awk '{sum+=$3;} END{print sum;}'

For a breakdown of memory usage for a specifc process (e.g. 222)...

pmap 222

For a breakdown of shared memory usage...

ipcs -b

Processes by % Memory Usage...

ps -o pid,user,%mem,command ax | sort -b -k3 -r

ps -eo pid,cmd,%mem,%cpu --sort=-%mem

ps -eo pid,euser,cmd,vsz --sort=-vsz

ps -eo pid,euser,comm,vsz --sort=-vsz | grep grid | awk '{sum+=$4;} END{print sum;}'

VSZ = Virtual Memory Size - indicates the maximum amount of memory a process would use if it loaded all of its functions and libraries into physical memory. Linux uses "demand" paging which means pages are only loaded into memory when the application uses them. i.e. VSZ does not show how much memory a process is actually using.
RSS = Resident Set Size -indicates used memory. But shared libraries are counted for each process.

Using top...

top

See the MEM% column
Shift+m - sort by memory usage.m - toggle memory displayc - toggle full command

Using htop...

htop

See the MEM% column
F6 then select %MEM using cursor keys to sort processes by memory usage

htop

Top 15 Memory Hogs

svmon -Pt15 | perl -e 'while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}'

Sorts on Resident Set Size

ps aux | sort -n -k 6

ps alxww | sort -n -k 11

topas

c - to cycle through CPU detailsd - to cycle through Disk details
Topas Monitor for host:myaixhost1               EVENTS/QUEUES    FILE/TTYTue Oct 18 17:27:55 2022   Interval:2           Cswitch   94537  Readch   197.5M                                                Syscall  329.8K  Writech 7896.2KCPU     User% Kern% Wait% Idle%   Physc  Entc%  Reads     25978  Rawin         0Total    50.3  31.5  10.8   7.4    3.81 190.66  Writes     2468  Ttyout      387                                                Forks         3  Igets       106Network    BPS  I-Pkts  O-Pkts    B-In   B-Out  Execs         4  Namei      2062Total     975K   363.0   1.00K   39.6K    935K  Runqueue   8.00  Dirblk        0                                                Waitqueue  10.0Disk    Busy%      BPS     TPS  B-Read  B-Writ                   MEMORYTotal    32.4     223M   29.0K    210M   12.3M  PAGING           Real,MB   32768                                                Faults    36019  % Comp     88FileSystem          BPS    TPS  B-Read  B-Writ  Steals    54725  % Noncomp  11Total              204M  27.6K    198M   6.87M  PgspIn        0  % Client   11                                                PgspOut       0Name           PID  CPU%  PgSp Owner            PageIn    53722  PAGING SPACEoracle     29622312  8.5 20.1M oracle           PageOut    3150  Size,MB   26240oracle     55050446  7.9 21.7M oracle           Sios      56910  % Used     44oracle     50724986  7.7 10.5M oracle                            % Free     56oracle     59703408  6.8 19.5M oracle           NFS (calls/sec)oracle     36372720  4.7 19.4M oracle           SerV2         0  WPAR Activ    0lrud         262152  3.8  640K root             CliV2         0  WPAR Total    0oracle     57409682  3.2 25.1M oracle           SerV3         0  Press: "h"-helporacle     17367170  2.6 11.6M oracle           CliV3         0         "q"-quitoracle     46137462  2.6 18.8M oracle           SerV4         0oracle     33095710  2.3 17.3M oracle           CliV4         0oracle     63242434  2.3 18.4M oracleoracle     62980230  2.2 18.9M oraclejava       18874548  2.1  329M oracleoracle     39649322  2.1 41.2M oraclevtiol        786456  2.1 1.06M rootoracle     36765752  2.1 18.8M oracleoracle     43122878  2.0 33.2M oracleaioserve   66846838  1.2  448K oracleoracle     62914612  0.8 18.0M oracleoracle      9109524  0.5 22.4M oracle

Bibliography

https://www.golinuxcloud.com/check-memory-usage-per-process-linux/https://www.golinuxcloud.com/how-to-find-memory-leaks/https://www.cyberciti.biz/tips/how-much-ram-does-my-linux-system.html 
https://linuxize.com/post/free-command-in-linux/ https://linuxhint.com/check_memory_usage_process_linux/ 
https://stackoverflow.com/questions/1971422/linux-how-to-put-a-load-on-system-memoryhttps://stackoverflow.com/questions/7880784/what-is-rss-and-vsz-in-linux-memory-management
https://blogs.oracle.com/linux/anticipating-your-memory-needshttps://www.fosslinux.com/43296/linux-ps-command-with-examples.htmhttps://man7.org/linux/man-pages/man1/ps.1.htmlhttps://www.kernel.org/doc/html/latest/filesystems/proc.htmlhttps://linuxconfig.org/ps-output-difference-between-vsz-vs-rss-memory-usagehttps://linuxconfig.org/how-to-monitor-ram-usage-on-linuxhttps://phoenixnap.com/kb/vmstat-commandhttps://www.linuxatemyram.com/
https://unix.stackexchange.com/questions/99334/how-to-fill-90-of-the-free-memoryhttps://unix.stackexchange.com/questions/478443/how-shall-i-understand-the-output-of-freehttps://unix.stackexchange.com/questions/261247/how-can-i-get-the-amount-of-available-memory-portably-across-distributionshttps://unix.stackexchange.com/questions/586249/swap-grows-slowly-with-80-available-memory
https://www.ibm.com/support/pages/memory-analysis-aixhttps://www.ibm.com/docs/en/aix/7.2?topic=usage-memory-determination-ps-commandhttps://developer.ibm.com/articles/using-svmon-to-display-available-memory-on-aix/
https://tecadmin.net/add-swap-partition-on-ec2-linux-instance/
TODO:smemslabtophttps://unix.stackexchange.com/questions/361408/calculate-lpar-cpu-utilization-aix
https://support.oracle.comVirtual Memory Paging Reported Incorrectly on 11.2.0.3 (Doc ID 1515849.1)
AIX Memory Leakshttps://www.ibm.com/support/pages/memory-leak-diagnosis-aix