UNIX Storage Diagnostics

run blk new|usr sys idl wai stl| used free buff cach| read writ| read writ5.0 0 0| 43 1 55 0 0|28.5G 7116M 7883M 18.4G|1.00 6.00 | 512B 29k5.0 0 0| 44 1 55 0 0|28.5G 7119M 7883M 18.4G| 0 0 | 0 0 5.0 0 0| 44 1 55 0 0|28.5G 7116M 7883M 18.4G|15.0 20.0 |1410k 154k5.0 0 0| 44 1 54 0 0|28.5G 7115M 7883M 18.4G|1.00 7.00 | 512B 37k5.0 0 0| 43 2 54 0 0|28.5G 7115M 7883M 18.4G| 0 2.00 | 0 548k5.0 0 0| 43 2 55 0 0|28.5G 7115M 7883M 18.4G|1.00 7.00 | 512B 64k5.0 0 0| 43 1 55 0 0|28.5G 7114M 7883M 18.4G|3.00 12.0 | 17k 73k5.0 0 0| 44 1 55 0 0|28.5G 7114M 7883M 18.4G| 0 0 | 0 0 5.0 0 0| 47 2 52 0 0|28.5G 7120M 7883M 18.4G|1.00 14.0 | 512B 189k5.0 0 0| 44 1 55 0 0|28.5G 7120M 7883M 18.4G| 0 8.00 | 0 88k6.0 0 0| 45 1 54 0 0|28.5G 7115M 7883M 18.4G|10.0 0 |1280k 0 5.0 0 0| 43 1 55 1 0|28.5G 7120M 7883M 18.4G|1.00 4.00 | 512B 21k6.0 0 0| 44 1 55 0 0|28.5G 7123M 7883M 18.4G| 0 0 | 0 0 5.0 0 0| 44 2 54 0 0|28.5G 7123M 7883M 18.4G|1.00 2.00 | 512B 4608B6.0 0 0| 44 1 55 0 0|28.5G 7123M 7883M 18.4G| 0 56.0 | 0 364k

-p Enable Process Stats

-c Enable CPU Stats

-m Enable Memory Stats

-r Enable I/O Stats

-d Enable Disk Stats

fio

Random Write Test

sudo fio --name=myjob \

--ioengine=libaio \

--readwrite=randwrite \

--size=5120M \

--numjobs=4 \

--runtime=240 \

--group_reporting

Arbitrary job name

Linux native asynchronous I/O

Random writes

Total size of file I/O for each thread of this job

How many clones of this job to run

How long job will run for (in seconds)

Displays aggregate totals for all jobs

iostat

sysstat

iostat -xcdmN 5

-x Extended statistics

-c CPU statistics

-d Device utilisation

-m Megabytes per second

-N Device mapper names

5 refresh every 5 seconds

For full details:man iostat

%user%nice%system%iowait%steal%idle
r/sw/srMB/swMB/srrqm/swrqm/s%rrqm%wrqmr_awaitw_awaitaqu-szrareq-szwareq-szscvtm%util

percentage of CPU utilization that occurred while executing at the user level (application)percentage of CPU utilization that occurred while executing at the user level with nice prioritypercentage of CPU utilization that occurred while executing at the system level (kernel)percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O requestpercentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processorpercentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request
read requests per secondwrite requests per secondmegabytes read per secondmegabytes written per secondread requests merged per second that were queued to the devicewrite requests merged per second that were queued to the devicepercentage of read requests merged together before being sent to the devicepercentage of write requests merged together before being sent to the deviceaverage time (in milliseconds) for read requests issued to the device to be served. Includes the time spent by the requests in queue and the time spent servicing them.average time (in milliseconds) for write requests issued to the device to be served. Includes the time spent by the requests in queue and the time spent servicing them.average queue length of the requests that were issued to the deviceaverage size (in kilobytes) of the read requests that were issued to the deviceaverage size (in kilobytes) of the write requests that were issued to the deviceaverage service time (in milliseconds) for I/O requests that were issued to the device. This field will be removed in a future sysstat version.Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100% for devices serving requests serially. But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits.

iostat -htx 1 1

-h Human Readable

-t Include time in output

-x Extended statistics

1 refresh every 1 second

1 for one iteration only

iotop

CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO %

task_delayacct is set to off by default from Linux Kernel 5.14 onwards.uname --kernel-release

To dynamically set it to on...sudo sysctl kernel.task_delayacct=1
To disable...sudo sysctl kernel.task_delayacct=0
Note that turning this on may have some effect on system performance

The PRIO column shows the Thread priority...idle - "A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero." (1)be/0 - Best Effort priority 0be/4 - Best Effort priority 4rt/4 - Real Time priority 4Priority levels range from 0 to 7 with a lower number being higher priority.

blktrace

fatrace

sudo fatrace

inotify

iwatch

Disk I/O (AIX)

nmon

d

Disk I/O Graphs

│ Disk-KBytes/second-(K=1024,M=1024*1024) ─────────────────────────────────────────────────││Disk Busy Read Write 0----------25-----------50------------75--------100 ││ Name KB/s KB/s | | | | | ││cd0 0% 0 0| | ││hdisk2 0% 0 0| | ││hdisk10 98% 88556 408|RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR> ││hdisk9 97% 147755 7714|RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRWW> ││hdisk8 0% 0 0| | ││hdisk7 1% 56 0|R> | ││hdisk0 0% 0 0| | ││hdisk6 0% 0 5| > | ││hdisk5 37% 1849 7248|RRRRWWWWWWWWWWWWWWW > | ││hdisk1 0% 0 0| | ││Totals 238217 15374+-----------|------------|-------------|----------+ │

nmon

D

DiskIO +Service times

│ Disk-KBytes/second-(K=1024,M=1024*1024) ─────────────────────────────────────────────────││Disk Busy Read Write Transfers Size Peak% Peak KB/s qDepth ││ Name KB/s KB/s /sec KB Read+Write or N/A ││cd0 0% 0.0 0.0 0.0 0.0 0% 0.0 -- ││hdisk2 0% 0.0 0.0 0.0 0.0 0% 0.0 -- ││hdisk10 98% 144680.3 2011.8 21592.3 6.8 100% 355240.8 0 ││hdisk9 91% 117476.5 6493.5 18165.0 6.8 100% 306984.4 0 ││hdisk8 0% 0.0 0.0 0.0 0.0 0% 0.0 -- ││hdisk7 0% 16.0 0.0 3.5 4.6 5% 2029.8 -- ││hdisk0 0% 0.0 0.0 0.0 0.0 0% 0.0 -- ││hdisk6 39% 696.2 12663.2 1350.9 9.9 39% 13359.4 0 ││hdisk5 34% 2225.8 5889.5 1378.4 5.9 100% 22158.5 -- ││hdisk1 0% 0.0 0.0 0.0 0.0 0% 0.0 -- ││Totals(MB/s) Read=258.9 Write=26.4 Size(GB)=0 Free(GB)=0 │

D

│ Disk-KBytes/second-(K=1024,M=1024*1024) ─────────────────────────────────────────────────││Disk Size Free Disk Disk Volume Disk ││ Name GB GB Paths Adapter Group Description ││cd0 0 0 0 vscsi0 None Virtual SCSI Optical Served by VIO Serv││hdisk2 0 0 1 vscsi1 None Virtual SCSI Disk Drive ││hdisk10 3624 229 2 vscsi0 orau02vg Virtual SCSI Disk Drive ││hdisk9 3067 0 2 vscsi0 orau02vg Virtual SCSI Disk Drive ││hdisk8 192 12 2 vscsi0 orafravg Virtual SCSI Disk Drive ││hdisk7 64 1 2 vscsi0 rootvg Virtual SCSI Disk Drive ││hdisk0 0 0 2 vscsi0 old_rootvg Virtual SCSI Disk Drive ││hdisk6 192 2 2 vscsi0 oravg Virtual SCSI Disk Drive ││hdisk5 360 0 2 vscsi0 orafravg Virtual SCSI Disk Drive ││hdisk1 15 5 2 vscsi0 oravg Virtual SCSI Disk Drive ││Totals(MB/s) Read=214.1 Write=68.6 Size(GB)=7513 Free(GB)=249 │

D

│ Disk - Service times and Queues ─────────────────────────────────────────────────││ Disk Service Read Service Write Service Wait ServQ WaitQ ServQ ││ Name milli-seconds milli-seconds milli-seconds milli-seconds Size Size Full ││cd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ││hdisk2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ││hdisk10 0.7 0.4 0.4 0.0 0.1 0.0 0.0 ││hdisk9 0.7 0.3 0.4 0.0 0.3 0.0 0.0 ││hdisk8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ││hdisk7 0.5 0.2 0.3 0.0 0.0 0.0 0.0 ││hdisk0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ││hdisk6 0.7 0.3 0.5 0.0 0.0 0.0 0.0 ││hdisk5 0.6 0.3 0.4 0.0 0.0 0.0 0.0 ││hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ││Totals(MB/s) Read=200.6 Write=17.0 Size(GB)=7513 Free(GB)=249 │

D

│ Disk-KBytes/second-(K=1024,M=1024*1024) ─────────────────────────────────────────────────││Disk Busy Read Write 0----------25-----------50------------75--------100 ││ Name KB/s KB/s | | | | | ││cd0 0% 0 0| | ││hdisk2 0% 0 0| | ││hdisk10 92% 90578 6115|RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRWWW > ││hdisk9 98% 222680 5767|RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR> ││hdisk8 0% 0 0| | ││hdisk7 0% 1024 2| > | ││hdisk0 0% 0 0| | ││hdisk6 0% 3 7| > | ││hdisk5 8% 636 2876|RWWWW > ││hdisk1 0% 0 0| | ││Totals 314921 14767+-----------|------------|-------------|----------+ │

nmon

o

Disks %Busy Map

│ Disk-Busy-Map ───────────────────────────────────────────────────────────────────────────││ Key(%): @=90 #=80 X=70 8=60 O=50 0=40 o=30 +=20 -=10 .=5 _=0% ││hdisks numbers-> 1 2 3 4 ││ 01234567890123456789012345678901234567890123456789 ││ 0 to 49 ___ .___@@ │

nmon

a

Disk Adapter

│ Disk-Adapter-I/O ────────────────────────────────────────────────────────────────────────││Name %busy read write xfers Disks Adapter-Type ││vscsi0 224.0 196619.6 16385.0 KB/s 32769.9 9 Virtual SCSI Client AA ││vscsi1 0.0 256.0 32.0 KB/s 32.0 9 Virtual SCSI Client A ││TOTALS 2 adapters 196875.6 16417.0 KB/s 32801.9 10 TOTAL(MB/s)=208.3 │

e

ESS I/O

│ ESS-I/O ─────────────────────────────────────────────────────────────────────────────────││ ││Name Size(GB) AvgBusy read-KB/s write-KB/s TotalMB/s xfers/s vpaths=0 ││No ESS vpath found or not found by odm commands │

V

Volume Group stats

│ Volume-Group-I/O ────────────────────────────────────────────────────────────────────────││Name Disks AvgBusy Read|WriteKB/s TotalMB/s xfers/s BlockSizeKB ││orau02vg 2 97.7% 302775.9|9595.4 305.1 46712.4 6.7 ││orafravg 2 7.5% 638.0|2471.8 3.0 406.5 7.7 ││rootvg 1 0.5% 36.0|2.0 0.0 6.0 6.3 ││old_rootvg 1 0.0% 0.0|0.0 0.0 0.0 0.0 ││oravg 2 0.0% 12.5|36.2 0.0 15.5 3.1 ││ VGs= 5 TOTALS 8 13.2% 303462.3|12105.4 308.2 47140.4 │

AIO Servers (AIX)

A

Asynchronous I/O Processes

│ Asynchronous-I/O-Processes ──────────────────────────────────────────────────────────────││ Total AIO processes= 281 Actually in use= 111 CPU used= 75.9% [Use zero ││ All time peak= 287 Recent peak= 142 Peak= 85.0% to reset] │

ioo -FL | more

NAME CUR DEF BOOT MIN MAX UNIT TYPE     DEPENDENCIES--------------------------------------------------------------------------------aio_active 1 1 boolean S--------------------------------------------------------------------------------aio_maxreqs 128K 128K 128K 4K 1M numeric D--------------------------------------------------------------------------------aio_maxservers 30 30 30 1 20000 numeric D     aio_minservers--------------------------------------------------------------------------------aio_minservers 3 3 3 0 20000 numeric D     aio_maxservers--------------------------------------------------------------------------------aio_server_inactivity 300 300 300 1 86400 seconds D--------------------------------------------------------------------------------dk_lbp_buf_size 512 512 512 512 4K int D--------------------------------------------------------------------------------dk_lbp_enabled 1 1 1 0 1 boolean D--------------------------------------------------------------------------------dk_lbp_num_bufs 64 64 64 1 1K int D--------------------------------------------------------------------------------j2_atimeUpdateSymlink 0 0 0 0 1 boolean D--------------------------------------------------------------------------------j2_chunkTruncate 0 0 0 0 1 boolean D--------------------------------------------------------------------------------j2_dynamicBufferPreallocation                          16 16 16 0 256 16K slabs D--------------------------------------------------------------------------------j2_inodeCacheSize 200 200 200 1 1000 D--------------------------------------------------------------------------------j2_maxPageReadAhead 128 128 128 0 64K 4KB pages D--------------------------------------------------------------------------------j2_maxRandomWrite 0 0 0 0 64K 4KB pages D--------------------------------------------------------------------------------j2_metadataCacheSize 200 200 200 1 1000 D--------------------------------------------------------------------------------j2_minPageReadAhead 2 2 2 0 64K 4KB pages D--------------------------------------------------------------------------------j2_nPagesPerWriteBehindCluster                          32 32 32 0 64K D--------------------------------------------------------------------------------j2_nRandomCluster 0 0 0 0 64K 16KB clusters D--------------------------------------------------------------------------------j2_numChunkTrunc 10 10 10 1 100 numeric D--------------------------------------------------------------------------------j2_recoveryMode 1 1 1 0 1 boolean D--------------------------------------------------------------------------------j2_syncByVFS 0 0 0 0 86400 seconds D--------------------------------------------------------------------------------j2_syncConcurrency 4 4 4 1 128 numeric D--------------------------------------------------------------------------------j2_syncDelayReport 600 600 600 10 64K seconds D--------------------------------------------------------------------------------j2_syncPageCount 0 0 0 0 64K 4KB pages D--------------------------------------------------------------------------------j2_syncPageLimit 16 16 16 1 64K iterations D--------------------------------------------------------------------------------lvm_bufcnt 9 9 9 1 64 128KB/buffer D--------------------------------------------------------------------------------maxpgahead 8 8 8 0 4K 4KB pages D     minpgahead--------------------------------------------------------------------------------maxrandwrt 0 0 0 0 2G-1 4KB pages D--------------------------------------------------------------------------------numclust 1 1 1 0 2G-1 16KB/cluster D--------------------------------------------------------------------------------numfsbufs 196 196 196 1 2G-1 M--------------------------------------------------------------------------------pd_npages 4K 4K 4K 1 512K 4KB pages D--------------------------------------------------------------------------------posix_aio_active 0 0 boolean S--------------------------------------------------------------------------------posix_aio_maxreqs 128K 128K 128K 4K 1M numeric D--------------------------------------------------------------------------------posix_aio_maxservers 30 30 30 1 20000 numeric D     aio_minservers--------------------------------------------------------------------------------posix_aio_minservers 3 3 3 0 20000 numeric D     aio_maxservers--------------------------------------------------------------------------------posix_aio_server_inactivity                          300 300 300 1 86400 seconds D--------------------------------------------------------------------------------spec_accessupdate 0 0 0 0 2 numeric D--------------------------------------------------------------------------------##Restricted tunables--------------------------------------------------------------------------------aio_fastpath 1 1 1 0 1 boolean D--------------------------------------------------------------------------------aio_fsfastpath 1 1 1 0 1 boolean D--------------------------------------------------------------------------------aio_kprocprio 39 39 39 0 254 numeric D--------------------------------------------------------------------------------aio_multitidsusp 1 1 1 0 1 boolean D--------------------------------------------------------------------------------aio_sample_rate 5 5 5 1 86400 seconds D--------------------------------------------------------------------------------aio_samples_per_cycle 6 6 6 1 128K numeric D--------------------------------------------------------------------------------iodone_distr_all_bufs 0 0 0 0 1 boolean D--------------------------------------------------------------------------------iodone_distr_disable 0 0 0 0 1 boolean B--------------------------------------------------------------------------------iodone_distr_sib_check 1 1 1 0 1 boolean B--------------------------------------------------------------------------------iodone_distr_trace_iodone 0 0 0 0 1 boolean D--------------------------------------------------------------------------------j2_maxPageLocks 64 64 64 8 16K numeric D--------------------------------------------------------------------------------j2_maxUsableMaxTransfer 512 512 512 1 4K pages M--------------------------------------------------------------------------------j2_nBufferPerPagerDevice 512 512 512 512 256K M--------------------------------------------------------------------------------j2_nonFatalCrashesSystem 0 0 0 0 1 boolean D--------------------------------------------------------------------------------j2_syncMappedLimit 0 0 0 0 1M megabytes D--------------------------------------------------------------------------------j2_syncModifiedMapped 1 1 1 0 1 boolean D--------------------------------------------------------------------------------j2_syncdLogSyncInterval 1 1 1 0 4K iterations D--------------------------------------------------------------------------------j2_unmarkComp 0 0 0 0 1 boolean D--------------------------------------------------------------------------------jfs_clread_enabled 0 0 0 0 1 boolean D--------------------------------------------------------------------------------jfs_use_read_lock 1 1 1 0 1 boolean D--------------------------------------------------------------------------------memory_frames 8M 8M 4KB pages S--------------------------------------------------------------------------------minpgahead 2 2 2 0 4K 4KB pages D     maxpgahead--------------------------------------------------------------------------------pcibus_dma_memory_protect n/a 0 0 0 0 boolean D--------------------------------------------------------------------------------pcibus_eeh_perm_timeout n/a 0 0 0 0 seconds D--------------------------------------------------------------------------------pcibus_max_dma_window n/a 0 0 0 0 gigabytes B--------------------------------------------------------------------------------pgahd_scale_thresh 0 0 0 0 6553K 4KB pages D--------------------------------------------------------------------------------posix_aio_fastpath 1 1 1 0 1 boolean D--------------------------------------------------------------------------------posix_aio_fsfastpath 1 1 1 0 1 boolean D--------------------------------------------------------------------------------posix_aio_kprocprio 39 39 39 0 254 numeric D--------------------------------------------------------------------------------posix_aio_sample_rate 5 5 5 1 86400 seconds D--------------------------------------------------------------------------------posix_aio_samples_per_cycle                          6 6 6 1 128K numeric D--------------------------------------------------------------------------------pv_min_pbuf 512 512 512 512 2G-1 D--------------------------------------------------------------------------------sync_release_ilock 0 0 0 0 1 boolean D--------------------------------------------------------------------------------
n/a means parameter not supported by the current platform or kernel
Parameter types:    S = Static: cannot be changed    D = Dynamic: can be freely changed    B = Bosboot: can only be changed using bosboot and reboot    R = Reboot: can only be changed during reboot    C = Connect: changes are only effective for future socket connections    M = Mount: changes are only effective for future mountings    I = Incremental: can only be incremented    d = deprecated: deprecated and cannot be changed
Value conventions:    K = Kilo: 2^10 G = Giga: 2^30 P = Peta: 2^50    M = Mega: 2^20 T = Tera: 2^40 E = Exa: 2^60

ioo -o aio_maxreqs

iostat -A | more

Legacy AIO Servers

iostat -P | more

POSIX AIO Servers

System configuration: lcpu=20 drives=10 ent=4.25 paths=17 vdisks=2 maxserver=600
aio: avgc avfc maxgc maxfc maxreqs avg-cpu: % user % sys % idle % iowait physc % entc 5586.7 0.0 210 0 131072 12.9 8.3 21.1 57.7 0.9 22.1
Disks: % tm_act Kbps tps Kb_read Kb_wrtncd0 0.0 0.0 0.0 0 0hdisk2 0.0 0.0 0.0 0 0hdisk10 21.3 34431.3 2264.2 248027820600 3412123516hdisk9 21.9 47080.6 3040.9 330712945491 13100502192hdisk8 0.0 21.7 0.1 95690628 62895220hdisk7 1.3 185.2 15.8 1202817237 149950408hdisk0 0.0 0.0 0.0 0 0hdisk6 1.1 128.7 19.2 330604275 609391187hdisk5 5.1 2398.9 138.2 7024666038 10493528608hdisk1 0.0 0.0 0.0 5087 40

pstat -a | head -3; pstat -a | egrep ' aioserver'

Legacy AIO Servers

pstat -a | head -3; pstat -a | egrep 'posix_aioserver'

POSIX AIO Servers

PROC TABLE:
SLT ST PID PPID PGRP UID EUID TCNT NAME 57 a 39000a 1 39000a 211 211 1 aioserver 67 a 430074 1 430074 211 211 1 aioserver 69 a 4500bc 1 4500bc 211 211 1 aioserver 92 a 5c006e 1 5c006e 211 211 1 aioserver 93 a 5d0094 1 5d0094 211 211 1 aioserver 97 a 610030 1 610030 211 211 1 aioserver100 a 64005e 1 64005e 211 211 1 aioserver103 a 670066 1 670066 211 211 1 aioserver105 a 690048 1 690048 211 211 1 aioserver...

ps -ek|grep -v grep|grep –v posix_aioserver|grep -c aioserver

IOCP (AIX)

IO Completion Ports

lslpp -l bos.iocp.rte

lsdev -Cc iocp

VIO (AIX)

To determine if your I/O is handled by a Virtual I/O Server...

lsdev -Cc disk

Disks labelled as "Virtual SCSI Disk Drive" are managed by a VIO Server

lspath

lspath -E -l hdisk10 -i 0 -a priority

lsattr -El hdisk10

lsattr -El vscsi0

lsmpio -l hdisk10

Bibliography

https://www.geeksforgeeks.org/dstat-command-in-linux-with-examples/https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation/https://linoxide.com/linux-how-to/measure-disk-performance-fio/https://scoutapm.com/blog/understanding-disk-i-o-when-should-you-be-worriedhttps://unixutils.com/monitoring-disk-iops-in-linux/https://www.percona.com/blog/2014/06/25/why-util-number-from-iostat-is-meaningless-for-mysql-capacity-planning/https://unix.stackexchange.com/questions/60091/where-is-the-documentation-for-what-sda-sdb-dm-0-dm-1-meanhttps://unix.stackexchange.com/questions/86875/determining-specific-file-responsible-for-high-i-o
IOPShttps://unixutils.com/monitoring-disk-iops-in-linux/
iotophttps://superuser.com/questions/610581/iotop-complains-config-task-delay-acct-not-enabled-in-kernel-only-for-specifichttps://www.kernel.org/doc/html/latest/accounting/delay-accounting.html#usagehttps://www.mankier.com/8/iotop-chttps://unix.stackexchange.com/questions/238444/iotop-and-ionice-p-show-a-different-class-priority-for-the-same-process(1) https://linux.die.net/man/1/ionice
AIOhttps://www.linkedin.com/pulse/aio-aix-fast-path-great-performance-mark-rayhttps://www.ibm.com/docs/en/aix/7.2?topic=aio-identifying-number-servers-used-currentlyhttps://www.ibm.com/docs/en/aix/7.3?topic=aio-identifying-number-servershttps://www.ibm.com/docs/en/aix/7.2?topic=aio-tunable-values-asynchronous-iohttps://community.oracle.com/mosc/discussion/2718030/aioservers-on-aixhttps://support.oracle.com - How does Oracle use AIO servers and what determines how many are used? (Doc ID 443368.1)https://docs.oracle.com/en/database/oracle/oracle-database/12.2/axdbi/checking-asynchronous-input-output-processes.htmlhttps://community.spiceworks.com/t/aioserver/913418/2https://docs.oracle.com/en/database/oracle/oracle-database/19/unxar/administering-oracle-database-on-ibm-aix-on-power-systems-64-Bit.html
iostathttps://manpages.debian.org/testing/sysstat/iostat.1.en.htmlhttps://linux.die.net/man/1/iostathttps://unix.stackexchange.com/questions/104192/iostat-await-vs-svctm

IOCPhttps://www.ibm.com/docs/en/db2/11.1?topic=parallelism-configuring-iocp-aix