UNIX File Management

 pwd, cd, mkdir, rmdir, pushd, popd
vim, head, tail, more, less
zip, gzip, compress
cp, mv, rsync, gcp
chmod, chgrp, chown, aclput, aclget

TODO

Listing Files & Directories

ls

ls -a

ls -A

ls -d

ls -m

ls -r

ls -l

ll

ls -l --block-size=M

ls --full-time

ll

ls -lh

ls -la

ls -lt

ls -lart

ls -lS

ls -lrS

ls -R

ls --group-directories-first

ls --help

ls --version

find .

find . -exec ls -l {} \;

find . -type f -mtime +3285

find . -newer myfile

List files and directories in current directory

List files and directories including those starting with . (hidden)

as above but ignore the special paths . (current dir) and .. (parent dir)

List directories, not their contents

Comma separated list of files and directories

Reverse sort order

Long list - shows information about each file

Long list - synonym for above

As above but show sizes in MB instead of bytes

As above but with full iso timestamp

Long list - synonym for above

Long list with human readable file sizes

Long list with all files (including those starting with . that are normally hidden)

Long list sorted by modification time (ascending)

Long list, all files, in reverse order, by modification time (descending)

Long list sorted by file size (largest first)

Long list sorted by file size (smallest first)

Recursive (include contents of subdirectories)

Show directories at start of list followed by files

The man page (above information is abbreviated, man page contains full information)

Show version

Show relative path for all files in current directory and all subdirectories

Perform ls -l for all files in current directory and all subdirectories

Find all regular files in current directory and all subdirectories where file was last modified over 9 years ago.

Find all files in current directory and all subdirectories where file was modified more recently than myfile

To create an empty file with a specified timestamp for use with the -newer option see the "Generating Test Files" section later on this page.

lsd

exa

snap install lsd

apt install exa

Compatible with ls but with icons

Similar to ls and lsd but with some other options

ls -l

lsd -l

exa -l --icons

exa --long --tree

exa --long --grid

exa -abghHliS

File metadata

wc myfile

wc -c myfile

wc -m myfile

wc -l myfile

wc -L myfile

wc -w myfile

cat myfile | wc

wc --bytes myfile

wc --chars myfile

wc --lines myfile

wc --max-line-length myfile

wc --words  myfile

Show number of lines, words and bytes in myfile

Show number of bytes in myfile

Show number of characters in myfile

Show number of lines in myfile

Show maximum line length in myfile

Show number of words in myfile

file myfile.txt

file myfile.sh

file myfile.zip

file mydir

myfile.txt: ASCII text

myfile.sh: POSIX shell script, ASCII text executable

myfile.zip: Zip archive data, at least v2.0 to extract

mydir: directory

du myfile

du -h myfile

du -h *

du -hsc pattern

du -sg *

Disk Usage of myfile

Disk Usage of myfile with human readable sizing 

Disk Usage of all files in current directory

Total Disk Usage of all files matching pattern

Total Disk Usage of all files and directories (and all files and subdirectories therein) in current directory in GB (AIX only)

dust *

Disk Usage of all files in current directory, with histogram

snap install dust

find . -type f -name 'pattern' -exec du -ch {} + | grep total$

find . -type f -name "pattern" -printf "%s\n" | gawk -M '{t+=$1}END{print t}'

{ find . -type f -name "pattern" -printf "%s+"; echo 0; } | bc

Total Disk Usage of all files matching pattern

Total Disk Usage of all files matching pattern

Total Disk Usage of all files matching pattern

find /u01/in -type d | xargs du -sh

Directory tree with disk usage of each subdirectory (human readable sizes)

tree

tree --help

tree -C

tree -h

tree -a

tree -f

tree -pug

tree -P pattern

tree --prune

Directory tree

Help

Use colours to indicate file types

Include human readable size size

Include hidden files

Include file path

Include permissions, user and group

Include only file/directories that match pattern

Prune empty directories from output

yum install tree

dnf install tree

apt install tree

zypper in tree

tree

tree -d

.├── alert│   └── log.xml├── cdump├── incident├── incpkg├── lck│   ├── AM_1096102193_3488045378.lck│   ├── AM_1096102262_3454819329.lck│   ├── AM_1744845641_3861997533.lck│   └── AM_3216668543_3129272988.lck├── log│   ├── debug│   └── test├── metadata│   ├── ADR_CONTROL.ams│   ├── ADR_INVALIDATION.ams│   ├── INC_METER_IMPT_DEF.ams│   └── INC_METER_PK_IMPTS.ams├── metadata_dgif├── metadata_pv├── stage├── sweep└── trace    └── sqlnet.log
14 directories, 10 files
.├── alert├── cdump├── incident├── incpkg├── lck├── log│   ├── debug│   └── test├── metadata├── metadata_dgif├── metadata_pv├── stage├── sweep└── trace
14 directories

tree -Ppughf *.ams --prune

.└── [drwxr-xr-x oracle   oinstall 4.0K]  ./metadata    ├── [-rw-r----- oracle   oinstall  64K]  ./metadata/ADR_CONTROL.ams    ├── [-rw-r----- oracle   oinstall  64K]  ./metadata/ADR_INVALIDATION.ams    ├── [-rw-r----- oracle   oinstall  64K]  ./metadata/INC_METER_IMPT_DEF.ams    └── [-rw-r----- oracle   oinstall  64K]  ./metadata/INC_METER_PK_IMPTS.ams
1 directory, 4 files

Counting Files Created by Year

touch -amt 202101010000 /tmp/$$-01012021.txt

touch -amt 202201010000 /tmp/$$-01012022.txt

touch -amt 202301010000 /tmp/$$-01012023.txt

touch -amt 202401010000 /tmp/$$-01012024.txt

touch -amt 202501010000 /tmp/$$-01012025.txt

echo "2021: " $(find . -newer /tmp/$$-01012021.txt ! -newer /tmp/$$-01012022.txt | wc -l)

echo "2022: " $(find . -newer /tmp/$$-01012022.txt ! -newer /tmp/$$-01012023.txt | wc -l)

echo "2023: " $(find . -newer /tmp/$$-01012023.txt ! -newer /tmp/$$-01012024.txt | wc -l)

echo "2024: " $(find . -newer /tmp/$$-01012024.txt ! -newer /tmp/$$-01012025.txt | wc -l)

echo "2025: " $(find . -newer /tmp/$$-01012025.txt | wc -l)

rm /tmp/$$-*.txt

(Tested on AIX)

Who has File Open?

fuser myfile

Returns a list of process ids for processes with the file open.

myfile:  7471242 8650968 14024712 14221336 7930360 11403542 11469080 16777704 24052122

lsof myfile

Checking File Existence

Reporting whether file exists or not...

if [[ -f myfile.txt ]]

then

   echo "exists"

else

   echo "missing"

fi

Reporting whether file exists...

[[ -f myfile.txt ]] && echo "exists"

Reporting whether file is missing...

[[ ! -f myfile.txt ]] && echo "missing"

Reporting missing files from a provided list...

for file in $(cat ${FILELIST})

do

  [ ! -f ${file} ] && echo ${file}

done >missingfiles.txt

Variation that captures progress...

>missingfiles.txt

>processed.txt

for file in $(cat ${FILELIST})

do

  [ ! -f ${file} ] && echo ${file} >>missingfiles.txt

  echo ${file} >>processed.txt

done

To report progress use...

echo "$(cat processed.txt|wc -l)|$(cat ${FILELIST}|wc -l)" | awk 'BEGIN { FS = "|" } { $3 = $1 / $2 * 100 "%" } { print $3 }'

Checking for Difference

diff myfile1 myfile2

diff -c myfile1 myfile2

UNIX-Windows-Cribsheet

Examples

myfile1

aaaaaaaa

bbbbbbbb

cccccccc

dddddddd

DDDDDDDD

eeeeeeee

ffffffff

myfile2

aaaaaaaa

bbbbbbBB

cccccccc

DDDDDDDD

eeeeeeee

EEEEEEEE

ffffffff


Line 1 remains the same

Line 2 is changed

Line 3 remains the same

Line 4 is deleted

Line 5 remains the same

Line 6 is added

Line 7 stays the same

  diff myfile1 myfile2

2c2

< bbbbbbbb

---

> bbbbbbBB

4d3

< dddddddd

6a6

> EEEEEEEE

Line 2 is changed

< indicates state before change


> indicates state after change

Line 4 is deleted (and becomes line 3)

< indicates state before deletion

Line 6 is added

> indicates state after addition

diff -c myfile1 myfile2

*** myfile1     2023-04-19 13:33:52.988318584 +0100

--- myfile2     2023-04-19 13:34:19.610844453 +0100

***************

*** 1,7 ****

  aaaaaaaa

! bbbbbbbb

  cccccccc

- dddddddd

  DDDDDDDD

  eeeeeeee

  ffffffff

--- 1,7 ----

  aaaaaaaa

! bbbbbbBB

  cccccccc

  DDDDDDDD

  eeeeeeee

+ EEEEEEEE

  ffffffff

myfile1 is represented by ***

myfile2 is represented by ---


stanza contains lines 1 to 7 of myfile1


! indicates this row will be changed


- indicates this row is deleted



stanza contains lines 1 to 7 of myfile 2


! indicates this row has changed





+ indicates this row is added


diff -e myfile1 myfile2

6a

EEEEEEEE

.

4d

2c

bbbbbbBB

.

Commands are suitable for the 'ed' editor.

Hard/Symbolic Links

To have the file /u01/myfile.txt also be accessible using /u01/myfile2.txt...

ln /u01/myfile.txt /u01/myfile2.txt

To have the file /u01/myfile.txt appear to be on /u02 as myfile.txt...

ln -s /u01/myfile.txt /u02/myfile.txt

This is a symbolic link (as specified by the -s); only one file exists but it can be accessed from two distinct paths

Notes

Generating Test Files

To generate a 77GiB test file...

time dd if=/dev/zero of=myfile1.tst bs=4k iflag=fullblock,count_bytes count=77G status=progress

To generate a 77GiB test file with random data...

time dd if=/dev/urandom of=myfile1.tst bs=4k iflag=fullblock,count_bytes count=77G status=progress

Note that this is more cpu intensive than using /dev/zero

To create an empty file (or change the timestamp of an existing file) with the current timestamp...

touch myfile

To create an empty file (or change the timestamp of an existing file) with a specific timestamp...

touch -d "20230101" myfile

touch -d "20230101 1415" myfile

touch --date "1972-03-31 10:15" myfile

touch --date "next Thursday" myfile



-rw-r--r-- 1 root root 0 Feb 14 17:32 myfile


-rw-r--r-- 1 root root 0 Jan  1 00:00 myfile

-rw-r--r-- 1 root root 0 Jan  1 14:15 myfile

-rw-r--r-- 1 root root 0 Mar 31  1972 myfile

-rw-r--r-- 1 root root 0 Feb 16  2023 myfile

Moving Files

mv myfile mynewfile

This example moves files older than 14 days from a main directory (/u01/out) to a subdirectory (/u01/out/oldfiles) without affecting any files in other subdirectories of the main directory...

mkdir -p /u01/out/oldfiles

for file in $(find /u01/out ! -path /u01/out -prune -type f -mtime +14)

do

  ls -l ${file}

  mv ${file} /u01/out/oldfiles

done

Known issues

In ksh on AIX you may see this error if there are a lot of files in the directory...: 

0403-029 There is not enough memory available now. 

The workaround is to use ksh93 instead of ksh.

Deleting Files

rm myfile

find . -type f -mtime +3285 -exec ls -l {} \; -exec rm {} \;

3285 = approx 9 years (9*365)2920 = approx 8 years (8*365)2555 = approx 7 years (7*365)2190 = approx 6 years (6*365)366 = 1 year (allowing for leap year)

Truncating Files

> myfile

Bibliography

https://www.ibm.com/support/pages/ksh-%E2%80%9C0403-029-there-not-enough-memory-available-now%E2%80%9D https://unix.stackexchange.com/questions/109900/find-prune-does-not-ignore-specified-path https://en.wikipedia.org/wiki/Ln_(Unix)https://detailed.wordpress.com/2017/10/28/understanding-ls-command-output/https://devconnected.com/how-to-check-if-file-or-directory-exists-in-bash/https://www.unix.com/hp-ux/278999-shell-awk-script-percentage.htmlhttps://github.com/ibraheemdev/modern-unixhttps://github.com/bootandy/dusthttps://github.com/ogham/exa
findhttps://unix.stackexchange.com/questions/298590/using-find-non-recursivelyhttps://unix.stackexchange.com/questions/24557/how-do-i-stop-a-find-from-descending-into-found-directorieshttps://unix.stackexchange.com/questions/41550/find-the-total-size-of-certain-files-within-a-directory-branchhttps://www.unix.com/shell-programming-and-scripting/193857-find-w-multiple-expressions.htmlhttps://stackoverflow.com/questions/5119946/find-exec-with-multiple-commandshttps://stackoverflow.com/questions/71562728/find-with-shell-with-different-and-or-conditionshttps://stackoverflow.com/questions/56221518/whats-meaning-of-print0-in-following-commandhttps://askubuntu.com/questions/191044/how-to-find-files-between-two-dates-using-find
Viewing Fileshttps://www.ibm.com/docs/fr/aix/7.1?topic=p-pg-command
Generating Test Fileshttps://stackoverflow.com/questions/257844/quickly-create-a-large-file-on-a-linux-systemhttps://man7.org/linux/man-pages/man1/touch.1.html
rmhttps://man7.org/linux/man-pages/man1/rm.1.html
wchttps://linux.die.net/man/1/wc
duhttps://stackoverflow.com/questions/71074558/how-to-use-tree-or-du-command-to-get-directory-content-sizes
treehttps://www.tecmint.com/linux-tree-command-examples/
Hex Editorshttps://www.tecmint.com/best-hex-editors-for-linux/