UNIX File Management

Infrastructure

UNIX

Listing Files & Directories

File metadata

Counting Files Created by Year

Who has File Open?

Checking File Existence

Checking for Difference

Examples

Hard/Symbolic Links

Generating Test Files

UNIX Directories

pwd, cd, mkdir, rmdir, pushd, popd

sh File Handling

vim, head, tail, more, less

UNIX File Checksums

UNIX File Compression

zip, gzip, compress

UNIX File Copy

cp, mv, rsync, gcp

UNIX Filesystems

UNIX File Permissions

chmod, chgrp, chown, aclput, aclget

logrotate

TODO

skulker

Listing Files & Directories

ls -a

ls -A

ls -d

ls -m

ls -r

ls -l

ls -l --block-size=M

ls --full-time

ls -lh

ls -la

ls -lt

ls -lart

ls -lS

ls -lrS

ls -R

ls --group-directories-first

ls --help

ls --version

find .

find . -exec ls -l {} \;

find . -type f -mtime +3285

find . -newer myfile

List files and directories in current directory

List files and directories including those starting with . (hidden)

as above but ignore the special paths . (current dir) and .. (parent dir)

List directories, not their contents

Comma separated list of files and directories

Reverse sort order

Long list - shows information about each file

Long list - synonym for above

As above but show sizes in MB instead of bytes

As above but with full iso timestamp

Long list - synonym for above

Long list with human readable file sizes

Long list with all files (including those starting with . that are normally hidden)

Long list sorted by modification time (ascending)

Long list, all files, in reverse order, by modification time (descending)

Long list sorted by file size (largest first)

Long list sorted by file size (smallest first)

Recursive (include contents of subdirectories)

Show directories at start of list followed by files

The man page (above information is abbreviated, man page contains full information)

Show version

Show relative path for all files in current directory and all subdirectories

Perform ls -l for all files in current directory and all subdirectories

Find all regular files in current directory and all subdirectories where file was last modified over 9 years ago.

Find all files in current directory and all subdirectories where file was modified more recently than myfile

UNIX File Permissions

To create an empty file with a specified timestamp for use with the -newer option see the "Generating Test Files" section later on this page.

lsd

exa

snap install lsd

apt install exa

Compatible with ls but with icons

Similar to ls and lsd but with some other options

ls -l

lsd -l

exa -l --icons

exa --long --tree

exa --long --grid

exa -abghHliS

File metadata

wc myfile

wc -c myfile

wc -m myfile

wc -l myfile

wc -L myfile

wc -w myfile

cat myfile | wc

wc --bytes myfile

wc --chars myfile

wc --lines myfile

wc --max-line-length myfile

wc --words myfile

Show number of lines, words and bytes in myfile

Show number of bytes in myfile

Show number of characters in myfile

Show number of lines in myfile

Show maximum line length in myfile

Show number of words in myfile

file myfile.txt

file myfile.sh

file myfile.zip

file mydir

myfile.txt: ASCII text

myfile.sh: POSIX shell script, ASCII text executable

myfile.zip: Zip archive data, at least v2.0 to extract

mydir: directory

du myfile

du -h myfile

du -h *

du -hsc pattern

du -sg *

Disk Usage of myfile

Disk Usage of myfile with human readable sizing

Disk Usage of all files in current directory

Total Disk Usage of all files matching pattern

Total Disk Usage of all files and directories (and all files and subdirectories therein) in current directory in GB (AIX only)

dust *

Disk Usage of all files in current directory, with histogram

snap install dust

find . -type f -name 'pattern' -exec du -ch {} + | grep total$

find . -type f -name "pattern" -printf "%s\n" | gawk -M '{t+=$1}END{print t}'

{ find . -type f -name "pattern" -printf "%s+"; echo 0; } | bc

Total Disk Usage of all files matching pattern

find /u01/in -type d | xargs du -sh

Directory tree with disk usage of each subdirectory (human readable sizes)

tree

tree --help

tree -C

tree -h

tree -a

tree -f

tree -pug

tree -P pattern

tree --prune

Directory tree

Help

Use colours to indicate file types

Include human readable size size

Include hidden files

Include file path

Include permissions, user and group

Include only file/directories that match pattern

Prune empty directories from output

yum install tree

dnf install tree

apt install tree

zypper in tree

tree

tree -d

.├── alert│ └── log.xml├── cdump├── incident├── incpkg├── lck│ ├── AM_1096102193_3488045378.lck│ ├── AM_1096102262_3454819329.lck│ ├── AM_1744845641_3861997533.lck│ └── AM_3216668543_3129272988.lck├── log│ ├── debug│ └── test├── metadata│ ├── ADR_CONTROL.ams│ ├── ADR_INVALIDATION.ams│ ├── INC_METER_IMPT_DEF.ams│ └── INC_METER_PK_IMPTS.ams├── metadata_dgif├── metadata_pv├── stage├── sweep└── trace └── sqlnet.log
14 directories, 10 files

.├── alert├── cdump├── incident├── incpkg├── lck├── log│ ├── debug│ └── test├── metadata├── metadata_dgif├── metadata_pv├── stage├── sweep└── trace
14 directories

tree -Ppughf *.ams --prune

.└── [drwxr-xr-x oracle oinstall 4.0K] ./metadata ├── [-rw-r----- oracle oinstall 64K] ./metadata/ADR_CONTROL.ams ├── [-rw-r----- oracle oinstall 64K] ./metadata/ADR_INVALIDATION.ams ├── [-rw-r----- oracle oinstall 64K] ./metadata/INC_METER_IMPT_DEF.ams └── [-rw-r----- oracle oinstall 64K] ./metadata/INC_METER_PK_IMPTS.ams
1 directory, 4 files

Counting Files Created by Year

touch -amt 202101010000 /tmp/$$-01012021.txt

touch -amt 202201010000 /tmp/$$-01012022.txt

touch -amt 202301010000 /tmp/$$-01012023.txt

touch -amt 202401010000 /tmp/$$-01012024.txt

touch -amt 202501010000 /tmp/$$-01012025.txt

echo "2021: " $(find . -newer /tmp/$$-01012021.txt ! -newer /tmp/$$-01012022.txt | wc -l)

echo "2022: " $(find . -newer /tmp/$$-01012022.txt ! -newer /tmp/$$-01012023.txt | wc -l)

echo "2023: " $(find . -newer /tmp/$$-01012023.txt ! -newer /tmp/$$-01012024.txt | wc -l)

echo "2024: " $(find . -newer /tmp/$$-01012024.txt ! -newer /tmp/$$-01012025.txt | wc -l)

echo "2025: " $(find . -newer /tmp/$$-01012025.txt | wc -l)

rm /tmp/$$-*.txt

(Tested on AIX)

Who has File Open?

fuser myfile

Returns a list of process ids for processes with the file open.

myfile: 7471242 8650968 14024712 14221336 7930360 11403542 11469080 16777704 24052122

lsof myfile

Checking File Existence

Reporting whether file exists or not...

if [[ -f myfile.txt ]]

then

echo "exists"

else

echo "missing"

Reporting whether file exists...

[[ -f myfile.txt ]] && echo "exists"

Reporting whether file is missing...

[[ ! -f myfile.txt ]] && echo "missing"

Reporting missing files from a provided list...

for file in $(cat ${FILELIST})

[ ! -f ${file} ] && echo ${file}

done >missingfiles.txt

Variation that captures progress...

>missingfiles.txt

>processed.txt

for file in $(cat ${FILELIST})

[ ! -f ${file} ] && echo ${file} >>missingfiles.txt

echo ${file} >>processed.txt

done

To report progress use...

echo "$(cat processed.txt|wc -l)|$(cat ${FILELIST}|wc -l)" | awk 'BEGIN { FS = "|" } { $3 = $1 / $2 * 100 "%" } { print $3 }'

Checking for Difference

diff myfile1 myfile2

diff -c myfile1 myfile2

UNIX-Windows-Cribsheet

Examples

myfile1

aaaaaaaa

bbbbbbbb

cccccccc

dddddddd

DDDDDDDD

eeeeeeee

ffffffff

myfile2

aaaaaaaa

bbbbbbBB

cccccccc

DDDDDDDD

eeeeeeee

EEEEEEEE

ffffffff

Line 1 remains the same

Line 2 is changed

Line 3 remains the same

Line 4 is deleted

Line 5 remains the same

Line 6 is added

Line 7 stays the same

diff myfile1 myfile2

2c2

< bbbbbbbb

---

> bbbbbbBB

4d3

< dddddddd

6a6

> EEEEEEEE

Line 2 is changed

< indicates state before change

> indicates state after change

Line 4 is deleted (and becomes line 3)

< indicates state before deletion

Line 6 is added

> indicates state after addition

diff -c myfile1 myfile2

*** myfile1 2023-04-19 13:33:52.988318584 +0100

--- myfile2 2023-04-19 13:34:19.610844453 +0100

***************

*** 1,7 ****

aaaaaaaa

! bbbbbbbb

cccccccc

- dddddddd

DDDDDDDD

eeeeeeee

ffffffff

--- 1,7 ----

aaaaaaaa

! bbbbbbBB

cccccccc

DDDDDDDD

eeeeeeee

+ EEEEEEEE

ffffffff

myfile1 is represented by ***

myfile2 is represented by ---

stanza contains lines 1 to 7 of myfile1

! indicates this row will be changed

- indicates this row is deleted

stanza contains lines 1 to 7 of myfile 2

! indicates this row has changed

+ indicates this row is added

diff -e myfile1 myfile2

EEEEEEEE

bbbbbbBB

Commands are suitable for the 'ed' editor.

Hard/Symbolic Links

To have the file /u01/myfile.txt also be accessible using /u01/myfile2.txt...

ln /u01/myfile.txt /u01/myfile2.txt

To have the file /u01/myfile.txt appear to be on /u02 as myfile.txt...

ln -s /u01/myfile.txt /u02/myfile.txt

This is a symbolic link (as specified by the -s); only one file exists but it can be accessed from two distinct paths

Notes

Hard links can only exist on the same filesystem whereas symbolic links can link across different filesystems.
Hard links cannot be used for directories whereas symbolic links can.
Hard links share an inode.

UNIX Filesystems

Generating Test Files

To generate a 77GiB test file...

time dd if=/dev/zero of=myfile1.tst bs=4k iflag=fullblock,count_bytes count=77G status=progress

To generate a 77GiB test file with random data...

time dd if=/dev/urandom of=myfile1.tst bs=4k iflag=fullblock,count_bytes count=77G status=progress

Note that this is more cpu intensive than using /dev/zero

To create an empty file (or change the timestamp of an existing file) with the current timestamp...

touch myfile

To create an empty file (or change the timestamp of an existing file) with a specific timestamp...

touch -d "20230101" myfile

touch -d "20230101 1415" myfile

touch --date "1972-03-31 10:15" myfile

touch --date "next Thursday" myfile

-rw-r--r-- 1 root root 0 Feb 14 17:32 myfile

-rw-r--r-- 1 root root 0 Jan 1 00:00 myfile

-rw-r--r-- 1 root root 0 Jan 1 14:15 myfile

-rw-r--r-- 1 root root 0 Mar 31 1972 myfile

-rw-r--r-- 1 root root 0 Feb 16 2023 myfile

Moving Files

mv myfile mynewfile

This example moves files older than 14 days from a main directory (/u01/out) to a subdirectory (/u01/out/oldfiles) without affecting any files in other subdirectories of the main directory...

mkdir -p /u01/out/oldfiles

for file in $(find /u01/out ! -path /u01/out -prune -type f -mtime +14)

ls -l ${file}

mv ${file} /u01/out/oldfiles

done

Known issues

In ksh on AIX you may see this error if there are a lot of files in the directory...:

0403-029 There is not enough memory available now.

The workaround is to use ksh93 instead of ksh.

Deleting Files

rm myfile

find . -type f -mtime +3285 -exec ls -l {} \; -exec rm {} \;

3285 = approx 9 years (9*365)2920 = approx 8 years (8*365)2555 = approx 7 years (7*365)2190 = approx 6 years (6*365)366 = 1 year (allowing for leap year)

Truncating Files

> myfile

Bibliography

https://www.ibm.com/support/pages/ksh-%E2%80%9C0403-029-there-not-enough-memory-available-now%E2%80%9D https://unix.stackexchange.com/questions/109900/find-prune-does-not-ignore-specified-path https://en.wikipedia.org/wiki/Ln_(Unix)https://detailed.wordpress.com/2017/10/28/understanding-ls-command-output/https://devconnected.com/how-to-check-if-file-or-directory-exists-in-bash/https://www.unix.com/hp-ux/278999-shell-awk-script-percentage.htmlhttps://github.com/ibraheemdev/modern-unixhttps://github.com/bootandy/dusthttps://github.com/ogham/exa
findhttps://unix.stackexchange.com/questions/298590/using-find-non-recursivelyhttps://unix.stackexchange.com/questions/24557/how-do-i-stop-a-find-from-descending-into-found-directorieshttps://unix.stackexchange.com/questions/41550/find-the-total-size-of-certain-files-within-a-directory-branchhttps://www.unix.com/shell-programming-and-scripting/193857-find-w-multiple-expressions.htmlhttps://stackoverflow.com/questions/5119946/find-exec-with-multiple-commandshttps://stackoverflow.com/questions/71562728/find-with-shell-with-different-and-or-conditionshttps://stackoverflow.com/questions/56221518/whats-meaning-of-print0-in-following-commandhttps://askubuntu.com/questions/191044/how-to-find-files-between-two-dates-using-find
Viewing Fileshttps://www.ibm.com/docs/fr/aix/7.1?topic=p-pg-command
Generating Test Fileshttps://stackoverflow.com/questions/257844/quickly-create-a-large-file-on-a-linux-systemhttps://man7.org/linux/man-pages/man1/touch.1.html
rmhttps://man7.org/linux/man-pages/man1/rm.1.html
wchttps://linux.die.net/man/1/wc
duhttps://stackoverflow.com/questions/71074558/how-to-use-tree-or-du-command-to-get-directory-content-sizes
treehttps://www.tecmint.com/linux-tree-command-examples/
Hex Editorshttps://www.tecmint.com/best-hex-editors-for-linux/

Oracle Internals