Refreshing Important Unix/Linux Commands

This is refresher on the Unix/Linux commands which i use quite often. Hence, wanted to share them with fellow IT professionals.

Disk Usage

du -ah   display disk usage of all files and directories
du -ahc display disk usage of all file and directories and display total usage
du -mh   display disk usage in MB
du -sh   display the total used size in form of a summary
du -ahc —exclude=“*.txt”   exclude files from disk usage output
du -ahc —time   display the modification time in disk usage output

Secure Copy

Copy the file “snowy.txt” from a remote host to the local host
$ scp your_username@remotehost:snowy.txt /some/local/directory

Copy the file “snowy.txt” from the local host to a remote host
$ scp snowy.txt your_username@remotehost:/some/remote/directory

Copy the directory “tardis” from the local host to a remote host’s directory “bar”
$ scp -r tardis your_username@remotehost:/some/remote/directory/bar

Copy the file “snowy.txt” from remote host “rh1” to remote host “rh2”
$ scp your_username@rh1:/some/remote/directory/snowy.txt \

Copying the files “tardis.txt” and “bar.txt” from the local host to your home directory on the remote host
$ scp tardis.txt bar.txt your_username@remotehost:~

Copy the file “snowy.txt” from the local host to a remote host using port 2264
$ scp -P 2264 snowy.txt your_username@remotehost:/some/remote/directory

Copy multiple files from the remote host to your current directory on the local host
$ scp your_username@remotehost:/some/remote/directory/\{a,b,c\} .
$ scp your_username@remotehost:~/\{tardis.txt,bar.txt\} .

File Transfer Protocol (FTP)

Connect to FTP server and change local/remote directories
ftp IP/Hostname
ftp> bin
ftp> hash
Change remote server current directory
ftp> pwd
257 “/myftpserver” is current directory.
ftp> cd dir1
250 CWD command successful. “/myftpserver/dir1” is current directory.
ftp> pwd
257 “/myftpserver/dir1” is current directory.
Change local machine current directory
ftp> pwd
ftp> lcd /tmp
Local directory now /tmp
ftp> pwd
Download Files from FTP Server
ftp> ls
ftp> get FILENAME
ftp> mget *.html (download multiple files using mget)
ftp> mls *.html (view the file names before downloading)
Upload files to FTP Server
ftp> mput *.html
Close FTP connection
ftp> close
ftp> bye


AWK Concepts

Files with TAB seperated columns
print 1st column of a file

awk ‘{print $1}’ messages
display two columns with a separator
awk ‘{print $1″ “$2}’ messages
print all the columns from a file, this is same like “cat messages”
awk ‘{print $0}’ messages

For “,” (csv) separated columns use “-F” or “FS” seperators
awk -F “,” ‘{print $1}’ BillingFile_20170103-0919-54.csv
awk ‘{print $1}’ FS=”,” BillingFile_20170103-0919-54.csv

Do no display the first column (Headers in a file)
awk ‘NR!=1{print $1″ “$2}’ FS=”,” BillingFile_20170103-0919-54.csv
Display all rows after row 20
awk ‘NR>20{print $1″ “$2}’ FS=”,” BillingFile_20170103-0919-54.csv
NR>=20 || NR<20 || NR<=20

match column value then filer data (matc column 3 with value 13)
awk ‘$3==’13′{print $0}’ messages | grep “vol006_smb02”
using OR
awk ‘$1==’13′ || $3==’10000′{print $0}’ messages
using AND (&&)

search for a specific string
awk -F “,” ‘/NAS/{print $0}’ BillingFile_20170103-0919-54.csv
excluding specific string
awk -F “,” ‘!/NAS/{print $0}’ BillingFile_20170103-0919-54.csv
awk -F “,” ‘!/au2106/ && !/AU2106/ && NR!=1{print $1}’ BillingFile_20170103-0919-54.csv
display all records where a string matches in a particular column
awk ‘$1~/Fri/{print $0}’ messages
search a string at beginning or end 
awk -F=”,” ‘$1!~/^N/{print $0}’ BillingFile_20170103-0919-54.csv
end: N$
using “if” statement

awk -F “,” ‘{if ($3>=20000) print $0}’ test.csv
awk -F “,” ‘BEGIN{salary=2}{if($3==10000){ $3=$3*2; print $0;}}’ test.csv
awk -F “,” ‘BEGIN{count=0}{if($3==10000) {count++;print $0}}END{print “Total Records: ” count}’ test.csv

Analyze messages file from NetApp Storage Controller

Download multiple messages files from storage controller on unix/linux host’s /tmp
$ cd /tmp
Consolidate multiple file in to one
$ cat $(ls -t) > outputfile.txt

Read a file, use “awk” to select a column in the file, sort the column contents and get unique values
$ cat outputfile.txt | awk ‘{print $6}’| sort | uniq -c | sort -rn
Read a file, use “grep” to search for string then use “awk” to select a column from the searched rows and display unique string with their count of occurrence
$ cat outputfile.txt | grep SNOWY02:wafl.volume.snap.autoDelete:info| awk ‘{print $9}’| sort | uniq -c | sort -rn
Read a file, use “grep” to search for string then use “awk” to select a column from the searched rows and display unique strings with customised output
$ cat SNOWY02_messages.txt | grep SNOWY02:wafl.volume.snap.autoDelete:info| awk ‘{print $9″ was deleted on “$1,$2,$3}’| uniq -c | sort -rn
Display anything within square brackets [ ]
cat snowy_02.txt | sed ‘s/.*\[\([^]]*\)\].*/\1/g’ | sort| uniq -c | sort -rn

Connect your Mac to Storage System’s Serial Console

Search for Serial Port
ls /dev/tty*
If that doesn’t work try this:
ioreg -c IOSerialBSDClient  | grep usb

Use screen from Mac Terminal to connect to storage system’s serial console
screen /dev/cu.usbserial 9600
Capture Screen Terminal Output on mac terminal:
script -a -t 0 out.txt screen /dev/ttyUSB0 115200
•script built in app to “make a typescript of terminal session”
•-a append to output file
•-t 0 time between writing to output file is 0 seconds, so out.txt is updated for every new char
•out.txt is just the output file name
•screen /dev/ttyUSB0 115200 – command from question for connecting to external device
you can then use tail to see that the file is updating
tail -100 out.txt

Activate Web Server on Mac

Mac OS comes installed with Python language. Python as a built in Web server which can be used to upload firmware files to storage controllers.
python -m SimpleHTTPServer 8000

Leave a Reply

Your email address will not be published. Required fields are marked *