Linux commands - reference notes

Whether you are a scientist or a software developer, you should learn the basics of the Linux operating system (OS) as it can often make your life much easier when it comes to writing software. I like Windows and use it daily, but with some things, I find Linux more natural to use.

I enjoy using Linux because of its shell - command-line interpreter. Unfortunately, I am not an eidetiker, and often I have to rely on notes or re-learning to understand what some commands do (primarily when someone else wrote the code). I have some notes scattered all over the system, and I keep running into a hole when I don’t have one reference.

I succumbed to my frustration over my procrastination to put Linux command line notes in one place for almost two years now. Note: I will keep updating this page from time to time.

Note 2: If you want to try Linux, you could start with Ubuntu or CentOS, although latter various people recommend the latter if you’re going to become a Linux Administrator.

Note 3: If you are not comfortable with the command line, then try Hackerrank shell exercises to practice. Happy learning.

Linux directory structure

When I first started working with Linux I was confused where things live, but I found it useful to know the basics of:

1. linux directory structure
2. location of OS components
3. application directory structures

Linux directory structure looks something like this:

/
|--- /bin
|--- /etc
|--- /opt
|--- /tmp
|--- /usr
|---/home
|      |-- kamil
|      |-- magda (my partner doesn't use Linux 😢)
|--- /var

where:

` / - "root" is on the top of file system hierarchy in Linux (like c:\ in Windows ) /bin - folder containing binaries and other executables e.g. compiled commands that you can use in shell /etc - system configuration files - how systems behaves e.g. power settings /home - home directories e.g. user data like Documents, Pictures etc. /opt - optional or third party software - not bundled with OS /tmp - temporary space, mostly cleared on reboot /usr - user related programs (but if you need to know more, just do some soul searching as this is not so straighforward as explained <a href="https://askubuntu.com/questions/130186/what-is-the-rationale-for-the-usr-directory"> here</a>) /var` - variable data, mostly that change often, for example, log files

Some other locations:

/proc - provides information about running processes on the OS
/sbin - system administration binaries
/usr/bin - binaries and other executable programs
/usr/lib - libraries
/usr/local - locally installed software that is not part of the base OS
/usr/sbin - system administration binaries
/var/log - log files

some linux operating sytems also use:

/srv - contains data which is served by the system
/srv/www - web server files
/srv/ftp - ftp files
/sys - used to display and sometimes configure the devices known to the Linux kernel

Some services can have their accounts and can have different home directory, for example:

~ftp

or

/srv/ftp

Begginer scenario: I want to list the contents of my current location, change directory or find out more about commands

Typing ~ will get your account home directory.

To find our where you are use:

pwd

To list contents of your current location:

ls

To change directories:

cd [directory]

cd by itself will change to previous directory that you can inspect with echo $OLDPWD (echo - displays a line of text or environment variable, in this case, to your console)

The best tools to find out more about commands that your are using are:

man [Your Command] 

will display a manual for your command, and:

apropos [keyword] 

will list suggestions for any utilities on the system that might relate to keyword. For example, apropos 'find' will list to all command or files that have ‘find’ keyword in their name or description.

Alternative to the above we can type man -k SEARCH_TERM.

To find out more about given command we can use following options: --help or -h (-h can sometimes stand for human readable so watch out for this)

If you want to navigate to top of man page press g or G to go to the bottom.

When you are learning you might find history useful:

history [<number>]

where is optional, e.g.: `history 15` gets you 15 last commands that you run in shell.

To clear console we use:

clear

And finally to exit current session in console we type:

exit

Simple file introspection and manipulation

To view contents of a file we use:

cat [filename] 

it concatenates and displays contents of filename.

If you want to page over the file contents:

cat [filename] | less

If we want to find out a type of file when we don’t know extension, we use:

file [filename]

To peak into the head

head filename

or the tail of the file use:

tail filename

It is useful to inspect logs by following the file changes with -f option, for example:

tail -f /var/log/filename.log

cp

Copy file from source to destination interactively:

cp -i  source_dir destination

Recursive copy:

cp -r source_dir destination

Bundling collection of files to an archive

General tar command:

tar [-] c|x|t f tarfile [pattern]

that (c) creates, (x) extracts or (t) lists contents of a tar archive using pattern, if supplied.

Other parameters that can be supplied:

v be verbose
z use compression
f file ust this file

How to tar?

Create a tar directory:

tar cf [archive name].tar [filename or directory to tar]

tar xf [archive name].tar

How to use compress

tar zcf [archive name].tgz [filename or directory to tar and compress]

alternatively use [filename].tar.gz naming convention

Application related commands

To find bin directory where programs are stored use, e.g.:

which firefox

To execute commands outside $PATH:

/full/path/to/command

or if you inside /full/path/to/

./command-in-this-dir

[TODO] add more notes

More directory related commands

Building directory, -p if the target directory is going to be in a subfolder that doesn’t exist:

mkdir [-p] directory - create a directory
rmdir [-p] directory - remove a directory
rm -rf directory - recursively removes directory

To list directory files with persmissions and hidden files:

ls -la

(l) long format listing including all files (r) reverse sorted by (t) time including hidden (a) files

ls -latr

(R) recursively

ls -R

long format listing and last modification of (d) directory

ls -ld

(F) reveal file types

ls -F 

example output of the above:

where / indicates that file is a directory @ is a link and * is an executable.

In the context of the above: . means this directory .. means parent directory

Note on symbolic links:

- a link that points to the actual file or directory
- use the link as if it were the file
- a link can be used to create a shortcut
- use for long file directory names
- use to indicate the current version of software

tree

tree can be an useful utility to visualise your folder hierarchy . On Ubuntu you can get it by running:

sudo apt-get install tree

Sample use:

tree . 
tree -d
tree -C

sample output:

File/folder permissions

Say that you list folder content with

ls -lRF

Symbol type explanation

In the above image lets consider README.md file:

-rw-rw-r-- 1 kamilw kamilw    0 Jun 30 13:50 README.md

In the first position we have file type, that is either: - regular file d directory l symbolic link

2nd-4th User permissions
5th-7th Group permissions
8th-10th Other - all users permissions
that are either:

r read w write x execute - if not enabled for user

To display a user’s groups

groups

or

id -Gn

Changing permissions on filename

e.g.

chmod g+w [filename]

Multiple permission changes example (adding user read write and execute, and removing execute to the group):

chmod u+rwx,g-x [filename]

To change filename’s permissions to read for all (-r–r–r–):

chmod a=r [filename]

Octal permissions

Symbolic to octal mapping:

New files belong to your primary group.

To change the group:

chgrp

Common issues with permissions

- permissiosn on a directory can effect the files in the directory
- if the file permission look correct, try checking directory permissions
- then work your way up to the root

File creation mask

File creation mask can be set in your /etc/profile, but if you are more defensive then just append in ~/.bashrc using:

umask [-S] [mode]

• sets the file creation maks to mode, if given
• use -S for symbolic notation

The file creation mask determines default permissions. If no mask were used permissiosn would be:

- 777 for directories
- 666 for files

When you want to give read, write, execute then use chmod 7. umask takes away permission i.e. umask 7 will remove read, write, execute

[TODO] add more explanation

Finding files

Case sensitive:

find [directory] -name "string to find"

Case insensitive:

find [directory] -iname "string to find"

5 day old but less than 8 days in current directory

find . -mtime +5 -mtime -8

Can also apply command to result

find . -mtime +0 -mtime -8 -ls

Find by size

find . -size +1G

[TODO] few more examples

locate command

locate [pattern]

• lists files that match pattern
• faster than the find command
• queries an index
• results are not in real time
• may not be enabled on all systems

Vi

I usually stay with pico or nano to edit config files on the system. There are times when we need to use vi editor. It can be tricky to get into it. Vi comprises of:

• command mode
• insert mode
• line mode

vi command mode and navigation

k up one line j down on line h left one char l right one char w rith one word b left one word ^ go to beginning of a line $ go to end of a line

Vi insert mode

i insert at the cursor position I insert at the beginning of the line a append after the cursor position A append at the end of the line

Vi line mode

:w writes (saves) the file :w! forces file to be saved :q quit :q! quit without save :wq! write and quit :x same as :wq

:n positions the cursor at line n :$ positions the cursor on the last line

:set nu turn on the line numbering :set nonu turn off line numbering

:help [subcommand] get help

Repeat commands by preceding it with number

8k move up a line 8 times 40i insert 40 times 40i_ insert "_" characters

x delete a char dw delete a word dd delete a line D delete from current position

r replace the current char cw change current word cc change the current line c$ change teh text from current position C same as c$ ~ reverses the case of a character

yy yank (copy) the current line y yank the

p paste the most recent deleted or yanked text

u Undo ctr+r redo

/ forward search `?` reverse search

Diskspace usage

du estimates file usage

du -k display sizes in Kilobytes

du -h display sizes in human readable format

Yum

[TODO] some intricacies in CentOS

Handy command to list all yum install commands issued:-

sqlite3 /var/lib/yum/history/history-2018-02-21.sqlite "select tid, cmdline from trans_cmdline"

Attempting to determine more about mysterious “cracklib” binary…

sqlite3 /var/lib/yum/history/history-2016-12-16.sqlite "select tid, cmdline from trans_cmdline" | grep cracklib

rpm -q --whatrequires cracklib

rpm -q --whatrequires cracklib-dicts

Sed - Text Substitition

sed -i ’s/old-string/new-string/g’ /file.txt

Replace Env variables

Replace bash strings in input file i.e. $string1 or ${string2} with env variables

envsubst < input_file > output_file 

List, interrogate, kill system processes

To list current processes:

ps

To list processes by hierarchy:

ps axf 

Find out what the full command behind the process

ps auxwww | grep [pid]

where pid is process id e.g. 1122

To find and kill processes on Linux OS use:

pgrep / pkill

If you want to find process containing keyword fire:

pgrep fire 

it shows pids for fire* but not bash fire*

pgrep -f fire 

shows pids for * fire*.

pkill is similar to the above, but kills found processes, eg: pkill -9 -f batch.sh

if, single square brackets, double square brackets

Single [ Bracket i.e. [] is POSIX shell compliant to to enclose a conditional expression.

Double [[ Brackets i.e. [[]] is an enhanced (or extension) version of standard POSIX version, this is supported by bash and other shells such as zsh (which I am usually using) and ksh.

In bash, for numeric comparison we use eq, ne, lt and gt, with double brackets for comparison we can use ==, !=, <, and > literally.

Examples:

If [ 10 -gt 20 ]

is the same as:

If [[ 10 > 20 ]]

Double parentheses (( )) are used for arithmetic operations, for example:

((a++))

((meaning = 42))

for ((i=0; i<10; i++))

echo $((a + b + (14 * c)))

If (( $numeric > 30 )); then…

Numbers/calculation

expr 50 * 25

Arrays

# initiate an array object
declare -A countries

# populate the array object
countries[uk]='London'
countries[us]='Washington DC'
countries[fr]='Paris'
countries[de]='Berlin'
countries[it]='Rome'

# iterate through the object's contents and output to console
for country in "${!countries[@]}"

do
    echo "key :" $country
    echo "value:" ${countries[$country]}
done

date

date --rfc-3339=seconds | sed 's/ /T/'

BASH commands

Ctrl +r reverse lookup search (bash history)

Ctrl +l clear screen

Ctrl +w cut last word

Ctrl +u cut line

Ctrl +y paste cut word/line

Magical braces

If you have ImageMagick and want to convert jpg to png:

convert [filename].{jpg,png}

Rename file by appending .bak extension:

mv file{,.bak}

{1..5} expands to 1 2 3 4 5

!! to run last command

!!

sudo !!

Use last parameter on previous command:

!$ 

or

$_ 

For example, copy first some files to target:

cp some-files ../path/to/folder

now I want to change to target directory:

cd !$

or

cd $_

Execute command with history line number nnn

!nnn 

Other useful tools

ts - add a timestamp comm - find common lines in two files watch - rerun command every 2 secs file - determine file type of specified file ncdu - file what is using up disk space column - sort into columns pv - pipe viewer - gives stats on what is going through pipe cal - tiny calendar xsel/xclip - system clipboard

jq

If you need to slice and dice json objects use jq. More on here.

Examples:

aws ec2 describe-images --filter Name=owner-id,Values=[AWS ID] Name=tag:Name,Values=PS-Docker-AMI | jq -r .Images[0].ImageId


aws ec2 describe-instances --filters "Name=tag:Name,Values=theo-spark" "Name=instance-state-name,Values=running" | jq -r '.Reservations[0].Instances[0].InstanceId'

Bash script for pushing cherry picked updates to git branches

[TODO] explain

#!/bin/bash

ARRAY=( dev1 dev2 dev3 )

for i in "${ARRAY[@]}"; do

    git checkout $i

    sleep 3

    git cherry-pick staging

    sleep 3

    git push

    sleep 3

done

Bash example for programatically providing input to a process (fdisk)

[TODO] explain

(
    echo n
    echo p
    echo 1
    echo   
    echo   
    echo w

) | fdisk /dev/nvme1n1


mkfs.xfs -f /dev/nvme1n1p1
mkdir -p /mnt/mysql
echo "/dev/nvme1n1 /mnt/mysql xfs  defaults  0   0" >> /etc/fstab
mount -a