Week3.md

Week 3 Notes

Combining Commands and Files

• Executing Multiple Commands
  • command1; command2; command3;
    • Each command will be executed one after the other.
  • command1 && command2
    • command2 will be executed only if command 1 succeeds
    • If the return code is 0 it is true and if it is greater than 0 it is false
    • ls && date -Q && wc -l /etc/profile will display the dir listing followed by error that -Q is invalid; wc is not executed.
  • command1 || command2
    • command2 will not be executed if command1 succeeds
    • ls /blah || date will display current date after "No such file or directory"
    • ls || date will display just the directory listing
    • command2 is like a Plan B if command1 doesn't succeed.
  • Example ls /blah ; date ; wc -l /etc/profile ;
  • If we use parenthesis ie (ls /blah ; date ; wc -l /etc/profile ;) the command gets executed in a subshell and is returned back to the shell we are using.
  • We can use echo $BASH_SUBSHELL to return an integer which tells us at what level of execution we are.
    • (echo $BASH_SUBSHELL) will report a value of 1
    • (ls; (date; echo $BASH_SUBSHELL)) will report a value of 2
  • Launching too many subshells could be expensive computationally.
• File Descriptors
  • Every command in linux has 3 file descriptors - stdin (0) , stdout (1), stderr (2).
    • stdin is a pointer to a stream that is coming from the keyboard or use input
    • stdout or stderr usually points to the screen where the display or output is made.
    • the three pointers are looking at only the stream of characters.
    • they can be directed to a file or a command, or the default behaviour can be left as it is.
  • Combining a command and a file
    • command > file1
      • stdout is redirected to file1
      • file1 will be created if it does not exist
      • if file1 exists, its contents will be overwritten
      • example : ls -1 /usr/bin > file1 - displays no output on the screen because there is no error
      • ls -1 /blah > file1 - displays an error. file1 is overwritten and is now 0 Bytes.
      • hwinfo > hwinfo.txt
      • trying this command in a folder where there is no w permissions will generate an error
      • The cat command tries to read from the provided file name if not given it tries to read from stdin (keyboard)
        • cat > file1 will allow you to type content. The feature could be used to create text files on the command line. You can come out using the Ctrl+D option.
        • cat file1 displays the content of file1
        • cat takes input from the keyboard and displays it on the screen (line by line; when you press enter) - Finish by pressing Ctrl+D to signify end of file.
    • command >> file1
      • contents will be appended to file1
      • new file1 will be created if it does not exist.
      • Example : date >> file2 ; wc -l /etc/profile >> file2 ; file /usr/bin/znew >> file2 ;
      • cat >> file1 to append text to a file from command line. Come out using Ctrl + D

Redirections

• combining command and file (continued ..)
  • (contd..)
    • command 2> file1
      • redirects stderr to file1
      • file1, if it exists, will be overwritten.
      • file1 will be created if it does not exist.
      • Example ls $HOME /blah 2> error.txt
    • command > file1 2> file2
      • stdout is redirected to file1
      • stderr is redirected to file2
      • Contents of file1 and file2 will be overwritten.
      • The output is in one file and the errors are in another file.
      • Example : ls $HOME /blah > output.txt 2> error.txt
      • ls -R /etc > output.txt 2> error.txt - permission related errors in error.txt
    • command < file1
      • stdin is redirected - a command expecting input from the keyboard could take the input from a file.
      • Example : wc /etc/profile behaves similar to wc < /etc/profile
    • command > file1 2>&1
      • command output will be redirected to file1
      • 2> indicates stderr and that is being redirected to &1 (first stream) which is stdout
      • contents of file1 will be overwritten
      • Example : ls $ HOME /blah > file1 output alone is sent to file1. Error on screen
      • Example : ls $ HOME /blah > file1 2>&1 output and error is sent to file1.
    • command1 | command2 Pipe
      • stdout output of command 1 is sent to stdin of command2 as input
      • Example ls /usr/bin | wc -l
    • command1 | command2 > file1
      • command1 and command2 are combined and the stdout of command2 is sent to file1. Errors are still shown on the screen.
      • Example ls /usr/bin | wc -l > file1 - file1 has the number of lines counted by wc
    • command > file1 2> /dev/null
      • /dev/null file - A sink for output to be discarded. Like a "black hole"
      • We normally don't do anything with the /dev folder as there are sensitive system files there.
      • If you are confident that the script is running well and you do not want to display any error on the screen, you can redirect the stderr to /dev/null
      • stderr is redirected to /dev/null
      • Example : ls $HOME /blah > file1 2> /dev/null
      • Example : ls -R /etc > file1 2> /dev/null - file1 contains the output except errors
    • command1 | tee file1
      • Used in sitiations where you want to have a copy of the output in a file as well as on the screen.
      • The tee command reads from stdin and writes to stdout and file/s.
      • Example : ls $HOME | tee file1 also ls $HOME | tee file1 file2 for creating multiple copies
      • diff file1 file2 comapares files line by line
        • no output if the files are identical
      • Example : ls $HOME /blah | tee file1 file2 | wc -l - Here tee keeps copy of output in a file and also sends output to wc -l for further processing.
      • Example : ls $HOME /blah 2> /dev/null | tee file1 file2 | wc -l to supress errors. Note location of 2> is since the error is generated there.

Shell Variables - Part 1

• Creation, inspection, modification, lists
• Creating a variable
  • myvar="value string"
    • myvar can't start with a number, but you can mix alphanumeric and _
    • No space around the =
    • "value string" is the number,string or command. Output of a command can be assigned to myvar by enclosing the command in back-ticks.
• Exporting a variable
  • export myvar="value string" or
  • myvar="value string" ; export myvar
  • This makes the value of the variable available to a shell that is spawned by the current shell.
• Using variable values
  • echo $myvar
  • echo ${myvar}
    • can manipulate the value of the variable by inserting some commands within the braces.
  • echo "${myvar}_something"
• Removing a variable
  • unset myvar
  • Removing value of a variable myvar=
• Test is a variable is set
  • [[ -v myvar ]] ; echo $?
    • 0 : success (variable myvar is set)
    • 1 : failure (variable myvar is not set)
  • [[ -z ${myvar+x} ]] ; echo $? (the x can be any string)
    • 0 : success (variable myvar is not set)
    • 1 : failure (variable myvar is set)
• Substitute default value
  • If the variable myvar is not set, use "default" as its default value
  • echo ${myvar:-"default"}
    • if myvar is set display its value
    • else display "default"
• Set default value
  • If the variable myvar is not set then set "default" as its value
  • echo ${myvar:="default"}
    • if myvar is set display its value
    • else set "default" as its value and display its new value
• Reset value if variable is set
  • If the variable myvar is set, then set "default" as its value
  • echo ${myvar:+"default"}
    • if myvar is set, then set "default" as its value and display the new value
    • else display null
• List of variable names
  • echo ${!H*}
    • displays the list of names of shell variables that start with H
• Length of string value
  • echo ${#myvar}
    • Display length of the string value of the variable myvar
    • if myvar is not set then display 0
• Slice of a string value
  • echo ${myvar:5:4} (5 is the offset and 4 is the slice length)
    • Display 4 characters of the string value of the variable myvar after skipping first 5 characters.
  • if the slice length is larget than the length of the string then only what is available in the string will be displayed.
  • the offset can also be negative. However you need to provide a space after the : to avoid confusion with the earlier usage of the :- symbol. The offset would come from the right hand side of the string.
• Remove matching pattern
  • echo ${myvar#pattern} - matches once
  • echo ${myvar##pattern} - matches maximum possible
  • Whatever is matching the pattern will be removed and the rest of it will be displayed on the screen.
• Keep matching pattern
  • echo ${myvar%pattern} - matches once
  • echo ${myvar%%pattern} - matches maximum possible
• Replace matching pattern
  • echo ${myvar/pattern/string} - match once and replace with string
  • echo ${myvar//pattern/string} - match max possible and replace with string
• Replace matching pattern by location
  • echo ${myvar/#pattern/string} - match at begining and replace with string
  • echo ${myvar/%pattern/string} - match at the end and replace with string
• Changing case
  • echo ${myvar,} - Change the first character to lower case.
  • echo ${myvar,,} - Change all characters to lower case.
  • echo ${myvar^} - Change first character to uppercase
  • echo ${myvar^^} - Change all characters to upper case
  • The original value of the variable is not changed. Only the display will be modified as the trigger commands are within braces.
• Restricting value types
  • declare -i myvar - only integers assigned
  • declare -l myvar - Only lower case chars assigned
  • declare -u myvar - Only upper case chars assigned
  • declare -r myvar - Variable is read only
  • Once a variable is set as read only you may have to restart the bash to be able to set it
• Removing restrictions
  • declare +i myvar - integer restriction removed
  • declare +l myvar - lower case chars restriction removed
  • declare +u myvar - upper case chars restriction removed
  • declare +r myvar - Can't do once it is read-only
• Indexed arrays
  • declare -a arr
    • Declare arr as an indexed array
  • $arr[0]=”value”
    • Set value of element with index 0 in the array
  • echo ${arr[0]}
    • Value of element with index 0 in the array
  • echo ${#arr[@]}
    • Number of elements in the array. The @ symbol is a wild character to run through all the elements in the array
  • echo ${!arr[@]}
    • Display all indices used
  • echo ${arr[@]}
    • Display values of all elements of the array
  • unset ‘arr[2]’
    • Delete element with index 2 in the array
  • arr+=(“value”)
    • Append an element with a value to the end of the array
• Associative arrays
  • declare -A hash
    • declare hash as an associative array
  • $hash[“a”]=”value”
    • set the value of element with index a in the array
  • echo ${hash[“a”]}
    • value of element with index a in the array
  • echo ${#hash[@]}
    • number of elements in the array
  • echo ${!hash[@]}
    • display all indices used
  • echo ${hash[@]}
    • display values of all elements of the array
  • unset ‘hash[“a”]’
    • delete an element with index a in the array
  • Can do everything in the indexed array except append because there is nothing called the end of the array as there is no sequence for the elements of a hash
• Examples
  • true always returns exit code 0
  • false always returns exit code 1 (Check with echo $?)
  • To check whether a variable is present
    • [[ -v myvar ]] ; echo $? returns 1 if the variable is not present in the memory
    • [[ -z ${myvar+x} ]] ; echo $? returns 0 if variable is not present and 1 if it is present. x is a string that will be used as a replacement if the variable was not present.
  • Use of Braces
    • myvar=FileName
    • echo $myvar
    • echo "$myvar.txt" prints FileName.txt
    • echo "$myvar_txt" does not print anything as the variable myvar_txt does not exist
    • echo "${myvar}_txt" prints Filename_txt
    • Braces are useful in stating clearly the name of the variable.
    • Can also be used outside quotes echo ${myvar}
  • Does the variable we have created get passed on to the shell or any other program created within the shell
    • myvar=3.14 ; echo $myvar
    • bash one more level of bash
    • ps --forest to show that we are one level below
    • echo $myvar not present
    • Use export myvar=3.14 to ensure this variable is available to all spawned sub shells.
    • Change value of variable within the child shell
    • modification of value is not reflected in the value of the variable in the parent shell
    • even if you do export of the variable within the child shell it will not change the value within the parent shell.
  • Use of back-ticks
    • mydate=`date` value of mydate will be output of date.
    • mydate=`echo Sunday that is today` ; echo $mydate
  • Manipulations for variables within the shell environment
    • We would like to have echo display a default value if variable is not available
    • echo ${myvar:-hello} the - indicates if the value is not present what is the display value
    • echo ${myvar:-"myvar is not set"}
    • Set the value if it was not set already
    • echo ${myvar:=hello} if absent / not set then set it to the value after =
    • If it is present it will not change
    • echo ${myvar:?"myvar is not set"} displays a little more information and a debug message. bash: myvar: myvar is not set
    • Unset the value of a variable using unset myvar
    • echo ${myvar:+HELLO} displays the message if the variable is present
  • Inspecting all the variables in the shell environment
    • printenv
    • env
    • echo ${!H*} displays the names of variables begining with 'H' - ! indicates names of the variables instead of value.
  • Counting characters
    • mydate=`date` stores the output of the date command in mydate
    • echo ${#mydate} prints the length of the value present in mydate.
    • length of a non-existing variable is zero
  • Features of using colon : within braces {}
    • Extracting part of a string from the value of a particular variable.
      • echo ${mydate:6:10}
      • echo ${myvar:3:3} will print def for myvar=abcdefg ie: 3 characters after the offset (position 3)
    • Using negative offset
      • echo ${myvar: -3:3} and echo ${myvar: -3:4} will print efg for myvar=abcdefg
        • note - is to be preceeded with a blank to avoid confusion
        • asking for more characters, will print just what is available
      • echo ${myvar: -3:2} will print ef for myvar=abcdefg
    • Extracting a portion of the date
      • Output of date is Tuesday 25 January 2022 09:10:20 PM IST
      • Output of date +"%d %B %Y" is 25 January 2022
      • if mydate=`date` then echo ${mydate:8:16} will also print 25 January 2022
    • Extracting patterns from a string
      • myvar=filename.txt.jpg
      • echo ${myvar#*.} minimal matching displays txt.jpg
      • myvar=filename.somethingelse.jpeg
      • echo ${myvar##*.} maximal matching displays jpeg
      • echo ${myvar%*.} displays filename.somethingelse
        • the % is used to indicate what has not been matched. (minimal)
      • echo ${myvar%%*.} displays filename
        • the % is used to indicate what has not been matched. (maximal)
      • Can be combined echo ${myvar%%.*}.${myvar##*.} to get filename.jpeg
    • Replacing what has been matched
    • Pattern matching in Linux usually goes with a pair of forward slashes.
    • Convert all e to E in a string
      • echo ${myvar/e/E} replaces only the first occurance of e
      • echo ${myvar//e/E} replaces all occurances of e
    • Replace characters at the begining of a string
      • echo ${myvar/#f/F} replaces the occurance of f in the begining of the string with F. The # indicates the begining of the string
    • Replace characters at the end of a string
      • echo ${myvar/%g/G} replaces the occurance of g at the end of the string with G. The % indicates the end of the string.
    • Replace jpeg with jpg, only if it is at the end of a string
      • echo ${myvar/%jpeg/jpg}
    • Modifying and storing it in a variable
      • myvar1=`echo ${myvar//jpeg/jpg}`
    • Generic command to remove day from date
      • echo ${mydate#*day}
    • Upper case to lower case and vice-versa
      • echo ${mydate,} changes first character to lowercase
      • echo ${mydate,,} converts all characters to lowercase
      • echo ${mydate^} changes first character to uppercase
      • echo ${mydate^^} changes all characters to uppercase
    • Restricting values that can be assigned to shell variables using declare
      • declare is a shell builtin
      • + to unset a restriction and - to set it (Note : counterintutuve )
      • -a for indexed arrays (need not be ordered indexes)
      • -A for associative arrays (dictionaries)
      • -i for integers
      • -u for uppercase conversion on assignment
      • Integer restriction
        • declare -i mynum
        • mynum=10 will assign 10 to mynum
        • mynum=hello will assign 0 to mynum
      • lowercase restriciton
        • declare -l myvar
        • myvar=hello assigns hello to myvar
        • myvar=BELLOW converts BELLOW to lowercase and assigns it to myvar.
      • removing a restriction
        • declare +l myvar
        • the value is still contained after removing the restriction but you can now store upper case characters as well
      • declaring a read-only variable
        • declare -r myvar
        • once a variable has been set as read only, you cannot change its value and you cannot remove the read-only restriction using +r
        • declare +r myvar gives the error bash: declare: myvar: readonly variable
    • Arrays
      • declare -a arr
      • arr[0]=Sunday
      • arr[1]=Monday
      • echo ${arr[0]}
      • echo ${arr[1]}
      • echo ${#arr[@]} gives number of elements in the array
      • echo ${arr[@]} displays all values
      • echo ${!arr[@]} displays the indices`
      • You can have any index without filling up intermediate indices. Indices are not necessarily contiguous.
      • arr[100]=Friday is also valid
      • Removing an element from an array = unset 'arr[100]'
      • Appending to an array arr+=(Tuesday)
      • Populating an array in one go arr=(Sunday Monday Tuesday) . The indices are sequential
    • Associative Arrays / Hashes
      • declare -A hash
      • hash[0]="Amal"
      • hash["mm12b001"]="Charlie"
      • echo ${!hash[@]} to get indices
      • echo ${hash["mm12b001"]}
    • File names in a shell variable
      • myfiles=(`ls`)
      • echo ${myfiles[@]}