linux and bash pt 3

1 files changed, 557 insertions, 0 deletions

diff --git a/content/weblog/2019-01-11_intro-to-linux-and-bash-pt3/index.md b/content/weblog/2019-01-11_intro-to-linux-and-bash-pt3/index.md
new file mode 100644
index 0000000..eaf2757
--- /dev/null
+++ b/content/weblog/2019-01-11_intro-to-linux-and-bash-pt3/index.md
@@ -0,0 +1,557 @@
++++
+title = "Intro to Linux and the Bash command line, pt III"
+date = 2019-01-11T08:23:00Z
++++
+
+New year, new post. In this third, and most probably final part of these
+tutorial/guide series I will be mentioning some useful commands and programs
+usually present in most standard linux installations. I will be talking
+especially about programs/commands to manipulate text output from programs and
+files. I will also talk a little bit about regular expressions, a powerful tool
+to perform searches inside text strings.
+
+<!-- more -->
+
+## Filters
+
+These programs that perform operations on input text and then write them to
+standard output are commonly known as filters. You may already be familiar with
+one of these commands, which the first one that I'm going to talk about.
+
+### cat
+
+This command allows you the see the content of a text file (or files). It
+stands for con*cat*enate, and not for the house pet. The most basic use of this
+command is to view the contents of a text file, just by typing cat followed by
+the path of the file we wish to see. However, as it name implies, it also has
+the ability to concatenate the contents of multiple text files, for example
+text file 'sample.txt'
+
+```sh
+user@host:~/Documents/notes$ cat sample.txt
+Pepe    cool
+Tide Pods   lame
+Uganda Knuckles cool
+Thanos  cool
+JPEG    ok
+Despacito   lame
+Bowsette    cool
+Harold  cool
+Sans    coolest
+Minions lamest
+NPC cool
+```
+
+And we want to concatenate its content with file 'sample2.txt' to standard
+output
+
+```sh
+user@host:~/Documents/notes$ cat sample.txt sample2.txt
+Pepe    cool
+Tide Pods   lame
+Uganda Knuckles cool
+Thanos  cool
+JPEG    ok
+Despacito   lame
+Bowsette    cool
+Harold  cool
+Sans    coolest
+Minions lamest
+NPC cool
+Troll Face  old
+Can haz chezburger  really old
+ROFLcopter  super old
+Dancing baby    ancient
+```
+
+As usual, this command accepts different options, like for example the -n
+option to display line
+
+```sh
+user@host:~/Documents/notes$ cat -n sample.txt
+     1  Pepe    cool
+     2  Tide Pods   lame
+     3  Uganda Knuckles cool
+     4  Thanos  cool
+     5  JPEG    ok
+     6  Despacito   lame
+     7  Bowsette    cool
+     8  Harold  cool
+     9  Sans    coolest
+    10  Minions lamest
+    11  NPC cool
+```
+
+As always, you can check other options by looking up cat and any other command
+with man, as explained in the previous part.
+
+### head
+
+This is a really simple command, it shows the first n lines of a text
+file/output. To use it, type head, followed by -n, then the number of lines to
+show, and then the file. For example, let's say we want to see the first 5
+lines of sample.txt
+
+```sh
+user@host:~/Documents/notes$ head -n 5 sample.txt
+Pepe    cool
+Tide Pods   lame
+Uganda Knuckles cool
+Thanos  cool
+JPEG    ok
+```
+
+If we use the command without passing the number of lines we wish to see, it
+outputs the first 10 lines by default. For this just type head followed by the
+path of the file.
+
+### tail
+
+Basically the same as head, except it shows the last lines. Let's say we want
+to see the last three lines
+
+```sh
+user@host:~/Documents/notes$ tail -n 3 sample.txt
+Sans    coolest
+Minions lamest
+NPC cool
+```
+
+As with head, the default is to output 10 lines.
+
+### sort
+
+This command is as obvious as it seems. It sorts output. For example
+
+```sh
+user@host:~/Documents/notes$ sort sample.txt
+Bowsette    cool
+Despacito   lame
+Harold  cool
+JPEG    ok
+Minions lamest
+NPC cool
+Pepe    cool
+Sans    coolest
+Thanos  cool
+Tide Pods   lame
+Uganda Knuckles cool
+```
+
+### sed
+
+This one is a really powerful utility to transform and manipulate text,
+however, to keep this tutorial short, I will only be showing a couple of the
+most used cases. sed stands for "stream editor".
+
+The way to use sed, is to pass it a kind of script (a sed script) that tells it
+what to do with the text. The first and one of the most basic uses of sed, is
+to basically perform the same task as head, to get the first n number of lines.
+For example, let's say we want the first 7 lines of sample.txt
+
+```sh
+user@host:~/Documents/notes$ sed '7q' sample.txt
+Pepe    cool
+Tide Pods   lame
+Uganda Knuckles cool
+Thanos  cool
+JPEG    ok
+Despacito   lame
+Bowsette    cool
+```
+
+Of course what I've just told it you it does is a simplification of what it
+really does. Most accurately, the command or script that we passed to sed tells
+it to output the first seven lines, and the q tells it to stop after that.
+
+Another basic use of sed, and arguably the most common one, is to perform
+search and replace operations on text. The basic syntax for this operations is
+'s/<search>/<replace>/' where <search> is the term you want to search for, and
+<replace> is the term you wish to replace it with.
+
+By default it will replace only the first occurrence in each line, however, we
+can specify which or how many occurrences we want to replace by adding a number
+and/or letter to the end. For example, if we add a two
+('s/<search>/<replace>/2') it will replace only second occurrences of each
+line.
+
+But what if we want to replace each and every occurrence in all of the text?
+For that we would use the letter g at the end. Let's say for example, that we
+want to replace all occurrences of "cool" in our sample.txt file, for "dank".
+In this case we would type something like this
+
+```sh
+user@host:~/Documents/notes$ sed 's/cool/dank/g' sample.txt
+Pepe    dank
+Tide Pods   lame
+Uganda Knuckles dank
+Thanos  dank
+JPEG    ok
+Despacito   lame
+Bowsette    dank
+Harold  dank
+Sans    dankest
+Minions lamest
+NPC dank
+```
+
+A thing to keep in mind, is that you should be enclosing the sed script in
+single quotes. Of course these are only some of the most basic uses of this
+command.
+
+### grep
+
+This is the last program to manipulate text output that I want to mention. I
+will demonstrate its basic use in this section, but I will show you a little
+bit more about it in the next section when I will be writing about regular
+expressions.
+
+Back to grep, it is a program that basically searches a pattern that you give
+it, and it will print to you the lines that contain that pattern. For example,
+let's say that we want to see only the cool (or dank) memes in our file to be
+displayed
+
+```sh
+user@host:~/Documents/notes$ grep 'cool' sample.txt
+Pepe    cool
+Uganda Knuckles cool
+Thanos  coolcharacter
+Bowsette    cool
+Harold  cool
+Sans    coolest
+NPC cool
+```
+
+This line of text that we passed it, is actually the most basic form of regular
+expression, of which we will be looking into detail next.
+
+## Regular expressions
+
+A regular expression, or regex for short, is a string of text, that define a
+search pattern for a larger set of text. Regexes are used in many programs,
+such as in text editors, and search engines, and can be also of great use in
+the terminal
+
+### An intermission
+
+Before going into actual regular expressions in grep, I want to mention a
+couple of characters that can make your life easier when dealing with files in
+the terminal. They are called wildcards, and they are the asterisk (*) and the
+question mark (?). If you've ever wondered why you can't use those characters
+in any of your files' names, that's why.
+
+I'll start by explaining the asterisk. When you use the asterisk, you are
+asking to look at or take all files that contain the any number of any
+combination of symbols in the place where you put it. For example, we could be
+looking at files that start with sa
+
+```sh
+user@host:~/Documents/notes$ ls sa*
+saturday.txt sample.txt sample2.txt sample.png
+```
+
+Or another example, we could be looking for files that just contain sa in their
+name
+
+```sh
+user@host:~/Documents/notes$ ls *sa*
+asado.png saturday.txt sample.txt sample2.txt sample.png
+```
+
+Now the question mark. The question mark indicates that there should be a
+character in its place, just any character. Let's say that we want to see all
+files with name "sample" that have a three character extension
+
+```sh
+user@host:~/Documents/notes$ ls sample.???
+sample.txt sample.png
+```
+
+Wildcards come really handy when you need to manipulate multiple files with
+similar names. If the files that you wish to manipulate don't really have
+similar names, you might want to use curly braces to indicate a list of files
+to manipulate, separated by commas. For example
+
+```sh
+user@host:~/Documents/notes$ rm {monday.txt,december1999.txt,saturday.txt}
+```
+
+### Back to regex
+
+Now I'll explain some things about regular expressions, and I'll demonstrate
+some basic uses with grep. Here are some basic concepts
+
+* `.` - The dot means a single character (any character). e.g. 'be.r' would match
+  bear, beer, befr, etc.
+* `*` - The preceding element matches 0 or more times. e.g. 'an*t' would match
+  at, ant, annt, annnt, etc.
+* `+` - The preceding element matches one or more times. e.g. 'an+t' would match
+  ant, annt, annnt, etc.
+* `?` - The preceding element matches 0 or one time. e.g. 'an?t' would match at,
+  and ant.
+* `{n}` - The preceding element matches exactly n times.
+* `{min, }` - The preceding element matches at least min times.
+* `{min, max}` - The preceding element matches at least min times, and no more
+  than max times.
+* `|` - The pipe, logical OR operator. e.g. 'gray|grey' would match gray and grey
+* `()` - The parenthesis group multiple characters as one element. e.g.
+  'gr(a|e)y' would match gray and grey.
+* `[abc]` - It matches if a character is one of those inside the brackets.
+* `[^abc]` - It matches if none of the characters is one of those inside the
+  brackets.
+* `[a-d]` - A range of characters. i.e. a, b, c, or d.
+* `^` - Matches the beginning of the line.
+* `$` - Matches the end of the line.
+
+So now let's suppose for a practical example with grep, that we want to find
+all lines that have "cool" or "ok" in them. In this case we would use the "|"
+pipe symbol. However, if we use normal grep, we would have to escape the pipe
+symbol like this "\|". That's why it is better that we use "grep -E" to enable
+extended regex, or its shorter alias "egrep". It would look something like this
+
+```sh
+user@host:~/Documents/notes$ egrep 'cool|ok' sample.txt
+Pepe    cool
+Uganda Knuckles cool
+Thanos  cool
+JPEG    ok
+Bowsette    cool
+Harold  cool
+Sans    coolest
+NPC cool
+```
+
+Let's suppose, for another example, that we want to match those lines with a 't' as the last character
+
+```sh
+user@host:~/Documents/notes$ egrep 't$' sample.txt
+Sans    coolest
+Minions lamest
+```
+
+I have already mentioned and shown you the use of regexes with grep (and/or
+egrep). Now I would like to show a more practical example with sed. Yes, sed
+uses its own script language to alter text input, however, it also makes use of
+regular expressions.
+
+Let's suppose that we have a file that looks like this
+
+```sh
+user@host:~/Documents/notes$ cat shortcuts
+# Some shortcuts
+
+d       ~/Documents
+D       ~/Downloads
+m       ~/Music
+pp      ~/Pictures
+vv      ~/Videos
+
+
+s       ~/.scripts # My scripts
+cf      ~/.config # My configs
+```
+
+As we can see there is a lot of whitespace, and although comments might be of
+help to humans, they are of no use to machine. Let's begin by getting rid of
+the comments, for that first need to remember the search and replace command of
+sed, 's/<search>/replace/g', since we basically want to get rid of any
+comment-looking string and replace it with, well, nothing. Now we have to think
+of a regex that will match comments, for that '#.*' will do. What regex means
+is, match '#' and everything after it. Now let's put it together, and
+
+```sh
+user@host:~/Documents/notes$ sed 's/#.*//g' shortcuts
+
+
+d       ~/Documents
+D       ~/Downloads
+m       ~/Music
+pp      ~/Pictures
+vv      ~/Videos
+
+
+s       ~/.scripts
+cf      ~/.config
+```
+
+Bam, there it is. However, we still have the blank lines left, and, if you pay
+close attention, the comments have been deleted, but, the spaces that used to
+be before some of the comments are still there.
+
+So first, let's improve our current sed command, if we want to match 0 or more
+spaces (zero because not every comment has a space before it) we would use the
+`*` symbol, but what symbol would we use for spaces? Well, that's an easy one, in
+sed we escape spaces like this '\s', so now our sed command looks like this
+'s/\s*#.*//g'.
+
+Let's take care of the last part, getting rid of blank lines. For this we would
+need to issue a separate command, but fortunately we can stack commands in one
+line with a semicolon (;). Now that we know that we need a way to match empty
+lines with a regex, that's very easy - '^$' just match the beginning and the
+end of line together, after that, we add a sed command for deleting lines which
+I haven't mentioned (d), and our one liner is ready...
+
+```sh
+user@host:~/Documents/notes$ sed 's/\s*#.*//g; /^$/d' shortcuts
+d       ~/Documents
+D       ~/Downloads
+m       ~/Music
+pp      ~/Pictures
+vv      ~/Videos
+s       ~/.scripts
+cf      ~/.config
+```
+
+Of course, issuing this command will not replace the original file, it will
+simply output the result to the terminal screen. If you want to overwrite the
+original file with the result of the sed command, you can pass sed the '-i'
+option.
+
+## Piping and redirecting output
+
+This post is already getting too long, however there's one more useful thing
+about *nix systems that I'd like to mention - the pipeline. The pipeline in
+Unix and Unix-like OSs is a chain of redirected output to the input of another
+program. Along with that, there are operators to redirect standard output to
+files (and viceversa).
+
+### Redirecting to and from files
+
+Let's suppose that we want to repeat the last example, and want to clean the
+file of comments and blank lines. We already now how to overwrite that file,
+however, what if we want to save it to another file using common Unix operators
+in bash. For that we can use the '>' and '>>' operators. For example, let's we
+want to save the result to a second file called "shortcuts_clean"
+
+```sh
+user@host:~/Documents/notes$ sed 's/\s*#.*//g; /^$/d' shortcuts > shortcuts_clean
+```
+
+Since there was no "shortcuts_clean" file, it has been created automatically.
+However, if the file had already existed, it would have overwritten it, unless
+we had used the '>>' operator, in that case, it would have appended the output
+to the already existent file.
+
+Just as there's '>' to redirect TO files, there's also the '<' to redirect from
+files to a program's standard input. However, must of the times you would just
+pass the name/path of the file to the program as an argument.
+
+### Piping
+
+Now that we know how to redirect from and to files, we can learn how to
+redirect from one program to another, with pipes. The pipe operator in *nix
+systems is the vertical bar symbol (|). Let's suppose that we want to see the
+first three files in our current directory, for that, we can pipe the output of
+ls into head, like this
+
+```sh
+user@host:~/Documents/notes$ ls | head -n 3
+asado.png
+monday.txt
+sample.txt
+```
+
+Now let's get back to our sample.txt file. Let's imagine that we first want to
+sort our lines, and we want to preserve only those lines that contain "cool" or
+"lame". Then let's suppose we want to modify to contain legit terms, and not
+some antiquated boomer slang, so we want to replace cool with dank, and lame
+with normie. Finally we want that to be output to a file instead of the screen.
+Whew! Sounds like a lot of stuff to do, but it is quite simple, and it looks
+like this
+
+```sh
+user@host:~/Documents/notes$ egrep 'cool|lame' sample.txt | sort | sed 's/cool/dank/g;s/lame/normie/g' > memes.txt
+```
+
+So if we now take a look at the file...
+
+```sh
+user@host:~/Documents/notes$ cat memes.txt
+Bowsette    dank
+Despacito   normie
+Harold  dank
+Minions normiest
+NPC dank
+Pepe    dank
+Sans    dankest
+Thanos  dank
+Tide Pods   normie
+Uganda Knuckles dank
+```
+
+And that's basically it.
+
+## Post scriptum
+
+Before ending it for good, I want to show some other programs that might be of
+use in the Bash command line.
+
+### less
+
+This command might come in handy when there's another command that outputs a
+lot of text that overfills the terminal screen. You can pipe (as we have just
+learned) the output of that command to less, so that you can navigate with your
+arrow keys, or better yet with vim keys (hjkl). You can also search for terms
+by typing slash (/), just like with man.
+
+### tar
+
+This program is used in Linux to create and extract archives with the .tar
+format, usually also compressing them using gunzip (.gz).
+
+There usually two ways you will be using the program. One to extract files from
+a compressed archive
+
+```sh
+user@host:~/Documents/notes$ tar -xzvf oldnotes.tar.gz
+```
+
+And the other to archive and compress files
+
+```sh
+user@host:~/Documents/notes$ tar -czvf allnotes.tar.gz *
+```
+
+To learn more about the different options of this program, I recommend you
+check the man pages of tar ('man tar').
+
+### ssh and scp
+
+You may have already heard about ssh, which stands for "secure shell", even if
+you are new to Linux or Unix/Unix-like systems. This program is used to connect
+to other computers over a network (or the internet for instance), especially to
+servers.
+
+Let's suppose that you have a server with ip address 180.80.8.20 and your user
+is tux
+
+```sh
+user@host:~$ ssh tux@180.80.8.20
+```
+
+Of course, here we have assumed that the standard ssh port (22) is being used,
+otherwise you will have to specify it by passing -p followed by the port
+number.
+
+Now let's talk about scp, which stands for "secure copy". This command uses the
+same protocol as ssh, and it's used to copy files from one computer to another
+over a network. Let's suppose that you want to copy a file from your current
+computer to the server we used in the previous example
+
+```sh
+user@host:~$ scp somefile tux@180.80.8.20:/home/tux/directory/
+```
+
+If we were trying to do it the other way around, that is, from the remote
+computer to your local computer, it would look like this
+
+```sh
+user@host:~$ scp tux@180.80.8.20:/home/tux/directory/somefile directory/
+```
+
+Just as with ssh, if you are not using standard port 22, you need to say to the
+program to which port you are trying to connect, except that in the case of
+scp, the flag is '-P' instead of '-p', and goes right after "scp".
+
+Well, that's it for this tutorial/guide series, I really hope it was of use to
+you.