diff --git a/lessons/03_searching_files.md b/lessons/03_searching_files.md index 9961792..0dc8ad4 100644 --- a/lessons/03_searching_files.md +++ b/lessons/03_searching_files.md @@ -8,18 +8,18 @@ date: 2015-07-30 ## Learning objectives -* Practice searching for characters or patterns in a text file using the `grep` command -* Learn about output redirection -* Explore how to use the pipe (`|`) character to chain together commands ++ Use grep to search for character strings and surrounding text in a text file (such as a fastq file) ++ Redirect output of grep to another file by overwriting or appending ++ Chain commands together using the pipe (|) character ## Searching files -We showed a little how to search within a file using `less`. We can also +We showed a one way to search within a file using `less`. We can also search within files without even opening them, using `grep`. Grep is a command-line -utility for searching plain-text data sets for lines matching a string or regular expression. +utility for searching plain-text files for lines matching a string or regular expression. Let's give it a try! -Suppose we want to see how many reads in our file have really bad, with 10 consecutive Ns +Suppose we want to see how many reads in a fastq file are poor quality, with 10 consecutive Ns. Let's search for the string NNNNNNNNNN in file `SRR098026.fastq` in the `untrimmed_fastq` folder: ```bash @@ -27,10 +27,9 @@ $ cd ~/dc_sample_data/untrimmed_fastq/ $ grep NNNNNNNNNN SRR098026.fastq ``` -We get back a lot of lines. What is we want to see the whole fastq record for each of these read. -We can use the '-B' argument for grep to return the matched line plus one before (-B 1) and two -lines after (-A 2). Since each record is four lines and the last second is the sequence, this should -give the whole record. +A lot of lines are printed to the shell! What if we want to see the whole fastq record for each of these read? +We can use the arguments '-B' and '-A' for the grep command to return plus one line before (-B1) and two +lines after (-A2) the line that matches. Since each fastq record is four lines long, and because our grep string matches the second line of the record, this command will give the entire record. ```bash $ grep -B1 -A2 NNNNNNNNNN SRR098026.fastq @@ -47,10 +46,9 @@ for example: **Exercise** 1) Search for the sequence GNATNACCACTTCC in SRR098026.fastq. -In addition to finding the sequence, have your search also return -the name of the sequence. +Return the name of the record with the read that contains this sequence. -2) Search for that sequence in both fastq files. +2) Search for the above sequence in both fastq files. **** ## Redirection @@ -74,21 +72,19 @@ from all the files in to another file called 'bad_reads.txt' $ grep -B1 -A2 NNNNNNNNNN SRR098026.fastq > bad_reads.txt ``` -The prompt should sit there a little bit, and then it should look like nothing -happened. But type `ls`. You should have a new file called bad_reads.txt. Take -a look at it and see if it has what you think it should. +The prompt will advance forward one line, and then will pause for the time it requires to run. The shell prompt will then reappear and it will look like nothing happened. But type `ls`. You will now have a new file called bad_reads.txt. Take +a look at it using the 'less' command and see if it has what you think it should. -If we use '>>', it will append to rather than overwrite a file. This can be useful for +If we use '>>', it will append to an existing file (or make a new one if the file doesn't exist) rather than overwrite a file. This can be useful for saving more than one search, for example: ```bash $ grep -B1 -A2 NNNNNNNNNN SRR097977.fastq >> bad_reads.txt ``` -There's one more useful redirection command that we're going to show, and that's -called the pipe command, and it is `|`. It's probably not a key on +There's one more useful redirection command that we're going to introduce. The pipe command is the `|` character. It's probably not a key on your keyboard you use very much. What `|` does is take the output that -scrolling by on the terminal and then can run it through another command. +is printed to the terminal and passes the output to another command. When it was all whizzing by before, we wished we could just slow it down and look at it, like we can with `less`. Well it turns out that we can! We pipe the `grep` command through `less` @@ -97,7 +93,7 @@ the `grep` command through `less` $ grep -B1 -A2 NNNNNNNNNN SRR098026.fastq | less ``` -Now we can use the arrows to scroll up and down and use `q` to get out. +Now we can use the arrows to scroll up and down and use `q` to exit. We can also do something tricky and use the command `wc`. `wc` stands for `word count`. It counts the number of lines or characters. So, we can use @@ -115,19 +111,18 @@ just want the number of lines, we can use the `-l` flag for `lines`. $ grep -B1 -A2 NNNNNNNNNN SRR098026.fastq | wc -l ``` -Redirecting is not super intuitive, but it's really powerful for stringing -together these different commands, so you can do whatever you need to do. +Redirecting is really powerful for stringing +together different commands, so you can do whatever you need to do. -The philosophy behind these command line programs is that none of them +These command line programs don't really do anything all that impressive. BUT when you start chaining them together, you can do some really powerful things really -efficiently. If you want to be proficient at using the shell, you must -learn to become proficient with the pipe and redirection operators: +efficiently. Redirection operators are very userful for proficiency with the shell! `|`, `>`, `>>`. ## Practicing searching and redirection -Finally, let's use the new tools in our kit and a few new ones to example our SRA metadata file. +Finally, let's use the new tools in our kit and a few new ones with our SRA metadata file. ```bash $ cd ../sra_metadata/ @@ -152,7 +147,7 @@ $ head -n 1 SraRunTable.txt BioSample_s InsertSize_l LibraryLayout_s Library_Name_s LoadDate_s MBases_l MBytes_l ReleaseDate_s Run_s SRA_Sample_s Sample_Name_s Assay_Type_s AssemblyName_s BioProject_s Center_Name_s Consent_s Organism_Platform_s SRA_Study_s g1k_analysis_group_s g1k_pop_code_s source_s strain_s -That's only the first line but it is a lot to take in. 'cut' is a program that will extract columns in tab-delimited +That's only the first line but it is a lot to take in. 'cut' is a program that will extract columns in tab-delimited files. It is a very good command to know. Lets look at just the first four columns in the header using the '|' redirect and 'cut'