Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 23 additions & 28 deletions lessons/03_searching_files.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,29 +8,28 @@ date: 2015-07-30

## Learning objectives

* Practice searching for characters or patterns in a text file using the `grep` command
* Learn about output redirection
* Explore how to use the pipe (`|`) character to chain together commands
+ Use grep to search for character strings and surrounding text in a text file (such as a fastq file)
+ Redirect output of grep to another file by overwriting or appending
+ Chain commands together using the pipe (|) character

## Searching files

We showed a little how to search within a file using `less`. We can also
We showed a one way to search within a file using `less`. We can also
search within files without even opening them, using `grep`. Grep is a command-line
utility for searching plain-text data sets for lines matching a string or regular expression.
utility for searching plain-text files for lines matching a string or regular expression.
Let's give it a try!

Suppose we want to see how many reads in our file have really bad, with 10 consecutive Ns
Suppose we want to see how many reads in a fastq file are poor quality, with 10 consecutive Ns.
Let's search for the string NNNNNNNNNN in file `SRR098026.fastq` in the `untrimmed_fastq` folder:

```bash
$ cd ~/dc_sample_data/untrimmed_fastq/
$ grep NNNNNNNNNN SRR098026.fastq
```

We get back a lot of lines. What is we want to see the whole fastq record for each of these read.
We can use the '-B' argument for grep to return the matched line plus one before (-B 1) and two
lines after (-A 2). Since each record is four lines and the last second is the sequence, this should
give the whole record.
A lot of lines are printed to the shell! What if we want to see the whole fastq record for each of these read?
We can use the arguments '-B' and '-A' for the grep command to return plus one line before (-B1) and two
lines after (-A2) the line that matches. Since each fastq record is four lines long, and because our grep string matches the second line of the record, this command will give the entire record.

```bash
$ grep -B1 -A2 NNNNNNNNNN SRR098026.fastq
Expand All @@ -47,10 +46,9 @@ for example:
**Exercise**

1) Search for the sequence GNATNACCACTTCC in SRR098026.fastq.
In addition to finding the sequence, have your search also return
the name of the sequence.
Return the name of the record with the read that contains this sequence.

2) Search for that sequence in both fastq files.
2) Search for the above sequence in both fastq files.
****

## Redirection
Expand All @@ -74,21 +72,19 @@ from all the files in to another file called 'bad_reads.txt'
$ grep -B1 -A2 NNNNNNNNNN SRR098026.fastq > bad_reads.txt
```

The prompt should sit there a little bit, and then it should look like nothing
happened. But type `ls`. You should have a new file called bad_reads.txt. Take
a look at it and see if it has what you think it should.
The prompt will advance forward one line, and then will pause for the time it requires to run. The shell prompt will then reappear and it will look like nothing happened. But type `ls`. You will now have a new file called bad_reads.txt. Take
a look at it using the 'less' command and see if it has what you think it should.

If we use '>>', it will append to rather than overwrite a file. This can be useful for
If we use '>>', it will append to an existing file (or make a new one if the file doesn't exist) rather than overwrite a file. This can be useful for
saving more than one search, for example:

```bash
$ grep -B1 -A2 NNNNNNNNNN SRR097977.fastq >> bad_reads.txt
```

There's one more useful redirection command that we're going to show, and that's
called the pipe command, and it is `|`. It's probably not a key on
There's one more useful redirection command that we're going to introduce. The pipe command is the `|` character. It's probably not a key on
your keyboard you use very much. What `|` does is take the output that
scrolling by on the terminal and then can run it through another command.
is printed to the terminal and passes the output to another command.
When it was all whizzing by before, we wished we could just slow it down and
look at it, like we can with `less`. Well it turns out that we can! We pipe
the `grep` command through `less`
Expand All @@ -97,7 +93,7 @@ the `grep` command through `less`
$ grep -B1 -A2 NNNNNNNNNN SRR098026.fastq | less
```

Now we can use the arrows to scroll up and down and use `q` to get out.
Now we can use the arrows to scroll up and down and use `q` to exit.

We can also do something tricky and use the command `wc`. `wc` stands for
`word count`. It counts the number of lines or characters. So, we can use
Expand All @@ -115,19 +111,18 @@ just want the number of lines, we can use the `-l` flag for `lines`.
$ grep -B1 -A2 NNNNNNNNNN SRR098026.fastq | wc -l
```

Redirecting is not super intuitive, but it's really powerful for stringing
together these different commands, so you can do whatever you need to do.
Redirecting is really powerful for stringing
together different commands, so you can do whatever you need to do.

The philosophy behind these command line programs is that none of them
These command line programs don't
really do anything all that impressive. BUT when you start chaining
them together, you can do some really powerful things really
efficiently. If you want to be proficient at using the shell, you must
learn to become proficient with the pipe and redirection operators:
efficiently. Redirection operators are very userful for proficiency with the shell!
`|`, `>`, `>>`.

## Practicing searching and redirection

Finally, let's use the new tools in our kit and a few new ones to example our SRA metadata file.
Finally, let's use the new tools in our kit and a few new ones with our SRA metadata file.

```bash
$ cd ../sra_metadata/
Expand All @@ -152,7 +147,7 @@ $ head -n 1 SraRunTable.txt

BioSample_s InsertSize_l LibraryLayout_s Library_Name_s LoadDate_s MBases_l MBytes_l ReleaseDate_s Run_s SRA_Sample_s Sample_Name_s Assay_Type_s AssemblyName_s BioProject_s Center_Name_s Consent_s Organism_Platform_s SRA_Study_s g1k_analysis_group_s g1k_pop_code_s source_s strain_s

That's only the first line but it is a lot to take in. 'cut' is a program that will extract columns in tab-delimited
That's only the first line but it is a lot to take in. 'cut' is a program that will extract columns in tab-delimited
files. It is a very good command to know. Lets look at just the first four columns in the header using the '|' redirect
and 'cut'

Expand Down