Solomon ShortNew problems demand new solutions. New solutions create new problems.
Assembly of Pacific Biosciences data with MIRA3
MIRA 4.0 officially supports only CCS or error-corrected-CLR
This is here to set the stage: at the moment, MIRA can only make good
use of PacBio reads which have an error rate of roundabout 5%. This
means you will have to use either:
CCS (Circular Consensus Sequence) reads with at least 3 to 4 passes.
CLR (Continuous Long Reads) which were error corrected either with
PacBio CCS reads, self-corrected with itself, or some other
high-quality sequencing technology (Illumina comes to mind)
Impatient people can directly jump to
which contains
walkthroughs using data made publicly available by PacBio.
Unofficially, MIRA also supports uncorrected reads. However, the
assemblies are currently (as of 4.0) not good enough and using MIRA with
this kind of data is not recommended for general use.
WARNING
During the course of 2011, PacBio has made available on
their DevNet site
quite a number of documents and introductory videos. A must read for
everyone working with this sequencing technology.
Introduction
As of January 2014, Pacific Biosciences looks like the new undisputed
star in the de-novo sequencing arena. Illumina may try to counter it via
Moleculo, but first reports point at a few shortcomings of Moleculo
which Pacific Biosciences does not have.
Some reading requirements
This guide assumes that you have basic working knowledge of Unix
systems, know the basic principles of sequencing (and sequence
assembly) and what assemblers do.
While there are step by step walk-throughs on how to setup your
data for Sanger, 454 and Solexa in other MIRA guides, this guide is
(currently) a bit more terse. You are expected to read at some point in
time:
the mira_reference help file to look up
some command line options. for hybrid assemblies of PacBio data with Sanger, 454, Solexa
the corresponding mira_usage,
mira_454 or mira_solexa
help files to look up how to prepare the different data sets.Assembly of PacBio data with MIRAPreparing data
MIRA will happily read data in several different formats (FASTA,
FASTQ, etc.). For the sake of simplicity, this guide will use FASTQ as
demonstration format.
Unpaired reads
This is actually quite simple. Just put your reads as FASTQ in a
file and you are done. No need to bother about read naming
conventions or similar things. Like so:
@readname_001
ACGTTGCAGGGTCATGCAGT...
@readname_002
...Setting up files and directoriesLaunching MIRA
In the most basic incantation, you will need to tell MIRA just five things:
the name of your project.
whether you want a "genome" or "EST" assembly
whether it is a denovo or mapping assembly
which quality level (draft, normal or accurate)
the data you want to use
The above needs to be written together in a manifest file, which is a
file listing the MIRA configuration and the data it should use. Have a
look at the walkthrough below and at the MIRA reference to get an idea
on how to write such a file (it's simple, really). Then start MIRA
like this:
miramanifest.conf >&log_assembly.txt
MIRA has -- at the last count -- more than 150 parameters one can use
to fine tune almost every aspect of an assembly, from data loading
options to results saving, from data preprocessing to results
interpretation, from simple alignment parameters to parametrisation of
internal misassembly decision rules ... and much more. Many of these
parameters can be even set individually for each sequencing technology
they apply to. Example given: in an assembly with Solexa, Sanger, 454
and PacBio data, the minimum read length for Solexa could be set to
30, while for 454 it could be 80, Sanger 100 and PacBio 150. Please
refer to the reference manual for a full overview on how to use
quick switches and extended
switches in manifest files.
Walkthroughs: real data sets from PacBio
We'll use some data provided by PacBio for the
E. coli O104:H4 outbreak in 2011, see for
more info.
Error corrected CLR for E. coli C227-11$
That data set is quite interesting: PacBio took CLR reads (the reads
with only ~85% accuracy) and mapped CCS reads (presumably >90%
accuracy) to them to correct errors of the CLR reads. The resulting
error corrected CLR data is of pretty good
quality, not only from the quality values but when assembled, the
number of sequencing errors in the reads which can be spotted in the
alignments is obviously quite low.
Preparing a directory structure
Note: this is how I set up a project, feel free to implement whatever
structure suits your needs.
$mkdir c227-11-clrc$cd c227-11-clrcarcadia:c227-11-clrc$mkdir origdata data assemblies
Your directory should now look like this:
arcadia:c227-11-clrc$ls -l
drwxr-xr-x 2 bach users 48 2011-08-19 20:21 assemblies
drwxr-xr-x 2 bach users 48 2011-08-19 20:21 data
drwxr-xr-x 2 bach users 48 2011-08-19 20:21 origdata
"c227-11-clrc" is an arbitrary name I just chose by concatenating
the name of the bug and "-clrc" to indicate that this project has
CLR sequences which were Corected. But you can name this whatever you
want: foobar, blafurbsel, ...
Explanation of the structure:
the origdata directory will contain the 'raw'
result files that one might get from sequencing. In our case it
will be the .tar.gz file with the data in
FASTQ format from the DevNet site.
the data directory will contain the
preprocessed sequences for the assembly, ready to be used by MIRA
the assemblies directory will contain
assemblies we make with our data (we might want to make more than
one).
Getting the data and preparing it
Head over to PacBio DevNet and fetch the data set for the E. coli
C227-11 CLR corrected data set. Put it into the
origdata directory created a few moments ago.
Now, let's extract the data to the data directory:
arcadia:c227-11-clrc$cd dataarcadia:data$tar xvzf ../origdata/e-coli-c227-11-corrected-fastq-1.2.2beta.tgz
e-coli-c227-11-corrected-fastq-1.2.2beta/
e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq
One thing you would quickly find out but which I tell now to save
time: at the moment, PacBio seems to love ultra long read
names. Here are the first 10 from the current data set:
arcadia:data$grep ^@m e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq | head -10
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10040/0_5174/c0
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10040/0_5174/c1
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1017/0_1636/c0
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1054/0_4073/c0
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1054/0_4073/c1
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1054/4121_4891/c0
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10548/0_5766/c0
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10548/0_5766/c1
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10640/0_2393/c0
@m110618_035655_42142_c100158802555500000315044108071130_s1_p0/11000/0_3285/c0
How sweet! Read names with 80 and more characters AND having the "/"
character as component, the later being a recipe for disaster sooner
or later in some post-processing pipelines.
MIRA has absolutely no problem with the above: neither with long read
names nor with the "/" character in the name. However, long read names
are a problem for example for gap4 (an assembly
viewer) and the "/" character might lead to confusion with the
standard UNIX directory separator, many programs which can be used
to post-process assemblies simply disallow that character.
For the sake of simplicity and compatibility, let's rename all
sequences. For this we'll use miraconvert,
which is a binary of the MIRA program package:
arcadia:data$miraconvert
-R c227-11-clrc
e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq
c227-11-clrc.fastq
Loading from fastq, saving to: fastq
Loading data from FASTQ ...
Counting sequences in FASTQ file: found 73496 sequences.
Localtime: Sat Aug 20 20:36:26 2011
Unusual offset of 34, guessing this file to be a Sanger-type FASTQ format.
Using calculated FASTQ quality offset: 33
Localtime: Sat Aug 20 20:36:26 2011
Loading data from FASTQ file:
[0%] ....|.... [10%] ....|.... [20%] ....|.... [30%] ....|.... [40%] ....|....
[50%] ....|.... [60%] ....|.... [70%] ....|.... [80%] ....|.... [90%] ....|.... [100%]
Done.
Loaded 73496 reads, Localtime: Sat Aug 20 20:36:35 2011
done.
Data conversion process finished, no obvious errors encountered.
The above command has been split in multiple lines for better overview
but should be entered in one line.
The parameters to miraconvert say
-R c227-11-clrc: sequences
should be renamed, all starting with "c227-11-clrc" and to which
miraconvert will append an underscore and
a counter.
e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq:
that's the full name of our input file.
c227-11-clrc.fastq: that's
the name of our output file.
Your directory should now look like this ...
arcadia:data$ls -l
-rw-r--r-- 1 bach bach 257844076 2011-08-20 21:04 c227-11-clrc.fastq
drwxr-x--- 2 bach bach 4096 2011-07-22 04:49 e-coli-c227-11-corrected-fastq-1.2.2beta
... and as we do not need the subdirectory with the extracted data from PacBio anymore, let's get rid of it:
arcadia:data$rm -rf e-coli-c227-11-corrected-fastq-1.2.2betaarcadia:data$ls -l
-rw-r--r-- 1 bach bach 257844076 2011-08-20 21:04 c227-11-clrc.fastq
Perfect, we're done here.
Starting the assembly
Good, we're almost there. Let's switch to the
assembly directory and create a subdirectory for our
first assembly test.
arcadia:data$cd ../assemblies/arcadia:assemblies$mkdir 1sttestarcadia:assemblies$cd 1sttest
This directory is quite empty and the PacBio data is not
present. And it does not need to be as we'll tell MIRA in the manifest
configuration file where to find the data and what to do with it.
# A manifest file can contain comment lines, these start with the #-character
# First part of a manifest: defining some basic things
# In this example, we just give a name to the assembly
# and tell MIRA it should assemble a genome de-novo in accurate mode
project = c227-11-clrc
job = genome,denovo,accurate
parameters = -CO:mrpg=5
# The second part defines the sequencing data MIRA should load and assemble
# The data is logically divided into "readgroups", for more information
# please consult the MIRA manual, chapter "Reference"
readgroup = MyPacBioCLRCReads
technology = pcbiohq
data = ../../data/c227-11-clrc.fastq
Starting the assembly is now just a matter of one line:
arcadia:1sttest$mira manifest.conf >&log_assembly.txt
Some 3 to 4 hours later, you should have a nice and shiny assembly of
your data.
Now, that was easy, wasn't it? In the above example - for assemblies
having only PacBio data and if you followed the walkthrough on how to
prepare the data - everything you might want to adapt in the first
time are the following options:
project= (for naming your assembly project)
job= (perhaps to change the quality of the assembly to 'draft'
parameters= -CO:mrpg= X
At the moment, this parameter needs to be adjusted by YOU
regarding the expected average coverage of error corrected reads
in your project. The value X should be set
as in X = AvgCov / 5 (alternatively divide
by 4 or 6).
Of course, you are free to change any of the > 150 options via
the extended parameters MIRA has ... but this is covered in the MIRA
main reference manual.
Working with the results of the assembly
There is a whole chapter in the manual dedicated to this, you are expected to read it :-)
However, for the impatient, here's a quick rundown on what I am
going to show as example in this section: loading results into an
assembly viewer.
Looking at the assembly in gap4
I'm very fond of gap4, so I'll use it to show ho the assembly looks like:
arcadia:c227-11-clrc_results$caf2gap -project c227-11 -ace c227-11-clrc_LargeContigs_out.caf >&/dev/nullarcadia:c227-11-clrc_results$ls -l C*
-rw-r--r-- 1 bach bach 543539856 2011-08-20 22:35 C227-11.0
-rw-r--r-- 1 bach bach 39512896 2011-08-20 22:35 C227-11.0.aux
arcadia:c227-11-clrc_results$gap4 C227-11.0
And while it's difficult to judge an assembly only from a
screen shot, I made one: 22 contigs, none smaller than 4kb and
pretty good certainty your bases are correct. Pray tell, isn't that
beautiful?
Result of the assembly of E. coli C227-11 with
error corrected CLR reads.
CCS reads for E. coli C227-11
Head over to PacBio DevNet and fetch the data set for the E. coli
C227-11 CCS data set.
For the rest ... well, it's pretty much the same as for the CLR data
set. Just one little difference: in the .tgz you
downloaded, PacBio has split the data set into multiple FASTQ files
(for whatever reason). You will need to give those names in the
manifest file before starting to work with that. Yep, and that's it.
Hybrid assemblies: PacBio plus Sanger/454/Solexa/IonTorrent
TO BE EXPANDED: no real walkthrough yest, just a few hints.
Prepare your PacBio data like explained in this guide. I.e.: get it
error corrected into FASTQ files.
Prepare your other data (Sanger, 454, Solexa, IonTorrent or any combination of
it) like explained in the respective MIRA guides.
Write a manifest which includes all data files as readgroups and start MIRA.