Solomon ShortThere is no such thing like overkill.
Assembly of Illumina data with MIRA3
Introduction
This guide assumes that you have basic working knowledge of Unix systems, know
the basic principles of sequencing (and sequence assembly) and what assemblers
do.
While there are step by step instructions on how to setup your Solexa data and
then perform an assembly, this guide expects you to read at some point in time
the MIRA reference manual file to look up some
command line options as well as general information on what tags
MIRA uses in assemblies, files it generates etc.pp
the short usage introduction to MIRA3 so that
you have a basic knowledge on how to set up projects in mira for
Sanger sequencing projects.
Caveats when assembling Solexa data
Even very short Solexa reads (< 50bp) are great for mapping assemblies. I
simply love them as you can easily spot differences in mutant organisms
... or boost the quality of a newly sequenced genome to Q60.
Regarding de-novo assemblies ... well, from an assembler's point of
view, very short reads are a catastrophe, regardless of the sequencing
technology.
Repeats. The problem of repetitive sequences (e.g. rRNA stretches in
bacteria) gets worse the shorter the read lengths get.
Amount of data. As MIRA is by heart an assembler to resolve
difficult repetitive problems as they occur in Sanger and 454 reads, it
drags along quite a lot of ancillary information which is useless in
Solexa assemblies ... but still eats away memory.
Things look better for the now available 'longer' Solexa reads. Starting
with a length of 75bp and paired-end data, de-novo for bacteria is not
that bad at all. The first Solexas with a length of ~110 bases are
appearing in public, and from a contig building these are about as good
for de-novo as the first 454 GS20 reads were (from a quality perspective
they are even way better).
Rule of thumb: in genome sequencing, the longer, the better.
If you have to pay a bit more to get longer reads (e.g. Solexa 100mers
instead of 75mers), go get the longer reads. With these, the results
you generate are way(!) better than with 36, 50 or even 75mers
... both in mapping and de-novo. Don't try to save a couple of hundred
bucks in sequencing, you'll pay dearly afterwards in assembly.
For RNASeq, things are a bit more complicated
Rule of thumb: in RNASeq de-novo sequencing, the longer, the better.
Like for genome sequencing, go get longer reads. Your assemblies will
be definitively be better.
Even if you can get bacteria sequenced with ridiculously high coverage
like 500x or 1000x, this amount of data is simply not needed. Even
more important - though counterintuitive - is the fact that due to
non-random sequence dependent sequencing errors, a too high coverage
may even make the assembly worse.
Another rule of thumb: when having more than enough data, reduce the
data set so as to have an average coverage of approximately 100x. In
some rare cases (high GC content), perhaps 120x to 150x, but certainly
not more.
When reducing a data set, do NOT, under
no circumstances not, try fancying selection of reads by some arbitrary
quality or length criteria. This will introduce a terrible bias in your
assembly due to non-random sequence-dependent sequencing errors and
non-random sequence dependent base quality assignment. More on this in
the next section.
Typical highlights and lowlights of Solexa sequencing data
Note: This section contains things I've
seen in the past and simply jotted down. You may have different
observations.
Highlights
Quality
For 36mers and the MIRA proposed-end-clipping, even in the old
pipeline I get about 90 to 95% reads matching to a reference without
a single error. For 72mers, the number is approximately 5% lower,
100mers another 5% less. Still, these are great numbers.
Improved base calling pipeline of Illumina
The new base calling pipeline (1.4 or 2.4?) rolled out by Illumina
in Q1/Q2 2009 typically yields 20-50% more data from the very same
images. Furthermore, the base calling is way better than in the old
pipeline. For Solexa 76 mers, after trimming I get only 1% real
junk, between 85 and 90% of the reads are matching to a reference
without a single error. Of the remaining reads, roughly 50% have one
error, 25% have two errors, 12.5% have three errors etc.
It is worthwhile to re-analyse your old data if the images are
still around.
Lowlights
Long homopolymers
Long homopolymers (stretches of identical bases in reads) can be a
slight problem for Solexa. However, it must be noted that this is a
problem of all sequencing technologies on the market so far (Sanger,
Solexa, 454). Furthermore, the problem in much less pronounced in
Solexa than in 454 data: in Solexa, first problem appear may appear
in stretches of 9 to 10 bases, in 454 a stretch of 3 to 4 bases may
already start being problematic in some reads.
The GGCxG and GGC motifs
GGCxG or even GGC motif in the
5' to 3' direction of reads. This one is particularly annoying and
it took me quite a while to circumvent in MIRA the problems it
causes.
Simply put: at some places in a genome, base calling after a
GGCxG or GGC motif is
particularly error prone, the number of reads without errors
declines markedly. Repeated GGC motifs worsen
the situation. The following screen shots of a mapping assembly
illustrate this.
The first example is a the GGCxG motif (in form
of a GGCTG) occurring in approximately one third
of the reads at the shown position. Note that all but one read
with this problem are in the same (plus) direction.
The Solexa GGCxG problem.
The next two screen shots show the GGC, once for
forward direction and one with reverse direction reads:
The Solexa GGC problem, forward example
The Solexa GGC problem, reverse example
Places in the genome that have GGCGGC.....GCCGCC
(a motif, perhaps even repeated, then some bases and then an
inverted motif) almost always have very, very low number of good
reads. Especially when the motif is GGCxG.
Things get especially difficult when these motifs occur at sites
where users may have a genuine interest. The following example is a
screen shot from the Lenski data (see walk-through below) where a
simple mapping reveals an anomaly which -- in reality -- is an IS
insertion (see )
but could also look like a GGCxG motif in forward
direction (GGCCG) and at the same time a
GGC motif in reverse direction:
A genuine place of interest almost masked by the
GGCxG problem.
Strong GC bias in some Solexa data (2nd half 2009 until advent of TruSeq kit at end of 2010)
Here I'm recycling a few slides from a couple of talks I held in 2010.
Things used to be so nice and easy with the early Solexa data I worked
with (36 and 44mers) in late 2007 / early 2008. When sample taking was
done right -- e.g. for bacteria: in stationary phase -- and the
sequencing lab did a good job, the read coverage of the genome was
almost even. I did see a few papers claiming to see non-trivial GC
bias back then, but after having analysed the data I worked with I
dismissed them as "not relevant for my use cases." Have a look at the
following figure showing exemplarily the coverage of a 45% GC
bacterium in 2008:
Example for no GC coverage bias in 2008 Solexa data.
Example for no GC coverage bias in 2008 Solexa data. Apart from a
slight smile shape of the coverage --
indicating the sample taking was not 100% in stationary phase of the
bacterial culture -- everything looks pretty nice: the average
coverage is at 27x, and when looking at potential genome
duplications at twice the coverage (54x), there's nothing apart a
single peak (which turned out to be a problem in a rRNA region).
Things changed starting sometime in Q3 2009, at least that's when I
got some data which made me notice a problem. Have a look at the
following figure which shows exactly the same organism as in the
figure above (bacterium, 45% GC):
Example for GC coverage bias starting Q3 2009 in Solexa data.
Example for GC coverage bias starting Q3 2009 in Solexa
data. There's no smile shape anymore -- the
people in the lab learned to pay attention to sample in 100%
stationary phase -- but something else is extremely disconcerting:
the average coverage is at 33x, and when looking at potential genome
duplications at twice the coverage (66x), there are several dozen
peaks crossing the 66x threshold over a several kilobases (in one
case over 200 Kb) all over the genome. As if several small genome
duplications happened.
By the way, the figures above are just examples: I saw over a dozen
sequencing projects in 2008 without GC bias and several dozen in 2009
/ 2010 with GC bias.
Checking the potential genome duplication sites, they all looked
"clean", i.e., the typical genome insertion markers are
missing. Poking around at possible explanations, I looked at GC
content of those parts in the genome ... and there was the
explanation:
Example for GC coverage bias, direct comparison 2008 / 2010 data.
Example for GC coverage bias, direct comparison 2008 / 2010
data. The bug has 45% average GC, areas with above average read
coverage in 2010 data turn out to be lower GC: around 33 to 36%. The
effect is also noticeable in the 2008 data, but barely so.
Now as to actually why the GC bias suddenly
became so strong is unknown to me. The people in the lab use the same
protocol since several years to extract the DNA and the sequencing
providers claim to always use the Illumina standard protocols.
But obviously something must have changed. Current ideas about possible reasons include
changed chemistries from Illumina leading perhaps to bias during DNA amplification
changed "standard" protocols
other ...
It took Illumina some 18 months to resolve that problem for the
broader public: since data I work on were done with the TruSeq kit,
this problem has vanished.
However, if you based some conclusions or wrote a paper with Illumina
data which might be affected by the GC bias (Q3 2009 to Q4 2010), I
suggest you rethink all the conclusion drawn. This should be
especially the case for transcriptomics experiments where a difference
in expression of 2x to 3x starts to get highly significant!
Mapping assemblies
This part will introduce you step by step how to get your data together for a
simple mapping assembly.
I'll make up an example using an imaginary bacterium: Bacillus chocorafoliensis (or short: Bchoc).
In this example, we assume you have two strains: a wild type strain of
Bchoc_wt and a mutant which you perhaps got from mutagenesis or other
means. Let's imagine that this mutant needs more time to eliminate a given
amount of chocolate, so we call the mutant Bchoc_se ... SE for
slow eater
You wanted to know which mutations might be responsible for the observed
behaviour. Assume the genome of Bchoc_wt is available to you as it was
published (or you previously sequenced it), so you resequenced Bchoc_se
with Solexa to examine mutations.
Copying and naming the sequence data
You need to create (or get from your sequencing provider) the sequencing data
in either FASTQ or FASTA + FASTA quality format. The following walkthrough
uses what most people nowadays get: FASTQ.
Create a new project directory (e.g. myProject) and a subdirectory of this which will hold the sequencing data (e.g. data).
arcadia:/path/tomkdir myProjectarcadia:/path/tocd myProjectarcadia:/path/to/myProject$mkdir data
Put the FASTQ data into that data directory so that it now looks perhaps like this:
arcadia:/path/to/myProject$ls -l data
-rw-r--r-- 1 bach users 263985896 2008-03-28 21:49 bchocse_lane6.solexa.fastq
-rw-r--r-- 1 bach users 264823645 2008-03-28 21:51 bchocse_lane7.solexa.fastq
Copying and naming the reference sequence
The reference sequence (the backbone) can be in a number of different
formats: GFF3, GenBank, FASTA, CAF. The first three have the advantage
of being able to carry additional information like, e.g.,
annotation. In this example, we will use a GenBank file like the ones
one can download from the NCBI. So, let's assume that our wild type
strain is in the following file:
NC_someNCBInumber.gbk.
You do not need to copy the reference sequence to your directory, but
I normally copy also the reference file into the directory with my
data as I want to have, at the end of my work, a nice little
self-sufficient directory which I can archive away and still be sure
that in 10 years time I have all data I need together.
arcadia:/path/to/myProject$cp /somewhere/NC_someNCBInumber.gbk dataarcadia:/path/to/myProject$ls -l data
-rw-r--r-- 1 bach users 6543511 2008-04-08 23:53 NC_someNCBInumber.gbk
-rw-r--r-- 1 bach users 263985896 2008-03-28 21:49 bchocse_lane6.solexa.fastq
-rw-r--r-- 1 bach users 264823645 2008-03-28 21:51 bchocse_lane7.solexa.fastq
We're almost finished with the setup. As I like to have things neatly separated, I always create a directory called assemblies which will hold my assemblies (or different trials) together. Let's quickly do that:
arcadia:/path/to/myProject$mkdir assembliesarcadia:/path/to/myProject$mkdir assemblies/1sttrialarcadia:/path/to/myProject$cd assemblies/1sttrial
Writing a simple manifest file
A manifest file is a configuration file for MIRA which tells it what type of assembly it should do and which data it should load. In this case we have unpaired sequencing data which we want to map to a reference sequence, the manifest file for that is pretty simple:
# Example for a manifest describing a mapping assembly with
# unpaired Illumina data
# First part: defining some basic things
# In this example, we just give a name to the assembly
# and tell MIRA it should map a genome in accurate mode
# As special parameter, we want to use 4 threads in parallel (where possible)
project = MyFirstAssembly
job = genome,mapping,accurate
parameters = -GE:not=4
# The second part defines the sequencing data MIRA should load and assemble
# The data is logically divided into "readgroups"
# first, the reference sequence
readgroup
is_reference
data = ../../data/NC_someNCBInumber.gbk
technology = text
strain = bchoc_wt
# now the Illumina data
readgroup = SomeUnpairedIlluminaReadsIGotFromTheLab
data = ../../data/*fastq
technology = solexa
strain = bchoc_se
Please look up the parameters of the manifest file in the main manual. The ones
above basically say: make an accurate mapping of Solexa reads
against a genome; in one pass; the name of the backbone strain is
'bchoc_wt'; the
base qualities for the backbone are to be assumed Q30; for Solexa
data: assign default strain names for reads which have not loaded
ancillary data with strain info and that default strain name should
be 'bchoc_se'.
Starting the assembly
Starting the assembly is now just a matter of a simple command line:
arcadia:/path/to/myProject/assemblies/1sttrial$mira manifest.conf >&log_assembly.txt
For a bacterial project having a backbone of ~4 megabases and with
~4.5 million Solexa 36mers, MIRA needs some ~21 minutes on my
development machine.
A yeast project with a genome of ~20 megabases and ~20 million 36mers
needs 3.5 hours and 28 GiB RAM.
For this example - if you followed the walk-through on how to prepare the data
- everything you might want to adapt in the first time are the following thing in the manifest file:
options:
project= (for naming your assembly project)
strain_name= to give the names of your reference and mapping strain
-GE:not= to tell MIRA how many processors it should use
Of course, you are free to change any option via the extended parameters, but
this will be the topic of another FAQ.
Mapping with multiple strains
MIRA will make use of ancillary information present in the manifest
file. One of these is the information to which strain (or organism or
cell line etc.pp) the generated data belongs.
You just need to tell in the manifest file which data comes from which
strain. Let's assume that in the example from above, the "lane6" data
were from a first mutant named bchoc_se1 and the
"lane7" data were from a second mutant
named bchoc_se2. Here's the manifest file you
would write then:
# Example for a manifest describing a mapping assembly with
# unpaired Illumina data
# First part: defining some basic things
# In this example, we just give a name to the assembly
# and tell MIRA it should map a genome in accurate mode
# As special parameter, we want to use 4 threads in parallel (where possible)
project = MyFirstAssembly
job = genome,mapping,accurate
parameters = -GE:not=4
# The second part defines the sequencing data MIRA should load and assemble
# The data is logically divided into "readgroups"
# first, the reference sequence
readgroup
is_reference
data = ../../data/NC_someNCBInumber.gbk
technology = text
strain = bchoc_wt
# now the Illumina data
readgroup = DataForSE1
data = ../../data/bchocse_lane6.solexa.fastq
technology = solexa
strain = bchoc_se1readgroup = DataForSE2
data = ../../data/bchocse_lane7.solexa.fastq
technology = solexa
strain = bchoc_se2
While mapping (or even assembling de-novo) with multiple strains is
possible, the interpretation of results may become a bit daunting in
some cases. For many scenarios it might therefore be preferable to
successively use the data sets in own mappings or assemblies.
This is really the only change you need to do for this example.
Manifest for paired-end data sets
When using paired-end data in mapping, you must decide whether you want
use the MIRA feature to create long 'coverage equivalent reads'
(CERs) which saves a lot of memory (both in the assembler and
later on in an assembly editor). However, you then
loosepaired-endinformation!
or whether you want to keeppaired-endinformation! at the expense of larger memory requirements both in MIRA
and in assembly editors afterwards.
or a mix of the two above
The Illumina pipeline generally gives you two files for paired-end
data: a project-1.fastq and
project-2.fastq. The first file containing the
first read of a read-pair, the second file the second read. Depending
on the preprocessing pipeline of your sequencing provider, the names
of the reads are either the very same in both files or already have
a /1 or /2 appended.
MIRA can read all FASTQ variants produced by various Illumina
pipelines, be they with or without the /1 and /2 already appended to
the names. You generally do not need to do any name mangling before
feeding the data to MIRA.
The following manifest file is an example for mapping a 500 bp
paired-end and a 3kb mate-pair library of a strain
called bchoc_se1 against a GenBank reference
file containing a strain called bchoc_wt:
# Example for a manifest describing a mapping assembly with
# paired Illumina data, not merging reads and therefore keeping
# all pair information
# First part: defining some basic things
# In this example, we just give a name to the assembly
# and tell MIRA it should map a genome in accurate mode
# As special parameter, we want to use 4 threads in parallel (where possible)
# and switch off merging of Solexa reads
project = MyFirstAssembly
job = genome,mapping,accurate
parameters = -GE:not=4 SOLEXA_SETTINGS -CO:msr=no
# The second part defines the sequencing data MIRA should load and assemble
# The data is logically divided into "readgroups"
# first, the reference sequence
readgroup
is_reference
data = ../../data/NC_someNCBInumber.gbk
technology = text
strain = bchoc_wt
# now the Illumina data
readgroup = DataForPairedEnd500bpLib
data = ../../data/project500bp-1.fastq ../../data/project500bp-2.fastq
technology = solexa
strain = bchoc_se1
template_size = 250 750 infoonly
segment_placement = ---> <--- infoonlyreadgroup = DataForMatePair3kbLib
data = ../../data/project3kb-1.fastq ../../data/project3kb-2.fastq
technology = solexa
strain = bchoc_se1
template_size = 2500 3500 infoonly
segment_placement = <--- ---> infoonly
Please look up the parameters used in the main manual. The ones
above basically say: make an accurate mapping of Solexa reads
against a genome. Additionally do not merge short short Solexa
reads to the contig.
For the paired-end library, assume a DNA template template size of
250 to 750 bp and the segment orientation of the read pairs follows
the forward / reverse scheme. That information should be treated as
"information only" by MIRA, i.e., it is not used for deciding whether
a pair is well mapped.
For the mate-pair library, assume a DNA template template size of
2500 to 3500 bp and the segment orientation of the read pairs follows
the reverse / forward scheme. That information should be treated as
"information only" by MIRA, i.e., it is not used for deciding whether
a pair is well mapped.
Comparing this command line with a command line for unpaired-data, two
parameters were added in the section for Solexa data:
-CO:msr=no tells MIRA not to merge reads that
are 100% identical to the backbone. This also allows to keep the
template information for the reads.
infoonly attribute actually switches
off checking of template sizes and segment
placement when inserting reads into the backbone. At first glance
this might seem counter-intuitive, but it's absolutely necessary
to spot, e.g., genome re-arrangements or indels in data analysis
after the assembly.
The reason is that if template size and segment placement checking
were on, the following would happen at, e.g. sites of
re-arrangement: MIRA would map the first read of a read-pair
without problem. However, it would very probably reject the second
read because it would not map at the specified distance or
orientation from its partner. Therefore, in mapping assemblies
with paired-end data, checking of the template size must be
switched off to give post-processing programs a chance to spot
re-arrangements.
Note: for other influencing factors
you might want to change depending on size of Solexa reads, see
section above on mapping of unpaired data.
Places of interest in a mapping assembly
This section just give a short overview on the tags you might find
interesting. For more information, especially on how to configure gap4
or consed, please consult the mira usage document
and the mira manual.
In file types that allow tags (CAF, MAF, ACE), SNPs and other
interesting features will be marked by MIRA with a number of tags. The
following sections give a brief overview. For a description of what
the tags are (SROc, WRMc etc.), please read up the section "Tags used
in the assembly by MIRA and EdIt" in the main manual.
Screen shots in this section are taken from the walk-through with
Lenski data (see below).
Where are SNPs?
the SROc tag will point to most
SNPs. Should you assemble sequences of more than one strain (I
cannot really recommend such a strategy), you also might
encounter SIOc and SAOc tags.
"SROc" tag showing a SNP position in a Solexa mapping
assembly.
"SROc" tag showing a SNP/indel position in a Solexa mapping
assembly.
the WRMc tags might sometimes
point SNPs to indels of one or two bases.
Where are insertions, deletions or genome re-arrangements?
Large deletions: the MCVc tags
point to deletions in the resequenced data, where no read is
covering the reference genome.
"MCVc" tag (dark red stretch in figure) showing a genome
deletion in Solexa mapping assembly.
Insertions, small deletions and re-arrangements: these are
harder to spot. In unpaired data sets they can be found looking
at clusters of SROc, SRMc, WRMc, and / or UNSc tags.
An IS150 insertion hiding behind a WRMc and a SRMc tags
more massive occurrences of these tags lead to a rather colourful
display in finishing programs, which is why these clusters are
also sometimes called Xmas-trees.
A 16 base pair deletion leading to a SROc/UNsC xmas-tree
An IS186 insertion leading to a SROc/UNsC xmas-tree
In sets with paired-end data, post-processing software (or
alignment viewers) can use the read-pair information to guide
you to these sites (MIRA doesn't set tags at the moment).
Other tags of interest
the UNSc tag points to areas
where the consensus algorithm had troubles choosing a base. This
happens in low coverage areas, at places of insertions (compared
to the reference genome) or sometimes also in places where
repeats with a few bases difference are present. Often enough,
these tags are in areas with problematic sequences for the
Solexa sequencing technology like, e.g., a
GGCxG or even GGC motif in
the reads.
the SRMc tag points to places
where repeats with a few bases difference are present. Here too,
sequence problematic for the Solexa technology are likely to
have cause base calling errors and subsequently setting of this
tag.
Comprehensive spreadsheet tables (for Excel or OOcalc)
Biologists are not really interested in SNPs coordinates, and why
should they? They're more interested where SNPs are, how good they
are, which genes or other elements they hit, whether they have an
effect on a protein sequence, whether they may be important etc. For
organisms without intron/exon structure or splice variants, MIRA can
generate pretty comprehensive tables and files if an annotated
GenBank file was used as reference and strain information was given
to MIRA during the assembly.
Well, MIRA does all that automatically for you if the reference
sequence you gave was annotated.
For this, miraconvert should be used with the
asnp format as target and a CAF file as input:
$miraconvert -t asnp input.caf output
Note that it is strongly suggested to perform a quick manual cleanup
of the assembly prior to this: for rare cases (mainly at site of
small indels of one or two bases), mira will not tag SNPs with a SNP
tag (SROc, SAOc or SIOc) but will be fooled into a tag denoting
unsure positions (UNSc). This can be quickly corrected manually. See
further down in this manual in the section on post-processing.
After conversion, you will have four files in the directory which
you can all drag-and-drop into spreadsheet applications like
OpenOffice Calc or Excel.
The files should be pretty self-explanatory, here's just a short overview:
output_info_snplist.txt is a simple list of
the SNPs, with their positions compared to the reference
sequence (in bases and map degrees on the genome) as well as the
GenBank features they hit.
output_info_featureanalysis.txt is a much
extended version of the list above. It puts the SNPs into
context of the features (proteins, genes, RNAs etc.) and gives a
nice list, SNP by SNP, what might cause bigger changes in
proteins.
output_info_featuresummary.txt looks at the
changes (SNPs, indels) from the other way round. It gives an
excellent overview which features (genes, proteins, RNAs,
intergenic regions) you should investigate.
There's one column (named 'interesting') which pretty much
summarises up everything you need into three categories: yes,
no, and perhaps. 'Yes' is set if indels were detected, an amino
acid changed, start or stop codon changed or for SNPs in
intergenic regions and RNAs. 'Perhaps' is set for SNPs in
proteins that change a codon, but not an amino acid (silent
SNPs). 'No' is set if no SNP is hitting a feature.
output_info_featuresequences.txt simply
gives the sequences of each feature of the reference sequence
and the resequenced strain.
HTML files depicting SNP positions and deletions
I've come to realise that people who don't handle data from NextGen
sequencing technologies on a regular basis (e.g., many biologists)
don't want to be bothered with learning to handle specialised
programs to have a look at their resequenced strains. Be it because
they don't have time to learn how to use a new program or because
their desktop is not strong enough (CPU, memory) to handle the data
sets.
Something even biologist know to operate are browsers. Therefore,
miraconvert has the option to load a CAF file of a mapping
assembly at output to HTML those areas which are interesting to
biologists. It uses the tags SROc, SAOc, SIOc and MCVc and outputs
the surrounding alignment of these areas together with a nice
overview and links to jump from one position to the previous or
next.
This is done with the '-t hsnp' option of miraconvert:
$miraconvert -t hsnp input.caf outputNote: I recommend doing this only
if the resequenced strain is a very close relative to the reference
genome, else the HTML gets pretty big. But for a couple of hundred
SNPs it works great.
WIG files depicting contig coverage
miraconvert can also dump a coverage file in
WIG format (using '-t wig'). This comes pretty handy for searching
genome deletions or duplications in programs like the Affymetrix
Integrated Genome Browser (IGB, see ).
Tables for gene expression values / genome deletions & duplications
When having data mapped against a reference with annotations (either from GenBank formats or GFF3 formats), miraconvert can generate tables depicting either expression values (in RNASeq/EST data mappings) or probable genome multiplication and deletion factors (in genome mappings). For this to work, you must use a MAF or CAF file as input, specify fcov as output format and the reference sequence must have had annotations during the mapping with MIRA.
TODO: add examplemiraconvert -t fcov mira_out.maf myfeaturetable
Walkthrough: mapping of E.coli from Lenski lab against E.coli B REL606
Sorry, needs to be re-written for the relatively new SRR format
distributed at the NCBI ... and changes in MIRA 3.9.x. Please come
back later.
De-novo Solexa only assemblies
This is actually quite straightforward and not much different from
mapping assemblies: you need to tell MIRA were the data files are,
whether or not there are special conditions which apply (paired-end info
etc.) and that it is a de-novo assembly that you want and off you go.
# Example for a manifest describing a de-novo assembly with
# paired Illumina data
# First part: defining some basic things
# In this example, we just give a name to the assembly
# and tell MIRA it should assemble a genome in accurate mode
# As special parameter, we want to use 4 threads in parallel (where possible)
project = MyFirstAssembly
job = genome,denovo,accurate
parameters = -GE:not=4
# The second part defines the sequencing data MIRA should load and assemble
# The data is logically divided into "readgroups". This example uses a
# paired-end library with 500bp and a mate-pair library with 3.5kb
readgroup = MyPairedEnd500bpLib
data = ../../data/project500bp-1.fastq ../../data/project500bp-2.fastq
technology = solexa
strain = bchoc_se1
template_size = 250 750
segment_placement = ---> <---readgroup = MyMatePair3kbLib
data = ../../data/project3kb-1.fastq ../../data/project3kb-2.fastq
technology = solexa
strain = bchoc_se1
template_size = 2500 3500
segment_placement = <--- --->
Then you start the assembly:
arcadia:/path/to/myProject$mira manifest.conf >&log_assembly.txt
Of course, you can add any other switch you want like, e.g., changing
the number of processors used, adding default strain names etc.pp
De-novo hybrid assemblies (Solexa + ...)
Two strategies can be thought of to assemble genomes using a combination
of Solexa and other (longer) reads: either using all reads for a full
de-novo assembly or first assembling the longer reads and use the
resulting assembly as backbone to map Solexa reads. Both strategies have
their pro and cons.
All reads de-novo
Throwing all reads into a de-novo assembly is the most straightforward
way to get 'good' assemblies. This strategy is also the one which - in
most cases - yields the longest contigs as, in many projects, parts of
a genome not covered by one sequencing technology will probably be
covered by another sequencing technology. Furthermore, having the
consensus covered by more than one sequencing technology make base
calling a pretty robust thing: if MIRA finds disagreements it cannot
resolve easily, the assembler at least leaves a tag in the assembly to
point human finishers to these positions of interest.
The downside of this approach however is the fact that the sheer
amount of data in Solexa sequencing projects makes life difficult for
de-novo assemblers, especially for MIRA which is keeping quite some
additional information in memory in de-novo assemblies and tries to
use algorithms as exact as possible during contig
construction. Therefore, MIRA sometimes still runs into data sets
which make it behave quite badly with respect to assembly time and
memory consumption (but this is being constantly improved).
Full de-novo hybrid assemblies can be recommended only for bacteria at
the moment, although lower eukaryotes should also be feasible on
larger machines.
This is actually quite straightforward and not much different from
mapping assemblies: you need to tell MIRA were the data files are,
whether or not there are special conditions which apply (paired-end
info etc.) and that it is a de-novo assembly that you want and off you
go.
# Example for a manifest describing a de-novo assembly with
# shotgun 454 data and paired Illumina data
# First part: defining some basic things
# In this example, we just give a name to the assembly
# and tell MIRA it should assemble a genome in accurate mode
# As special parameter, we want to use 4 threads in parallel (where possible)
project = MyFirstAssembly
job = genome,denovo,accurate
parameters = -GE:not=4
# The second part defines the sequencing data MIRA should load and assemble
# The data is logically divided into "readgroups". This example uses a
# 454 shotgun library; a Solexa paired-end library with 500bp and a Solexa
# mate-pair library with 3.5kb
# First the 454 data. Note that, in this example, no XML data is loaded for
# 454 (showing that this is less critical for shotgun 454 data)
readgroup = MyShotgun454Lib
data = ../../data/project_454data.fastq
technology = 454
strain = bchoc_se1
# Now Illumina data
readgroup = MyPairedEnd500bpLib
data = ../../data/project500bp-1.fastq ../../data/project500bp-2.fastq
technology = solexa
strain = bchoc_se1
template_size = 250 750
segment_placement = ---> <---readgroup = MyMatePair3kbLib
data = ../../data/project3kb-1.fastq ../../data/project3kb-2.fastq
technology = solexa
strain = bchoc_se1
template_size = 2500 3500
segment_placement = <--- --->
Then you start the assembly:
arcadia:/path/to/myProject$mira manifest.conf >&log_assembly.txt
Long reads first, then Solexa
This strategy works in two steps: first assembling long reads, then
mapping short reads to the full alignment (not just a consensus
sequence). The result will be an assembly containing 454 (or Sanger)
and Solexa reads.
Step 1: assemble the 'long' reads (454 or Sanger or both) just as
you would when assembling 454 or Sanger data.
Step 2: filter the results. Fetch 'long' contigs from the assembly
before which are larger than 500 bases and have a given minimum
coverage. The Idea is to get more or less all contigs which
represent valid genome sequence of your project.
$miraconvert -x 500 mira_out.maf hybrid_backbone.maf
You might eventually want to add an additional filter for minimum
average coverage. If your project has an average coverage of 24,
you should filter for a minimum average coverage of 33% (coverage
8, you might want to try out higher coverages) like this:
$miraconvert -x 500 -y 8 mira_out.maf hybrid_backbone.maf
Step 3: map the Solexa data. Copy the hybrid backbone to a new
empty directory, add in the Solexa data, start a mapping assembly
using the MAF as input for the backbone / reference. If you
assembled the 454 / Sanger data with strain info, the Solexa data
should also get those (as described above).
The readgroup section of your manifest file would then look like this: (need to write)
Post-processing of assemblies
This section is a bit terse, you should also read the chapter on
working with results of MIRA3.
Post-processing mapping assemblies
When working with resequencing data and a mapping assembly, I always
load finished projects into an assembly editor and perform a quick
cleanup of the results.
For close relatives of the reference strain this doesn't take long as
MIRA will have set tags (see section earlier in this document) at all
sites you should have a look at. For example, very close mutant
bacteria with just SNPs or simple deletions and no genome
reorganisation, I usually clean up in 10 to 15 minutes. That gives the
last boost to data quality and your users (biologists etc.) will thank
you for that as it reduces their work in analysing the data (be it
looking at data or performing wet-lab experiments).
Assume you have the following result files in the result directory of
a MIRA assembly:
arcadia:/path/to/myProject/newstrain_d_results$ls -l
-rw-r--r-- 1 bach bach 655176303 2009-06-08 14:56 newstrain_out.caf
...
The general workflow I use is to convert the CAF file to a gap4 database
and start the gap4 editor:
arcadia:newstrain_d_results$caf2gap -project NEWSTRAIN -ace newstrain_out.caf >& /dev/nullarcadia:newstrain_d_results$gap4 NEWSTRAIN.0
Then, in gap4 or gap5, I
quickly search for the UNSc and WRMc tags and check whether they
could be real SNPs that were overseen by MIRA. In that case, I
manually set a SROc (or SIOc) tag in gap4 via hot-keys that were
defined to set these tags.
sometimes also quickly clean up reads that are causing trouble in
alignments and lead to wrong base calling. These can be found at
sites with UNSc tags, most of the time they have the 5' to 3'
GGCxG motif which can cause trouble to Solexa.
look at sites with deletions (tagged with MCVc) and look whether I
should clean up the borders of the deletion.
After this, I convert the gap4 database back to CAF format:
$gap2caf -project NEWSTRAIN >newstrain_edited.caf
But beware: gap4 does not have the same consensus calling routines as
MIRA and will have saved it's own consensus in the new CAF. In fact,
gap4 performs rather badly in projects with multiple sequencing
technologies. So I use miraconvert from the MIRA package to recall
a good consensus (and save it in MAF as it's more compact and a lot
faster in handling than CAF):
$miraconvert -r c newstrain_edited.caf newstrain_edited_recalled.maf
And from this file I can then convert with miraconvert to any
other format I or my users need: CAF, FASTA, ACE, WIG (for coverage
analysis) etc.pp.
I can also also generate tables and HTML files with SNP analysis
results (with the "-t asnp" and "-t
hsnp" options of miraconvert)
Post-processing de-novo assemblies
As the result file of MIRA de-novo assemblies contains everything down
to 'contigs' with just two reads, it is advised to first filter out
all contigs which are smaller than a given size or have a coverage
lower than 1/3 to 1/2 of the overall coverage.
Filtering is performed by miraconvert using CAF file as
input. Assume you have the following file:
arcadia:/path/to/myProject/newstrain_d_results$ls -l
...
-rw-r--r-- 1 bach bach 655176303 2009-06-08 14:56 newstrain_out.maf
...
Let's say you have a hybrid assembly with an average coverage of
50x. I normally filter out all contigs which have an average coverage
less than 1/3 and are smaller than 500 bases. These are mostly junk
contiglets remaining from the assembly and can be more or less safely
ignored. This is done the following way:
arcadia:newstrain_d_results$miraconvert -x 500 -y 17 newstrain_out.maf newstrain_filterx500y17.caf
From there on, convert the filtered CAF file to anything you need to
continue finishing of the genome (gap4 database, ACE, etc.pp).
Using SRA data from the Short Read Archive at the NCBI
Make sure you have the SRA toolkit installed, see
The SRA format needs to be converted into something MIRA understands,
FASTQ being an ideal candidate. Therefore the program used for this will
be fastq-dump, which unfortunately will not output
names as needed by default, but fortunately has useful options to define
your own names via the --defline-seq
and --defline-qual command line options.
When using paired-end or mate-pair data, one disconcerting thing about
the SRA tools is that they will always concatenate by default the two
reads into a single sequence, which is of course not what any sane
person would expect. Here too, there's a command line option to get
around this: --split-spot
The following shows how I would recommend extracting (and naming) reads
from the SRA:
$ls -l
...
$fastq-dump --split-spot --defline-seq '@$ac_$si/$ri' --defline-qual '+' SRR012345.sra
Known bugs / problems
These are actual for version 3 of MIRA and might or might not have been
addressed in later version.
Bugs:
mapping of paired-end reads with one read being in non-repetitive
area and the other in a repeat is not as effective as it should
be. The optimal strategy to use would be to map first the
non-repetitive read and then the read in the repeat. Unfortunately,
this is not yet implemented in MIRA.