Bastien Chevreux bach@chevreux.org
Solomon Short New problems demand new solutions. New solutions create new problems.
Assembly of Pacific Biosciences data with MIRA3 MIRA 4.0 officially supports only CCS or error-corrected-CLR This is here to set the stage: at the moment, MIRA can only make good use of PacBio reads which have an error rate of roundabout 5%. This means you will have to use either: CCS (Circular Consensus Sequence) reads with at least 3 to 4 passes. CLR (Continuous Long Reads) which were error corrected either with PacBio CCS reads, self-corrected with itself, or some other high-quality sequencing technology (Illumina comes to mind) Impatient people can directly jump to which contains walkthroughs using data made publicly available by PacBio. Unofficially, MIRA also supports uncorrected reads. However, the assemblies are currently (as of 4.0) not good enough and using MIRA with this kind of data is not recommended for general use. WARNING During the course of 2011, PacBio has made available on their DevNet site quite a number of documents and introductory videos. A must read for everyone working with this sequencing technology. Introduction As of January 2014, Pacific Biosciences looks like the new undisputed star in the de-novo sequencing arena. Illumina may try to counter it via Moleculo, but first reports point at a few shortcomings of Moleculo which Pacific Biosciences does not have. Some reading requirements This guide assumes that you have basic working knowledge of Unix systems, know the basic principles of sequencing (and sequence assembly) and what assemblers do. While there are step by step walk-throughs on how to setup your data for Sanger, 454 and Solexa in other MIRA guides, this guide is (currently) a bit more terse. You are expected to read at some point in time: the mira_reference help file to look up some command line options. for hybrid assemblies of PacBio data with Sanger, 454, Solexa the corresponding mira_usage, mira_454 or mira_solexa help files to look up how to prepare the different data sets. Assembly of PacBio data with MIRA Preparing data MIRA will happily read data in several different formats (FASTA, FASTQ, etc.). For the sake of simplicity, this guide will use FASTQ as demonstration format. Unpaired reads This is actually quite simple. Just put your reads as FASTQ in a file and you are done. No need to bother about read naming conventions or similar things. Like so: @readname_001 ACGTTGCAGGGTCATGCAGT... @readname_002 ... Setting up files and directories Launching MIRA In the most basic incantation, you will need to tell MIRA just five things: the name of your project. whether you want a "genome" or "EST" assembly whether it is a denovo or mapping assembly which quality level (draft, normal or accurate) the data you want to use The above needs to be written together in a manifest file, which is a file listing the MIRA configuration and the data it should use. Have a look at the walkthrough below and at the MIRA reference to get an idea on how to write such a file (it's simple, really). Then start MIRA like this: mira manifest.conf >&log_assembly.txt MIRA has -- at the last count -- more than 150 parameters one can use to fine tune almost every aspect of an assembly, from data loading options to results saving, from data preprocessing to results interpretation, from simple alignment parameters to parametrisation of internal misassembly decision rules ... and much more. Many of these parameters can be even set individually for each sequencing technology they apply to. Example given: in an assembly with Solexa, Sanger, 454 and PacBio data, the minimum read length for Solexa could be set to 30, while for 454 it could be 80, Sanger 100 and PacBio 150. Please refer to the reference manual for a full overview on how to use quick switches and extended switches in manifest files. Walkthroughs: real data sets from PacBio We'll use some data provided by PacBio for the E. coli O104:H4 outbreak in 2011, see for more info. Error corrected CLR for E. coli C227-11$ That data set is quite interesting: PacBio took CLR reads (the reads with only ~85% accuracy) and mapped CCS reads (presumably >90% accuracy) to them to correct errors of the CLR reads. The resulting error corrected CLR data is of pretty good quality, not only from the quality values but when assembled, the number of sequencing errors in the reads which can be spotted in the alignments is obviously quite low. Preparing a directory structure Note: this is how I set up a project, feel free to implement whatever structure suits your needs. $ mkdir c227-11-clrc $ cd c227-11-clrc arcadia:c227-11-clrc$ mkdir origdata data assemblies Your directory should now look like this: arcadia:c227-11-clrc$ ls -l drwxr-xr-x 2 bach users 48 2011-08-19 20:21 assemblies drwxr-xr-x 2 bach users 48 2011-08-19 20:21 data drwxr-xr-x 2 bach users 48 2011-08-19 20:21 origdata "c227-11-clrc" is an arbitrary name I just chose by concatenating the name of the bug and "-clrc" to indicate that this project has CLR sequences which were Corected. But you can name this whatever you want: foobar, blafurbsel, ... Explanation of the structure: the origdata directory will contain the 'raw' result files that one might get from sequencing. In our case it will be the .tar.gz file with the data in FASTQ format from the DevNet site. the data directory will contain the preprocessed sequences for the assembly, ready to be used by MIRA the assemblies directory will contain assemblies we make with our data (we might want to make more than one). Getting the data and preparing it Head over to PacBio DevNet and fetch the data set for the E. coli C227-11 CLR corrected data set. Put it into the origdata directory created a few moments ago. Now, let's extract the data to the data directory: arcadia:c227-11-clrc$ cd data arcadia:data$ tar xvzf ../origdata/e-coli-c227-11-corrected-fastq-1.2.2beta.tgz e-coli-c227-11-corrected-fastq-1.2.2beta/ e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq One thing you would quickly find out but which I tell now to save time: at the moment, PacBio seems to love ultra long read names. Here are the first 10 from the current data set: arcadia:data$ grep ^@m e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq | head -10 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10040/0_5174/c0 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10040/0_5174/c1 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1017/0_1636/c0 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1054/0_4073/c0 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1054/0_4073/c1 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1054/4121_4891/c0 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10548/0_5766/c0 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10548/0_5766/c1 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10640/0_2393/c0 @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/11000/0_3285/c0 How sweet! Read names with 80 and more characters AND having the "/" character as component, the later being a recipe for disaster sooner or later in some post-processing pipelines. MIRA has absolutely no problem with the above: neither with long read names nor with the "/" character in the name. However, long read names are a problem for example for gap4 (an assembly viewer) and the "/" character might lead to confusion with the standard UNIX directory separator, many programs which can be used to post-process assemblies simply disallow that character. For the sake of simplicity and compatibility, let's rename all sequences. For this we'll use miraconvert, which is a binary of the MIRA program package: arcadia:data$ miraconvert -R c227-11-clrc e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq c227-11-clrc.fastq Loading from fastq, saving to: fastq Loading data from FASTQ ... Counting sequences in FASTQ file: found 73496 sequences. Localtime: Sat Aug 20 20:36:26 2011 Unusual offset of 34, guessing this file to be a Sanger-type FASTQ format. Using calculated FASTQ quality offset: 33 Localtime: Sat Aug 20 20:36:26 2011 Loading data from FASTQ file: [0%] ....|.... [10%] ....|.... [20%] ....|.... [30%] ....|.... [40%] ....|.... [50%] ....|.... [60%] ....|.... [70%] ....|.... [80%] ....|.... [90%] ....|.... [100%] Done. Loaded 73496 reads, Localtime: Sat Aug 20 20:36:35 2011 done. Data conversion process finished, no obvious errors encountered. The above command has been split in multiple lines for better overview but should be entered in one line. The parameters to miraconvert say -R c227-11-clrc: sequences should be renamed, all starting with "c227-11-clrc" and to which miraconvert will append an underscore and a counter. e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq: that's the full name of our input file. c227-11-clrc.fastq: that's the name of our output file. Your directory should now look like this ... arcadia:data$ ls -l -rw-r--r-- 1 bach bach 257844076 2011-08-20 21:04 c227-11-clrc.fastq drwxr-x--- 2 bach bach 4096 2011-07-22 04:49 e-coli-c227-11-corrected-fastq-1.2.2beta ... and as we do not need the subdirectory with the extracted data from PacBio anymore, let's get rid of it: arcadia:data$ rm -rf e-coli-c227-11-corrected-fastq-1.2.2beta arcadia:data$ ls -l -rw-r--r-- 1 bach bach 257844076 2011-08-20 21:04 c227-11-clrc.fastq Perfect, we're done here. Starting the assembly Good, we're almost there. Let's switch to the assembly directory and create a subdirectory for our first assembly test. arcadia:data$ cd ../assemblies/ arcadia:assemblies$ mkdir 1sttest arcadia:assemblies$ cd 1sttest This directory is quite empty and the PacBio data is not present. And it does not need to be as we'll tell MIRA in the manifest configuration file where to find the data and what to do with it. # A manifest file can contain comment lines, these start with the #-character # First part of a manifest: defining some basic things # In this example, we just give a name to the assembly # and tell MIRA it should assemble a genome de-novo in accurate mode project = c227-11-clrc job = genome,denovo,accurate parameters = -CO:mrpg=5 # The second part defines the sequencing data MIRA should load and assemble # The data is logically divided into "readgroups", for more information # please consult the MIRA manual, chapter "Reference" readgroup = MyPacBioCLRCReads technology = pcbiohq data = ../../data/c227-11-clrc.fastq Starting the assembly is now just a matter of one line: arcadia:1sttest$ mira manifest.conf >&log_assembly.txt Some 3 to 4 hours later, you should have a nice and shiny assembly of your data. Now, that was easy, wasn't it? In the above example - for assemblies having only PacBio data and if you followed the walkthrough on how to prepare the data - everything you might want to adapt in the first time are the following options: project= (for naming your assembly project) job= (perhaps to change the quality of the assembly to 'draft' parameters= -CO:mrpg= X At the moment, this parameter needs to be adjusted by YOU regarding the expected average coverage of error corrected reads in your project. The value X should be set as in X = AvgCov / 5 (alternatively divide by 4 or 6). Of course, you are free to change any of the > 150 options via the extended parameters MIRA has ... but this is covered in the MIRA main reference manual. Working with the results of the assembly There is a whole chapter in the manual dedicated to this, you are expected to read it :-) However, for the impatient, here's a quick rundown on what I am going to show as example in this section: loading results into an assembly viewer. Looking at the assembly in gap4 I'm very fond of gap4, so I'll use it to show ho the assembly looks like: arcadia:c227-11-clrc_results$ caf2gap -project c227-11 -ace c227-11-clrc_LargeContigs_out.caf >&/dev/null arcadia:c227-11-clrc_results$ ls -l C* -rw-r--r-- 1 bach bach 543539856 2011-08-20 22:35 C227-11.0 -rw-r--r-- 1 bach bach 39512896 2011-08-20 22:35 C227-11.0.aux arcadia:c227-11-clrc_results$ gap4 C227-11.0 And while it's difficult to judge an assembly only from a screen shot, I made one: 22 contigs, none smaller than 4kb and pretty good certainty your bases are correct. Pray tell, isn't that beautiful?
Result of the assembly of <emphasis>E. coli</emphasis> C227-11 with error corrected CLR reads.
CCS reads for E. coli C227-11 Head over to PacBio DevNet and fetch the data set for the E. coli C227-11 CCS data set. For the rest ... well, it's pretty much the same as for the CLR data set. Just one little difference: in the .tgz you downloaded, PacBio has split the data set into multiple FASTQ files (for whatever reason). You will need to give those names in the manifest file before starting to work with that. Yep, and that's it.
Hybrid assemblies: PacBio plus Sanger/454/Solexa/IonTorrent TO BE EXPANDED: no real walkthrough yest, just a few hints. Prepare your PacBio data like explained in this guide. I.e.: get it error corrected into FASTQ files. Prepare your other data (Sanger, 454, Solexa, IonTorrent or any combination of it) like explained in the respective MIRA guides. Write a manifest which includes all data files as readgroups and start MIRA.