Debian Codegraph @

Documentation/BaitFisher-BaitFilter-Manual.tex

raw
1 \documentclass[a4paper,pdflatex,11pt]{article}
2
3 \usepackage{epsf,epsfig}
4 %\RequirePackage{amsfonts}[1995/01/01]
5 %\usepackage{bbm} %% needed for mathbbm{1}
6 \usepackage{parskip}
7 \usepackage{titlesec}
8
9 \def\mdot{\mathaccent"17}
10
11 \usepackage{color}
12 \definecolor{darkred}{rgb}{0.3,0,0}
13 \definecolor{darkgreen}{rgb}{0,0.5,0}
14 \definecolor{darkblue}{rgb}{0,0,0.5}
15
16 \usepackage[backref=page,hyperindex=true,
17 pdftitle={},pdfauthor={Christoph Mayer}]{hyperref}
18
19 %% !1! %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
20 \hypersetup{colorlinks,linkcolor=darkred,filecolor=darkgreen,urlcolor=darkblue,citecolor=darkred}
21 %\hypersetup{colorlinks,linkcolor=black,filecolor=black,urlcolor=black,citecolor=black}
22 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
23
24 %hyperref:
25 % \href={URL}{text}
26 % href="Name" end Name
27
28
29 %\oddsidemargin 0.2cm
30 \topmargin 0cm
31 \textwidth 160mm
32 \textheight 230mm
33
34 %\if@twoside
35 \oddsidemargin 0cm \evensidemargin 0cm %\marginparwidth 100pt
36
37 % \oddsidemargin 0.5cm \evensidemargin -0.2cm %\marginparwidth 100pt
38 %\else
39 % \oddsidemargin 0.5cm \evensidemargin 0.5cm \marginparwidth 83pt
40 %\fit
41
42 \setlength{\parindent}{0pt}
43
44 \newcommand{\shortpage}{\enlargethispage{-\baselineskip}}
45 \newcommand{\vshortpage}{\enlargethispage{-2\baselineskip}}
46 \newcommand{\vvshortpage}{\enlargethispage{-3\baselineskip}}
47 \newcommand{\vvvshortpage}{\enlargethispage{-4\baselineskip}}
48 \newcommand{\longpage}{\enlargethispage{\baselineskip}}
49 \newcommand{\vlongpage}{\enlargethispage{2\baselineskip}}
50
51 \renewcommand{\thesection}{\Roman{section}}
52
53 %\titleformat{\section}{\normalfont\large\bfseries}{\thesection}{1em}{}
54
55
56 \newcounter{conditionscount}
57 \newenvironment{conditions}%
58 {\begin{list}{\rm (C\arabic{conditionscount}) ~}{\usecounter{conditionscount}%
59 % \setlength{\labelsep}{0pt}%
60 \setlength{\leftmargin}{50pt}%
61 % \setlength{\labelwidth}{0pt}%
62 % \setlength{\listparindent}{0pt}
63 }}
64 {\end{list}}
65 \newcommand{\Cref}[1]{(C\ref{#1})}
66
67
68 \newcounter{romanlistcount}
69 \newenvironment{romanlist}%
70 {\begin{list}{\rm (\roman{romanlistcount}):}{\usecounter{romanlistcount}%
71 % \setlength{\labelsep}{0pt}%
72 \setlength{\leftmargin}{40pt}%
73 % \setlength{\rightmargin}{40pt}%
74 \setlength{\topsep}{1mm}
75 \setlength{\parsep}{0mm}
76 \setlength{\itemsep}{3mm}
77 \setlength{\labelwidth}{30pt}%
78 % \setlength{\listparindent}{0pt}
79 }}
80 {\end{list}}
81 \newenvironment{condenseditemize}%
82 {\begin{list}{\textbullet}{%
83 % \setlength{\labelsep}{0pt}%
84 \setlength{\leftmargin}{30pt}%
85 % \setlength{\rightmargin}{40pt}%
86 \setlength{\topsep}{2mm}
87 \setlength{\parsep}{0mm}
88 \setlength{\itemsep}{2mm}
89 \setlength{\labelwidth}{30pt}%
90 % \setlength{\listparindent}{0pt}
91 }}
92 {\end{list}}
93 \newenvironment{steplist}%
94 {\begin{list}{\em Step: \arabic{romanlistcount}:~~}{\usecounter{romanlistcount}%
95 % \setlength{\labelsep}{0pt}%
96 \setlength{\leftmargin}{70pt}%
97 % \setlength{\rightmargin}{40pt}%
98 \setlength{\topsep}{1mm}
99 \setlength{\parsep}{0mm}
100 \setlength{\itemsep}{0mm}
101 \setlength{\labelwidth}{70pt}%
102 % \setlength{\listparindent}{0pt}
103 }}
104 {\end{list}}
105
106 \def \bc{\begin{center}}
107 \def \ec{\end{center}}
108
109
110
111
112 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
113 \begin{document} %% beginning of document %%%%%
114 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
115 \thispagestyle{empty}
116 {\centering
117 {\large\bf The BaitFisher package manual\\[4mm]}
118 {\large BaitFisher version 1.2.7\\}
119 {\large BaitFilter version 1.0.6\\[5mm]}
120
121 {\large August 2017\\[5mm]}
122
123 %\vspace{2cm}
124 {\large Christoph Mayer and Manuela Sann\\}
125 {\large Forschungsmuseum Alexander Koenig, Bonn, Germany\\}
126 \vspace{1cm}
127 }
128
129 \section{About the BaitFisher package}
130
131
132 The BaitFisher package consists of two programs: BaitFisher and
133 BaitFilter.
134
135 {\em BaitFisher} has been designed to construct hybrid enrichment baits from
136 multiple sequence alignments (MSAs) or annotated features within MSAs. The
137 main goal of BaitFisher is to automatically adapt the number of necessary baits to the
138 variability of the underlying alignment.
139 In conserved regions of the MSA, BaitFisher designs fewer baits, in more variable regions
140 it designs more baits.
141 This approach makes use of the fact
142 that hybrid enrichment baits can differ to some extends from the target
143 region, which they should hybridise agains in order to capture and enrich them.
144 By specifying a maximum allowed distance between the designed baits and the sequences in the
145 MSAs the user can control the allowed bait-to-target distance and the
146 degree of reduction in the number of baits that are designed. See the
147 BaitFisher paper Mayer, Sann et al. 2016 for details.
148
149 {\em BaitFilter} was designed to post-process the baits that have been designed with BaitFisher.
150 BaitFilter can be invoked in multiple steps to conduct one of the following tasks:
151 (i) Determine whether baits bind
152 unspecific to a reference genome, i.e. determined whether the bait is similar to more than one
153 region in the reference genome. (ii) Filter baits that only
154 have partial length matches to a reference genome. (iii) Determine
155 the optimal bait region in a MSA. The optimal bait region
156 can be the most conserved region in the MSA or the region with the
157 highest number of sequences without gaps or ambiguous nucleotides in the alignment.
158 (iv) Convert the (filtered) set of baits into a format which can be uploaded at the bait
159 producing company.
160
161 Since performance was one of the major design goals, the BaitFisher and
162 BaitFilter programs are both written in C++.
163
164 \textbf{When using BaitFisher please cite:}
165
166 \href{https://academic.oup.com/mbe/article/33/7/1875/2579186/BaitFisher-A-Software-Package-for-Multispecies}
167 {Mayer, C., Sann, M., Donath, A., Meixner, M., Podsiadlowski, L., Peters,
168 R.S., Petersen, M., Meusemann, K., Liere, K., Wägele, J.-W., Misof, B.,
169 Bleidorn, C., Ohl, M., Niehuis, O., 2016. BaitFisher: A Software Package for
170 Multispecies Target DNA Enrichment Probe Design. \emph{Mol. Biol. Evol.} 33, 1875 - 1886.
171 doi:10.1093/molbev/msw056.}
172
173 \section{Where to obtain the BaitFisher package}
174
175 A Debian package with BaitFisher will be available soon. This will allow you to install BaitFisher
176 on Debian Linux directly from you Linux package manager.\\
177
178 The source code for the BaitFisher package can be downloaded from:\\
179 \href{https://github.com/cmayer/BaitFisher-package.git}
180 {\emph{https://github.com/cmayer/BaitFisher-package.git}}
181
182 When downloading BaitFisher from github, we recommend to download the latest release version:
183 \href{https://github.com/cmayer/BaitFisher-package/releases}
184 {\emph{https://github.com/cmayer/BaitFisher-package/releases}}
185 or the latest development version:
186 \href{https://github.com/cmayer/BaitFisher-package/archive/master.zip}
187 {\emph{https://github.com/cmayer/BaitFisher-package/archive/master.zip}}.
188
189
190 \section{Compiling and installing BaitFisher and BaitFilter from source}
191
192 \emph{System requirements:}
193
194 BaitFisher can be compiled on all platforms, which provide a C++
195 compiler that includes the following system header files: unistd.h,
196 sys/stat.h, sys/types.h, dirent.h. These header files are standard
197 header files on all unix systems such as Linux and Mac Os X. I also
198 managed to compile BaitFisher and BaitFilter on Windows using the Mingw
199 compiler (www.mingw.org).
200
201 \emph{Compiling BaitFisher and BaitFilter:}
202
203 Please, unpack the BaitFisher archive you downloaded.
204
205 On the command line change to the BaitFisher-package-master directory.
206 Type \verb+make+ on the command line. This compiles the BaitFisher as well as
207 the BaitFilter program. The executables of these to programs are
208 called: BaitFisher-vxxx and BaitFilter-yyy, where xxx and yyy are version numbers
209 of the two programs in the release you downloaded.
210 The binary executables can be copied on
211 your computer to any place you like. To use these executables, they have
212 to be addressed with their path, or they have to be placed in a directory which is included
213 in your systems path variable.
214
215 \section{Terms you should be familiar with when reading the manual }
216
217 \textbf{Multiple sequence alignment (MSA):}
218 In this context an MSA will always be an alignment of multiple
219 nucleotide sequences. Baits can only be designed for nucleotide
220 alignments and not for amino acid alignments.
221
222 \textbf{Target MSAs:}
223 MSAs for which baits shall be designed.
224
225
226 \textbf{Target region:}
227 DNA region of interest in the genome.
228
229 \textbf{Bait:}
230 An artificially assembled RNA sequence that binds (i.e. hybridises) to parts of the target
231 region. It can be used to capture sheared genomic DNA fragments.
232 A bait is able to hybridise agains sequence fragments which
233 are identical or almost identical to its reverse complement.
234 Usually, a difference of 15\% to 20\% in the nucleotides between bait and target are not
235 a problem and the bait still binds to the target. (See Mayer, Sann et al. 2016 for details.)
236
237 \textbf{Tiling design:}
238 The probability of capturing a given locus increases with the number of baits that are
239 placed into the locus.
240 For this reason, it is common practice not only to use a single bait per locus but a
241 so called tiling design consisting of multiple baits.
242
243 With a tiling design one specifies how successive baits are arranged and how many baits are used at a single locus
244 (see Figure \ref{label_Fig_TilingDesing}).
245 A tiling design is completely defined by the following information: the bait length,
246 the number of successive baits, the offset between starting coordinates.
247
248 %For example in Sann et al. 2017 it has been shown that the sequence coverage of
249 %assembled target loci increases with the number of baits in the tiling design.
250 %(Information will be added as soon as the paper is published.)
251 %Please site Sann et al. 2017 if this information helped you to determine
252 %a useful number of baits in the tiling design of your work.
253
254 \begin{figure}
255 \centering
256 \includegraphics[width=10cm]{Tiling-design.png}
257
258 \caption{Target region and tiling design. In this example the tiling design
259 consists of seven successive 120-bp bait windows. The complete tiling design
260 spans 240 bp. Starting coordinates of the baits are separated by an
261 offset of 20 bp. A region for which baits have been constructed
262 according to a given tiling design shall be referred to as a bait
263 region.\label{label_Fig_TilingDesing}}
264 \end{figure}
265
266
267 \textbf{Bait region:}
268 A genomic DNA region for which baits have been successfully determined
269 for a given tiling design. The length of the bait region is identical to
270 the length of the tiling design (see Figure \ref{label_Fig_TilingDesing}).
271 While a tiling design is only a description of how baits will be constructed, a bait region
272 is a genomic locus at which designing baits for a full tiling design was successful.
273
274
275
276 \textbf{Bait window and bait tiling stack sets:}
277 For a given locus of length of the tiling design, BaitFisher will cut the MSA into
278 N bait windows, where N is the number of baits in the tiling design.
279 The first bait window starts at the beginning of the the locus.
280 The second bait window starts with on offset specified in the tiling design, e.g.\ 20bp
281 in the example above.
282 The starting points of all further bait windows are separated from its successor
283 by the same offset.
284 In order to design Baits for a full tiling design,
285 BaitFisher will design baits for each of the N bait windows.
286 The collection of all baits in one bait windows is called a tiling stack set of baits,
287 or short a ``tiling stack''.
288 The number of bait windows and the number of tiling stacks are both equal to the number of
289 successive baits in the tiling design.
290
291 The number of baits that BaitFisher designs for a given bait window and therefore
292 for a given tiling stack
293 varies and will depend on the similarity of the sequences in the bait window.
294 If the sequences in the bait window are highly similar, the number of required baits will be small.
295 In the extreme case, one bait can be sufficient to represent all sequences in a bait window.
296 If the sequences are very dissimilar, the number of required baits will be high.
297 In the extreme case, the number of required baits will be equal to the number of sequences
298 in the bait window.
299
300 In the above example we have 7 bait windows and 7 tiling stack sets
301 of baits in each bait region.
302
303
304 \textbf{Continuous vs. non continuous sequence data:}
305 When designing baits for genomic loci, the underlying assumption is that
306 the sequences in the MSA are continuous, i.e.
307 the sequences in the MSA of the target region are highly similar in full length to the DNA
308 in the genomes of the organisms for which DNA shall be enriched.
309 The sequences in the MSA are called non continuous, if sequence segments have been removed
310 or inserted with respect to the homologous sequence in the genome of interest.
311
312 There are important cases in which sequence data might be non continuous:
313 E.g.\ for MSAs of transcriptomic sequences, it is not guaranteed that
314 the sequences are continuous in the genome, since the splicing machinery might have
315 already altered the original sequences by removing introns.
316 If introns have been spliced out in the transcript sequences,
317 the MSA of transcript sequences contains exon-exon boundaries, which do not
318 exist in the genome since introns separate the exons.
319 This can be important when designing baits. Baits which span the artificial exon-exon boundary,
320 are expected to be less efficient or even worthless compared to other baits.
321
322
323 The BaitFisher package implements two strategies to deal with this problem.
324 (i) BaitFisher implements a method to cut transcript sequences into exons or CDS regions
325 with the help of an annotated genome.
326 (ii) BaitFilter can be used to search for baits which do not match in full length or almost full length
327 to a reference genome.
328 More specifically, BaitFilter can be asked to remove bait regions
329 in which tiling stack exist in which not at least one bait exists which
330 matches with at least a specified proportion of its length with the reference sequence. For more details see the
331 \textbf{-\/-blast-min-hit-coverage-of-baits-in-tiling-stack} option, which can be invoked together with one of the following modes:
332 blast-l, blast-c, blast-f, blast-a.
333
334
335 \textbf{Reference genome:}
336 If alignment cutting is requested, an annotated genome is required. See
337 Section \ref{section_alignment_cutting} for details. A genome is also required if a blast search of the baits against a genome is requested.
338 This genome will be called a reference genome, since it serves as a reference either for the features we want to excise or as a reference for
339 for the blast search.
340
341 \section{The algorithm behind BaitFisher}
342
343 The algorithm behind BaitFisher is described in detail in
344 Mayer, Sann et al.\ 2016.
345
346
347 \section{BaitFishers two major modes: full MSA mode and MSA cutting mode}
348
349 BaitFisher can design baits for MSAs.
350
351 Furthermore, BaitFisher can be used to excise annotated features from MSAs with the aid of
352 an annotated genome. E.g.\ it can excise exons or CDS regions from transcript alignments.
353
354 \subsection{Full MSA mode}
355
356 This is the standard mode. The user simply specifies the folder containing the multiple sequence alignment files
357 and the tiling design in a configuration file and BaitFisher will design baits for all alignment files and for every
358 possible starting point of a bait region.
359 In this mode the user can e.g.\ design baits for COI or 18S sequences or for genes excised by the user.
360 %%%%%%%%% %Strategies for designing baits in different genomic regions are discussed in Section ... .
361
362 \subsection{Alignment cutting mode} \label{section_alignment_cutting}
363
364 BaitFisher can excise features such as exons or CDS regions from given alignments using an annotation file.
365 The most important example is to excise exons or CDS regions from transcript alignments.
366
367 The main reason the user might want to excise e.g.\ exons or CDS regions from transcript alignments is that
368 transcript alignments might not contain continuous DNA. A transcript alignment might or might not contain
369 introns. If the introns have already been removed by the splicing machinery of the organism the transcripts originate from,
370 then the DNA is not continuous any more. If baits are designed across an exon-exon boundary in the transcript file in which
371 an intron is missing, then there is no such locus in the genome. As a consequence, the efficiency of the baits
372 spanning this exon-exon boundary are expected to be reduced since they cannot bind in full length to the target.
373 In this case one might prefer to design baits only for
374 annotated exons or CDS regions and not for the whole alignment.
375 With this approach, we ensure to design baits for sequence segments which are continuous in the genome.
376
377 Alignment cutting can only be performed, if the user has an annotated
378 reference genome in form of a fasta file, containing the genome sequence
379 assembly and a gff file, containing the annotation. Moreover, each MSA
380 for which baits have to be designed, has to have a sequence that stems
381 from the reference genome. The name of this sequence has to encode the
382 taxon name as well as the gene name in the annotation.
383
384 The name of the reference species, which is used in the MSA, has to be specified in the BaitFisher configuration file.
385
386 If alignment cutting is requested, BaitFisher conducts the following steps:
387
388 \begin{itemize}
389 \item For each MSA (usually a gene), BaitFisher
390 searches for the corresponding annotation in the gff file.
391 This is only possible if the name of the gene is encoded in the sequence name of a sequence in the MSA which belongs to the reference genome.
392
393 \item With the genomic coordinates of the gene features that BaitFisher finds in the gff file it excises the features from the genome and
394
395 \item aligns them to the transcript sequence that comes from the reference genome.
396 \item The alignment coordinates at which the genomic feature starts and ends within the reference sequence
397 allow to determine the feature coordinates in the MSA.
398 \item The feature can
399 now be excised and BaitFisher will confine the bait design to the excised features.
400 \end{itemize}
401
402 An example analysis comes with the BaitFisher package. See Section (\ref{label_Example_alignment_cutting})
403 for details.
404
405 {\bf More detailed description of the alignment cutting procedure:}\\
406 For being able to cut the alignments into features, BaitFisher requires
407 the following: First, each alignment is only allowed to contain a single
408 gene. The alignment has to contain the sequence of the reference species
409 and the name of the gene has to be encoded in the sequence name of this
410 reference species. The gene name is required to locate the annotation of
411 this gene in the gff file. Since the sequence names in the fasta file contain several
412 pieces of information, it must be possible to split the sequence names into these pieces
413 with a specific delimitation character.
414 After splitting the sequence name with the delimitation character, there
415 has to be one field that contains the name of the gene written exactly in the same way as
416 the gene is called in the gff file. Another field has to
417 contain the taxon name. The taxon name is required (i) to identify the
418 reference sequence and (ii) to identify the taxon name if required taxa
419 are specified (see below). The delimitation character and field numbers
420 of gene and taxon name can be specified in the parameter file. After
421 extracting the gene name from the sequence name of the reference
422 species, BaitFisher uses the gene name to find all features that belong
423 to this gene. In the gff file each feature is described on a single
424 line. The last tab delimited field in a gff file is the so call
425 attribute field. Usually, the name of the gene is given as an attribute
426 of the form ``Parent = <GeneID>'', where the ``='' symbol can be left
427 out. The keyword parent is often used to indicate the name of the next
428 outer element a feature belongs to, which in this case is the gene.
429 Other keywords are possible. The keyword under which BaitFisher can find
430 the geneID can be specified in the parameter file. The geneID has to
431 match exactly the geneID found in the sequence name of the MSA. Only
432 then will BaitFisher be able to find all features that belong to the
433 geneID of the MSA BaitFisher works on. After BaitFisher determined all
434 features of a specified geneID, it will filter those features who's
435 feature name match the name specified in the parameter file. Typical
436 feature names for which baits are constructed are ``CDS'' and ``exon''.
437 After BaitFisher successfully identified features for the MSA it is
438 working on, it uses the gff file coordinates to extract the feature sequences
439 from the specified genome. Each of these sequence segments are aligned to the
440 reference sequence in the MSA. The coordinates at which the feature
441 begins and ends in the transcript sequence will allow BaitFisher to
442 identify the coordinates of the feature in the MSA. In the next step
443 BaitFisher will extract the feature from the MSA. It will save the
444 feature in a separate fasta file for future reference for the user and
445 it will use it to design baits.
446
447 No baits will be designed if BaitFisher cannot find a sequence name that
448 encodes the correct name of the reference taxon or if no annotated
449 features can be found in the gff file for this specific gene.
450
451
452
453 \section{Special mode -- required taxa}
454
455 BaitFisher can only design baits for sequences that do not contains gaps
456 or ambiguous characters in the bait window.
457 In a bait windows in which some sequences do not fulfil this requirement,
458 BaitFisher will only include sequences in the bait design process
459 which have defined bases in the whole window.
460 Sequences which do not fulfil this requirement will simply not be considered in the
461 bait design of this particular bait window.
462
463 %%% Add a graphics here.
464
465
466 This is an acceptable approach, as long as the missing sequences are not highly important.
467 If, however, the missing sequences are the most important relatives of those
468 species for which baits are designed, one might not want to design baits for regions
469 in which no sequence information is present for the most important species.
470
471 If the user considers specific sequences in the MSAs to be of particular
472 importance she/he can specify the corresponding taxon names as required in
473 the parameter file. If one of the required taxa is missing from an MSA
474 or if its sequence contains gaps or ambiguous characters in the bait
475 window, BaitFisher will not design baits for
476 the region of the MSA that contains the missing information.
477
478 If required taxa are specified, the sequence names in all MSAs have to
479 contain taxon names, in order to be able to identify
480 the sequences. BaitFisher identifies taxon names by splitting sequence
481 names according to the
482 {\verb sequence-name-field-delimiter} specified in the parameter file
483 and extracting the taxon name in the field specified by the
484 {\verb sequence-name-taxon-field_number} option.
485
486 The required taxon string has the following format: Groups of taxa of
487 which at least one representative has to be present are enclosed in
488 parentheses. Taxon names within one group are separated by commas.
489 Multiple groups, separated by comma, can be specified.
490
491 Example: (Taxon1, Taxon2, Taxon3),(Taxon4),(Taxon5, Taxon6)
492
493 In this example a bait window is only valid if either Taxon1, Taxon2 or
494 Taxon3 are present, if Taxon4 is present and finally, if either Taxon5 or Taxon6 are present in full length of the bait window.
495
496 %%% Continue here XXXXXXXXXXXXX
497
498 \section{Program options, the configuration file and input file formats}
499
500 In order to design baits for one or multiple MSAs, BaitFisher requires
501 several pieces of information:
502
503 \begin{enumerate}
504 \def\labelenumi{\roman{enumi}.}
505 \item
506 Details about the tiling design, i.e. the bait length, the number of
507 baits per tiling design, and the offset between the starting
508 coordinates
509 \item
510 Directory that contains all input files
511 \item
512 Directory into which output files shall be written
513 \item
514 Whether or not the alignments shall be cut according to an
515 annotated reference genome.
516 \item
517 (Optional) If the alignment shall be cut, BaitFisher requires
518 following information:
519
520
521 (a) The path and filename of the reference genome and the annotation
522 file.
523
524 (b) The name of the feature (spelled exactly as in the gff file e.g CDS)
525 for which baits shall be constructed.
526
527 (c) The name of the reference species with associated sequence in the
528 MSA and related reference genome.
529
530 (d) The name of the attribute field in the gff file that contains the
531 gene name (usually one of: parent, parent-ID, spelled exactly like in
532 the gff file).
533
534 (e) A character (field separator) by which the sequence names in the MSA
535 can be separated to extract the taxon name and the gene name. This is
536 required first, to find the sequence that corresponds to the reference
537 genome and second, to find for each gene MSA the corresponding
538 annotation in the gff file.
539
540 (f) The position at which the taxon name is found in the fasta sequence
541 name after splitting the name according to the field separator.
542
543 (g) The position at which the gene name is found in the fasta sequence
544 name after splitting the name according to the field separator.
545
546 \item (Optional) A string that contains required taxa. Baits will only be
547 constructed for potential bait regions in which all required taxa are
548 present.
549
550 \end{enumerate}
551
552 \textbf{BaitFisher program options:}
553
554 The BaitFisher parameter file is the place where all program options,
555 i.e. the above information has to be specified. The parameter file has
556 to be a plain text file. A word document or other non-pure text formats
557 cannot be read by BaitFisher. The parameter file is allowed to contain
558 ``comments''. Comments are indicated by the ``\#'' symbol. More
559 specifically, when reading the parameter file, BaitFisher ignores in
560 each line all text after the occurrence of the first ``\#'' symbol. The
561 text after the first ``\#'' can be used by the user to add his own
562 comments. The order in which the required or optional options are
563 specified is irrelevant. A fully specified parameter file can be found
564 in the example analysis.
565
566 \subsection{Required options:}
567
568 \textbf{directory-transcript-files}
569
570 \begin{quote}
571 Example: \verb+directory-transcript-files /Users/myname/data/baits/nt-data+
572
573 The path to the directory that contains all input MSA files. The path
574 can be an absolute path or relative to the current working directory. All input
575 files have to be aligned sequence files in the fasta format.
576 \end{quote}
577
578 \textbf{bait-length}
579
580 \begin{quote}
581 Example: \verb+bait-length 120+
582
583 The length of the baits that are constructed. A commonly used value is
584 120 bp. Some bait producing suppliers only produce baits of specific
585 lengths, e.g. 120 bp.
586 \end{quote}
587
588 \textbf{cluster-threshold}
589
590 Example : \verb+cluster-threshold 0.15+
591
592 In the process of clustering similar baits, BaitFisher creates cluster
593 with a maximum sequence distance specified by the cluster-threshold. The
594 bait sequence will be determined as an artificial sequence in the centre
595 of each cluster. So the maximum distance of the baits to the sequences
596 in the MSA should be roughly half the value specified by this option.
597
598 Typical values are 0.1 to 0.2, which corresponds to 10\%-20\% sequence
599 dissimilarly within clusters of sequences. If baits are constructed for
600 target sequences very similar to the sequences in the MSAs, this
601 parameter controls that bait-to-target distance.
602
603 \textbf{bait-offset}
604
605 \begin{quote}
606 Example: \verb+bait-offset 20+
607
608 The offset between starting coordinates of baits in tiling design. A
609 typical value is 20.
610 \end{quote}
611
612 \textbf{bait-number}
613
614 Example: \verb+bait-number 7+
615
616 The number of baits in the tiling design. Typical values are between 1
617 and 7.
618
619 \textbf{output-directory}
620
621 Example: \verb+output-directory /Users/myname/data/baits/bait-results+
622
623 Path to the directory BaitFisher will use for output files. The path can
624 be absolute or relative to the current working directory. BaitFisher
625 will write to this directory the loci- baits.txt (see Section XX for
626 more information). If alignment cutting is requested, BaitFisher will
627 also write all excised alignments to this directory in fasta format.
628 This directory has to be created by the user before starting BaitFisher.
629
630 \subsection{Optional options:}
631
632 \textbf{required-taxonomic-groups-string}
633
634 Example:
635
636 \verb+required-taxonomic-groups-string (Taxon1, Taxon2),(Taxon3)+
637
638 If the user wants to construct baits only for loci at which the MSA has
639 full sequence information for specific taxa, they can be specified here.
640 See Section VI for more information.
641
642 \textbf{sequence-name-field-delimiter}
643
644 \begin{quote}
645 Example: \verb+sequence-name-field-delimiter |+
646
647 If the user specifies required taxon names or if the alignment has to be
648 cut according to annotated features, the taxon, and for alignment
649 cutting also the gene name, has to be specified in each sequence name in
650 the input fasta file. More specifically, the sequence name has to be
651 separated by the sequence-name-field-delimiter into different parts.
652 One of these parts must be the taxon name, if alignment cutting is
653 requested, another part must contain the gene name. This option must be
654 a single character. This parameter is ignored if no required taxa are
655 specified and no alignment cutting is requested.
656 \end{quote}
657
658 \textbf{sequence-name-taxon-field\_number}
659
660 \begin{quote}
661 Example: \verb+sequence-name-taxon-field_number 2+
662
663 If either a required taxon string is specified, or if alignment cutting
664 is requested, BaitFisher will divide all sequence names of input files
665 according to the sequence-name-field-delimiter. This option specifies
666 the index of the field that contains the taxon name. The first index is
667 index 1. This parameter is ignored if no required taxa are specified and
668 no alignment cutting is requested. Default value: 2
669 \end{quote}
670
671 \textbf{sequence-name-geneID-field\_number}
672
673 \begin{quote}
674 Example: \verb+sequence-name-geneID-field_number 3+
675
676 If alignment cutting is requested, BaitFisher will divide all sequence
677 names of input files according to the sequence-name-field-delimiter.
678 This option specifies the index of the field that contains the gene
679 name. The first index is index 1. This parameter is ignored if no
680 alignment cutting is requested. Default value: 3
681 \end{quote}
682
683 \textbf{Hamming-center-sequence-computation-mode}
684
685 \begin{quote}
686 Example: \verb+Hamming-center-sequence-computation-mode heuristic+
687
688 BaitFisher implements an exact and a heuristic mode to compute the
689 one-center sequence. The exact mode is slightly better in finding the
690 artificial sequence that minimises the maximum distance to all sequences
691 in a cluster. For a large number of sequences in a cluster the
692 computation time can be enormous, so that only the heuristic mode can be
693 recommended. The heuristic mode is much faster and only slightly less
694 accurate. Allowed values: exact, heuristic. Default: heuristic.
695 \end{quote}
696
697 \textbf{verbosity}
698
699 \begin{quote}
700 Example: \verb+verbosity 3+
701
702 The integer value specifies how much runtime output BaitFisher writes to
703 the terminal (i.e. to standard output and standard error). If this value
704 is set to 0 BaitFisher runs in silent mode and only informs about
705 critical errors that force it to exit. With a value of 1, BaitFisher
706 also writes warning to the the standard output. With a value of 2
707 BaitFisher provides regular progress messages. All higher values are
708 mainly for debugging purposes. The output reaches a maximum level at a
709 value of 10000. Default: 5.
710 \end{quote}
711
712 \textbf{Options that have to be specified if alignment cutting was set
713 to yes. If no alignment cutting is requested, these options are
714 ignored.}
715
716 \textbf{reference-genome-file}
717
718 \begin{quote}
719 Example:
720
721 \verb+reference-genome-file /Users/myname/data/ref-genome/Apis.fas+
722
723 The filename of the fasta file that contains the genome assembly of the
724 reference genome. This option is only required if alignment cutting is
725 set to yes.
726 \end{quote}
727
728 \textbf{gff-file-reference-genome}
729
730 \begin{quote}
731 Example:
732
733 \verb+gff-file-reference-genome /Users/myname/data/ref-genome/Apis.gff+
734
735 Name of the gff file with annotation information for the reference
736 genome specified by the reference-genome-file option. This option is
737 only required if alignment cutting is set to yes.
738 \end{quote}
739
740 \textbf{reference-genome-name}
741
742 \begin{quote}
743 Example: \verb+reference-genome-name Apis_mellifera+
744
745 Taxon name of the reference genome. In order to be able to cut out
746 features, BaitFisher has to find a sequence with this taxon name in each
747 MSA it has been given to designs baits. MSAs which have no sequence that
748 contains this taxon name in the taxon name field of its sequence name
749 cannot be processed. A warning will written to the standard output, if
750 no sequence with this taxon name is found and if the verbosity
751 $\ge 2$.
752 \end{quote}
753
754 \textbf{Options that should be specified if alignment cutting was set to
755 yes. If no alignment cutting is requested, these options are ignored.}
756
757 \textbf{gff-feature-name}
758
759 \begin{quote}
760 Example: \verb+gff-feature-name CDS+
761
762 Name of the gff feature that shall be used for alignment cutting.
763 Typical names are: CDS, exon. Default: CDS. This name has to be spelled
764 and capitalised exactly as the name of the feature in the gff file.
765 \end{quote}
766
767 \textbf{gff-record-attribute-name-for-geneID}
768
769 \begin{quote}
770 Example: \verb+gff-record-attribute-name-for-geneID Parent+
771
772 Name of the gff attribute that specifies the geneID. Each line in the
773 gff file consist of 7/9 tab delimited fields. The last field contains
774 any number of attributes. For features such as CDS, exon, introns, etc,
775 the attribute string has to contains an attribute with the geneID.
776 Typically the name of this attribute is Parent. Default: parent. Apart
777 from capitalisation of this name, the name as to be identical to the one
778 used in the gff file. The geneID found in the gff file has to match the
779 geneID found in the gene name field of the input sequences.
780 \end{quote}
781
782 \textbf{remove-reference-sequence}
783
784 \begin{quote}
785 Example: \verb+remove-reference-sequence yes+
786
787 Allowed values: yes or no. If the sequences from the reference genome
788 are added to the MSAs only for the purpose to allow the alignment to be
789 cut into its features, it is usually not intended by the user to design
790 baits for the reference sequence. In this case the reference sequence
791 should be removed. If the reference sequence belongs naturally to the
792 sequences for which baits are designed, it should not be removed.
793 \end{quote}
794
795 \section{Starting a BaitFisher analysis on the command line}
796
797
798 If the BaitFisher program has not been placed into one of the
799 directories included in the system path, it has to be called with the
800 full path to where the binary file is located. If it is placed in the
801 system path, it can simply be called with its name. The only but
802 required command line parameter is the name of the parameter file.
803
804 BaitFisher-v1.2.7 name-of-parameter-file
805
806 Make sure that the output directory specified in the parameter file
807 exists, since it will not be created automatically.
808
809 \section{The output format}
810
811 In the directory specified in the parameter file, BaitFisher writes a
812 bait file with the standard name: baits-loci.txt. This file contains one
813 line of output for each bait region for which baits could be designed
814 successfully.
815
816 For each bait region BaitFisher produces the following output on one
817 line. Column numbers refer to the tab delimited fields the line of
818 output.
819
820 $1_{st}$ column: Name of alignment file this bait region
821 belongs to.
822
823 $2_{nd}$ column: Gene name in case of alignment cutting, or
824 the name of the alignment again.
825
826 $3_{rd}$ column: Feature number of requested kind (e.g.
827 CDS) in case of alignment cutting, or 0 otherwise.
828
829 $4_{th}$ column: Maximum number of features of requested
830 kind (e.g. CDS) in case of alignment cutting, or 0 otherwise.
831
832 $5_{th}$ column: Starting coordinate of bait region in the
833 alignment
834
835 $6_{th}$ column: Number of sequences which where fully
836 available in the bait region. Note: If you requested a tiling design
837 with just one bait per bait region, the maximum number can be the number
838 of sequences. If the number is lower than the number of sequences, some
839 sequences contain gaps or ambiguous characters in this bait region. If a
840 tiling design was requested that has more successive baits per bait
841 region, say N, the number of sequences which where available will have
842 an upper limit of N*number\_of\_ sequences\_in\_the\_MSA. Again, the
843 maximum value N*number\_of\_sequences\_in\_the\_MSA will be reached if
844 all sequences have no gaps or ambiguous characters.
845
846 This column can be interpreted as the amount of evidence the baits in
847 the bait region are based on. If a large alignment contains only one
848 valid sequence in a certain region of the MSA, BaitFisher will still
849 construct baits for this region, unless required taxa are missing. After
850 bait design, regions with insufficient information content for the bait
851 design can only be spotted by looking for suspiciously small numbers of
852 sequences in this column.
853
854 $7_{th}$ column: Number of baits required for the full
855 tiling design in this bait region. An upper limit for this number is the
856 number of sequences from column 6. The greater the difference is between
857 column 7 and 6, the more the user can benefit from clustering sequences
858 and computing only one center sequence per cluster.
859
860 $8_{th}$ column: Baits for this bait region. This last
861 field describing the bait region contains several bits of information:
862 For each bait is first shows the ``stack number''. For a tiling design
863 with N successive baits we have N successive bait windows, in the
864 following called bait stacks. The numbers are integer values in the
865 range 1 \ldots{} N. Followed by the stack number comes the bait
866 sequence. After each bait sequence there are two real numbers:
867 The CG content of the bait and the maximum distance of the bait to the
868 sequences in its cluster. Multiple baits are separated by commas. The
869 number of baits that will be found per line is equal to the number
870 specified in column 7.
871
872 Obviously, with one bait region per possible starting position,
873 BaitFisher produces a highly redundant output. The reason is that some
874 starting positions might be more efficient in terms of ``required number
875 of baits'' or ``number of sequences the result is based on''. The
876 redundant output allows the user to choose for each locus of interest
877 the optimal bait region. This topic is addressed in Section XI.
878
879 \section{Example analyses}
880
881 The BaitFisher package includes two example analyses, one with and one
882 without alignment cutting.
883 The example with alignment cutting is also an example in which a required taxa string
884 has been specified.
885 Both examples are included in the BaitFisher package, in the
886 Example folder. The example with
887 alignment cutting requires a full genome as well as a gff file.
888 Since they are too large, we included reduced files and provide download links for the
889 original files.
890
891 \subsection{Example without alignment cutting:}
892
893 In order to run this example analysis, compile and install BaitFisher as described above.
894 On the command line, change to the
895 Example/Example-without-alignment-cutting folder. In this folder you
896 find the parameter.txt file which contains all BaitFisher options for
897 this example run. In the alignment folder you find three gene MSAs for
898 which baits shall be designed.
899 In order to run the analysis type
900 run-example.sh on the command line.
901 This requires that the BaitFisher program can be found in the relative path
902 ../../BaitFisher-v1.2.7. If the executable has been moved somewhere else, either modify the
903 command in the run-example.sh file or recreate a copy of the executable in the main BaitFisher folder.
904
905 The example analysis should finish within a few seconds.
906
907 This run produces a single output file in the folder BaitFisher-results.
908 The first few lines in this output file, which result from processing the input file
909 EOG5DV4JW.fas look as follows:
910
911 EOG5DV4JW.fas EOG5DV4JW.fas 0 0 1 173 19 1 ATG \dots
912
913 EOG5DV4JW.fas EOG5DV4JW.fas 0 0 2 173 18 1 TGG \dots
914
915 EOG5DV4JW.fas EOG5DV4JW.fas 0 0 3 173 19 1 GGC \dots
916
917 EOG5DV4JW.fas EOG5DV4JW.fas 0 0 4 173 19 1 GCT \dots
918 \ldots
919
920 A full description of the tab-delimited columns is given above. Since
921 the columns 3 and 4 only play a role if alignment cutting is performed,
922 the columns are filled with zeros in this example. Column 5 contains the
923 starting position of the bait region in the alignment. We see that
924 BaitFisher computes baits for each possible starting position of the bait region.
925 Column 6 is the number of complete sequences the baits in this bait region are based on.
926 We observe that all starting positions have the same sequence coverage in the bait
927 region. (See Section \ref{} for details.) %%%XXXXXXXXXXXXXXXXXXXX
928 Column 7 shows the number of required baits for this bait
929 region. The smallest number of required baits is found for starting position 2.
930 In this example we have chosen a rather conserved gene, with the
931 result that BaitFisher can drastically reduce the number of baits
932 required in comparison to the number of baits required if baits are
933 constructed for each sequence. The proportion of baits we can save here
934 is is: (1-18/173)*100\% = 89.6\%.
935
936 \subsection{Example with alignment cutting:}
937
938 \label{label_Example_alignment_cutting}
939
940 On the command line, change to the directory\\
941 \verb+Example/Example-with-alignment-cutting-and-required-taxa+.
942
943 In principle a BaitFisher analysis with alignment cutting requires a whole genome and the corresponding annotation file.
944 Since we will only analyse five transcript files containing five genes in this example, we
945 reduced the whole genome file to the scaffolds required in this example and the corresponding annotation file
946 to the annotation corresponding to the genes in this example. This reduction allows us to make this example
947 to be included in the git-hub repository.
948
949
950 The reference genome we use in this example is the {\it Nasinia vitripennis} genome.
951 The reduced genome can be found in the Nvit-genome folder of this example. The reduced genome
952 is a modified version of the whole genome which can be downloaded together with the annotation file
953 here
954 \footnote{\url{http://www.hymenopteragenome.org/drupal/sites/hymenopteragenome.org.nasonia/files/data/Nvit_1.0_scaffolds.fa.gz} and
955 \url{http://hymenopteragenome.org/nasonia/nasonia_genome_consortium/data/Nvit_OGSv1.2.gff2.gz}}.
956
957 The annotation file has to be in the gff-format and has to correspond to the fasta file.
958 To get an idea how the annotation file looks like the first few lines are shown:
959
960 {
961 \small
962 \begin{verbatim}
963 SCAFFOLD4 OGSv12_RefSeq transcript 3975063 3977809 . - . GenePrediction NV11525-RA
964 SCAFFOLD4 OGSv12_RefSeq exon 3975063 3976592 . - . GenePrediction NV11525-RA
965 SCAFFOLD4 OGSv12_RefSeq exon 3977689 3977809 . - . GenePrediction NV11525-RA
966 SCAFFOLD4 OGSv12_RefSeq mRNA 3975384 3977706 . - . GenePrediction NV11525-RA
967 SCAFFOLD4 OGSv12_RefSeq CDS 3975384 3976592 . - 0 GenePrediction NV11525-RA
968 SCAFFOLD4 OGSv12_RefSeq CDS 3977689 3977706 . - 0 GenePrediction NV11525-RA
969 \end{verbatim}
970 }
971
972 This gff-file is always tab delimited and contains nine columns. The last column can contain several key/value pairs.
973 A key and its value can be separated by a space or the ``='' symbol. Several key
974 value pairs are separated by semi-colons. In this example the last and nineth
975 column consists of a single key-value pair given by the string ``GenePrediction gene-name'',
976 where gene-name is the name of the gene in the official gene set. Note that in the gff-file extract above
977 the tabs between the columns and the space in the last column cannot be distinguished.\\
978
979 {\bf In brief:}\\
980 There are two important correspondences between the gff file and the remaining data.
981 Both have to be fulfilled to make alignment cutting into features possible. \\
982
983 (i) The sequence names in the genomic fasta file have to correspond to the names given
984 in the first column of the gff file. Furthermore, the coordinates in the fourth and fifth column
985 of the gff file have to correspond to the coordinates of the gene in the fasta file.
986 This correspondence should be fulfilled if the two files are obtained from the same source
987 and are based on the same assembly and annotation. Annotation versions and assembly versions can differ
988 such as in the case of the Nasonia genome.\\
989
990 (ii) The second correspondence is a bit more tricky. The gene name found in the gff file,
991 in the above example this is {\it NV11525-RA}, has to be encoded in the sequence name of the reference
992 species in the transcript files.
993
994 {\bf In detail:}\\
995 While the correspondence (i) is usually fulfilled if the sequences are downloaded from the same source,
996 the second correspondence needs more explanation:
997
998 BaitFisher has to know which transcript or gene files corresponds to which
999 gene names in the gff file. In order to make this connection, the transcript files have to contain
1000 a sequence originating from the reference species, in our example from {\it Nasonia}.
1001 This sequence has to have a sequence name in which
1002 the gene name is encoded in a way, so that it can be extracted by BaitFisher.\\
1003
1004 Let us give an example:
1005 One of the transcript files in the alignments folder is the file:\\
1006 \verb+EOG5B2RVS.linsi.nt_without_bees_ants_rmgo.fas+.
1007
1008 This looks as follows:
1009 \begin{verbatim}
1010 >EOG5B2RVS|HSALT_3.3|Nysson_niger|C277098-531
1011 ATG...
1012 >EOG5B2RVS|AECHI_3.8|Dinetus_pictus|scaffold1253-531
1013 ATG...
1014 ...
1015 >EOG5B2RVS|NVITR_1.2|Nasonia_vitripennis|NV16710-RA
1016 ATG...
1017 >EOG5B2RVS|LHUMI|Sphecius_convallis|C324842-531
1018 ATG...
1019 >EOG5B2RVS|LHUMI|Stizoides_tridentatus|C357502-531
1020 ATG...
1021 >EOG5B2RVS|AECHI_3.8|Psenulus_fuscipennis|C384528-531
1022 ATG...
1023 \end{verbatim}
1024
1025 Note that each sequence name is delimited by the ``$|$'' character into 4 fields.
1026 Important fields are the third and fourth field in this example: The third field contains
1027 the species name and the fourth field contains the gene name for species for which a whole genome is
1028 available.
1029
1030 Alignment cutting is possible if one of the sequences has the name of the reference species
1031 in the third field. In our example this is the name \verb+Nasonia_vitripennis+.
1032 This sequence name has to have a gene name in the fourth field which corresponds to one of the
1033 genes found in the gff file. In this example this is the gene name \verb+NV16710-RA+.
1034
1035 Without these correspondences, BaitFisher would not know how to find the features in the gene files
1036 and how to excise them. Further down we will discuss some approaches which allow us to obtain transcript files with the
1037 appropriate sequence names.
1038
1039 For the moment we assume we have corresponding transcript or gene files, a reference genome and
1040 the annotation file and we will have a look at the parameters which have to be set in the
1041 parameter file. The following options specified in the parameter file of this example analysis
1042 are all directly related to the alignment cutting:\\
1043
1044 This option switches alignment cutting on:\\
1045 \verb+alignment-cutting yes+\\
1046
1047 A sequence of the reference species has to be included in every transcript or gene alignment.
1048 The reference species might not be of further interest for the project and one might not want
1049 to design baits for the reference species. In this case one can tell BaitFisher
1050 to remove the reference species from the alignment before the bait design procedure.
1051 If the reference species shall be removed, BaitFisher will also remove gap only columns
1052 from the alignment, which can arise if the presence of the reference species lead to gaps in all
1053 other sequences. Removing gap only columns can change to coordinates in the transcript or gene
1054 alignments.
1055 The user will find the final feature alignments for which baits have been designed in the specified
1056 output folder.
1057
1058 With this command, BaitFisher removes the reference species from all transcript alignments.\\
1059 \verb+remove-reference-sequence yes+.
1060
1061 The use might have transcript alignments in which the sequence name and the gene name are encoded
1062 but maybe with a different field delimiter or in different fields.
1063
1064 The following parameters specify the field delimiter and the field numbers in the parameter file:
1065
1066 \begin{verbatim}
1067 sequence-name-field-delimiter | # Default: "|". Must be a single character
1068 sequence-name-taxon-field_number 3 # Default: 2
1069 sequence-name-geneID-field_number 4 # Default: 3
1070 \end{verbatim}
1071
1072 GFF files tend to look very different. Different genome projects unfortunately use different names for
1073 the same things. BaitFisher e.g.\ has to find the gene name in the gff file.
1074 As described above, this gene name can be found in the ninth column of the gff file, where it occurs
1075 as in key value pair. In our example the key is \verb+GenePrediction+.
1076 In other projects the key for the gene name can be
1077 different. Examples are \verb+name+ or \verb+ParentID+.
1078 The key word after which the gene name can be found
1079 can be specified in the parameter file. In this example:\\
1080 \verb+gff-record-attribute-name-for-geneID GenePrediction+\\
1081
1082 The user might want to excise different features. In the gff file, the feature is given in the third column.
1083 Reasonable features for our purpose are \verb+exon+ or \verb+CDS+, since only these two are
1084 generally annotated and are guaranteed to yield sequences that are continuous in the genome.
1085 The name of the features that shall be excised can be specified in the parameter file.\\
1086 \verb+gff-feature-name CDS+\\
1087
1088 Finally, BaitFisher has to know were to find the genome file and the gff file.
1089 Furthermore, it has to know the name of the reference species in order to identify the
1090 reference sequence in the transcript or gene alignments.
1091
1092 These three parameters are specified in this example as follows:
1093 \begin{verbatim}
1094 gff-file-reference-genome Nvit-genome/Nvit_OGSv1.2_reduced.gff2
1095 reference-genome-file Nvit-genome/Nvit_1.0_scaffolds_reduced.fa
1096 reference-genome-name Nasonia_vitripennis
1097 \end{verbatim}
1098
1099 The algorithm of BaitFisher can be outlined as follows:\\
1100 (i) Read every transcript or gene alignment into memory.
1101 (ii) Determine the reference sequence by its name.
1102 (iii) Determine the gene name specified in the sequence name of the reference species.
1103 Look up the gene name in the gff file and extract all features which have this gene name
1104 and which are of the requested feature type (e.g. exon).
1105 (iv) For each feature found in the gff file in step (iii) excise the genomic nucleotide sequence of
1106 the feature of the reference species from the genome file using the sequence name
1107 (``SCAFFOLD4'' in the above example) and the
1108 coordinates of the feature in the genome file. Align the feature sequence that has been excised
1109 to the sequence of the reference species in the transcript or gene alignment. The feature should be identical to part of the transcript alignment.
1110 The start and end coordinates of the feature in the reference sequence
1111 will tell us where the feature starts and ends in the transcript alignment.
1112 With these coordinates, BaitFisher excises the feature from the transcript/gene alignment.
1113 If the user requested to remove the reference species from the alignments,
1114 it will be removed. All gap only columns resulting from this step will be removed as well.
1115 If the feature alignment is long enough to host a full bait region, the bait design will be started.\\
1116
1117 {\bf Obtaining transcript or gene alignments with sequence names which encode the species names and the gene names:}\\
1118 The most difficult part in the alignment cutting mode is to obtain files with corresponding names.
1119 If transcripts are put together with the Orthograph software, the sequence names should be in the format
1120 required by BaitFisher. Otherwise script have to be written to modify in the input files.
1121
1122
1123
1124 \section{BaitFilter and how to proceed after obtaining the BaitFisher output file}
1125
1126 After BaitFisher has designed baits for all possible starting positions,
1127 we have to do some post processing of the output file. This post processing
1128 can be done with the BaitFilter program or with custom scripts written by the user.
1129
1130 \textbf{BaitFilter} is a command line program that has the following
1131 command line options. Some options are required and cannot be omitted.
1132 These options are indicated below.
1133
1134 \textbf{-h, -\/-help}
1135
1136 Displays usage information and exits.
1137
1138 \textbf{-\/-version}
1139
1140 Displays version information and exits.
1141
1142 \textbf{-\/-, -\/-ignore\_rest}
1143
1144 Ignores the rest of the labeled arguments following this flag.
1145
1146 \textbf{-i \textless string\textgreater , -\/-input-bait-file-name
1147 \textless string\textgreater }
1148
1149 (required) Name of the input bait locus file. This is the bait file
1150
1151 obtained from the BaitFisher program.
1152
1153 \textbf{-o \textless string\textgreater , -\/-output-bait-file-name
1154 \textless string\textgreater }
1155
1156 (required) Name of the output bait file. This is the file the
1157
1158 contains the filtered bait loci and the baits.
1159
1160 \textbf{-c \textless string\textgreater , -\/-convert
1161 \textless string\textgreater }
1162
1163 \begin{quote}
1164 Allows the user to produce the final output file for the bait producing
1165 company. In this mode, BaitFilter reads the input bait file and instead
1166 of doing a filtering step, it produces a costume bait file that can be
1167 uploaded to the baits producing company. In order to avoid confusion, a
1168 filtering step cannot be done in the same run as the conversion. If you
1169 want to filter a bait file and convert the output, you will need to call
1170 this program twice, first to do the filtering and second to do the
1171 conversion. Allowed conversion parameters currently are: ``four-column-upload'',
1172 which is accepted by several companies. The original format name for the
1173 ``four-column-upload'' format was ``Agilent-Germany'', which is still a silently accepted
1174 format name.
1175
1176 The four-column-upload format contains information potentially of interest for the researcher.
1177 The output contains four columns delimited by tabs. The four columns are: TargetID, ProbeID, Sequence, and Replication.
1178 The TargetID and the ProbeID are automatically generated and contain information about the bait locus.
1179 The information differs slightly in the two possible cases: alignment cutting is used or not used.\\
1180
1181 {\bf Example with alignment cutting:}\\
1182
1183 If alignment cutting has been used, the output file consists of lines in the following format:\\
1184 {
1185 \tiny
1186 \verb+GB42203-PA_9 Project1_GB42203-PA_9_11_1_1_1 AACAGTTTACGCACTCCTGATTTAACTTGGGAAAAAGTAAGATCTCAAGTGGACCATGTAATCTGGCCAGATGGAAAGCGT+\\[-1mm]
1187 \verb+ATCGTCCTCCTAGCAGAAGGTCGTTTGGTCAACTTATCT 1+
1188 }
1189
1190 If alignment cutting is used, the TargetID contains the gene name, in this example ``GB42203-PA'', followed by \verb+_9+, where
1191 the number is number of the feature. In this case it is the ninth CDS region of the GB42203-PA gene.
1192
1193 %os << gene_name << "_" << feature_num << '\t'
1194 % << probeIDprefix
1195 % << gene_name << "_"
1196 % << feature_num << "_"
1197 % << num_features << "_"
1198 % << start << "_"
1199 % << t_index << "_"
1200 % << count << '\t'
1201 % << (**i1).get_bait()
1202 % << "\t1" << std::endl;
1203
1204 If a probe ID-prefix is specified when calling BaitFilter, this string is prepended to all probeIDs in the
1205 output file. In the above example, the specified probe ID-prefix was \verb+Project1_+.
1206 If alignment cutting is used, the probe ID-prefix is followed by the gene name, which in turn is followed by the string
1207 \verb+_9_11_1_1_1+.
1208 The numbers separated by underscores are:
1209 \begin{itemize}
1210 \item the number of the feature the bait is in (here the 9th CDS region),
1211 \item the total number of features in this gene (here the gene contains 11 CDS regions),
1212 \item the start coordinate within the feature (here it is base 1),
1213 \item the tiling index, i.e. the index of the bait window within the tiling design (here it is bait window 1).
1214 \item the number of the bait in the bait window (here this is the 1st bait in this bait window).
1215 \end{itemize}
1216
1217 After the ProbeID comes the bait string, which is followed by the replication number which is always set to one
1218 by BaitFilter.
1219
1220 {\bf Example without alignment cutting:}\\
1221 If no alignment cutting has been used, the output file consists of lines in the following format:\\
1222 {
1223 \tiny
1224 \verb+file1.fas_0 Project2_file1.fas_0_0_45_2_3 ATTTTTGCCTTATGAGCAGGTATACTAGGAACCTCTCTCAGAATATTAATCCGATTAGAACTAAGAAATCCTGGATCTTGAATTAA+\\[-1mm]
1225 \verb+CAATGATCAAATCTACAATTCCACTATTACTGCT 1+\\
1226 }
1227
1228 In this case, the TargetID is the sequence name followed by \verb+_0+, where the \verb+_0+ is appended only for consistency reasons.
1229 If the user wants to replace the TargeID with a different string, this has to be done in a text editor.
1230
1231 The ProbeID is composed of the probe ID-prefix, the file name and the string \verb+_0_0_45_2_3+. Since a feature number and the number of features in this alignment
1232 is not applicable if no alignment cutting is performed, these two numbers are 0. They are part of the output only for consistency reasons.
1233 The remaining numbers have the same
1234 meaning as above. In this example, the bait starts at position 45 of the alignment, the bait is in the 2nd bait window of the tiling design. Within this bait window,
1235 this is the 3rd bait.
1236
1237 The \verb+ID-prefix+ option and its purpose are explained in more detail below.\\
1238
1239
1240 New output formats can be added upon request. Please contact the
1241 author Christoph Mayer, Email:
1242 \href{mailto:[email protected]}{\emph{[email protected]}}.
1243 \end{quote}
1244
1245 \textbf{-S}
1246
1247 \begin{quote}
1248 Do not filter any baits but only compute stats for the input file. No
1249 output file needs to specified and if an output file is specified, it is
1250 ignored. In all filter modes this status option in turned on
1251 automatically. This option is useful if stats, but no analysis is
1252 needed.
1253 \end{quote}
1254
1255 \textbf{-m \textless string\textgreater , -\/-mode
1256 \textless string\textgreater }
1257
1258 \begin{quote}
1259 Appart form the input file this is the most essential option. This
1260 option specifies which filter mode BaitFilter uses. The following modes
1261 are available.
1262
1263 "ab": Retain only the best bait locus for each alignment file (e.g.
1264 gene) when using the optimality criterion to minimize the total number
1265 of required baits.
1266
1267 "as": Retain only the best bait locus for each alignment file (e.g.
1268 gene) when using the optimality criterion to maximize the number of
1269 sequences the result is based on.
1270
1271 "fb": Retain only the best bait locus for each feature (e.g. CDS) when
1272 using the optimality criterion to minimize the total number of required
1273 baits. This mode can only be used if alignment cutting has been used in
1274 BaitFisher.
1275
1276 "fs": Retain only the best bait locus for each feature (e.g. CDS) when
1277 using the optimality criterion to maximize the number of sequences the
1278 result is based on. This mode can only be used if alignment cutting has
1279 been used in BaitFisher.
1280
1281 "blast-a": Remove all bait loci of ALIGNMENTs for which one or more
1282 baits have multiple good hits to a reference genome.
1283
1284 "blast-f": Remove all bait loci of FEATUREs for which one or more baits
1285 have multiple good hits to a reference genome.
1286
1287 "blast-l": Remove bait LOCI that contain a bait that hos multiple good
1288 hits to a reference genome.
1289
1290 "blast-c": \label{label-coverage-filter} Conduct a blast of the baits against a reference genome and do a coverage filter of bait regions.
1291 Requires the \textbf{-\/-blast-min-hit-coverage-of-baits-in-tiling-stack} option to be specified.
1292 See the description of the \textbf{-\/-blast-min-hit-coverage-of-baits-in-tiling-stack} for details.
1293
1294 "thin-b": Thin out a bait file to every Nth bait region, by determining the
1295 start position that minimizes the number of baits.
1296
1297 "thin-s": Thin out a bait file to every Nth bait region, by determining the
1298 start position that maximizes the number of sequences.
1299
1300 \textbf{-\/-blast-second-hit-evalue \textless real number\textgreater }
1301
1302 Maximum evalue for the second hit. A bait is characterised to bind
1303 ambiguously, if we have two good hits. This option is the e-value
1304 threshold for the second hit.
1305
1306 \textbf{-\/-blast-first-hit-evalue \textless real number\textgreater }
1307
1308 Maximum e-value for the first hit. A bait is characterised to bind
1309 ambiguously, if we have two good hits. This option is the e-value
1310 threshold for the first hit.
1311
1312 \textbf{-\/-blast-min-hit-coverage-of-baits-in-tiling-stack \textless real number\textgreater }
1313
1314 Specifies the minimum hit coverage for the best blast hit of the baits of a tiling stack against a reference genome.
1315 If one tiling stack of a bait region
1316 fails to have a bait with this minimum hit coverage, the whole bait region
1317 is discarded.
1318 The hit coverage is defined as the hit length of the blast hit divided by the length of the bait.
1319 This option is required in the mode "blast-c". If this option is not specified in the modes blast-l, blast-a, blast-f
1320 no hit coverage is checked.
1321
1322 Example: \verb+--blast-min-hit-coverage-of-baits-in-tiling-stack 0.95+
1323
1324 This requires that one bait has to exist in each tiling stack which has
1325 a hit to the reference genome where the hit covers 95\% of the bait. If
1326 the required baits do not exist, the bait region is discarded.
1327
1328 This parameter can only be used in the modes blast-c, blast-l, blast-f and blast-a. Since
1329 the order in which the coverage filter and the other filters are applied
1330 matters, the order is defined as follows:
1331
1332 For the mode options: blast-a, blast-f, blast-l the hit coverage is
1333 checked \emph{after} filtering for baits with multiple good hits to the
1334 reference genome. For the mode option blast-c only a coverage filter is being carried out.
1335
1336 \textbf{-\/-ref-blast-db \textless string\textgreater }
1337
1338 Base name to a blast data base file. This name is passed to the blast
1339 command. This is the name of the fasta file of your reference genome.
1340
1341 IMPORTANT: The makeblastdb program has to be called before starting the
1342 BaitFilter program. makeblastdb takes the fasta file and creates data
1343 base files out of it.
1344
1345 \textbf{ -b \textless string\textgreater, -\/-blast-result-file \textless string\textgreater }
1346
1347 Conducting a blast analysis of all baits against a reference genome can take a long time.
1348 If different filtering parameters, e.g. different coverage thresholds, are to be compared, the
1349 same blast analysis has to be done multiple times. With this argument, the
1350 blast analysis will be skipped and the specified blast result file will be
1351 used. This option has to be used with caution! No checks are done (so
1352 far) to ensure that the blast result file corresponds to the specified
1353 bait file. If a BaitFilter run was conducted which did a blast search,
1354 BaitFilter will not delete the blast result file after the run was
1355 completed. The result file with the name \verb+blast_result.txt+ will remain
1356 in the working directory. It can be moved or renamed and with this option
1357 it can be specified as the input file for further BaitFilter runs. If
1358 you have the slightest doubt whether you are using the correct blast
1359 result file, you should not use this option. This option is only
1360 allowed in modes that would normally do a blast search. This option
1361 cannot be specified together with the blast-executable,
1362 blast-evalue-cutoff, blast-extra-commandline, ref-blast-db options,
1363 since these are options specific to runs in which a blast search is
1364 conducted.
1365
1366
1367 \textbf{-\/-blast-extra-commandline \textless string\textgreater }
1368
1369 When invoking the blast command, extra command line parameters can be
1370 passed to the blast command. As an example with this option it is
1371 possible to specify the number of threads the blast command should use.
1372
1373 \textbf{-\/-blast-evalue-cutoff \textless real number\textgreater }
1374
1375 When conducting a blast search, a maximum evalue can be specified when
1376 calling the blast program. The effect is that hits with a higher
1377 evalue are not reported. BaitFilter always specifies such an evalue
1378 when calling the blast program. The default evalue passed by
1379 BaitFilter to the blast program is twice the
1380 --blast-second-hit-evalue. If a coverage filter is requested the
1381 default value is set to 0.001 if twice the value of
1382 --blast-second-hit-evalue is smaller than 0.001. This should guarantee
1383 that all hits necessary for the blast and/or coverage filter are
1384 found. If the user wants to set a different evalue threshold, this can
1385 be specified with this option. With version 1.0.6 of this program, the
1386 value is automatically changed to be larger or equal to 0.001 if the
1387 coverage filter is used. This makes the usage of this option unnecessary
1388 in most cases.
1389
1390 \textbf{-B \textless string\textgreater , -\/-blast-executable
1391 \textless string\textgreater }
1392
1393 Name of or path+name to the blast executable. Default: blastn. Minimum
1394 blast version number: Blast+ 2.2.x
1395
1396 \textbf{-t \textless positive integer\textgreater ,
1397 -\/-thinning-step-width \textless positive integer\textgreater }
1398
1399 Thin out the bait file by retaining only every Nth bait region. This
1400 option specified the step width N. If one of the modes thin-b, thin-s is
1401 active, this option is required, otherwise it is not allowed to set this
1402 parameter.
1403
1404 \textbf{-\/-ID-prefix \textless string\textgreater }
1405
1406 This option can only be specified in the \verb+-c+ or \verb+--convert+ mode,
1407 in which bait files in the BaitFisher format
1408 are converted to files that can be uploaded to bait composing companies.
1409
1410 When uploading bait files to bait producing companies,
1411 the ProbeID, i.e.\ the ID in the second column of the ``four-column-upload'' format
1412 has to be unique, even for different projects of the same user.
1413
1414 Since the gene name and position numbers might be identical in different projects,
1415 we introduced this option to allow the user to make ProbeIDs unique over
1416 different projects.
1417
1418 With this option the user is able to specify a
1419 prefix string for all probe IDs in the output file in the four-column-upload format.
1420 This prefix string is prepended to the ProbeID which is automatically generated by BaitFiler.
1421
1422 \textbf{-\/-verbosity \textless unsigned integer\textgreater }
1423
1424 The verbosity option controls the amount of information BaitFilter
1425 writes to the console while running. 0: report only error messages that
1426 lead to exiting the program.
1427 1: report also warnings, 2: report also progress, 3: report more
1428 detailed progress, \textgreater 10: debug output.10000: maximum
1429 possible diagnostic output. Default: 2.
1430 \end{quote}
1431
1432 Example applications:
1433
1434 \begin{enumerate}
1435 \def\labelenumi{\roman{enumi}.}
1436 \item
1437 We might want to search for baits that bind unspecific. This can
1438 be checked with the BaitFilter program, which in turn does a Blast
1439 search of all baits against a reference genome. If a bait has two or
1440 more good hits to different positions of the genome assembly, and if
1441 this is not an assembly artefact, there is some chance that the bait
1442 enriches two or more different loci. The user can choose to remove the
1443 bait region, the feature, the gene/alignment the bait belongs to. We
1444 propose to remove at least the bait-region in case one of its baits
1445 has a good chance to bind unspecific.
1446
1447 The command used to conduct this analysis is the following:
1448
1449 \begin{verbatim}
1450 BaitFilter-v1.0.6 -i loci_baits.txt
1451 --blast-first-hit-evalue 0.00000001
1452 --blast-second-hit-evalue 0.00001
1453 --ref-blast-db my-blast-db_of_reference_genome
1454 -o baits_filtered_by_blast-l
1455 -m blast-l
1456 --blast-extra-commandline "-num_threads 12"
1457 1> o_filter_blast.log 2> e_filter_blast.log
1458 \end{verbatim}
1459
1460 This command conducts a blast filter run with mode ``blast-l'', i.e.
1461 bait regions that have at least one bait with a blast hit e-value smaller than
1462 0.00000001 and a second good blast hit with an e-value smaller than 0.00001 are
1463 removed from the bait-file, since the bait is expected not to bind specifically.
1464 The blast program must with version 2.2.25+ or
1465 higher must be installed in the system path for this to work. If the
1466 Blast program is not in the system path, the full path can be specified
1467 with the --B option. The user has to invoke makeblastdb to make a blast
1468 data-base of the reference genome prior to starting BaitFilter. In this
1469 example the console output is redirect to two files. Namely, the
1470 standard output is redirect to the file \verb+o_filter_blast.log+ and the standard
1471 error is redirected to the file \verb+e_filter_blast.log+.
1472
1473 \item
1474 We might want to ensure that at least one bait per tiling stack maps in full
1475 length to a given reference genome. This check can be done if on the one hand we are not
1476 sure that the MSAs we use for bait design have indeed continuous gDNA and that alignment cutting
1477 cannot be done.
1478 The hit coverage can only be checked in conjunction with another blast filter mode. See
1479 comman line option above.
1480
1481 Example:
1482
1483 \begin{verbatim}
1484 BaitFilter-v1.0.6 --blast-min-hit-of-bait 0.85
1485 -i loci_baits.txt
1486 --blast-first-hit-evalue 0.00000001
1487 --blast-second-hit-evalue 0.00001
1488 --ref-blast-db my-blast-db_of_reference_genome
1489 -o baits_filtered_blas
1490 -m blast-l
1491 --blast-extra-commandline "-num_threads 12"
1492 1> o_filter_blast.log 2> e_filter_blast.log
1493 \end{verbatim}
1494
1495 In this example we apply two filter in one call to the BaitFilter program.
1496 BaitFilter first checks for multiple hits. Only bait regions in which all baits bind specifically
1497 are passed to the second filter in which baits
1498 are checked for a minimum hit coverage.
1499
1500 \item
1501 After applying the above filters, it is likely that we still have multiple bait regions
1502 per alignment or per feature and we probably want to reduce the number of bait regions to
1503 one per feature or one per gene or alignment.
1504 BaitFilter has two optimality criteria: Minimize the number of required baits in the
1505 target region or maximize the number of sequences used to design the baits.
1506
1507 Example:
1508 \begin{verbatim}
1509 BaitFilter-v1.0.6 -m fb -i baits_file -o baits_filtered
1510 1> o_BlastFilter 2> e_BlastFilter
1511 \end{verbatim}
1512
1513 This command only retains the best bait region per feature using the
1514 number of baits as the optimality criterion. As above, the
1515 console-output is redirected to two files.
1516
1517 \item With BaitFilter version 1.0.6 a new filter mode has been introduced.
1518 This mode allows you to keep not only one but multiple bait regions per feature, or alignment, if the
1519 length of the feature or alignment is long enough so that two or more bait regions can be situated
1520 in the alignment with a minimum distance.
1521
1522 Example:
1523
1524 Will be added soon.
1525
1526 \end{enumerate}
1527
1528 \section{BaitFilter example analyses}
1529
1530 BaitFilter example analyses are available for the two BaitFisher example
1531 analyses introduced above.
1532
1533 \textbf{Example without reference genome:}
1534
1535 You can run this example if BaitFilter has been compiled and if the
1536 BaitFilter example analysis in the folder
1537 Example/Example-without-alignment-cutting has been completed. To conduct
1538 the BaitFilter analysis, change on the command line from the
1539 Example/Example-without-alignment-cutting folder to the
1540 BaitFilter-analysis folder and type ./run-BaitFilter on the command line.
1541 This will produce four bait-files and eight log files. The first two
1542 output files are ``intermediate'' bait files. One of these contains the
1543 best bait regions when minimizing the number of baits and one contains
1544 the best bait regions when maximizing the number of sequences the baits
1545 are constructed from. Since we analysed three alignment files, the
1546 output files will contain baits for three bait regions, one for each of
1547 the alignment files.
1548
1549 The information BaitFilter write to standard error for the analysis
1550 finding the bait region with the smallest number of required baits
1551 reads:
1552
1553 \begin{verbatim}
1554 After determining the best bait region per alignment/gene under the criterion to
1555 minimize the number of baits we have
1556 Bait regions have been determined for this number of different:
1557 Alignment files 3
1558 Feature (is identical to number of files if not applicable: 3
1559 Bait regions: 3
1560 Total number of baits: 187
1561 Total number of sequences at loci: 479
1562 Proportion of baits saved: 60.96%
1563 Total time used: 0:0:2
1564 Your BaitFilter analysis finished successfully.
1565 \end{verbatim}
1566
1567 After producing the intermediate bait files, BaitFilter is invoked again
1568 to convert the intermediate bait file into a format that can be uploaded
1569 at bait producing companies.
1570
1571 BaitFiter again produces 2 log files in each run and the specified
1572 output file.
1573
1574 \textbf{Example with reference genome:}
1575
1576 Will be added soon.
1577
1578 \section{FAQ}
1579
1580 \begin{enumerate}
1581 \def\labelenumi{(\arabic{enumi})}
1582 \item
1583 \textbf{After sequencing the enriched library, the sequences will be
1584 assembled. This assembly will contain the whole or a part of the bait
1585 region. Can the regions be used in further analyses, e.g. phylogenetic
1586 analyses, or does it have to be removed in the same way a primer sites has to
1587 be removed when using the Sanger sequencing technique? }
1588 \end{enumerate}
1589
1590 \begin{quote}
1591 Bait regions do not have to be removed from the data set. In the case of
1592 primer regions in the Sanger sequencing method, it was more likely that
1593 the output sequence contains the exact primer sequence than the
1594 potentially different species sequence. Therefore primer regions had to
1595 be removed from the data set. \textbf{\\
1596 }For hybrid enrichment methods, bait regions do not have to be removed.
1597 The baits are digested in the lab and no remnants should be sequenced.
1598 The resulting sequence should originate 100\% from the target species,
1599 of no other problems arose. Therefore is impossible that the bait
1600 sequence ``overwrites'' the species sequence in the bait region.
1601 \end{quote}
1602
1603 \section{References}
1604
1605 \href{https://academic.oup.com/mbe/article/33/7/1875/2579186/BaitFisher-A-Software-Package-for-Multispecies}
1606 {Mayer, C., Sann, M., Donath, A., Meixner, M., Podsiadlowski, L., Peters,
1607 R.S., Petersen, M., Meusemann, K., Liere, K., Wägele, J.-W., Misof, B.,
1608 Bleidorn, C., Ohl, M., Niehuis, O., 2016. BaitFisher: A Software Package for
1609 Multispecies Target DNA Enrichment Probe Design. \emph{Mol. Biol. Evol.} 33, 1875 - 1886.
1610 doi:10.1093/molbev/msw056.}
1611
1612 %{Sann et al. 2017, submitted}
1613
1614 \end{document}