Header Leaderboard Ad

Collapse

bfast parallelization

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • rgarcia
    replied
    are mate pairs split when partitioning reads?

    Hello! I'm using this approach to parallelize the alignment of a run of SOLiD mate-pair reads. I converted the reads using solid2fastq, then partitioned the fastq files using the unix split command. I then successfully ran the match step on each partition.

    I then realised that solid2fastq placed F3 reads at the beginning of the fastq while R3 at the end, so after spliting the fastq, F3 reads will be separately aligned from R3 reads. It is here suggested that I separately generate a sam file from each partition.

    Does this mean that my reads won't be paired? Will samtools or picard pair my reads when I merge my sam files?

    Thanks!

    Originally posted by nilshomer View Post
    Note: "bfast match" searches the index, while "bfast index" creates an index.

    Here are the basic steps (remember to use multiple threads with the "-n" option).

    1.
    Partition the N reads into bins with R reads contained. Run these through "bfast match" to create ~X=N/R BMF files. Alternatively, if you have only one input file, use the "-s/-e" options to specify which reads to process for each invocation of "bfast match".

    2.
    For each BMF file from step 1, process them with "bfast localalign". You can also use "-s/-e" to further sub-divide each input file. Lets call the number of BAF files Y (since you may sub-divide, otherwise X equals Y).

    3.
    For each BAF file from step 2, process them with "bfast localalign" to get a SAM file.

    4.
    After all SAM files have been created, merge them with Picard or samtools (I recommend the former).

    This is the best way to partition the various steps of bfast, since step 1 ("bfast match") may take much more memory (for large references) than step 2 ("bfast localalign") and so if you had a heterogenous cluster you could submit the step 1 jobs to high memory nodes and step 2 jobs to low memory nodes etc. This is the spirit of the "bfast.submit.pl" script found with the BFAST distribution. The latter script is not well supported (there be dragons).

    Leave a comment:


  • nilshomer
    replied
    Originally posted by pengchy View Post
    To parallelize the running, whether I can split the reference genome into several, say 2, sections, and create index separately and then align separately then merge the bmf files?
    Thanks.
    Split the reads, merge the SAM files.

    Leave a comment:


  • pengchy
    replied
    To parallelize the running, whether I can split the reference genome into several, say 2, sections, and create index separately and then align separately then merge the bmf files?
    Thanks.

    Leave a comment:


  • nilshomer
    replied
    Originally posted by genome_anawk1 View Post
    Hi Nils,

    It is not obvious to me if one needs to use all the different index files / *.bif files (example 10 different files) at the same time for the three steps:
    1- bfast match
    2- bfast localalign
    3- bfast postprocess

    As posted previously, can I split jobs such that each of the ten bif files are processed separately. This will lead to 10xN separate jobs that run and consume less input time for reading in the dataset.

    Hope you can please suggest. Thanks in advance

    cheers,
    You should use all index files to get full sensitivity, but you could run one "bfast match" per index file on the input reads, then use "bmfmerge" (see the "butil" folder) to merge the index results (BMF files).

    Leave a comment:


  • genome_anawk1
    replied
    Hi Nils,

    It is not obvious to me if one needs to use all the different index files / *.bif files (example 10 different files) at the same time for the three steps:
    1- bfast match
    2- bfast localalign
    3- bfast postprocess

    As posted previously, can I split jobs such that each of the ten bif files are processed separately. This will lead to 10xN separate jobs that run and consume less input time for reading in the dataset.

    Hope you can please suggest. Thanks in advance

    cheers,

    Leave a comment:


  • genome_anawk1
    replied
    Hi Nils,
    Apologies, I fixed my previous query. I had copied the command incorrectly.
    cheers,
    new analyzer

    Leave a comment:


  • nilshomer
    replied
    Where's the "bfast match" command? You are definitely missing something. Try reading through the example in the manual's appendix.

    Leave a comment:


  • genome_anawk1
    replied
    Hi Nils,

    Thanks very much for your prompt response. Apologies, I had some incorrect commands earlier. I am making the fixes here.

    I did not use the "-s" or "-e" flags as I needed the algorithm to use color space information - the stress being on the accuracy. As the manual suggests (page 60), there was a trade off. Hence the long run time.

    My parallelization question is regarding the bif files (index files in color space). For example I have 10 index files and N (N~100) reads file called reads.j.fastq (j=1..N). My jobs are split as shown below:

    Using your set of masks, I create 10 bif files.
    using mask_1
    bfast index -f ref_genome.fa -m 111...11 -w 14 -i 1 -A 1

    ..
    using mask_k
    bfast index -f ref_genome.fa -m 110...11 -w 14 -i k -A 1

    ...
    (k=1 ..10 such unique masks as suggested in the manual).

    I am using the mouse genome, so is 10 an optimal basis for the hash masks. For now I am using "-w 14" but I guess that remains an open question for the mouse genome.

    Then get the 10 unique bif files. For the next few steps do I need to keep ALL 10 bif files as inputs (alongwith the ref genome in color space and nucleotide space) for bfast match, bfast localalign, and bfast postprocess ..
    Here is the parallelization that I now have ..

    dir_wk_1/
    bfast match -f ref_genome.fa -A 1 -r reads.1.fastq > bfast.matches_file.1.bmf
    bfast localalign -f ref_genome.fa -m bfast.matches_file.1.bmf -A 1 > bfast.aligned.file.1.baf
    bfast postprocess -f ref_genome.fa -i bfast.aligned.file.1.baf -A 1 > bfast.reported.file.1.sam


    dir_wk_2/
    bfast match -f ref_genome.fa -A 1 -r reads.2.fastq > bfast.matches_file.2.bmf
    bfast localalign ..
    bfast postprocess ...

    ...
    dir_wk_N/
    bfast match -f ref_genome.fa -A 1 -r reads.N.fastq > bfast.matches_file.N.bmf
    bfast localalign ..
    bfast postprocess ...

    ----------
    Specifically do the 10 bif files have to be referenced in each and every N subdirectory. Can I make 10xN separate runs - where each run has only ONE bif file (index file) referenced to it and one reads.j.fastq file referenced to it.

    Thanks very much, and apologies for the previous incorrect commands.
    cheers,
    new genome analyzer.
    Last edited by genome_anawk1; 05-14-2011, 07:12 AM.

    Leave a comment:


  • nilshomer
    replied
    Note: "bfast match" searches the index, while "bfast index" creates an index.

    Here are the basic steps (remember to use multiple threads with the "-n" option).

    1.
    Partition the N reads into bins with R reads contained. Run these through "bfast match" to create ~X=N/R BMF files. Alternatively, if you have only one input file, use the "-s/-e" options to specify which reads to process for each invocation of "bfast match".

    2.
    For each BMF file from step 1, process them with "bfast localalign". You can also use "-s/-e" to further sub-divide each input file. Lets call the number of BAF files Y (since you may sub-divide, otherwise X equals Y).

    3.
    For each BAF file from step 2, process them with "bfast localalign" to get a SAM file.

    4.
    After all SAM files have been created, merge them with Picard or samtools (I recommend the former).

    This is the best way to partition the various steps of bfast, since step 1 ("bfast match") may take much more memory (for large references) than step 2 ("bfast localalign") and so if you had a heterogenous cluster you could submit the step 1 jobs to high memory nodes and step 2 jobs to low memory nodes etc. This is the spirit of the "bfast.submit.pl" script found with the BFAST distribution. The latter script is not well supported (there be dragons).

    Leave a comment:


  • genome_anawk1
    started a topic bfast parallelization

    bfast parallelization

    Hello,

    I am a new bfast user and have successfully run the full bfast job which included all the details. Now to speed up the processing (parallelize at various levels) at various steps I need to know the following from the authors and experts.

    After using 10 unique set of indices suggested in the manual, I created 10 different index files.
    refgenome.fa.cs.<j>.1.bif (j=1,..10).
    I used the index files and the reference sequence and a unique reads.k.fastq file (k=1…N)
    to <br>
    search the indices (bfast index …)
    perform local alignment (bfast localalign)
    filter alignments (bfast postprocess)

    1) My question is : Can I further parallelize the searches for indices (bfast index) step.
    Can I run small jobs using independent index files for the above 3 steps and then merge the final *bam files after "bfast postprocess"?

    2) how can I generate similar sized "reads.k.fastq" files . I noticed that some files generated are larger than others - perhaps twice as large. The compute time takes twice as long too.

    Thanks in advance,
    cheers
    Last edited by genome_anawk1; 05-13-2011, 01:21 PM.

Latest Articles

Collapse

  • seqadmin
    A Brief Overview and Common Challenges in Single-cell Sequencing Analysis
    by seqadmin


    ​​​​​​The introduction of single-cell sequencing has advanced the ability to study cell-to-cell heterogeneity. Its use has improved our understanding of somatic mutations1, cell lineages2, cellular diversity and regulation3, and development in multicellular organisms4. Single-cell sequencing encompasses hundreds of techniques with different approaches to studying the genomes, transcriptomes, epigenomes, and other omics of individual cells. The analysis of single-cell sequencing data i...

    01-24-2023, 01:19 PM
  • seqadmin
    Introduction to Single-Cell Sequencing
    by seqadmin
    Single-cell sequencing is a technique used to investigate the genome, transcriptome, epigenome, and other omics of individual cells using high-throughput sequencing. This technology has provided many scientific breakthroughs and continues to be applied across many fields, including microbiology, oncology, immunology, neurobiology, precision medicine, and stem cell research.

    The advancement of single-cell sequencing began in 2009 when Tang et al. investigated the single-cell transcriptomes
    ...
    01-09-2023, 03:10 PM

ad_right_rmr

Collapse
Working...
X