Seqanswers Leaderboard Ad
Collapse
Announcement
Collapse
No announcement yet.
X
-
I do a lot of shell scripts and make files, and this Bpipe looks like it will make my life much easier.
I have a problem using the torque queue. The process does not return after successfully executing one statement. For example, if I give it...
Code:echo = { exec "echo this" exec "echo that" } Bpipe.run { echo }
This only happens when using torque. Incidentally, in order to get the queue to run had to export the QUEUE shell variable set to the name of the pbs queue.
Any ideas about how to get it to work?
Edit:
I figured out how to get it to work. I configured the queue to keep complete jobs for a minute. It had been removing jobs from the queue immediately after completion, so the job status would never be shown as "completed". Now it's fixed.Last edited by Alex Renwick; 04-20-2012, 07:58 AM.
Leave a comment:
-
Hi Simon,
thx for adding this issue to you develpment queue. I will you on this..
Tobias
Leave a comment:
-
Hi Tobias,
Originally posted by linusvanpelt View PostHi Simon,
I am trying to do some parallelization with bpipe and hope you can help me out on a problem. Like in this example from the Wiki
Code:Bpipe.run { chr(1..5) * [ hello ] }
...
Knowing that this is not working I was wondering if this could be implemented or resolved somehow?
Thanks for the feedback!
Simon
Leave a comment:
-
Hi Simon,
I am trying to do some parallelization with bpipe and hope you can help me out on a problem. Like in this example from the Wiki
Code:Bpipe.run { chr(1..5) * [ hello ] }
Code:@Transform("sam") align_stampy = { exec """ python $STAMPY_HOME/stampy.py --bwaoptions="-q10 $REFERENCE" -g $STAMPY_GENOME_INDEX -h $STAMPY_HASH_FILE -M $input1,$input2 -o $output --readgroup=ID:$rg_id,LB:$rg_lb,PL:$rg_pl,PU:$rg_pu,SM:$rg_sm --processpart=[COLOR="red"]$part[/COLOR] """ } Bpipe.run { [COLOR="Red"]part[/COLOR]("1/3", "2/3", "3/3") * [align_stampy] }
Thanks,
Tobias
Leave a comment:
-
Hi brentp,
Originally posted by brentp View Post
what if a given step needs multiple previous $output' s?
Code:exec "somecommand $input.vcf $input.bam"
Code:exec "somecommand most_recent_vcf.vcf most_recent_bam.bam"
Code:exec "somecommand $inputs.bam"
Code:exec "somecommand file1.bam file2.bam file3.bam ..."
Code:exec "for i in $inputs.bam; do samtools index $i; done"
Cheers,
Simon
Leave a comment:
-
Hi maubp,
Originally posted by maubp View PostThat's why you should use Unix pipes where possible (ideally without compressing the intermediate BAM files, use -u in samtools).
Does Bpipe support this? Perhaps it could using named pipes?
Named pipes is a really interesting idea. I think at the moment it would be problematic because Bpipe expects the process for a pipeline stage to terminate before it will initiate the next stage. But with some tweaks that could be relaxed, to allow this mode of operation.
I'll definitely put more thought into this - thanks for the discussion / ideas!
Leave a comment:
-
Originally posted by brentp View Postonly if you don't care about keeping around intermediate files, in case the pipeline adds a step in there or you change some parameters in a later step and don't want to re-run the entire pipeline.
Leave a comment:
-
Originally posted by genericforms View PostIf so, then this would negate the need for plug-ins.
Leave a comment:
-
Originally posted by maubp View PostThat's why you should use Unix pipes where possible (ideally without compressing the intermediate BAM files, use -u in samtools). Does Bpipe support this? Perhaps it could using named pipes?
If so, then this would negate the need for plug-ins.
Leave a comment:
-
Looks pretty useful, could you explain more about:
where, for example you have:
Code:@Transform("bai") index = { exec "samtools index $input" return input }
from the previous?
what if a given step needs multiple previous $output' s?
I guess it's not clear to me what's going on with the the $input and $output names.
Leave a comment:
-
Originally posted by genericforms View PostSo much time is wasted on writing a BAM file over and over.
Leave a comment:
-
I think this is a great tool for building pipelines. Thanks for sharing. I think it would be awesome if we could eventually adopt a similar approach to be analysis framework, where if tools are written as plugins, a BAM file could open ed once and then operated on with several plugins, and written once.
So much time is wasted on writing a BAM file over and over.
Leave a comment:
-
Bpipe: a new tool for running analysis pipelines
Hello all,
I would like to let everyone know about Bpipe, a new tool we have created to help run bioinformatics pipelines.
Many people will be familiar with tools like Galaxy and Taverna, etc. that help you run pipelines and give a graphical view of the pipeline, inputs, outputs and many other features to make analysis pipelines more robust and manageable. Bpipe is similar in many ways but aimed at users who are command line oriented. It lets you write your pipelines almost like how you write a shell script, but it automatically adds features such as:
- Transactional management of tasks - commands that fail get outputs cleaned up, log files saved and the pipeline cleanly aborted.
- Automatic connection of pipeline stages - Bpipe manages the file names for input and output of each stage in a systematic way so that you don't need to think about it
- Easy stopping or restarting - when a job fails it is easy to cleanly restart from the point of failure
- Audit trail - Bpipe keeps a journal of exactly which commands executed and what their inputs and outputs were
- Modularity - It's easy to make a library of pipeline stages (or commands) that you frequently use and mix and match them in different pipelines
- Parallelism - easily run many samples/files at the same time or split one sample and run analysis on many parts of it in parallel
- Integration with cluster resource managers - Bpipe supports PBS/Torque and more systems can be added easily
- Notifications - Bpipe can send you alerts by email or instant message to tell you when your pipeline finishes or even as each stage completes.
Bpipe is BSD licensed and available, along with documentation and examples, at http://bpipe.org. We also have a publication accepted in Bioinformatics which may be of interest as well.
Bpipe is very young and I hope to make many improvements, so I would love to have feedback from anybody here about it.
Thanks!
SimonTags: None
Latest Articles
Collapse
-
by seqadmin
The first FDA-approved CRISPR-based therapy marked the transition of therapeutic gene editing from a dream to reality1. CRISPR technologies have streamlined gene editing, and CRISPR screens have become an important approach for identifying genes involved in disease processes2. This technique introduces targeted mutations across numerous genes, enabling large-scale identification of gene functions, interactions, and pathways3. Identifying the full range...-
Channel: Articles
08-27-2024, 04:44 AM -
-
by seqadmin
Sequencing mRNA provides a snapshot of cellular activity, allowing researchers to study the dynamics of cellular processes, compare gene expression across different tissue types, and gain insights into the mechanisms of complex diseases. “mRNA’s central role in the dogma of molecular biology makes it a logical and relevant focus for transcriptomic studies,” stated Sebastian Aguilar Pierlé, Ph.D., Application Development Lead at Inorevia. “One of the major hurdles for...-
Channel: Articles
08-07-2024, 12:11 PM -
ad_right_rmr
Collapse
News
Collapse
Topics | Statistics | Last Post | ||
---|---|---|---|---|
Started by seqadmin, 08-27-2024, 04:40 AM
|
0 responses
16 views
0 likes
|
Last Post
by seqadmin
08-27-2024, 04:40 AM
|
||
New Single-Molecule Sequencing Platform Introduces Advanced Features for High-Throughput Genomics
by seqadmin
Started by seqadmin, 08-22-2024, 05:00 AM
|
0 responses
293 views
0 likes
|
Last Post
by seqadmin
08-22-2024, 05:00 AM
|
||
Started by seqadmin, 08-21-2024, 10:49 AM
|
0 responses
135 views
0 likes
|
Last Post
by seqadmin
08-21-2024, 10:49 AM
|
||
Started by seqadmin, 08-19-2024, 05:12 AM
|
0 responses
124 views
0 likes
|
Last Post
by seqadmin
08-19-2024, 05:12 AM
|
Leave a comment: