Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • A new release of the Snakemake workflow system

    Hi guys,
    I would like to announce version 2.4.8 of Snakemake.
    Snakemake is a pythonic text-based workflow system with a clean and easy to read language for defining your workflows. Snakemake is inspired by GNU Make. Workflows are defined by rules, that generate output files from input files. Rule dependencies and parallelization are automatically determined by Snakemake.
    In contrast to GNU Make, Snakemake allows to have multiple output files in a rule. Further, rules can use shell commands, Python, or R code. Snakemake provides many additional useful features like resource-aware scheduling, parameter- and version-tracking and detection of incomplete files.
    Finally, Snakemake has a generic cluster support, that works with any cluster or batch system that provides a qsub-like command and a shared filesystem.

    To give you an impression, this is how Snakemake rules look like:
    Code:
    rule targets:
        input:  'plots/dataset1.pdf', 
                'plots/dataset2.pdf'
    
    rule plot:
        input:  'raw/{dataset}.csv'
        output: 'plots/{dataset}.pdf'
        shell:  'somecommand {input} {output}'
    If you like Snakemake, please feel free to visit http://bitbucket.org/johanneskoester/snakemake.

  • #2
    Hi johanneskoester,

    I'm curious about learning more about snakemake, thanks for positing it!

    I'm quite familiar with python, R, bash but I'm not familiar at all with GNU Make and friends (other than executing it when I get some source code). So I must admit I fail to see where the advantage comes when building bioinformatics pipelines.

    For example, following this example on snakemake, this is how I would implement the same pipeline:

    Code:
    REF="/global/home/users/ebolotin/scratch/hg19/hg19"
    for fq in *.fastq.gz
    do
        bname=`basename $fq .fastq.gz`
        ## Prepare pipeline
        echo "cutadapt  -m 10 -a AGATCGGAAGAGCACACGTCTGAACTCC -o ${bname}.cut $fq &&
        bowtie2 -p 20 --very-sensitive -x $REF -U ${bname}.cut -S ${bname}.sam &&
        makeTagDirectory ${bname}.tag ${bname}.sam -keepAll -genome hg19" > ${bname}.sh
        ## Run the job:
        bash ${bname}.sh
        # OR
        # nohup ${bname}.sh &
        # OR
        # bsub [opts] < ${bname}.sh
    done
    Could you point out in what respect snakemake would make it preferable?

    Thanks!
    Dario

    Comment


    • #3
      Hi,
      sure. The solution you provide runs a for loop and spawns a bash job for each fastq.
      So, they will execute in parallel, all fine.
      When using Snakemake, you would have a similar effect at first sight. However, there are various advantages (some of them, but to the best of my knowledge not all, are also provided by other workflow systems):

      With Snakemake, you can define how many processes should be active at the same time, so that your machine is not flooded with jobs. Snakemake will schedule them in a way such that the utilization of the provided cores will be maximized. The scheduling is also aware of the number of threads each job uses.
      If a job fails, or you have to quit the execution, on the next invokation, Snakemake will determine what was already computed last time and only calculate the missing stuff.
      If an input file happens to be changed, Snakemake will propose to rerun the subsequent part of the pipeline automatically (in other words, Snakemake automatically detects if one of your files is outdated).
      You can run the very same workflow definition on a single machine or a cluster, without the need to redefine anything in the Snakefile.
      Following the well known pattern of input-output-code, the Snakemake rules are very easy to read, and help to separate your commands from the parameters.
      For each of the output files created during the workflow, Snakemake will store metadata like used parameters, commands, and input files etc. which is nice for documentation.

      Best,
      Johannes

      Comment


      • #4
        Interested in learning more about #SNAKEMAKE?

        Register now for the first 2-day #SNAKEMAKE Workshop in Berlin with Johannes Köster https://johanneskoester.bitbucket.io/

        Dates 3-4 May 2022 This course will be held online in response to the coronavirus outbreak


        You will learn how to create modern and reproducible #bioinformatic workflows

        Comment

        Latest Articles

        Collapse

        • seqadmin
          Exploring the Dynamics of the Tumor Microenvironment
          by seqadmin




          The complexity of cancer is clearly demonstrated in the diverse ecosystem of the tumor microenvironment (TME). The TME is made up of numerous cell types and its development begins with the changes that happen during oncogenesis. “Genomic mutations, copy number changes, epigenetic alterations, and alternative gene expression occur to varying degrees within the affected tumor cells,” explained Andrea O’Hara, Ph.D., Strategic Technical Specialist at Azenta. “As...
          07-08-2024, 03:19 PM
        • seqadmin
          Exploring Human Diversity Through Large-Scale Omics
          by seqadmin


          In 2003, researchers from the Human Genome Project (HGP) announced the most comprehensive genome to date1. Although the genome wasn’t fully completed until nearly 20 years later2, numerous large-scale projects, such as the International HapMap Project and 1000 Genomes Project, continued the HGP's work, capturing extensive variation and genomic diversity within humans. Recently, newer initiatives have significantly increased in scale and expanded beyond genomics, offering a more detailed...
          06-25-2024, 06:43 AM

        ad_right_rmr

        Collapse

        News

        Collapse

        Topics Statistics Last Post
        Started by seqadmin, 07-10-2024, 07:30 AM
        0 responses
        23 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 07-03-2024, 09:45 AM
        0 responses
        198 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 07-03-2024, 08:54 AM
        0 responses
        209 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 07-02-2024, 03:00 PM
        0 responses
        191 views
        0 likes
        Last Post seqadmin  
        Working...
        X