yet another framework?

[desaila]

Anyone who follows the web technologies is very familiar with the # of web frameworks there are, and the ever increasing number of them.

I was pretty excited when I saw the announcement, but was quickly disappointed to find out this doesn't do much, if anything, beyond what Picard and GATK already provide.

What are the advantages of using this framework over using Picard and GATK with BioJava? Development time will be faster in that scenario, there is more documentation, and they have a more liberal license (for now) for commercial users.

[zee]

This is a good point. How does openge make it easier us to run good genomics workflows. Perhaps an easyrun function for exome, RNASeq,etc, analysis will be a great thing to have in this system.

desaila, I think the current version of openge will provide some enhanced functions for doing thing with alignments, etc, since it is written in C.

Anybody else care to comment?

[oiiio]

@desaila, i'd like to chime in on your thread here, which is a good distinguishment question-

While there is overlap in the ultimate goals of Picard/GATK and the openge framework, as zee mentioned there are great differences in functionality. In terms of HPC, the openge binaries that were released are much more suited for parallelized performance (both CPU and GPU) in all of its available tools, and can also stream data similar to samtools and many our other favourite programs. Not mention that as a C program, the hassle of matching java versions is gone (for example, between your cluster nodes and your workspace), and can be opened on all 'nix distros with minimal worry.

Aside from performance and command line usability, there is also the workflow creation aspect of openge. Although still in development, an example of its use would be that a user could potentially put together a workflow for a specific type of genomic data that he has, and share it via some kind of devblog or SEQanswers thread with others who may need the exact same thing - saving this subset of analysts many hours. The openge 'wrapped' workflow would then be able to execute on these different machines without trouble. Not to mention advantages of using the openge wrappers, such as automated error handling, and other features of an appealing workflow manager.

Hopefully this can clear up your separation of the Picard/GATK and openge. You are also correct that there is more documentation, etc. but openge should be given its chance to grow as well It is a concept that has its own HPC functionality, but also a big potential for carrying a community.

[desaila]

@oiiio
What I don't understand, in your reply is how is OpenGE differentiating itself in it's parallelization model? GATK uses map/reduce and scales with the # of cpu's you have. It looks like this does the same thing by actually porting that functionality from GATK.

Let me clarify and say that I'm not trying to be pessimistic, and that I'm completely in support of this framework -- especially if they open up the commercial side of their license. However, it realistically does nothing that GATK and Picard don't already do. You can create workflows and things in GATK/Picard too.

Here's where I'm optimistic: the fact that Nvidia is touting this as a compute solution is implying that there will be cuda/openCL elements in the future and _that_ is certainly something GATK will not have.

[adaptivegenome]

@oiiio
What I don't understand, in your reply is how is OpenGE differentiating itself in it's parallelization model? GATK uses map/reduce and scales with the # of cpu's you have. It looks like this does the same thing by actually porting that functionality from GATK.

I would encourage you to take a look at the code. It appears you have posted only twice to SEQanswers prior to posting this thread. So, I have no way to gauge your technical level but I think you are misunderstanding the implementation. OpenGE implements multithreading that takes advantage of newer CPUs (and GPUs) that have lots of cores. This is not something that is ported from GATK and it is not map/reduce.

The comparisons to GATK and other tools are inevitable. It's an easy question to ask. We have no interest in replacing or competing with GATK or other tools though. We are looking to complement existing tools, but our primary focus or interest is:

(1) High performance code that maximizes compute power than be utilized on commodity hardware, compute nodes, etc.

(2) A workflow to integrate OpenGE with other tools (and vice versa). Here I can't say too much more (because I do not want to sell you on tales of the future) but the idea is to build graphical and sharable analysis pipelines. And I do not mean scripts. I mean fully managed workflows that track progress, detect and correct errors, etc.

I again encourage you to check out the code, follow the forum and give the code a chance if you want. Furthermore, we would welcome your thoughts and everyone else's on how to decide upon and prioritize future features.

[desaila]

I would encourage you to take a look at the code. It appears you have posted only twice to SEQanswers prior to posting this thread. So, I have no way to gauge your technical level but I think you are misunderstanding the implementation. OpenGE implements multithreading that takes advantage of newer CPUs (and GPUs) that have lots of cores. This is not something that is ported from GATK and it is not map/reduce.

https://github.com/adaptivegenome/op...thm_module.cpp and the .h

Correct me if I'm wrong, but inheriting classes of AlgorithmModule are simply asynchronous threads, or they might use a ThreadPool -- but browsing through the code it seems like the only thing that uses the ThreadPools is the read_sorter.{h, cpp}

So, asynchronous threads on a single box is the breakthrough here? If that's the case, GATK scales to the same levels as OpenGE does right now. Clearly that will change once this starts using CUDA.

[adaptivegenome]

So, asynchronous threads on a single box is the breakthrough here? If that's the case, GATK scales to the same levels as OpenGE does right now. Clearly that will change once this starts using CUDA.

I hope that future releases impress you more