NGS functionality for KNIME

[BAJ]

Hi,

I am very excited that I can share now with you my nodes and workflows that I created for NGS data analysis in KNIME.

KNIME is a workflow management system. Some of its features include:

* can handle many millions of rows on a desktop computer
* workflows can be executed from the command line
* integration with Galaxy/Mobyle possible
* workflows can be exchanged
* writing new functionality is relatively easy
* it is based on JAVA/Eclipse
* command line scripts can be organized
* no worries about naming intermediate files
* high content/ high through put problems can be already solved
* scripting in R, Perl, Python, Java, Matlab supported
* Hilighting/brushing supported
* open source
* commercial support available if desired
* support for statistics, flow control (if/while loops)
* supportive community
* creating professional looking reports


and now also:

* Reading / writing FastQ /SAM/BAM /BEDgraph files
* region of interest related tools
* AdapterRemoval
* and many more....


check it out and let me know what you think....
Installing KNIME:

http://knime.org/download-0

Installing community nodes:
http://tech.knime.org/community-contributions-info

To get a quick overview of how to use it with NGS data:
NGS nodes and descriptions:
http://tech.knime.org/community/next...ion-sequencing


Kind regards,

Bernd

[colindaven]

Looks interesting. Have you used this a lot already ? Which use cases? How easy is it to install ? Is there much memory overhead?

[BAJ]

I am using it for production in our NGS service facility. We are mainly concerned with anything but resequencing and SNPs.
It is very good for prototyping and then moving to production for tasks like preprocessing removing parts of a sequence, splitting, joining, stats.
There is actually negative memory overhead as KNIME stores tables on disk. So there is some overhead in compute time, but we are working on this.
You can easily parallelize things by building workflows that run in parallel.
Once you reduced your data set to something in the range of a few million you can easily work with it (or at least that is what I am doing). For data sets bigger than this it might be used from the command line or using command line executions from within kNIME...
Well, just give it a try and let me know if you run into problems.
Btw, installation is fairly easy... There are instructions on the web site, basically you have to unpack and start the application, then configure a proxy if necessary and install the additional nodes. Follow the links I provided

[BAJ]

One example on the memory:
I am currently running 6 nodes for reading BAM files in parallel, each table consists of some 300 M rows. The memory footprint is about 5 GB though I allocated 16 GB on a Linux machine...