Which file system is the best for NGS???

[ymc]

Does it really make a difference???

Or people just use the standard ones like NTFS, ext3???

[ymc]

Do people use Lustre? It seems to be designed for cluster computing.

[dawe]

Does it really make a difference???

Or people just use the standard ones like NTFS, ext3???

It does. If you are not running in a clustered environment (or if the cluster is small) I would suggest ZFS only, a recent build possibly. data deduplication, data compression, easy maintenance and a lot of other stuff. There may be problems in porting apps on OS which support ZFS.

[Torst]

Does it really make a difference???
Or people just use the standard ones like NTFS, ext3???

Well, if you are using NTFS, that implies you are using Windows, which is probably a mistake

As for Unix, we use XFS and sometimes EXT3 on our RAID volumes. And that means all our volumes. You wouldn't not use RAID would you?

Ultimately the filesystem won't be your bottleneck, it will be the underlying disk subsystem. If you can't get the data sets into RAM fast enough, you won't be using your CPUs at maximum. The filesystem won't help too much. Most of the files are large and read sequentially.

[ymc]

Well, if you are using NTFS, that implies you are using Windows, which is probably a mistake

As for Unix, we use XFS and sometimes EXT3 on our RAID volumes. And that means all our volumes. You wouldn't not use RAID would you?

Ultimately the filesystem won't be your bottleneck, it will be the underlying disk subsystem. If you can't get the data sets into RAM fast enough, you won't be using your CPUs at maximum. The filesystem won't help too much. Most of the files are large and read sequentially.

I do understand the bottleneck is most likely CPU and RAM for NGS. But since we are going to invest so much in the hardware, we might as well try to squeeze out every ounce of performance.

I am going to use RAID, most likely RAID10. I heard that the stripe size of a RAID interacts with the block size of a file system. How does this work? Anyone knows?

[RockChalkJayhawk]

Stay away from Fat32! You won't be able to save any file over 4GB.

[NextGenSeq]

You can use NTFS with Ubuntu but not Red Hat.

[ymc]

Is ZFS optimized for big sequential read and write????

[Torst]

I do understand the bottleneck is most likely CPU and RAM for NGS. But since we are going to invest so much in the hardware, we might as well try to squeeze out every ounce of performance.

My opinion is that spending 90% of your time getting an extra 10% is not worth it. Spend that effort on working smart - choosing the best tools, best methods, best protocols, smart pipelines.

I am going to use RAID, most likely RAID10. I heard that the stripe size of a RAID interacts with the block size of a file system. How does this work? Anyone knows?

In theory it is simple - you want to align logical volume accesses with the underlying physical volume accesses. In practice it is difficult, especially with 3rd-party hardware raid solutions, as the block layout is often hidden.

If you use Linux software raid, you have more control, and mkfs even queries the underlying md device to optimize the settings (I think). Read here for more: http://feedblog.org/2008/06/18/howto...trides-in-xfs/

[dawe]

Is ZFS optimized for big sequential read and write????

The good thing is that you can configure it to have good performances for almost any kind of files :-)