Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Which file system is the best for NGS???

    Does it really make a difference???

    Or people just use the standard ones like NTFS, ext3???

  • #2
    Do people use Lustre? It seems to be designed for cluster computing.

    Comment


    • #3
      Originally posted by ymc View Post
      Does it really make a difference???

      Or people just use the standard ones like NTFS, ext3???
      It does. If you are not running in a clustered environment (or if the cluster is small) I would suggest ZFS only, a recent build possibly. data deduplication, data compression, easy maintenance and a lot of other stuff. There may be problems in porting apps on OS which support ZFS.

      Comment


      • #4
        Originally posted by ymc View Post
        Does it really make a difference???
        Or people just use the standard ones like NTFS, ext3???
        Well, if you are using NTFS, that implies you are using Windows, which is probably a mistake

        As for Unix, we use XFS and sometimes EXT3 on our RAID volumes. And that means all our volumes. You wouldn't not use RAID would you?

        Ultimately the filesystem won't be your bottleneck, it will be the underlying disk subsystem. If you can't get the data sets into RAM fast enough, you won't be using your CPUs at maximum. The filesystem won't help too much. Most of the files are large and read sequentially.

        Comment


        • #5
          Originally posted by Torst View Post
          Well, if you are using NTFS, that implies you are using Windows, which is probably a mistake

          As for Unix, we use XFS and sometimes EXT3 on our RAID volumes. And that means all our volumes. You wouldn't not use RAID would you?

          Ultimately the filesystem won't be your bottleneck, it will be the underlying disk subsystem. If you can't get the data sets into RAM fast enough, you won't be using your CPUs at maximum. The filesystem won't help too much. Most of the files are large and read sequentially.
          I do understand the bottleneck is most likely CPU and RAM for NGS. But since we are going to invest so much in the hardware, we might as well try to squeeze out every ounce of performance.

          I am going to use RAID, most likely RAID10. I heard that the stripe size of a RAID interacts with the block size of a file system. How does this work? Anyone knows?

          Comment


          • #6
            Stay away from Fat32! You won't be able to save any file over 4GB.

            Comment


            • #7
              You can use NTFS with Ubuntu but not Red Hat.

              Comment


              • #8
                Is ZFS optimized for big sequential read and write????

                Comment


                • #9
                  Originally posted by ymc View Post
                  I do understand the bottleneck is most likely CPU and RAM for NGS. But since we are going to invest so much in the hardware, we might as well try to squeeze out every ounce of performance.
                  My opinion is that spending 90% of your time getting an extra 10% is not worth it. Spend that effort on working smart - choosing the best tools, best methods, best protocols, smart pipelines.

                  I am going to use RAID, most likely RAID10. I heard that the stripe size of a RAID interacts with the block size of a file system. How does this work? Anyone knows?
                  In theory it is simple - you want to align logical volume accesses with the underlying physical volume accesses. In practice it is difficult, especially with 3rd-party hardware raid solutions, as the block layout is often hidden.

                  If you use Linux software raid, you have more control, and mkfs even queries the underlying md device to optimize the settings (I think). Read here for more: http://feedblog.org/2008/06/18/howto...trides-in-xfs/

                  Comment


                  • #10
                    Originally posted by ymc View Post
                    Is ZFS optimized for big sequential read and write????
                    The good thing is that you can configure it to have good performances for almost any kind of files :-)

                    Comment

                    Latest Articles

                    Collapse

                    • seqadmin
                      Essential Discoveries and Tools in Epitranscriptomics
                      by seqadmin




                      The field of epigenetics has traditionally concentrated more on DNA and how changes like methylation and phosphorylation of histones impact gene expression and regulation. However, our increased understanding of RNA modifications and their importance in cellular processes has led to a rise in epitranscriptomics research. “Epitranscriptomics brings together the concepts of epigenetics and gene expression,” explained Adrien Leger, PhD, Principal Research Scientist...
                      04-22-2024, 07:01 AM
                    • seqadmin
                      Current Approaches to Protein Sequencing
                      by seqadmin


                      Proteins are often described as the workhorses of the cell, and identifying their sequences is key to understanding their role in biological processes and disease. Currently, the most common technique used to determine protein sequences is mass spectrometry. While still a valuable tool, mass spectrometry faces several limitations and requires a highly experienced scientist familiar with the equipment to operate it. Additionally, other proteomic methods, like affinity assays, are constrained...
                      04-04-2024, 04:25 PM

                    ad_right_rmr

                    Collapse

                    News

                    Collapse

                    Topics Statistics Last Post
                    Started by seqadmin, Today, 08:47 AM
                    0 responses
                    10 views
                    0 likes
                    Last Post seqadmin  
                    Started by seqadmin, 04-11-2024, 12:08 PM
                    0 responses
                    60 views
                    0 likes
                    Last Post seqadmin  
                    Started by seqadmin, 04-10-2024, 10:19 PM
                    0 responses
                    57 views
                    0 likes
                    Last Post seqadmin  
                    Started by seqadmin, 04-10-2024, 09:21 AM
                    0 responses
                    53 views
                    0 likes
                    Last Post seqadmin  
                    Working...
                    X