File compression tips

[bio_boris]

Hi all,

I have a large collection of FASTA files and tabular BLAST files ~30-60 TB that I would like to archive. Does anyone have any experience with saving storage space by sorting these files before compression? Or should I skip this and look into different compression formats other than GZIP?

edit
Will not be saved to an actual archive system.

Thanks!

[amitm]

hi bio_boris,
I have some experience with compressing fastq (not fasta though) using double compression of tar.gz
Though it would take lot of time (>1hr on a Linux wrkstn; fastq's of avg. 20Gb), about 3 fold compression is achieved.

[Brian Bushnell]

Sorting mapped files can save a lot of space, though sorting fasta, probably not so much, though it depends on the contents.

I recommend that you use pigz, which can create gzip files using all available CPU cores and thus is way faster than gzip for the same compression level. That means you can increase to a higher compression level at the same or better speed. Syntax is just like gzip and the files are still compatible with gzip. There are higher-compression formats, but gzip is more ubiquitous and better supported.

[Richard Finney]

BZIP2 is better than GZIP, at the cost of compression and decompression speed.

Many folks have bench marked the size and speed tradeoffs ...

https://duckduckgo.com/?q=bzip2+vs+gzip+benchmarks

Note: (via stackoverlow http://stackoverflow.com/questions/4...with-all-cores ) ...
find /source -type f -print0 | xargs -0 -n 1 -P $CORES gzip -9
to saturate your CPUs if you don't have pigz (which is a great name for a program ... "pigz" ... cool).

[GenoMax]

@bio_boris: If you are going to archive to tape then you should let the backup software take care of the compression.

[SES]

BZIP2 is better than GZIP, at the cost of compression and decompression speed.

I personally prefer bzip2 for that reason (the compression part), at least for archiving data. There is a program called pbzip2 that can be installed through your package manager and this speeds things up significantly by using all your CPUs, or any number you specify.

[manauwer]

pbzip2 is an efficient way to compress data.
the command here will do the task

tar cf outputfile_name --use-compress-prog=pbzip2 inputfile_or_directory