Decompressing a file - very basic question

[jk1124]

Hi everyone,

I have been running my sequencing analysis on a remote server. The sequencing data that I receive is compressed (.gz), so what I upload onto the server is a compressed file. I have been using gunzip to decompress these.

My concern is that often the process seems to take so long that I get a "Write failed" "broken pipe" error. The file no longer has the .gz extension, so it seems to have decompressed, but I have no decompressed file with which to compare size. So, a few questions:

1. Is there a way to make sure a file decompressed completely, without having a size to compare beforehand?

2. If a file does fail to decompress, does it remain in some kind of partially-decompressed state or does it revert?

3. Is there a faster way to decompress large files?

Thanks in advance.

[dsenalik]

What may be happening is your connection times out due to "inactivity" while waiting for the file to uncompress.

One general purpose solution may be to (depending on what operating system and client program you are using) is to set "keepalive" to something, like once every 60 seconds, so the connection does not die. Google "keepalive" and the program you are using. e.g. in Ubuntu here's the first Google hit from 2006 that has the instructions http://embraceubuntu.com/2006/02/03/...essions-alive/

for #1 and #2, in my experience if the decompress fails, everything reverts to the original condition.

for #3, probably not, but you can decompress multiple files at the same time (assuming you have more than one processor core) by detaching the jobs, e.g.

gunzip whoppinggreatfastqfile.fq.gz &

and if you detach from the terminal, then it will run to completion even if you disconnect, i.e.

nohup gunzip whoppinggreatfastqfile.fq.gz &

[rflrob]

You might also look into GNU screen, which lets you disconnect from a window, then reconnect later. It's like nohup, but better. People had been telling me for years to use it, and when I finally did, I wanted to slap myself backwards through time for not using it sooner.

[jk1124]

Great! Thanks for the advice.