question to scripting gurus

[yaximik]

I came across a simple one line command allowing take files in a folder one by one and pipe them into another command, but I cannot find this thread. Can someone help? Say, decompress hundreds of gziped files in a folder after downloading a database, or pipe to another process.

[dariober]

I came across a simple one line command allowing take files in a folder one by one and pipe them into another command, but I cannot find this thread. Can someone help? Say, decompress hundreds of gziped files in a folder after downloading a database, or pipe to another process.

Let's see if this puts you in the right direction...

To uncompress all the gzip files in the current directory this may suffice:

gunzip *.gz ## Use gunzip -r to descent in subdirectories

To do something with each gzipped file, e.g. print out the first 15 lines and write them to a file:

for gz in `find mydir/ -name '*.gz'`    
do
gunzip -c $gz | head -n 15 > ${gz}.head
done

Dario

[GenoMax]

This should do it: http://www.cyberciti.biz/faq/bash-loop-over-file/

There will be analogous commands for tcsh, if that is your favorite.

[sisch]

My first impression was, You are talking about the 'find' command.
The find command has much more functionality than a simple bash loop. However, for simplicity I use bash loops more often.

Here is the manpage of find

Best,
Simon

[EGrassi]

or ls | xargs ...

[syfo]

To uncompress all the gzip files in the current directory this may suffice:

gunzip *.gz ## Use gunzip -r to descent in subdirectories

To do something with each gzipped file, e.g. print out the first 15 lines and write them to a file:

for gz in `find mydir/ -name '*.gz'`    
do
gunzip -c $gz | head -n 15 > ${gz}.head
done

Dario

Yep.
And to get the results in the same file, you can do

for gz in *.gz ; do less $gz | head -15 ; done > outfile

[yaximik]

Thanks everyone. I realized I did not express the question clearly. Gzip is just one of applications, i rather asked about general looping over files:

while/for/if
    [I]something is in the folder[/I]
do
   [I]something to each file[/I]
until
   [I]all are processed[/I]
done

The beauty of what I saw was that it fit just in one line and was pretty general, so it can be applied to other commands that do not take '*' for multiple files.

[sisch]

In fact, you can also do for loop one-liners with ";" or even with & to fork them to background
For example looping over Fasta files (*.fa) and counting the headers (i.e. number of sequences) could be done on one line like so:

for f in *.fa; do grep -c ">"; done

If you want to parallel unzip a bunch of files you could fork the loop like so:

for f in *.gz; do gunzip ${f} & done; wait

This runs all gunzip calls in parallel and 'wait's for them to be finished (optional)

I learned the hard way: While those one-liners are nice and quick for scripting on a terminal, never use them in longer bash-scripts because a) you will at some point have a typo and then will have a very hard time finding it and b) there is essentially no need in saving space/shortening out commands in a bash script.

Best
Simon

[mbayer]

Hi yaximik,

I use this in my bash scripts:

#extract all the filenames to an array
files=`ls -l *.gz | awk '{ print $9 }'`

#iterate over these
for i in $files
do
	echo "gunzipping file $i"
	gunzip $i 
done

cheers

Micha

[yaximik]

Thanks a lot, everyone who provided inputs. That was very educational in general as I get a better idea how I can use simple loops. I wonder, would it be useful to have something like WiKiBits (or a better name) as a collection of little ingenious solutions?