Hello - I use to think I was good with a computer

[Jon_Keats]

Getting Started: Unix and Xcode

As I said in the first post I'm going to drop in a couple of posts over the next couple of days to outline my experiences to date.

In clinical training they have a mantra of "See one, Do one, Teach one" but on the research side it seems to be more "Need to do one, Figure one out, Maybe Teach one" so this will be my lame teaching attempt or at the very least a place others in our research group can get some basic instructions to replicate the pipeline I'm starting to put together. Hopefully this will be relevant to a number of people and will make some peoples life easier.

I've tested most of the following steps on both my laptop and the workstation we have in the lab (still waiting for Apple to release new Mac Pros… common Steve I'll buy an iPad if you release them in April). Obviously, I'm a Mac guy so these instructions are Mac oriented but should be comparable with any Unix/Linux environment, but that is only a guess.

*Workstation = Mac Pro with two dual core Intel Xeon5150 CPUs at 2.66Gz and 8Gb of 667MHz DDR2 RAM running Mac OSX Leopard 10.5.8*
*Laptop = MacBook Pro with an Intel Core 2 Duo CPU at 2.66GHz and 4Gb of 1067MHz DDR3 RAM running Mac OSX Snow Leopard 10.6.3*

Okay, so today you got some terabyte drives with Illumina data and you want to do something with it. The following instructions should get you ready to do something:

First thing to do is to familiarize yourself with Unix and the Terminal (Applications>Utilities>Terminal) application on your MAC. I would highly recommend working though at least the Unix portion of the "Unix and Perl for Biologists" course made public by Keith Bradham and Ian Korf at UC Davis (http://groups.google.com/group/unix-...for-biologists). I'd recommend going to their website and get the entire course package (http://korflab.ucdavis.edu/Unix_and_Perl/index.html) it is well worth a night or two of your time I promise.
If you are not going to do that you need to understand one or two commands to get going:

To get a manual on any command type "man command". Hit "space" to page down, "b" to back-up, and "q" to quit
To see what folder you are in currently type "pwd"
To see what folders and files exist in the current directory type "ls"
To move into a folder in the current directory type "cd myfolder" Note: you can move multiple levels downstream with "cd myfolder/myfolder2"
To go back one directory type "cd .." Note: you can move back multiple levels upstream with "cd ../.."
To copy a file from the current directory to a downstream folder "cp myfile myfolder/" or "cp myflie ../" to copy a file up a directory
To move a file from the current directory use "mv" in place of cp
A folder immediately downstream of the root directory (ie. absolute top of the tree) is always defined by "command /folder" (ie. if you type "cd /something" it looks for the folder "something" downstream of the root directory)
The current directory can always be noted by "./"
You will need to change the permissions of the compiled applications with "chmod 755 myfile". This makes the file readable and executable by everyone but only you can write, alternatively use 777 so anyone can do everything.

To run many of the applications Maq, BWA, Samtools, etc.. you will need to either place the applications in the PATH, define additional PATH locations, or you need to note the location of the application each time you call it. To find the current PATH directories used by Unix type "$PATH" and you should get a print out similar to the following:
-bash: /sw/bin:/sw/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/X11R6/bin

NOTE: These folders are directly below the root directory and represent places unix looks for applications to run. If you want to run any of the applications you will download and compile such as BWA you need to either type "./bwa" and the current directory must contain the bwa application. Assuming you have administrator rights to your machine, the way I initially got around this was to place the applications in one of the path directories as follows:

1) copy application to a PATH directory, type "sudo cp myfile /usr/bin" you will be prompted for you password (sudo = superuser … yes, today you are SUPERMAN)
2) make the file executable, type "sudo chmod /usr/bin/myfile"

NOTE: After running into an issue installing BFAST I'd suggest the following NOT the previous (Actually suggestion from Nils Homer...thanks)
1) create a directory in your home directory for the applications "mkdir -p $HOME/local/bin"
2) edit your .profile file so this directory is in your PATH directories
>open terminal
>type "ls -a" (You should see a file called .profile)
>open with nano "nano .profile"
*** Add the following lines to your .profile file, DO NOT remove things in the current version ***

export PATH=$HOME/local/bin:$PATH

> To save edits "control-O"
> To exit nano "control-X"

# Subsequently when you install things place the executable's in this directory so they are in a $PATH directory
# Either copy application to the directory $HOME/local/bin
# If using install script "./configure --prefix=$HOME/local"

This no longer requires sudo (Guess we shouldn't always be Superman)

Second you need to install Xcode on your Mac System so you can compile the various applications
- Download the current version, Xcode3.2, at (http://developer.apple.com/technolog...ols/xcode.html). You will have to become a member otherwise find your OS install discs and do it from the disc install option.
NOTE: This version is only compatible with Mac OSX Snow Leopard 10.6.x
- If you have a Leopard system go find the OS install discs (you need Disc 2) and install the package
> Mac OS X Install Disc 2 > open Xcode Tools folder > double click XcodeTools.mpkg

Third, for some applications like BFAST it will help to install "Fink" (...Another Nils suggestion) or "MacPorts" (seems more up to date)
- Download and install the current version from (http://www.finkproject.org/) or (http://www.macports.org/)
- You should install the package md5deep at least to install BFAST "fink install md5deep" or "port install md5deep"

Next step get the applications you need…

See the next post,

Jonathan

[Jon_Keats]

Thanks I'm glad to see people find my attempt at a bit of science geek humor funny, even with the typical spelling mistakes

[MQ-BCBB]

loved your post

I felt exactly like that not too long ago. And yes, I still remember the great feeling when I successfully loaded my data into IGV.

[drio]

Just catching up with SA posts.

This post is brilliant. Printing right now and posting in my cubicle. Instant classic.

[RockChalkJayhawk]

Jon,

I was is your exact situation 6 months ago. It gets better (slowly). The UNIX and Perl for Biologists was really helpful for me, as was this forum. You're headed in the right direction, just keep it up!

And I too agree that this is the best post on the site!

[ECO]

Nice post again Jonathan,

This is actually a great post/series of posts to kick off the "Basics in Bioinformatics" subforum that we've discussed in the Site Feedback forum. Don't be alarmed if I do some rearranging/forum creating later today.

[mangrove]

I agree... it is prizeworthy. Even at the price, the promise of all that sequence data is alluring, but no one I have talked to has gotten the data and NOT been overwhelmed. I hope this goes away as we get larger, faster computers, a guru to install all the programs (fortunately we have that), and eventually, the realization from NSF that lots of $ and many months will be required to actually use all those TBs. Good luck to us all (as Tiny Tim would say).

[nilshomer]

I wonder how many people are in the same boat as me.

1) Institute bought a couple of GAIIs
2) No one has money to use them
3) Institute has internal competition to pay for a couple of runs (makes the donors feel better about their donation if someone uses the machines), and you are lucky enough to get funded
4) You send a couple of samples off to never-never land and someone sends back a terabyte drive or two with "next-gen sequencing data"
5) You quickly realize people that use to do survival curves in your bioinformatics core don't really know that Illumina fastq is different from Sanger fastq and the analysis they provide is limited at best
5) Now what do you do?
6) Google > seqanswers > let the misery begins

So what have I learned this week,

A) My boss should have made me read and do the "Unix and Perl for Biologist" tutorial years ago. Google it if you are new and a bench/gene jockey (sanger sequencing/microarray person) like me with no unix experience it is an excellent use of a day
B) A place called SorgeForge exists
C) If I had a MAQ for my TOPHAT and a BOWTIE to go with my BWA I'd be better of than GERALD and his SAMTOOLS
D) just type "make" to compile...Opps that doesn't work if Xcode is not installed yet
E) No Mac OS comes with Xcode installed and if you have a Leopard machine, you better know where the OS install disks are as you can only install the new version for snow leopard that is not compatible...One would think that pancreatic cancer survivor Steve Jobs would try to make my life easier not harder
F) The genome is not the genome, ensembl is the place to get chromosomes but 1000 genomes is the place to get the genome.
G) BWA can align on my laptop...cool...next-gen/2nd gen alignment on a laptop and I though I needed a super computer

One step forward, one backward

PS - I generally believe in the KISS principle, so I'll try to come back and list my solutions as I bumble my way to something. But in a week I've learned enough Unix to actually like it and got a couple of lanes of PE data into IGV so I can take a look see

Can I nominate this as the best post on seqanswers? You really deserve a prize.

[Jon_Keats]

I wonder how many people are in the same boat as me.

1) Institute bought a couple of GAIIs
2) No one has money to use them
3) Institute has internal competition to pay for a couple of runs (makes the donors feel better about their donation if someone uses the machines), and you are lucky enough to get funded
4) You send a couple of samples off to never-never land and someone sends back a terabyte drive or two with "next-gen sequencing data"
5) You quickly realize people that use to do survival curves in your bioinformatics core don't really know that Illumina fastq is different from Sanger fastq and the analysis they provide is limited at best
5) Now what do you do?
6) Google > seqanswers > let the misery begins

So what have I learned this week,

A) My boss should have made me read and do the "Unix and Perl for Biologist" tutorial years ago. Google it if you are new and a bench/gene jockey (sanger sequencing/microarray person) like me with no unix experience it is an excellent use of a day
B) A place called SourceForge exists
C) If I had a MAQ for my TOPHAT and a BOWTIE to go with my BWA I'd be better of than GERALD and his SAMTOOLS
D) just type "make" to compile...Opps that doesn't work if Xcode is not installed yet
E) No Mac OS comes with Xcode installed and if you have a Leopard machine, you better know where the OS install disks are as you can only install the new version for snow leopard that is not compatible...One would think that pancreatic cancer survivor Steve Jobs would try to make my life easier not harder
F) The genome is not the genome, ensembl is the place to get chromosomes but 1000 genomes is the place to get the genome.
G) BWA can align on my laptop...cool...next-gen/2nd gen alignment on a laptop and I though I needed a super computer

One step forward, one backward

PS - I generally believe in the KISS principle, so I'll try to come back and list my solutions as I bumble my way to something. But in a week I've learned enough Unix to actually like it and got a couple of lanes of PE data into IGV so I can take a look see