concatenate??

[semna]

Hi
I have some bed files that I want to concatenate and also I want to have unique file as output(not overlapping). I tried with this script:
cat files | sort -u
But it seems that some overlapping. Anyone knows new script for this purpose?
Thanks in advance.

[steven]

you mean that you still have identical lines? that should not happen..
maybe because of leading/trailing spaces/tabs. i would try to remove them by inserting the command
sed 's/^[ \t]*//;s/[ \t]*$//'
between the cat and the sort.

[steven]

oh, if you mean "overlapping intervals", then try to use BEDtools to collapse/project/contig overlapping intervals -i don't remember the precise command though. does anyone else?

[semna]

Hi again,
let me explain more. I have some files like this:
chr7 27803437 27803887
chr7 36732616 36732791
chr7 81504256 81504424
chr7 101933606 101934986
chr7 57992467 57993280
chr7 81973420 81973558
chr7 43411272 43437604
I like to concatenate them and want to have unique lines as a result. because I know there are some repeated line that I don't want to come in my final ouput.
Thanks for your help

[steven]

well, then your "sort -u" should work fine. do you have an example of unfiltered duplicates?

[semna]

Hi steven
Thanks for your help. unfortunately I don't have. if this script is ok how can I test my output in order to sure about the result?
thanks

[steven]

yes, you can get duplicated lines by using.. "sort | uniq -d"
I would just trust "sort -u". Make sure all files come from the same source (windows/linux en of lines differ), have properly tab-separated fields (not some with spaces), have the same number of fields (not some with strand), and do not have trailing spaces.
I would use the sed command just in case, and test on the final bed file something like
cat bedfile | awk -F"\t" '{print NF}' | sort -u
should only return one number (3).

[semna]

Thanks very much steven. Actually I used your command :cat bedfile | awk -F"\t" '{print NF}' | sort -u . but the result was 1 and 3. So I think it means that something wrong. I found it in some part that there was not tab space. So I am trying to correct it. Honestly I am not good in bioinformatics.
Thanks

[steven]

ok, let's have a look at these problematic lines, just in case. try
awk -F "\t" 'NF!=3'
to check them, and if this is about an improper bed format with spaces instead of tabs, try
awk '{OFS="\t";print $0}'
to correct it.
let me know!

[semna]

First for second command I try but it's not corrected. but about the first one If I understand I use this:awk -F "\t" 'NF!=3' with my file but the result was just this:
track name='HMMSplicer Junctions' description='HMMSplicer Junctions' useScore=1
track name='HMMSplicer Non-cannonical Junctions' description='HMMSplicer Non-cannonical Junctions' useScore=1
and this is my output in some part:
chr6 383200 383444
chr6 383254 383453
chr6 383254 383456
chr6 38339289 38340680
chr6 38339296 38340680
chr6 38340702 38342087
chr6 383468 384211
chr6 383476 384207
chr6 38352485 38803482
chr6 38352516 38356426
chr6 38352518 38356428
chr6 38356435 38358938
some part is space but for the rest is tab. Thanks

[steven]

OK
looks like some of the bed have a header.
i'd then do
cat files.bed | awk 'NF==3{OFS="\t";print $1,$2,$3}' | sort -u > merged_unique.bed
and i would double check if there are lines starting with something else than "chr" with
awk '!/^chr/' merged_unique.bed
cheers,
s.

[semna]

thanks so much steven.it works.great

[steven]

cool. welcome.