Dear all
I hope there's someone here that can help me. I'm trying to run Crossbow on EMR via the command line, with a custom reference jar. The job gets through the align with bowtie step, but then fails at the Soapsnp step.
This is the stderr of the last task attempt:
Warning: No TOOLNAME file in tool directory: Bin
Soapsnp.pl: soapsnp: found: ./soapsnp, given:
Soapsnp.pl: s3cmd: found: /usr/bin/s3cmd, given:
Soapsnp.pl: jar: found: /usr/lib/jvm/java-6-sun/bin/jar, given:
Soapsnp.pl: hadoop: found: /home/hadoop/.versions/0.20.205/libexec/../bin/hadoop, given:
Soapsnp.pl: wget: found: /usr/bin/wget, given:
Soapsnp.pl: s3cfg:
Soapsnp.pl: soapsnp args: -2 -u -n -q
Soapsnp.pl: refdir:
Soapsnp.pl: snpdir:
Soapsnp.pl: partition length: 1000000
Soapsnp.pl: haploid ids: none
Soapsnp.pl: haploid arguments: -r 0.0001 -m
Soapsnp.pl: diploid arguments: -r 0.00005 -e 0.0001
Soapsnp.pl: base quality value: !
Soapsnp.pl: discard SNP bins: 0
Soapsnp.pl: dryrun: 0
Soapsnp.pl: ls -al
total 4
drwxr-xr-x 3 hadoop hadoop 4096 Oct 24 14:16 .
drwxr-xr-x 3 hadoop hadoop 17 Oct 24 14:16 ..
lrwxrwxrwx 1 hadoop hadoop 95 Oct 24 14:16 .job.jar.crc -> /mnt3/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/jars/.job.jar.crc
lrwxrwxrwx 1 hadoop hadoop 117 Oct 24 14:16 AWS.pm -> /mnt1/var/lib/hadoop/mapred/taskTracker/distcache/1326170264994024195_-1394371905_551961625/crossbow-emr/1.2.0/AWS.pm
lrwxrwxrwx 1 hadoop hadoop 120 Oct 24 14:16 Counters.pm -> /mnt2/var/lib/hadoop/mapred/taskTracker/distcache/5276300767665437818_572984849_551961625/crossbow-emr/1.2.0/Counters.pm
lrwxrwxrwx 1 hadoop hadoop 117 Oct 24 14:16 Get.pm -> /mnt1/var/lib/hadoop/mapred/taskTracker/distcache/-3285783341727707738_-295119297_551961625/crossbow-emr/1.2.0/Get.pm
lrwxrwxrwx 1 hadoop hadoop 91 Oct 24 14:16 META-INF -> /mnt3/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/jars/META-INF
lrwxrwxrwx 1 hadoop hadoop 120 Oct 24 14:16 Soapsnp.pl -> /mnt/var/lib/hadoop/mapred/taskTracker/distcache/5173834061477007000_-1158766529_551962625/crossbow-emr/1.2.0/Soapsnp.pl
lrwxrwxrwx 1 hadoop hadoop 115 Oct 24 14:16 Tools.pm -> /mnt/var/lib/hadoop/mapred/taskTracker/distcache/1759179445905053102_38002175_551962625/crossbow-emr/1.2.0/Tools.pm
lrwxrwxrwx 1 hadoop hadoop 118 Oct 24 14:16 Util.pm -> /mnt3/var/lib/hadoop/mapred/taskTracker/distcache/-3241708363905900832_2139188497_551962625/crossbow-emr/1.2.0/Util.pm
lrwxrwxrwx 1 hadoop hadoop 90 Oct 24 14:16 job.jar -> /mnt3/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/jars/job.jar
lrwxrwxrwx 1 hadoop hadoop 86 Oct 24 14:16 org -> /mnt3/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/jars/org
lrwxrwxrwx 1 hadoop hadoop 118 Oct 24 14:16 soapsnp -> /mnt3/var/lib/hadoop/mapred/taskTracker/distcache/3607859444418201245_902888079_273011625/crossbow-emr/1.2.0/soapsnp64
drwxr-xr-x 2 hadoop hadoop 6 Oct 24 14:16 tmp
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 2.
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 2.
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 3.
....
it complains about the same thing for 600000 lines
....
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 617744.
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 617745.
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 617746.
Read the last line of input
Soapsnp.pl: Genotyping chromosome chr0 5000000-6000000 using 617746 alignments: Wed Oct 24 14:18:54 UTC 2012
Soapsnp.pl: chromosome chr0 is diploid; using args "-r 0.00005 -e 0.0001"
Soapsnp.pl: head -4 .tmp.1000000.0005:
chr0 0005 004999899 - GATCATCCTCGCCTTGCAAAAATCCATCACCAACACGAAGATTTAGACCATTTTAAAACTTATAATTCTTTTCCAACCTACTTTACATTGTCCATTAAAAC CDCDDBDDBDDDECEEFFFFHHFHHJIHEIJHHIGJJJJJJIJJIHHJIIGHIJJIHDJJIHIJIJIIIHC4)FJIJIHHJJIHHJHGFHHHHFFFFFB@@ 0 28:T>C 2 r/2
chr0 0005 004999899 - GATCATCCTCGCCTTGCAAAAATCCATCACCAACACGAAGATTTAGACCATTTTAAAACTTATAATTCTTTTTCAACCTACTTTACATTGTCCATTAAAAC >C>CCBD?>DCCECEFFEFHHHHHEHCE@JJJJJHJJJJJJJJJJIHJIIGIIJJIIFJJJJJJJJIIJIJJJIIIJJHEJIJGFJJIGHHHHFFFFFCCC 0 - 2 r/2
chr0 0005 004999900 + ATCATCCTCGCCTTGCAAAAATCCATCACCAACACGAAGAGTTAGACCATTTTAAAACTTATAATTCTTTTTCAACCTACTTTACATTGTCCATTAAAACT CCCFFFFFHHHHHJJJJJJJJJJJJJJJJJJJJJJJJJJI/?FGIJIJJJIIJJJJJJIJJIHHHHHHHHFFFFFFEEEEECEDCDEEDEEDDEFDDDDDC 0 40:T>G 2 r/2
chr0 0005 004999902 + CATCCTCGCCTTGCAAAAATCCATCACCAACACGAAGATTTAGACCATTTTAAAACTTATAATTCTTTTTCAACCTACTTTACATTGTCCATTAAAACTAC CCCFFFFFHHGHDIJJJJJIJJJJJJJJJJJIJJIJJJJJJJJJJJJIJIHIGJJJJJJJJJJHHHHHHHFFFFFEDECEEDDDEEDEEDDEEDDDDDDDC 0 - 2 r/2
Soapsnp.pl: tail -4 .tmp.1000000.0005:
chr0 0005 005579007 - AATAGCAGTCTCTAAATCAATGTCTCCAGAAGGGACTTCCTTAAAGTCTGAGTCCTTCTTCAAAAATGGCTTTTGTCCATCAGTAGCAATCTTAGCATGTC A>5A;5A;-(;6>@A:C@BB@@@=3)=).;GCIHG@@@CCGHD>GBF<B?/9D;GF>FFBHFC::?9::3C<3FEC:??I:F<C,@@GDC+AA
;DA8;? 1 - 2 r/2
chr0 0005 005579012 - CAGTCTCTAAATCAATGTCTCCAGAAGGGACTTCCTTAAAGTCTGAGTCCTTCTTCAAAAATGGCTTTTGTCCATCAGTAGCAATCTTAGCATGTCTCTCC CCDCCDDDEDEDDDECACEFEBFFFGHHE?JJJJIJJJJJIJJJIGIIJHHJJJJJJJJIJJJIJJJJJJJJJJJJJJIJIJJJJIJJHHHHHFFDDA@@@ 1 - 2 r/2
chr0 0005 005579016 + CTCTAAATCAATGTCTCCAGAAGGGACTTCCTTAAAGTCTGAGTCCTTCTTCAAAAATGGCTTTTGTCCATCAGTAGCAATCTTAGCATGTCTCTCCAGAA @@<DDDDD?DBBFFE?CFH<AFCGBF=CFHIEHGIGGCHGCE@DDBFIIDGEIIIIIIIGCIGDHIFGGGIIIGICAEEHHHFFEEDDECCCD@CC>CACA 1 - 1 r/1
chr0 0005 005579019 - TAAATCAATGTCTCCAGAAGGGACTTCCTTAAAGTCTGAGTCCTTCTTCAAAAATGGCTTTTGTCCATCAGTAGCAATCTTAGCATGTCTCTCCAGAAGCA DEDC@ACC?;3?:A@C;EEDCDCB>@HEEAJIHEGGEGGF<GEBFB<FAEGIIHGHH@HGFHIHEECICDHGGEHAHHGIEEIIIIGEDHHFDDFEDD@B@ 1 - 2 r/2
Get.pm:ensureFetched: called on "S3N://crossbow-bucket/ref-pombe.jar"
Get.pm:ensureFetched: base name "ref-pombe.jar"
ls -al /mnt/8094/*ref-pombe.jar* /mnt/8094/.*ref-pombe.jar*
-rw-r--r-- 1 hadoop hadoop 0 Oct 24 13:46 /mnt/8094/.ref-pombe.jar.done
-rw-r--r-- 1 hadoop hadoop 0 Oct 24 13:46 /mnt/8094/.ref-pombe.jar.lock
-rw-r--r-- 1 hadoop hadoop 19422106 Oct 24 13:46 /mnt/8094/ref-pombe.jar
Pid 6220: Checking for done file /mnt/8094/.ref-pombe.jar.done
Pid 6220: done file /mnt/8094/.ref-pombe.jar.done was there already; continuing
Soapsnp.pl: Warning: /mnt/8094/snps/chr0.snps doesn't exist
Soapsnp.pl: ls -l /mnt/8094/snps
Soapsnp.pl: total 0
Soapsnp.pl: Warning: neither /mnt/8094/snps/chrchr0.snps nor /mnt/8094/snps/chr0.snps exist; not using known SNPs
Soapsnp.pl: ls -al /mnt/8094/snps
Soapsnp.pl: total 0
drwxr-xr-x 2 hadoop hadoop 6 Oct 22 14:11 .
drwxr-xr-x 6 hadoop hadoop 147 Oct 24 13:46 ..
Soapsnp.pl: ./soapsnp -i .tmp.1000000.0005 -d /mnt/8094/sequences/chr0.fa -o .tmp.snps -z '!' -L 101 -c -H -T .range_5000000_6000000 -r 0.00005 -e 0.0001 -2 -u -n -q >.soapsnp.6220.stdout 2>.soapsnp.6220.stderr
Soapsnp.pl: soapsnp returned 65280
Soapsnp.pl: command: ./soapsnp -i .tmp.1000000.0005 -d /mnt/8094/sequences/chr0.fa -o .tmp.snps -z '!' -L 101 -c -H -T .range_5000000_6000000 -r 0.00005 -e 0.0001 -2 -u -n -q >.soapsnp.6220.stdout 2>.soapsnp.6220.stderr
Soapsnp.pl: stdout from soapsnp:
Soapsnp.pl: stderr from soapsnp:
-i is set to .tmp.1000000.0005
-d is set to /mnt/8094/sequences/chr0.fa
-o is set to .tmp.snps
Standard Fastq System Set
-L is set to 101
-c is set
-T is set to .range_5000000_6000000
-r is set to 5e-05
-e is set to 0.0001
-2 is set
-u is set
-n is set
-q is set
Read 5579133 from 92987 lines of input FASTA sequence 14:18:54
Finished loading and binarizing chromosome 14:18:54
Finished parsing 0 known SNPs 14:18:54
Reading Chromosome and dbSNP information Done.
Unexpected Chromosome:chr0
Read target region done.
Training correction matrix in Crossbow format14:18:55
!0!
Assertion Failed: Chromosome: !chr0! NOT found
Soapsnp.pl: range: chr0 5000000 6000000
Soapsnp.pl: head -4 .tmp.snps:
Soapsnp.pl: tail -4 .tmp.snps:
Dying following soapsnp returning non-zero 65280 at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 350.
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:372)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:582)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:537)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:428)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
log4j:WARN Please initialize the log4j system properly.
The only non-standard thing is that I'm not using a prebuilt reference jar, I rolled my own ("ref-pombe.jar"). As far as I can follow the instructions in the Crossbow manual, it follows spec (including the cmap.txt that is not included in the instructions).
Could anyone see what is wrong with this?
Thanks!
I hope there's someone here that can help me. I'm trying to run Crossbow on EMR via the command line, with a custom reference jar. The job gets through the align with bowtie step, but then fails at the Soapsnp step.
This is the stderr of the last task attempt:
Warning: No TOOLNAME file in tool directory: Bin
Soapsnp.pl: soapsnp: found: ./soapsnp, given:
Soapsnp.pl: s3cmd: found: /usr/bin/s3cmd, given:
Soapsnp.pl: jar: found: /usr/lib/jvm/java-6-sun/bin/jar, given:
Soapsnp.pl: hadoop: found: /home/hadoop/.versions/0.20.205/libexec/../bin/hadoop, given:
Soapsnp.pl: wget: found: /usr/bin/wget, given:
Soapsnp.pl: s3cfg:
Soapsnp.pl: soapsnp args: -2 -u -n -q
Soapsnp.pl: refdir:
Soapsnp.pl: snpdir:
Soapsnp.pl: partition length: 1000000
Soapsnp.pl: haploid ids: none
Soapsnp.pl: haploid arguments: -r 0.0001 -m
Soapsnp.pl: diploid arguments: -r 0.00005 -e 0.0001
Soapsnp.pl: base quality value: !
Soapsnp.pl: discard SNP bins: 0
Soapsnp.pl: dryrun: 0
Soapsnp.pl: ls -al
total 4
drwxr-xr-x 3 hadoop hadoop 4096 Oct 24 14:16 .
drwxr-xr-x 3 hadoop hadoop 17 Oct 24 14:16 ..
lrwxrwxrwx 1 hadoop hadoop 95 Oct 24 14:16 .job.jar.crc -> /mnt3/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/jars/.job.jar.crc
lrwxrwxrwx 1 hadoop hadoop 117 Oct 24 14:16 AWS.pm -> /mnt1/var/lib/hadoop/mapred/taskTracker/distcache/1326170264994024195_-1394371905_551961625/crossbow-emr/1.2.0/AWS.pm
lrwxrwxrwx 1 hadoop hadoop 120 Oct 24 14:16 Counters.pm -> /mnt2/var/lib/hadoop/mapred/taskTracker/distcache/5276300767665437818_572984849_551961625/crossbow-emr/1.2.0/Counters.pm
lrwxrwxrwx 1 hadoop hadoop 117 Oct 24 14:16 Get.pm -> /mnt1/var/lib/hadoop/mapred/taskTracker/distcache/-3285783341727707738_-295119297_551961625/crossbow-emr/1.2.0/Get.pm
lrwxrwxrwx 1 hadoop hadoop 91 Oct 24 14:16 META-INF -> /mnt3/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/jars/META-INF
lrwxrwxrwx 1 hadoop hadoop 120 Oct 24 14:16 Soapsnp.pl -> /mnt/var/lib/hadoop/mapred/taskTracker/distcache/5173834061477007000_-1158766529_551962625/crossbow-emr/1.2.0/Soapsnp.pl
lrwxrwxrwx 1 hadoop hadoop 115 Oct 24 14:16 Tools.pm -> /mnt/var/lib/hadoop/mapred/taskTracker/distcache/1759179445905053102_38002175_551962625/crossbow-emr/1.2.0/Tools.pm
lrwxrwxrwx 1 hadoop hadoop 118 Oct 24 14:16 Util.pm -> /mnt3/var/lib/hadoop/mapred/taskTracker/distcache/-3241708363905900832_2139188497_551962625/crossbow-emr/1.2.0/Util.pm
lrwxrwxrwx 1 hadoop hadoop 90 Oct 24 14:16 job.jar -> /mnt3/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/jars/job.jar
lrwxrwxrwx 1 hadoop hadoop 86 Oct 24 14:16 org -> /mnt3/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/jars/org
lrwxrwxrwx 1 hadoop hadoop 118 Oct 24 14:16 soapsnp -> /mnt3/var/lib/hadoop/mapred/taskTracker/distcache/3607859444418201245_902888079_273011625/crossbow-emr/1.2.0/soapsnp64
drwxr-xr-x 2 hadoop hadoop 6 Oct 24 14:16 tmp
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 2.
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 2.
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 3.
....
it complains about the same thing for 600000 lines
....
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 617744.
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 617745.
Argument "chr0" isn't numeric in numeric ne (!=) at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 214, <STDIN> line 617746.
Read the last line of input
Soapsnp.pl: Genotyping chromosome chr0 5000000-6000000 using 617746 alignments: Wed Oct 24 14:18:54 UTC 2012
Soapsnp.pl: chromosome chr0 is diploid; using args "-r 0.00005 -e 0.0001"
Soapsnp.pl: head -4 .tmp.1000000.0005:
chr0 0005 004999899 - GATCATCCTCGCCTTGCAAAAATCCATCACCAACACGAAGATTTAGACCATTTTAAAACTTATAATTCTTTTCCAACCTACTTTACATTGTCCATTAAAAC CDCDDBDDBDDDECEEFFFFHHFHHJIHEIJHHIGJJJJJJIJJIHHJIIGHIJJIHDJJIHIJIJIIIHC4)FJIJIHHJJIHHJHGFHHHHFFFFFB@@ 0 28:T>C 2 r/2
chr0 0005 004999899 - GATCATCCTCGCCTTGCAAAAATCCATCACCAACACGAAGATTTAGACCATTTTAAAACTTATAATTCTTTTTCAACCTACTTTACATTGTCCATTAAAAC >C>CCBD?>DCCECEFFEFHHHHHEHCE@JJJJJHJJJJJJJJJJIHJIIGIIJJIIFJJJJJJJJIIJIJJJIIIJJHEJIJGFJJIGHHHHFFFFFCCC 0 - 2 r/2
chr0 0005 004999900 + ATCATCCTCGCCTTGCAAAAATCCATCACCAACACGAAGAGTTAGACCATTTTAAAACTTATAATTCTTTTTCAACCTACTTTACATTGTCCATTAAAACT CCCFFFFFHHHHHJJJJJJJJJJJJJJJJJJJJJJJJJJI/?FGIJIJJJIIJJJJJJIJJIHHHHHHHHFFFFFFEEEEECEDCDEEDEEDDEFDDDDDC 0 40:T>G 2 r/2
chr0 0005 004999902 + CATCCTCGCCTTGCAAAAATCCATCACCAACACGAAGATTTAGACCATTTTAAAACTTATAATTCTTTTTCAACCTACTTTACATTGTCCATTAAAACTAC CCCFFFFFHHGHDIJJJJJIJJJJJJJJJJJIJJIJJJJJJJJJJJJIJIHIGJJJJJJJJJJHHHHHHHFFFFFEDECEEDDDEEDEEDDEEDDDDDDDC 0 - 2 r/2
Soapsnp.pl: tail -4 .tmp.1000000.0005:
chr0 0005 005579007 - AATAGCAGTCTCTAAATCAATGTCTCCAGAAGGGACTTCCTTAAAGTCTGAGTCCTTCTTCAAAAATGGCTTTTGTCCATCAGTAGCAATCTTAGCATGTC A>5A;5A;-(;6>@A:C@BB@@@=3)=).;GCIHG@@@CCGHD>GBF<B?/9D;GF>FFBHFC::?9::3C<3FEC:??I:F<C,@@GDC+AA
;DA8;? 1 - 2 r/2chr0 0005 005579012 - CAGTCTCTAAATCAATGTCTCCAGAAGGGACTTCCTTAAAGTCTGAGTCCTTCTTCAAAAATGGCTTTTGTCCATCAGTAGCAATCTTAGCATGTCTCTCC CCDCCDDDEDEDDDECACEFEBFFFGHHE?JJJJIJJJJJIJJJIGIIJHHJJJJJJJJIJJJIJJJJJJJJJJJJJJIJIJJJJIJJHHHHHFFDDA@@@ 1 - 2 r/2
chr0 0005 005579016 + CTCTAAATCAATGTCTCCAGAAGGGACTTCCTTAAAGTCTGAGTCCTTCTTCAAAAATGGCTTTTGTCCATCAGTAGCAATCTTAGCATGTCTCTCCAGAA @@<DDDDD?DBBFFE?CFH<AFCGBF=CFHIEHGIGGCHGCE@DDBFIIDGEIIIIIIIGCIGDHIFGGGIIIGICAEEHHHFFEEDDECCCD@CC>CACA 1 - 1 r/1
chr0 0005 005579019 - TAAATCAATGTCTCCAGAAGGGACTTCCTTAAAGTCTGAGTCCTTCTTCAAAAATGGCTTTTGTCCATCAGTAGCAATCTTAGCATGTCTCTCCAGAAGCA DEDC@ACC?;3?:A@C;EEDCDCB>@HEEAJIHEGGEGGF<GEBFB<FAEGIIHGHH@HGFHIHEECICDHGGEHAHHGIEEIIIIGEDHHFDDFEDD@B@ 1 - 2 r/2
Get.pm:ensureFetched: called on "S3N://crossbow-bucket/ref-pombe.jar"
Get.pm:ensureFetched: base name "ref-pombe.jar"
ls -al /mnt/8094/*ref-pombe.jar* /mnt/8094/.*ref-pombe.jar*
-rw-r--r-- 1 hadoop hadoop 0 Oct 24 13:46 /mnt/8094/.ref-pombe.jar.done
-rw-r--r-- 1 hadoop hadoop 0 Oct 24 13:46 /mnt/8094/.ref-pombe.jar.lock
-rw-r--r-- 1 hadoop hadoop 19422106 Oct 24 13:46 /mnt/8094/ref-pombe.jar
Pid 6220: Checking for done file /mnt/8094/.ref-pombe.jar.done
Pid 6220: done file /mnt/8094/.ref-pombe.jar.done was there already; continuing
Soapsnp.pl: Warning: /mnt/8094/snps/chr0.snps doesn't exist
Soapsnp.pl: ls -l /mnt/8094/snps
Soapsnp.pl: total 0
Soapsnp.pl: Warning: neither /mnt/8094/snps/chrchr0.snps nor /mnt/8094/snps/chr0.snps exist; not using known SNPs
Soapsnp.pl: ls -al /mnt/8094/snps
Soapsnp.pl: total 0
drwxr-xr-x 2 hadoop hadoop 6 Oct 22 14:11 .
drwxr-xr-x 6 hadoop hadoop 147 Oct 24 13:46 ..
Soapsnp.pl: ./soapsnp -i .tmp.1000000.0005 -d /mnt/8094/sequences/chr0.fa -o .tmp.snps -z '!' -L 101 -c -H -T .range_5000000_6000000 -r 0.00005 -e 0.0001 -2 -u -n -q >.soapsnp.6220.stdout 2>.soapsnp.6220.stderr
Soapsnp.pl: soapsnp returned 65280
Soapsnp.pl: command: ./soapsnp -i .tmp.1000000.0005 -d /mnt/8094/sequences/chr0.fa -o .tmp.snps -z '!' -L 101 -c -H -T .range_5000000_6000000 -r 0.00005 -e 0.0001 -2 -u -n -q >.soapsnp.6220.stdout 2>.soapsnp.6220.stderr
Soapsnp.pl: stdout from soapsnp:
Soapsnp.pl: stderr from soapsnp:
-i is set to .tmp.1000000.0005
-d is set to /mnt/8094/sequences/chr0.fa
-o is set to .tmp.snps
Standard Fastq System Set
-L is set to 101
-c is set
-T is set to .range_5000000_6000000
-r is set to 5e-05
-e is set to 0.0001
-2 is set
-u is set
-n is set
-q is set
Read 5579133 from 92987 lines of input FASTA sequence 14:18:54
Finished loading and binarizing chromosome 14:18:54
Finished parsing 0 known SNPs 14:18:54
Reading Chromosome and dbSNP information Done.
Unexpected Chromosome:chr0
Read target region done.
Training correction matrix in Crossbow format14:18:55
!0!
Assertion Failed: Chromosome: !chr0! NOT found
Soapsnp.pl: range: chr0 5000000 6000000
Soapsnp.pl: head -4 .tmp.snps:
Soapsnp.pl: tail -4 .tmp.snps:
Dying following soapsnp returning non-zero 65280 at /mnt2/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201210241342_0002/attempt_201210241342_0002_r_000011_3/work/./Soapsnp.pl line 350.
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:372)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:582)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:537)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:428)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
log4j:WARN Please initialize the log4j system properly.
The only non-standard thing is that I'm not using a prebuilt reference jar, I rolled my own ("ref-pombe.jar"). As far as I can follow the instructions in the Crossbow manual, it follows spec (including the cmap.txt that is not included in the instructions).
Could anyone see what is wrong with this?
Thanks!