Crossbow on EMR (bowtie)

[huzefa_mehta]

Hello,

I am running Bowtie using Crossbow on EMR, and getting the following. Any clues would be appreciated.

Thanks,


-----------------------------------

2013-09-05 21:03:44,825 WARN org.apache.hadoop.conf.Configuration (main): DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
2013-09-05 21:03:45,772 INFO org.apache.hadoop.mapred.JobClient (main): Default number of map tasks: null
2013-09-05 21:03:45,772 INFO org.apache.hadoop.mapred.JobClient (main): Setting default number of map tasks based on cluster size to : 84
2013-09-05 21:03:45,772 INFO org.apache.hadoop.mapred.JobClient (main): Default number of reduce tasks: 0
2013-09-05 21:03:51,187 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2013-09-05 21:03:51,208 WARN com.hadoop.compression.lzo.LzoCodec (main): Could not find build properties file with revision hash
2013-09-05 21:03:51,208 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev UNKNOWN]
2013-09-05 21:03:51,227 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy (main): Snappy native library is available
2013-09-05 21:03:51,227 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy (main): Snappy native library loaded
2013-09-05 21:03:51,336 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input paths to process : 1
2013-09-05 21:03:51,925 INFO org.apache.hadoop.streaming.StreamJob (main): getLocalDirs(): [/mnt/var/lib/hadoop/mapred, /mnt1/var/lib/hadoop/mapred, /mnt2/var/lib/hadoop/mapred, /mnt3/var/lib/hadoop/mapred]
2013-09-05 21:03:51,925 INFO org.apache.hadoop.streaming.StreamJob (main): Running job: job_201309052100_0001
2013-09-05 21:03:51,925 INFO org.apache.hadoop.streaming.StreamJob (main): To kill this job, run:
2013-09-05 21:03:51,925 INFO org.apache.hadoop.streaming.StreamJob (main): /home/hadoop/bin/hadoop job -Dmapred.job.tracker=172.31.2.68:9001 -kill job_201309052100_0001
2013-09-05 21:03:51,926 INFO org.apache.hadoop.streaming.StreamJob (main): Tracking URL: http://ip-172-31-2-68:9100/jobdetail...309052100_0001
2013-09-05 21:03:52,930 INFO org.apache.hadoop.streaming.StreamJob (main): map 0% reduce 0%
2013-09-05 21:04:15,006 INFO org.apache.hadoop.streaming.StreamJob (main): map 4% reduce 0%
2013-09-05 21:04:18,015 INFO org.apache.hadoop.streaming.StreamJob (main): map 7% reduce 0%
2013-09-05 21:04:21,024 INFO org.apache.hadoop.streaming.StreamJob (main): map 9% reduce 0%
2013-09-05 21:04:24,032 INFO org.apache.hadoop.streaming.StreamJob (main): map 43% reduce 0%
2013-09-05 21:04:25,035 INFO org.apache.hadoop.streaming.StreamJob (main): map 55% reduce 0%
2013-09-05 21:04:27,040 INFO org.apache.hadoop.streaming.StreamJob (main): map 84% reduce 0%
2013-09-05 21:04:33,056 INFO org.apache.hadoop.streaming.StreamJob (main): map 95% reduce 0%
2013-09-05 21:04:36,064 INFO org.apache.hadoop.streaming.StreamJob (main): map 96% reduce 0%
2013-09-05 21:04:54,110 INFO org.apache.hadoop.streaming.StreamJob (main): map 100% reduce 100%
2013-09-05 21:04:54,112 INFO org.apache.hadoop.streaming.StreamJob (main): To kill this job, run:
2013-09-05 21:04:54,112 INFO org.apache.hadoop.streaming.StreamJob (main): /home/hadoop/bin/hadoop job -Dmapred.job.tracker=172.31.2.68:9001 -kill job_201309052100_0001
2013-09-05 21:04:54,113 INFO org.apache.hadoop.streaming.StreamJob (main): Tracking URL: http://ip-172-31-2-68:9100/jobdetail...309052100_0001
2013-09-05 21:04:54,113 ERROR org.apache.hadoop.streaming.StreamJob (main): Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201309052100_0001_m_000000
2013-09-05 21:04:54,113 INFO org.apache.hadoop.streaming.StreamJob (main): killJob...

[Ashwinub]

Hello Huzefa,

Did you manage to solve the issue? I am getting the similar error.

Thanks