Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
Collapse
 
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
Previous template Next
  • #1

    Crossbow 1.2.1 On Hadoop - help needed

    We have installed a Crossbow 1.2.1 on our 3 node Hadoop Cluster [running Intel Hadoop Distrib]. After installation, I ran the sample test script ./cb_hadoop –test and that validated that all components (Bowtie, SOAPsnp and SRA toolkit) were installed fine and usable by Crossbow.

    After that we tried to run a sample program from e_coli following the instructions here - http://bowtie-bio.sourceforge.net/cr...l.shtml#hadoop. This job involved connecting to a NIH site to download a .SRA file (3 GB) and then use that for processing in further steps. The file is being downloaded fine and written into Hadoop before the Alignment phase starts.

    The job runs for a while and then fails, here is the term output:
    Crossbow job
    ------------
    Hadoop streaming commands in: /tmp/crossbow-8syPJjsEPD/invoke.scripts/cb.70647.hadoop.sh
    Running...
    ==========================
    Stage 1 of 4. Preprocess
    ==========================
    Wed Jun 26 15:27:11 PDT 2013
    Warning: $HADOOP_HOME is deprecated.

    packageJobJar: [/usr/local/bin/crossbow-1.2.1/Get.pm, /usr/local/bin/crossbow-1.2.1/Counters.pm, /usr/local/bin/crossbow-1.2.1/Util.pm, /usr/local/bin/crossbow-1.2.1/Tools.pm, /usr/local/bin/crossbow-1.2.1/AWS.pm, /tmp/hadoop-root/hadoop-unjar1303404851548420422/] [] /tmp/streamjob4479954697968619538.jar tmpDir=null
    13/06/26 15:27:19 INFO mapred.FileInputFormat: Total input paths to process : 1
    13/06/26 15:27:19 INFO streaming.StreamJob: getLocalDirs(): [/hdfs/hd1/hadoop/mapred, /hdfs/hd2/hadoop/mapred, /hdfs/hd3/hadoop/mapred, /hdfs/hd4/hadoop/mapred, /hdfs/hd5/hadoop/mapred, /hdfs/hd6/hadoop/mapred, /hdfs/hd7/hadoop/mapred, /hdfs/hd8/hadoop/mapred, /hdfs/hd9/hadoop/mapred, /hdfs/hd10/hadoop/mapred, /hdfs/hd11/hadoop/mapred, /hdfs/hd12/hadoop/mapred]
    13/06/26 15:27:19 INFO streaming.StreamJob: Running job: job_201305151132_0043
    13/06/26 15:27:19 INFO streaming.StreamJob: To kill this job, run:
    13/06/26 15:27:19 INFO streaming.StreamJob: /usr/lib/hadoop/libexec/../bin/hadoop job -Dmapred.job.tracker=idh-hssdn1:54311 -kill job_201305151132_0043
    13/06/26 15:27:19 INFO streaming.StreamJob: Tracking URL: http://idh-hssdn1:50030/jobdetails.j...305151132_0043
    13/06/26 15:27:20 INFO streaming.StreamJob: map 0% reduce 0%
    13/06/26 15:27:28 INFO streaming.StreamJob: map 33% reduce 0%
    13/06/26 15:27:35 INFO streaming.StreamJob: map 67% reduce 0%
    13/06/26 15:27:48 INFO streaming.StreamJob: map 100% reduce 0%
    13/06/26 16:21:58 INFO streaming.StreamJob: map 100% reduce 100%
    13/06/26 16:21:58 INFO streaming.StreamJob: Job complete: job_201305151132_0043
    13/06/26 16:21:58 INFO streaming.StreamJob: Output: hdfs:///crossbow/intermediate/70647/preproc
    ==========================
    Stage 2 of 4. Align
    ==========================
    Wed Jun 26 16:22:00 PDT 2013
    Warning: $HADOOP_HOME is deprecated.

    packageJobJar: [/usr/local/bin/crossbow-1.2.1/Get.pm, /usr/local/bin/crossbow-1.2.1/Counters.pm, /usr/local/bin/crossbow-1.2.1/Util.pm, /usr/local/bin/crossbow-1.2.1/Tools.pm, /usr/local/bin/crossbow-1.2.1/AWS.pm, /tmp/hadoop-root/hadoop-unjar7668567870786726388/] [] /tmp/streamjob3181086113362656316.jar tmpDir=null
    13/06/26 16:22:08 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    13/06/26 16:22:08 WARN snappy.LoadSnappy: Snappy native library is available
    13/06/26 16:22:08 INFO snappy.LoadSnappy: Snappy native library loaded
    13/06/26 16:22:08 INFO mapred.FileInputFormat: Total input paths to process : 21
    13/06/26 16:22:08 INFO streaming.StreamJob: getLocalDirs(): [/hdfs/hd1/hadoop/mapred, /hdfs/hd2/hadoop/mapred, /hdfs/hd3/hadoop/mapred, /hdfs/hd4/hadoop/mapred, /hdfs/hd5/hadoop/mapred, /hdfs/hd6/hadoop/mapred, /hdfs/hd7/hadoop/mapred, /hdfs/hd8/hadoop/mapred, /hdfs/hd9/hadoop/mapred, /hdfs/hd10/hadoop/mapred, /hdfs/hd11/hadoop/mapred, /hdfs/hd12/hadoop/mapred]
    13/06/26 16:22:08 INFO streaming.StreamJob: Running job: job_201305151132_0044
    13/06/26 16:22:08 INFO streaming.StreamJob: To kill this job, run:
    13/06/26 16:22:08 INFO streaming.StreamJob: /usr/lib/hadoop/libexec/../bin/hadoop job -Dmapred.job.tracker=idh-hssdn1:54311 -kill job_201305151132_0044
    13/06/26 16:22:08 INFO streaming.StreamJob: Tracking URL: http://idh-hssdn1:50030/jobdetails.j...305151132_0044
    13/06/26 16:22:09 INFO streaming.StreamJob: map 0% reduce 0%
    13/06/26 16:22:33 INFO streaming.StreamJob: map 4% reduce 0%
    13/06/26 16:22:34 INFO streaming.StreamJob: map 5% reduce 0%
    13/06/26 16:22:36 INFO streaming.StreamJob: map 9% reduce 0%
    13/06/26 16:22:37 INFO streaming.StreamJob: map 10% reduce 0%
    13/06/26 16:22:39 INFO streaming.StreamJob: map 14% reduce 0%
    13/06/26 16:22:40 INFO streaming.StreamJob: map 15% reduce 0%
    13/06/26 16:22:43 INFO streaming.StreamJob: map 20% reduce 0%
    13/06/26 16:22:46 INFO streaming.StreamJob: map 23% reduce 0%
    13/06/26 16:22:47 INFO streaming.StreamJob: map 25% reduce 0%
    13/06/26 16:22:49 INFO streaming.StreamJob: map 28% reduce 0%
    13/06/26 16:22:50 INFO streaming.StreamJob: map 30% reduce 0%
    13/06/26 16:22:52 INFO streaming.StreamJob: map 32% reduce 0%
    13/06/26 16:22:53 INFO streaming.StreamJob: map 33% reduce 0%
    13/06/26 16:22:58 INFO streaming.StreamJob: map 29% reduce 0%
    13/06/26 16:22:59 INFO streaming.StreamJob: map 19% reduce 0%
    13/06/26 16:23:00 INFO streaming.StreamJob: map 10% reduce 0%
    13/06/26 16:23:01 INFO streaming.StreamJob: map 5% reduce 0%
    13/06/26 16:23:02 INFO streaming.StreamJob: map 0% reduce 0%
    13/06/26 16:23:14 INFO streaming.StreamJob: map 100% reduce 100%
    13/06/26 16:23:14 INFO streaming.StreamJob: To kill this job, run:
    13/06/26 16:23:14 INFO streaming.StreamJob: /usr/lib/hadoop/libexec/../bin/hadoop job -Dmapred.job.tracker=idh-hssdn1:54311 -kill job_201305151132_0044
    13/06/26 16:23:14 INFO streaming.StreamJob: Tracking URL: http://idh-hssdn1:50030/jobdetails.j...305151132_0044
    13/06/26 16:23:14 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201305151132_0044_m_000013
    13/06/26 16:23:14 INFO streaming.StreamJob: killJob...
    Streaming Command Failed!
    Non-zero exitlevel from Align streaming job
    elastic-mapreduce script completed with exitlevel 256


    This is what I see in the logs:

    org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /crossbow/intermediate/70647/align/_temporary/_attempt_201305151132_0044_m_000006_2/part-00006 File does not exist. Holder DFSClient_attempt_201305151132_0044_m_000006_2 does not have any open files.
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1802)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1793)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1848)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1836)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:719)
    at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1414)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1410)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1408)


    Any ideas how to resolve this problem? If needed I can post contents of other logs.

    Thanks,

    Abhi
    Tags: None
  • #2
    New error observed this morning trying to re-run the example - "Broken Pipe"!!!

    org.apache.hadoop.streaming.PipeMapRed: java.io.IOException: Broken pipe
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:282)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:568)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
    at org.apache.hadoop.mapred.MapTask.runOldMapper_aroundBody2(MapTask.java:444)
    at org.apache.hadoop.mapred.MapTask$AjcClosure3.run(MapTask.java:1)
    at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    at com.intel.bigdata.management.agent.HadoopTaskAspect.doPhaseCall(HadoopTaskAspect.java:166)
    at com.intel.bigdata.management.agent.HadoopTaskAspect.ajc$inlineAccessMethod$com_intel_bigdata_management_agent_HadoopTaskAspect$com_intel_bigdata_management_agent_HadoopTaskAspect$doPhaseCall(HadoopTaskAspect.java:1)
    at com.intel.bigdata.management.agent.HadoopTaskAspect.aroundMap(HadoopTaskAspect.java:38)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
    at org.apache.hadoop.mapred.MapTask.run_aroundBody0(MapTask.java:377)
    at org.apache.hadoop.mapred.MapTask$AjcClosure1.run(MapTask.java:1)
    at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    at com.intel.bigdata.management.agent.HadoopTaskAspect.aroundTaskRun(HadoopTaskAspect.java:95)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:351)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
    at org.apache.hadoop.mapred.Child.main(Child.java:260)

    Comment

    • #3
      Is this a Hadoop configuration error? Standard hadoop tests like Terasort, Wordcount work fine. Any help is greatly appreciated. Thanks.

      Comment

      • #4
        Hello,

        I was wondering if you were able to resolve this. I am running into this problem.

        Comment

        • #5
          My problem got solved when I used the required version of Bowtie 0.12.8. Please try that first.

          Comment

          Previous template Next

          Latest Articles

          Collapse
          • seqadmin
            Non-Coding RNA Research and Technologies
            by seqadmin


            Non-coding RNAs (ncRNAs) do not code for proteins but play important roles in numerous cellular processes including gene silencing, developmental pathways, and more. There are numerous types including microRNA (miRNA), long ncRNA (lncRNA), circular RNA (circRNA), and more. In this article, we discuss innovative ncRNA research and explore recent technological advancements that improve the study of ncRNAs.
            ​
            [Article Coming Soon!]...
            Today, 08:07 AM
          • seqadmin
            Recent Developments in Metagenomics
            by seqadmin





            Metagenomics has improved the way researchers study microorganisms across diverse environments. Historically, studying microorganisms relied on culturing them in the lab, a method that limits the investigation of many species since most are unculturable1. Metagenomics overcomes these issues by allowing the study of microorganisms regardless of their ability to be cultured or the environments they inhabit. Over time, the field has evolved, especially with the advent...
            09-23-2024, 06:35 AM
          • seqadmin
            Understanding Genetic Influence on Infectious Disease
            by seqadmin




            During the COVID-19 pandemic, scientists observed that while some individuals experienced severe illness when infected with SARS-CoV-2, others were barely affected. These disparities left researchers and clinicians wondering what causes the wide variations in response to viral infections and what role genetics plays.

            Jean-Laurent Casanova, M.D., Ph.D., Professor at Rockefeller University, is a leading expert in this crossover between genetics and infectious...
            09-09-2024, 10:59 AM
          View All

          ad_right_rmr

          Collapse

          News

          Collapse
          Topics Statistics Last Post
          Mechanical Forces in DNA Transcription Uncovered by Clemson Researchers by seqadmin
          Started by seqadmin, 10-02-2024, 04:51 AM
          0 responses
          14 views
          0 likes
          		 Last Post seqadmin
          by seqadmin
          10-02-2024, 04:51 AM  
          New Epigenetic Clock Links Cheek Cells to Mortality Risk by seqadmin
          Started by seqadmin, 10-01-2024, 07:10 AM
          0 responses
          24 views
          0 likes
          		 Last Post seqadmin
          by seqadmin
          10-01-2024, 07:10 AM  
          AI-Powered Blood Test Shows Promise for Early Ovarian Cancer Detection by seqadmin
          Started by seqadmin, 09-30-2024, 08:33 AM
          1 response
          31 views
          0 likes
          		 Last Post EmiTom
          by EmiTom
          Today, 06:46 AM  
          Stem Cell Research Suggests Human Cells May Enter Developmental Pause by seqadmin
          Started by seqadmin, 09-26-2024, 12:57 PM
          0 responses
          19 views
          0 likes
          		 Last Post seqadmin
          by seqadmin
          09-26-2024, 12:57 PM  
          View All
          Working...
          X