Unconfigured Ad

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • OstanNick
    Member
    • Apr 2014
    • 26

    #1

    Invalid Whitelist Format (filtering step)

    Hello,

    I successfully got an assembly using the whitelisting method for one of my eight BACs a couple days ago.

    I then generated whitelists for my other 7 BACs, and attempted to repeat the HGAP assembly process, however all of a sudden my whitelists are of an 'invalid format'.

    My workflow fails at the filter001_of_001 step & fetch adatperrpt fails.

    I went into the filter001_of_001 error log file and pinpointed the following message:

    Code:
    Invalid whitelist format for '' in file /opt/smrtanalysis/data/45/whitelist.txt.
    The whitelist is a .txt file, the same as my other whitelist, and has no different internal formatting, for example:

    Code:
    m130725_204700_42158_c100524682550000001823080609281380_s1_p0/0
    m130725_204700_42158_c100524682550000001823080609281380_s1_p0/1
    m130725_204700_42158_c100524682550000001823080609281380_s1_p0/2
    m130725_204700_42158_c100524682550000001823080609281380_s1_p0/3
    m130725_204700_42158_c100524682550000001823080609281380_s1_p0/4
    is the whitelist which is NOT working, and

    Code:
    m130726_012611_42158_c100524682550000001823080609281382_s1_p0/0
    m130726_012611_42158_c100524682550000001823080609281382_s1_p0/1
    m130726_012611_42158_c100524682550000001823080609281382_s1_p0/2
    m130726_012611_42158_c100524682550000001823080609281382_s1_p0/3
    m130726_012611_42158_c100524682550000001823080609281382_s1_p0/4
    is the whitelist which IS working.

    They are both named whitelist.txt and I have specified this in my run parameters, they just lie in different directories ../45/whitelist.txt and ../102/whitelist.txt.

    I have re-run my original whitelist run and it still works (protocol seems to be fine), but when I run the same protocol with a different whitelist I recieve the same error.

    Here are more relevant log files:


    Code:
    #!/bin/bash
    # Setting up SMRTpipe environment
    echo "Setting up ENV on $(uname -n)" for task filter_001of001
    
    SEYMOUR_HOME=/opt/smrtanalysis/current
    source $SEYMOUR_HOME/etc/setup.sh
    
    # Create the local TMP dir if it doesn't exist
    tmp_dir=$(readlink -m "/opt/smrtanalysis/tmpdir")
    if [ ! -e "$tmp_dir" ]; then
       stat=0
       mkdir -p $tmp_dir || stat=$?
       if [[ $stat -ne 0 ]]; then
           echo "SMRTpipe Unable to create TMP dir '/opt/smrtanalysis/tmpdir' on $(uname -n)" 1>&2
           exit 1
       else
           echo "successfully created or found TMP dir '/opt/smrtanalysis/tmpdir'"
       fi
    elif [[ ! -d "$tmp_dir" ]]; then
       echo "SMRTpipe TMP /opt/smrtanalysis/tmpdir must be a directory on $(uname -n)" 1>&2
       exit 1
    fi
    
    ########### TASK metadata #############
    # Task            : filter_001of001
    # Module          : P_Filter
    # Module Version  : 1.7.133178
    # TaskType        : None
    # URL             : task://016450/P_Filter/filter_001of001
    # createdAt       : 2014-07-02 13:32:29.396088
    # createdAt (UTC) : 2014-07-02 17:32:29.396095
    # ncmds           : 1
    # LogPath         : /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log
    # Script Path     : /opt/smrtanalysis/current/common/jobs/016/016450/workflow/P_Filter/filter_001of001.sh
    
    # Input       : /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn
    # Output      : /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv
    # Output      : /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions
    # Output      : /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn
    #
    ########### END TASK metadata #############
    
    # Writing to log file
    cat /opt/smrtanalysis/current/common/jobs/016/016450/workflow/P_Filter/filter_001of001.sh >> /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log;
    
    
    
    echo "Running task://016450/P_Filter/filter_001of001 on $(uname -a)"
    
    echo "Started on $(date -u)"
    echo 'Validating existence of Input Files'
    if [ -e /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn ]
    then
    echo 'Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn'
    else
    echo 'WARNING: Unable to find necessary input file, or dir /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn.'
    fi
    echo 'Successfully validated input files'
    
    # Task filter_001of001 commands:
    
    
    # Completed writing Task filter_001of001 commands
    
    
    # Task 1
    filter_plsh5.py --debug --filter='MinReadScore=0.8000,MinSRL=500,MinRL=100,ReadWhitelist=/opt/smrtanalysis/data/45/whitelist.txt' --trim='True' --outputDir=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions --outputSummary=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv --outputFofn=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn || exit $?
    echo "Task 1 completed at $(date)"
    
    
    
    rcode=$?
    echo "Finished on $(date -u)"
    echo "Task filter_001of001 with nproc 2 with exit code ${rcode}."
    exit ${rcode}# Writing stdout and stderr from Popen:
    Setting up ENV on nick for task filter_001of001
    Running task://016450/P_Filter/filter_001of001 on Linux nick 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
    Started on Wed Jul  2 17:32:43 UTC 2014
    Validating existence of Input Files
    Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn
    Successfully validated input files
    No handlers could be found for logger "pbfilter.filter_plsh5"
    Traceback (most recent call last):
      File "/opt/smrtanalysis/current/analysis/bin/filter_plsh5.py", line 8, in <module>
        load_entry_point('pbfilter==1.2', 'console_scripts', 'filter_plsh5.py')()
      File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 636, in main
        filters.extend(_to_filters(filter_str))
      File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 581, in _to_filters
        filters.append(SUPPORTED_FILTERS[filterName][2](threshold))
      File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 392, in __init__
        raise ValueError(emsg)
    ValueError: Invalid whitelist format for '' in file /opt/smrtanalysis/data/45/whitelist.txt.
    THE END OF MY MASTER.LOG
    Code:
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > ########### END TASK metadata #############
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Writing to log file
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > cat /opt/smrtanalysis/current/common/jobs/016/016450/workflow/P_Filter/filter_001of001.sh >> /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log;
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Running task://016450/P_Filter/filter_001of001 on $(uname -a)"
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Started on $(date -u)"
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'Validating existence of Input Files'
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > if [ -e /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn ]
    [ERROR] 2014-07-02 13:32:44,066 [SMRTpipe.engine.SmrtPipeTasks run 703] > then
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn'
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > else
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'WARNING: Unable to find necessary input file, or dir /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn.'
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > fi
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo 'Successfully validated input files'
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Task filter_001of001 commands:
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Completed writing Task filter_001of001 commands
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > # Task 1
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > filter_plsh5.py --debug --filter='MinReadScore=0.8000,MinSRL=500,MinRL=100,ReadWhitelist=/opt/smrtanalysis/data/45/whitelist.txt' --trim='True' --outputDir=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions --outputSummary=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv --outputFofn=/opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn || exit $?
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Task 1 completed at $(date)"
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > 
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > rcode=$?
    [ERROR] 2014-07-02 13:32:44,067 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Finished on $(date -u)"
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > echo "Task filter_001of001 with nproc 2 with exit code ${rcode}."
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > exit ${rcode}# Writing stdout and stderr from Popen:
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Setting up ENV on nick for task filter_001of001
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Running task://016450/P_Filter/filter_001of001 on Linux nick 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Started on Wed Jul  2 17:32:43 UTC 2014
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Validating existence of Input Files
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Successfully found /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Successfully validated input files
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > No handlers could be found for logger "pbfilter.filter_plsh5"
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > Traceback (most recent call last):
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/analysis/bin/filter_plsh5.py", line 8, in <module>
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > load_entry_point('pbfilter==1.2', 'console_scripts', 'filter_plsh5.py')()
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 636, in main
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > filters.extend(_to_filters(filter_str))
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 581, in _to_filters
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > filters.append(SUPPORTED_FILTERS[filterName][2](threshold))
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/pbfilter/filter_plsh5.py", line 392, in __init__
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > raise ValueError(emsg)
    [ERROR] 2014-07-02 13:32:44,068 [SMRTpipe.engine.SmrtPipeTasks run 703] > ValueError: Invalid whitelist format for '' in file /opt/smrtanalysis/data/45/whitelist.txt.
    [DEBUG] 2014-07-02 13:32:48,253 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 372] Grabbed task://016450/P_Filter/filter_001of001 with message started, runflag: 1 from Queue
    [INFO] 2014-07-02 13:32:48,254 [smrtpipe.status refreshTargets 419] Starting task://016450/P_Filter/filter_001of001
    [DEBUG] 2014-07-02 13:32:48,254 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 372] Grabbed task://016450/P_Filter/filter_001of001 with message fail from Queue
    [ERROR] 2014-07-02 13:32:48,254 [smrtpipe.status refreshTargets 413] *** Failed task task://016450/P_Filter/filter_001of001
    [INFO] 2014-07-02 13:32:48,255 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 323] Exiting Workflow while loop. No Jobs to be submitted and No alive threads.
    [INFO] 2014-07-02 13:32:48,255 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/genFrgFile, None
    [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/getUnitigs, None
    [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/runCaToUnitig, None
    [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/unitigConsensus, None
    [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssembleUnitig/writeRunCASpec, None
    [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/callConsensus, None
    [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/enrichAlnSummary, None
    [INFO] 2014-07-02 13:32:48,256 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/polishedJsonReport, None
    [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/topCorrectionsJsonReport, None
    [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/variantsJsonReport, None
    [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_AssemblyPolishing/zipPolishedFasta, None
    [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/adapterRpt, fail
    [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/getChemistry, done
    [INFO] 2014-07-02 13:32:48,257 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/overviewRpt, done
    [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Fetch/toFofn, done
    [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter.plsFofn.Scatter, done
    [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter.rgnFofn.Gather, None
    [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter.summary.Gather, None
    [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/filter_001of001, fail
    [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreadSummary, None
    [INFO] 2014-07-02 13:32:48,258 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreads.subreadFastq.Gather, None
    [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreads.subreads.Gather, None
    [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Filter/subreads_001of001, None
    [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_FilterReports/loadingRpt, None
    [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_FilterReports/statsRpt, None
    [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_FilterReports/subreadRpt, None
    [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/align, None
    [INFO] 2014-07-02 13:32:48,259 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/covGFF, None
    [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/gff2Bed, None
    [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/loadChemistry, None
    [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/repack, None
    [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/samBam, None
    [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/sort, None
    [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_Mapping/unmapped, None
    [INFO] 2014-07-02 13:32:48,260 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_MappingReports/coverageJsonReport, None
    [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_MappingReports/statsJsonReport, None
    [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/filterLongReadsByLength, None
    [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/hgapAlignForCorrection, None
    [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/hgapCorrection, None
    [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/hgapFilterM4, None
    [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_PreAssemblerDagcon/preAssemblerJsonReport, None
    [INFO] 2014-07-02 13:32:48,261 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 441] task status: task://016450/P_ReferenceUploader/runUploaderUnitig, None
    [DEBUG] 2014-07-02 13:32:48,262 [SMRTpipe.engine.SmrtPipeWorkflow refreshTargets 448] Number of task2thead 6
    [INFO] 2014-07-02 13:32:48,262 [smrtpipe.status execute 627] Found 2 failed tasks.
    [INFO] 2014-07-02 13:32:48,263 [smrtpipe.status execute 629] task adapterRpt FAILED 
    [DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 630] task adapterRpt FAILED with error task://016450/P_Fetch/adapterRpt Failed
    [DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 631] task adapterRpt FAILED with errors {}
    [INFO] 2014-07-02 13:32:48,263 [smrtpipe.status execute 629] task filter_001of001 FAILED 
    [DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 630] task filter_001of001 FAILED with error task://016450/P_Filter/filter_001of001 Failed
    [DEBUG] 2014-07-02 13:32:48,263 [SMRTpipe.engine.SmrtPipeWorkflow execute 631] task filter_001of001 FAILED with errors {'Unable to locate BaseCalls within hdf5 file': 'No Base Calls found in Primary; Pulse2Base may have failed.'}
    [ERROR] 2014-07-02 13:32:48,264 [SMRTpipe.engine.SmrtPipeWorkflow execute 637] task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
    [DEBUG] 2014-07-02 13:32:48,264 [SMRTpipe.engine.SmrtPipeWorkflow _update 602] Updating Workflow state
    [DEBUG] 2014-07-02 13:32:48,274 [SMRTpipe.engine.SmrtPipeTasks _extractComputeTime 864] <P_Fetch/adapterRpt> Found Started : Started on Wed Jul  2 17:32:35 UTC 2014
    [WARNING] 2014-07-02 13:32:48,275 [SMRTpipe.engine.SmrtPipeTasks computeTime 884] Unable to extract computeTime from /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Fetch/adapterRpt.log.
    [DEBUG] 2014-07-02 13:32:48,280 [SMRTpipe.engine.SmrtPipeTasks _extractComputeTime 864] <P_Filter/filter_001of001> Found Started : Started on Wed Jul  2 17:32:43 UTC 2014
    [WARNING] 2014-07-02 13:32:48,280 [SMRTpipe.engine.SmrtPipeTasks computeTime 884] Unable to extract computeTime from /opt/smrtanalysis/current/common/jobs/016/016450/log/P_Filter/filter_001of001.log.
    [DEBUG] 2014-07-02 13:32:48,318 [SMRTpipe.engine.SmrtPipeWorkflow _writeWorkflow 578] Writing DAG to /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.rdf
    [DEBUG] 2014-07-02 13:32:48,321 [SMRTpipe.engine.SmrtPipeWorkflow _writeWorkflow 578] Writing DAG to /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.dot
    [DEBUG] 2014-07-02 13:32:48,447 [SMRTpipe.engine.SmrtPipeWorkflow _writeWorkflow 578] Writing DAG to /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.dot
    [DEBUG] 2014-07-02 13:32:48,452 [SMRTpipe.SmrtWorker backticks 188] Running on nick with cmd dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg; addSVGLinks.py --size 1400,770 /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg
    [DEBUG] 2014-07-02 13:32:48,501 [SMRTpipe.SmrtWorker backticks 212] Successful output (Return code = 0) in 0.05 sec (0.00 min) of dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg; addSVGLinks.py --size 1400,770 /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.summary.svg
    [DEBUG] 2014-07-02 13:32:48,506 [SMRTpipe.SmrtWorker backticks 188] Running on nick with cmd dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg; addSVGLinks.py --size 2800,1540 --linkTasks /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg
    [DEBUG] 2014-07-02 13:32:48,557 [SMRTpipe.SmrtWorker backticks 212] Successful output (Return code = 0) in 0.05 sec (0.00 min) of dot -Tsvg /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.dot -o/opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg; addSVGLinks.py --size 2800,1540 --linkTasks /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg > /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new; mv /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg.new /opt/smrtanalysis/current/common/jobs/016/016450/workflow/Workflow.details.svg
    [DEBUG] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeWorkflow cleanChunkedFiles 607] cleaning up chunked Workflow Workflow files
    [WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_regions.chunk001of001.fofn for deleting.
    [DEBUG] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 202] Removing chunk file /opt/smrtanalysis/current/common/jobs/016/016450/input.chunk001of001.fofn
    [WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_summary.chunk001of001.csv for deleting.
    [WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_subreads.chunk001of001.fasta for deleting.
    [WARNING] 2014-07-02 13:32:48,558 [SMRTpipe.engine.SmrtPipeFiles cleanChunkedFiles 205] Could not find expected /opt/smrtanalysis/current/common/jobs/016/016450/data/filtered_subreads.chunk001of001.fastq for deleting.
    [ERROR] 2014-07-02 13:32:48,558 [SMRTpipe.SmrtPipeMain run 608] SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
    Traceback (most recent call last):
      File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/SmrtPipeMain.py", line 571, in run
        self._runTasks(pModules)
      File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/SmrtPipeMain.py", line 300, in _runTasks
        workflow.execute()
      File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/engine/SmrtPipeWorkflow.py", line 640, in execute
        raise SmrtExit(str(e))
    SmrtExit: SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
    [ERROR] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain exit 690] SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
    [INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain exit 692] Beginning exit process
    [INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain shutdown 716] beginning Shutdown Process
    [INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtPipeMain _generate_final_report 651] Writing final TOC page with error message : SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
    [DEBUG] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService merge_from_serialized_datastore 277] data.items.pickle merging item <DataItem name:Chemistry group:General >
    [DEBUG] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService merge_from_serialized_datastore 281] ('Chemistry', 'General')
    [DEBUG] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService generateFinalReport 722] jobInfo <JobInfo id:016450 name:45 Assembly 4 > dataUrl xml:/opt/smrtanalysis/current/common/jobs/016/016450/input.xml paramsFile /opt/smrtanalysis/current/common/jobs/016/016450/settings.xml
    [INFO] 2014-07-02 13:32:48,559 [SMRTpipe.SmrtReportService generateFinalReport 757] Skipping JNLP creation because no job id or no cmp.h5
    [DEBUG] 2014-07-02 13:32:48,560 [SMRTpipe.SmrtReportService _generateIndexHtml 699] Writing Report XML to /opt/smrtanalysis/current/common/jobs/016/016450/toc.xml
    [DEBUG] 2014-07-02 13:32:48,560 [SMRTpipe.SmrtReportService _generateIndexHtml 708] Generating html file from xsl /opt/smrtanalysis/current/analysis/etc/xsl/toc.xsl
    [DEBUG] 2014-07-02 13:32:48,564 [SMRTpipe.SmrtWorker backticks 188] Running on nick with cmd saxonb9 -xsl:/opt/smrtanalysis/current/analysis/etc/xsl/toc.xsl -s:/opt/smrtanalysis/current/common/jobs/016/016450/toc.xml -o:/opt/smrtanalysis/current/common/jobs/016/016450/index.html
    [DEBUG] 2014-07-02 13:32:49,368 [SMRTpipe.SmrtWorker backticks 212] Successful output (Return code = 0) in 0.80 sec (0.01 min) of saxonb9 -xsl:/opt/smrtanalysis/current/analysis/etc/xsl/toc.xsl -s:/opt/smrtanalysis/current/common/jobs/016/016450/toc.xml -o:/opt/smrtanalysis/current/common/jobs/016/016450/index.html
    [DEBUG] 2014-07-02 13:32:49,368 [SMRTpipe.SmrtReportService _toRdf 360] Adding report node <ReportNode group:General title:Workflow > to metadata.rdf.
    [DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _toRdf 360] Adding report node <ReportNode group:Diagnostic title:Json Overview Report > to metadata.rdf.
    [DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _toRdf 396] Adding Auxiliary Files to RDF.
    [DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _toRdf 437] No barcodeFasta to add to ReportService
    [DEBUG] 2014-07-02 13:32:49,369 [SMRTpipe.SmrtReportService _pullJsonAttributes 579] Json Graph attributes: [GraphAttribute name:SMRT Cells value:1 id:overview.ncells, GraphAttribute name:Movies value:1 id:overview.nmovies]
    [DEBUG] 2014-07-02 13:32:49,371 [SMRTpipe.SmrtReportService generateFinalReport 768] Writing Metadata XML to /opt/smrtanalysis/current/common/jobs/016/016450/metadata.rdf
    [INFO] 2014-07-02 13:32:49,373 [SMRTpipe.SmrtPipeMain shutdown 752] Completed shutdown gracefully
    [INFO] 2014-07-02 13:32:49,373 [SMRTpipe.SmrtPipeMain exit 702] Sending message Exiting smrtpipe last error: SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed
    [INFO] 2014-07-02 13:32:49,373 [smrtpipe.status failed 104] Sending Job 016450 failed message 'Exiting smrtpipe last error: SmrtExit task://016450/P_Fetch/adapterRpt Failed task://016450/P_Filter/filter_001of001 Failed' to url http://130.15.147.123:8080/smrtportal/api/jobs/016450/status
    [INFO] 2014-07-02 13:32:49,386 [SMRTpipe.HttpProgress _post_progress 151] Job Progress 'Failed' event POSTED to http://130.15.147.123:8080/smrtportal/api/jobs/016450/status
    [DEBUG] 2014-07-02 13:32:49,386 [SMRTpipe.HttpProgress _post_progress 152] The data string is progress=%7B%22message%22%3A+%22Exiting+smrtpipe+last+error%3A+SmrtExit+task%3A%2F%2F016450%2FP_Fetch%2FadapterRpt+Failed+task%3A%2F%2F016450%2FP_Filter%2Ffilter_001of001+Failed%22%2C+%22code%22%3A+%22Failed%22%2C+%22jobStage%22%3A+null%2C+%22value%22%3A+100%2C+%22moduleName%22%3A+null%7D
    [DEBUG] 2014-07-02 13:32:49,386 [SMRTpipe.SmrtPipeMain _releaseLock 368] -- Released lock (/opt/smrtanalysis/current/common/jobs/016/016450/smrt.lock) for this output directory.
    [DEBUG] 2014-07-02 13:32:49,386 [SMRTpipe.SmrtPipeMain run 636] Shutting down heartbeat service
    [INFO] 2014-07-02 13:32:49,386 [SMRTpipe.SmrtPipeMain run 641] completed SmrtPipeMain.run with exit code -1
    [INFO] 2014-07-02 13:32:49,386 [root main 336] Failed smrtpipe version v1.85.133289 with exit code -1 in 21.26 seconds (0.35 minutes)
    [INFO] 2014-07-02 13:33:28,170 [SMRTpipe.Heartbeat _startHeartbeat 65] Heartbeat attempting to stop
    I have no idea why this is happening, I have been comparing my files for hours now and can't find any differences.

    Any help is greatly appreciated!

    Thanks,
    Nick
  • GenoMax
    Senior Member
    • Feb 2008
    • 7142

    #2
    A mere possibility, but have you looked to see if there are any non-printable/control character differences between the two whitelist files?

    Comment

    • OstanNick
      Member
      • Apr 2014
      • 26

      #3
      Hey Genomax,

      I'm pretty sure both files are identical from a character point of view -- I used the same script to generate all of the files except I just hard code in my run name (mxxxx_xxxx_xxxxxxxxx).

      The only difference between the way I generated my original (working) whitelist and my 7 new (non-working white lists) was that:

      In my original, I copy/pasted all of the printed lines directly from the terminal, into a new 'whitelist.txt' file.

      In the new ones, I used the output [(script) > whitelist.txt] argument. So I just re-generated one of my new whitelists and copy/pasted instead. I'll let you know if it works once the current job in queue is finished.

      Comment

      • OstanNick
        Member
        • Apr 2014
        • 26

        #4
        Wow,

        For some reason, outputting it to a .txt file via the '>' when I ran my script caused it to mess up.

        Copy+paste directly from the terminal seemed to do the trick...

        Why would that ever occur?

        Comment

        • GenoMax
          Senior Member
          • Feb 2008
          • 7142

          #5
          Did you move the whitelist file between two OS's (e.g. OS X to Unix)?

          Comment

          • OstanNick
            Member
            • Apr 2014
            • 26

            #6
            No sir, everything has stayed on my Ubuntu OS (12.04)

            Comment

            • GenoMax
              Senior Member
              • Feb 2008
              • 7142

              #7
              So in what way was the file different between the copy/paste and script methods (e.g. http://alvinalexander.com/blog/post/...s-in-text-file)
              Last edited by GenoMax; 07-02-2014, 11:49 AM.

              Comment

              • OstanNick
                Member
                • Apr 2014
                • 26

                #8
                Neither of them had any non-ASCII characters

                Comment

                • rhall
                  Senior Member
                  • Aug 2012
                  • 324

                  #9
                  Glad you got it working, my guess from the error
                  Code:
                  Invalid whitelist format for ''
                  a blank line at the end of the file? Looking at the code, the line being read that is invalid is returned between the quotes:
                  Code:
                   emsg = "Invalid whitelist format for '{s}' in file {f}.".format(s=line.strip(), f=self._whitelist_file)

                  Comment

                  • OstanNick
                    Member
                    • Apr 2014
                    • 26

                    #10
                    I just checked all of them that I generated using the >output and none of them have a blank line anywhere. Very strange. However, I don't want to take up your time trying to solve something that's practically already fixed. I just generated all of mine manually via copy/paste.

                    Thanks,
                    Nick

                    Comment

                    • rhall
                      Senior Member
                      • Aug 2012
                      • 324

                      #11
                      It's not time wasted if we can find and fix a bug

                      If you want to attach examples of the text files that work and do not work (hopefully they won't get reformatted in the attachment) I can take a look at what is happening. Alternatively I can give you my email and we can transfer the files that way.

                      Comment

                      • OstanNick
                        Member
                        • Apr 2014
                        • 26

                        #12
                        Sure, I can e-mail them to you -- since the text files are >5mb and the limit here is <20kb

                        Comment

                        • OstanNick
                          Member
                          • Apr 2014
                          • 26

                          #13
                          I 'figured it out'. May have been a stupid thing to do on my end, but basically the whitelist must be generated as the smrtanalysis user. Make sure you do the following:

                          Code:
                          su smrtanalyis
                          Code:
                          > enter password
                          Code:
                          /usr/local/bin/python.decon$ python whitelist.py 147_woot.align 81740 > /opt/smrtanalysis/data/147/whitelist.txt
                          Note the space between the '>' and the beginning of the directory. It didn't work when I didn't include the space.

                          Once the file is created, do not change the permissions whatsoever (chmod/chown), or it will not work. I was changing the permissions on the file to be readable/writeable by all users which I think was the problem.

                          Cheers,
                          Nick

                          Comment

                          Latest Articles

                          Collapse

                          • SEQadmin2
                            Multiomic Liquid Biopsy: What DNA/RNA Panels Reveal That DNA Alone Can’t
                            by SEQadmin2



                            Tumors contain complex mixtures of cell populations that evolve in response to treatment and host-specific factors. Tissue biopsies provide only static snapshots of a tumor’s state. But what if we could follow that evolution over time using a minimally invasive technique?

                            Liquid biopsies can detect tumor-derived material circulating in the bloodstream, providing a way to study treatment response, detect emerging resistance, and discover new biomarkers. Circulating
                            ...
                            10-14-2026, 09:55 AM
                          • SEQadmin2
                            New Genomics Technologies Take Aim at Long-Standing Limits
                            by SEQadmin2


                            Researchers using sequencing and genomics tools often have to make trade-offs. They can choose between speed or scale, short reads or long-range information, or targeted panels or a view of the whole transcriptome. New technologies that have been released this year are built to address those tough choices.

                            We asked six companies the same four questions to learn about their latest products. The new technologies bring a lot to the table, including rethinking sequencing
                            ...
                            09-28-2026, 10:25 AM

                          ad_right_rmr

                          Collapse

                          News

                          Collapse

                          Topics Statistics Last Post
                          Started by SEQadmin2, 10-22-2026, 11:13 AM
                          0 responses
                          61 views
                          0 reactions
                          Last Post SEQadmin2  
                          Started by SEQadmin2, 10-22-2026, 10:48 AM
                          0 responses
                          43 views
                          0 reactions
                          Last Post SEQadmin2  
                          Started by SEQadmin2, 10-15-2026, 10:52 AM
                          0 responses
                          77 views
                          0 reactions
                          Last Post SEQadmin2  
                          Started by SEQadmin2, 10-14-2026, 05:39 AM
                          0 responses
                          74 views
                          0 reactions
                          Last Post SEQadmin2  
                          Working...