SMRT Pipe jobs exit with errors on single node (ubuntu 14)

[Gestahl]

Hi everybody,

I've managed to get SMRT Pipe running on my 2 core notebook (64bit, 8 GB RAM, Ubuntu 14), but when I start a job, they all exit after some seconds with errors (smrtpipe.log) like:

"SmrtExit task://Anonymous/P_Filter/filter.plsFofn.Scatter Failed" (P_Filter)
or
"Failed to resolve Workflow tasks (Workflow.resolve)
Unable to satisfy OR clause in task inpt definition for task://jobId/P_Mapping/align:
file://jobId/data/post_control_regions.fofn,file://jobId/data/filtered_regions.fofn,file://jobId/input.fofn" (P_Mapping)

My first thought was SMRT Pipe wants to do job distribution. (I set job management system to 'None' during installation)
I've looked inside the web.xml file, but the "jobsAreDistributed" parameter is set to 'false'.

Has anyone experienced similar secondary analysis problems? Could it be that a single notebook is simply not powerful enough to perform a P_Filter task?

I just want to do single jobs on a small bacterial genome with a more powerful desktop PC.

[gconcepcion]

Can you post some of the logs from the failed workflow directory?

There should be a corresponding log for each failed task which should give you more insight into what is occuring.

See job_dir/log/P_Filter/* etc.

[Gestahl]

I uploaded master and smrtpipe logs of a P_Filter and a P_Mapping task.
Please have a look!

[gconcepcion]

The relevant error message is in:
/opt/smrtanalysis/OUTPUTS/filtertest1/log/P_Filter/filter.plsFofn.Scatter.log

which is:
[ERROR] 2016-06-05 21:21:52,441 [SMRTpipe.engine.SmrtPipeTasks run 706] > Validating existence of Input Files
[ERROR] 2016-06-05 21:21:52,441 [SMRTpipe.engine.SmrtPipeTasks run 706] > WARNING: Unable to find necessary input file, or dir /opt/smrtanalysis/OUTPUTS/filtertest1/input.fofn.
[ERROR] 2016-06-05 21:21:52,441 [SMRTpipe.engine.SmrtPipeTasks run 706] > Successfully validated input files
[ERROR] 2016-06-05 21:21:52,442 [SMRTpipe.engine.SmrtPipeTasks run 706] > cat: /opt/smrtanalysis/OUTPUTS/filtertest1/input.fofn: No such file or directory
[ERROR] 2016-06-05 21:21:52,442 [SMRTpipe.engine.SmrtPipeTasks run 706] > Task 1 completed at Sun Jun 5 21:21:52 CEST 2016
[ERROR] 2016-06-05 21:21:52,442 [SMRTpipe.engine.SmrtPipeTasks run 706] > awk: fatal: cannot open file `/opt/smrtanalysis/OUTPUTS/filtertest1/input.fofn' for reading (No such file or directory)


Is your input.fofn in the right place?

[Gestahl]

Oh, this seems quite odd, because I created a 'my_inputs.xml' file which is located in my home directory.
The bax.h5 are found so I suppose this is working (see filter_smrtlpipe.log, lines 19 to 21).
After invoking the smrt shell, I moved inside the directory and defined the 'my_inputs.xml' file as the input (using "xml:my_inputs.xml"). I assumed this suffices.

I do not know why SMRT Pipe is actually looking inside the output folder.. (which I also defined)
Does the input file has to be provided as 'input.fofn' under all cicumstances?

[gconcepcion]

Can you zip up the entire contents of the log directory? You posted the *master.log & *smrtpipe.log for those two different jobs -but not the contents of the actual task directories - e.g. jobdir/log/P_Filter/*.log

Also, can you post my_input.xml?

It seems that you're missing the P_Fetch section that generates the input.fofn from your input.xml

[Gestahl]

P_Fetch? Holymoly! It was right in front of me the whole time. I didn't pay attention to it. Thought the input file definition was enough.

I will do an analysis including P_Fetch..

P.S.: see new attachment

[Gestahl]

The filter analysis runs smoothly now. Thank you very much for your help!

This means a lot to me since I am new to PacBio. It would've taken a massive amount of time to come up with P_Fetch. Thank you once more.

(Finished with message "Successfully exiting smrtpipe version xy with exit code 0 in 367.81 seconds")

[gconcepcion]

The filter analysis runs smoothly now. Thank you very much for your help!

This means a lot to me since I am new to PacBio. It would've taken a massive amount of time to come up with P_Fetch. Thank you once more.

(Finished with message "Successfully exiting smrtpipe version xy with exit code 0 in 367.81 seconds")

Great! Happy to help and glad you got it working!

[Gestahl]

Hi again.

My problems with SMRT Pipe continue. I edited my settings file and try to do P_Mapping after the P_Filter reports are generated.

I get the following console output:
"...
[INFO] 2016-07-11 20:49:35,636 Starting task://Anonymous/P_Mapping/align
[INFO] 2016-07-11 20:49:35,636 Starting task://Anonymous/P_Filter/subreadSummary
[INFO] 2016-07-11 20:49:35,637 Starting task://Anonymous/P_FilterReports/statsRpt
[INFO] 2016-07-11 20:49:35,637 Starting task://Anonymous/P_FilterReports/loadingRpt
[ERROR] 2016-07-11 20:49:35,637 *** Failed task task://Anonymous/P_Mapping/align
[INFO] 2016-07-11 20:53:09,790 Found 1 failed tasks.
[INFO] 2016-07-11 20:53:09,791 task align FAILED
Traceback (most recent call last):
File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/engine/SmrtPipeWorkflow.py", line 634, in execute
raise WorkflowError(_e_msg)
WorkflowError: task://Anonymous/P_Mapping/align Failed
Failed smrtpipe version v1.87.139483 with exit code -1 in 353.44 seconds (5.89 minutes)
"

I provided the corresponding logfile and the settings file I used, which is mostly an adaptation of the "RS_Modification_Detection" Protocol.

Possible source (I don't understand what's happening):
- line 201 in "smrtpipe.log", line 366 in "master.log":
[DEBUG] 2016-07-11 20:47:19,576 [SMRTpipe.engine.DistributableTasks __init__ 199] Creating Gather task with args: () kw: {'constructCommands': <function myGatherFunction at 0x2dd15f0>, 'outputs': {'toGather': URL file://jobId/data/aligned_reads.cmp.h5#presorted Path data/aligned_reads.cmp.h5}, '_taskTypeName': 'GatherNonEmptyCmpH5', 'errors': {'AssertionError: No alignments in the CmpH5 file(s).': 'Gather Error. No Reads Mapped to the reference; no further analysis is possible'}, 'nproc': 2}

Could someone please look into it and tell me what is going wrong? Could it be a hardware problem? Did I miss settings again?

If you require files from the results or data folders, let me know.

BR,
Benjamin

[gconcepcion]

Hi again.

My problems with SMRT Pipe continue. I edited my settings file and try to do P_Mapping after the P_Filter reports are generated.

I get the following console output:
"...
[INFO] 2016-07-11 20:49:35,636 Starting task://Anonymous/P_Mapping/align
[INFO] 2016-07-11 20:49:35,636 Starting task://Anonymous/P_Filter/subreadSummary
[INFO] 2016-07-11 20:49:35,637 Starting task://Anonymous/P_FilterReports/statsRpt
[INFO] 2016-07-11 20:49:35,637 Starting task://Anonymous/P_FilterReports/loadingRpt
[ERROR] 2016-07-11 20:49:35,637 *** Failed task task://Anonymous/P_Mapping/align
[INFO] 2016-07-11 20:53:09,790 Found 1 failed tasks.
[INFO] 2016-07-11 20:53:09,791 task align FAILED
Traceback (most recent call last):
File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/engine/SmrtPipeWorkflow.py", line 634, in execute
raise WorkflowError(_e_msg)
WorkflowError: task://Anonymous/P_Mapping/align Failed
Failed smrtpipe version v1.87.139483 with exit code -1 in 353.44 seconds (5.89 minutes)
"

I provided the corresponding logfile and the settings file I used, which is mostly an adaptation of the "RS_Modification_Detection" Protocol.

Possible source (I don't understand what's happening):
- line 201 in "smrtpipe.log", line 366 in "master.log":
[DEBUG] 2016-07-11 20:47:19,576 [SMRTpipe.engine.DistributableTasks __init__ 199] Creating Gather task with args: () kw: {'constructCommands': <function myGatherFunction at 0x2dd15f0>, 'outputs': {'toGather': URL file://jobId/data/aligned_reads.cmp.h5#presorted Path data/aligned_reads.cmp.h5}, '_taskTypeName': 'GatherNonEmptyCmpH5', 'errors': {'AssertionError: No alignments in the CmpH5 file(s).': 'Gather Error. No Reads Mapped to the reference; no further analysis is possible'}, 'nproc': 2}

Could someone please look into it and tell me what is going wrong? Could it be a hardware problem? Did I miss settings again?

If you require files from the results or data folders, let me know.

BR,
Benjamin

This is not a hardware problem, the problem is that "No Reads Mapped to the reference" as stated in the error message above. This is always because of one of two things:

1) The data is bad quality
2) The chosen reference is not appropriate for the data.

I'm assuming your data is probably of decent quality, and the more likely scenario here is that you're trying to map reads to an inappropriate reference. Have you verified the reference is correct?

[Gestahl]

That's interesting. An experiment had been conducted to de novo sequence a bacterial genome.
The reference sequence here is the de novo sequence of aforementioned experiment.
Reads are the reads created by de novo sequencing.
I just want to add base incorporation information to the sequence.

Could it be that my reference sequence ENTRY in the reference repository is flawed (I created it using the reference uploader)?

[jkaralius]

You have an illegal option in blasr:

[ERROR] 2016-07-11 20:49:31,421 [SMRTpipe.engine.SmrtPipeTasks run 706] > pbalign: error: unrecognized arguments: -x

Remove the -x in this section of your settings.xml:

<param name="pbalign_opts" hidden="true">
<value>--minAccuracy=0.75 --minLength=50 -x </value>
</param>

[gconcepcion]

You have an illegal option in blasr:

[ERROR] 2016-07-11 20:49:31,421 [SMRTpipe.engine.SmrtPipeTasks run 706] > pbalign: error: unrecognized arguments: -x

Remove the -x in this section of your settings.xml:

<param name="pbalign_opts" hidden="true">
<value>--minAccuracy=0.75 --minLength=50 -x </value>
</param>

Ahh. Yes. The third option is that the blasr mapping params are incorrect. I assumed he copied them from a working execution.

[Gestahl]

Mapping is in progress!
I really feel half-baked for taking mapping settings directly from the SMRT-Pipe-Reference Guide without checking.. sorry for bad information.
I took the body from a working pacbio protocoll and added stuff from the reference guide.

Thank you so much for having a look! This forum and its users are invaluable!