[Medhat]

Thanks for the guiding Dr. Hall,

That is very good, mean while we talked with the sequencing provider, and they will do resequencing and I hope that I will have better result.

meanwhile I am trying to get familiar with the protocols I used this protocol RS_Resquensing.1 whith the cells I have,
But it failed in the mapping process

[ERROR] 2014-09-04 20:48:10,531 [smrtpipe.status refreshTargets 413] *** Failed task task://016452/P_Mapping/align
[INFO] 2014-09-04 20:48:10,562 [smrtpipe.status execute 627] Found 1 failed tasks.
[INFO] 2014-09-04 20:48:10,562 [smrtpipe.status execute 629] task align FAILED
[ERROR] 2014-09-04 20:48:12,710 [SMRTpipe.SmrtPipeMain run 608] SmrtExit task://016452/P_Mapping/align Failed
Traceback (most recent call last):

Is it hardware problem , data or software?

[GenoMax]

Did process fail in spite of providing a reference for the RS_resequencing protocol?

[Medhat]

yes I provided a reference of the genome and yet it failed

[rhall]

You can check ./jobs/016/016452/logs/P_Mapping/align.log for a more meaningful log of the process that failed. How much memory does the machine have? running none distributed on a single machine can be quite memory intensive, particularly for a large genome.

[Medhat]

on my PC single machine which is 12 cpu 32G memory

in this directory:
/data/smrtAnalysis/current/common/jobs/016/016452/log/P_Mapping

log is :

# Task 1
pbalign.py "/data/smrtAnalysis/current/common/jobs/016/016452/input.fofn" "/data/smrtAnalysis/current/common/references/Maize" "/data/smrtAnalysis/current/common/jobs/016/016452/data/aligned_reads.cmp.h5" --seed=1 --minAccuracy=0.75 --minLength=50 --concordant --algorithmOptions="-useQuality" --algorithmOptions=' -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa' --hitPolicy=randombest --tmpDir=/data/smrtAnalysis/tmpdir --nproc=11 --regionTable=/data/smrtAnalysis/current/common/jobs/016/016452/data/filtered_regions.fofn || exit $?
echo "Task 1 completed at $(date)"
# Task 2
echo 'Alignment Complete' || exit $?
echo "Task 2 completed at $(date)"
# Task 3
loadPulses /data/smrtAnalysis/current/common/jobs/016/016452/input.fofn /data/smrtAnalysis/current/common/jobs/016/016452/data/aligned_reads.cmp.h5 -metrics DeletionQV,IPD,InsertionQV,PulseWidth,QualityValue,MergeQV,SubstitutionQV,DeletionTag -byread || exit $?
echo "Task 3 completed at $(date)"
# Task 4
echo 'LoadPulses Complete' || exit $?
echo "Task 4 completed at $(date)"



rcode=$?
echo "Finished on $(date -u)"
echo "Task align with nproc 11 with exit code ${rcode}."
exit ${rcode}# Writing stdout and stderr from Popen:
Setting up ENV on medhat for task align
Running task://016452/P_Mapping/align on Linux medhat 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Started on Thu Sep 4 16:06:40 UTC 2014
Validating existence of Input Files
Successfully found /data/smrtAnalysis/current/common/jobs/016/016452/input.fofn
Successfully found /data/smrtAnalysis/current/common/jobs/016/016452/data/filtered_regions.fofn
Successfully validated input files
2014-09-04 20:48:07,804 [ERROR] BlaserService returned a non-zero exit status. [INFO] 2014-09-04T18:06:46 [blasr] started.
Killed

[rhall]

After sourcing setup.sh:

source /data/smrtAnalysis/current/etc/setup.sh

Try running the pbalign command with debug turned on:

pbalign.py "/data/smrtAnalysis/current/common/jobs/016/016452/input.fofn" "/data/smrtAnalysis/current/common/references/Maize" "/data/smrtAnalysis/current/common/jobs/016/016452/data/aligned_reads.cmp.h5" --seed=1 --minAccuracy=0.75 --minLength=50 --concordant --algorithmOptions="-useQuality" --algorithmOptions=' -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa' --hitPolicy=randombest --tmpDir=/data/smrtAnalysis/tmpdir --nproc=11 --regionTable=/data/smrtAnalysis/current/common/jobs/016/016452/data/filtered_regions.fofn --debug

This should give us more info on why blasr crashed.

[Medhat]

OK but why i need to source the setup.sh ?

[GenoMax]

OK but why i need to source the setup.sh ?

That sets up appropriate environment variables so you can run the alignment job.

[Medhat]

medhat@medhat:/data/smrtAnalysis$ source /data/smrtAnalysis/current/etc/setup.sh

medhat@medhat:/data/smrtAnalysis$ pbalign.py "/data/smrtAnalysis/current/common/jobs/016/016452/input.fofn" "/data/smrtAnalysis/current/common/references/Maize" "/data/smrtAnalysis/current/common/jobs/016/016452/data/aligned_reads.cmp.h5" --seed=1 --minAccuracy=0.75 --minLength=50 --concordant --algorithmOptions="-useQuality" --algorithmOptions=' -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa' --hitPolicy=randombest --tmpDir=/data/smrtAnalysis/tmpdir --nproc=11 --regionTable=/data/smrtAnalysis/current/common/jobs/016/016452/data/filtered_regions.fofn --debug

> <string>(1)<module>()
(Pdb) continue

2014-09-09 15:59:07,050 [ERROR] BlaserService returned a non-zero exit status. [INFO] 2014-09-09T14:41:36 [blasr] started.
Killed

[rhall]

I though that would give more meaningful output, you could try running blasr directly:

blasr /data/smrtAnalysis/current/common/jobs/016/016452/input.fofn /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta  -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa -nproc 11

[Medhat]

Using this command

/data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/blasr /data/smrtAnalysis/current/common/jobs/016/016452/input.fofn /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta -minMatch 12 -bestn 10 -minPctIdentity 70.0 -sa /data/smrtAnalysis/current/common/references/Maize/sequence/Maize.fasta.sa -nproc 11 > ~/Downloads/blasr_run_log2.txt 2>&1

every thing is ok and this is the last line of the output

m140709_111451_42149_c100664292550000001823130112201495_s1_p0/163190/16535_18492 2 0 0 -6508 82.8514 133857058 133858856 237068873 16536 18488 18493 42573
[INFO] 2014-09-11T23:32:01 [blasr] ended.

[Medhat]

any new sugeestions?

[rhall]

After chatting with the developer of pbalign.py the blasr crash when being ran via pbalign is most likely related to a known memory issue. The workaround is to force the splitting up of the alignment job. Via command line, create a new directory and copy the settings.xml and input.xml file from the original SMRT Portal job directory:

source /data/smrtAnalysis/current/etc/setup.sh

smrtpipe.py --params=settings.xml xml:input.xml --dist -D NPROC=17 -D CLUSTER_MANAGER=BASH

All the reports are generated when running via command line instead of SMRT portal, just point a browser at <job dir>/index.html

If this works it is possible to make changes to SMRT Portal to distribute jobs by changing <smtanalysis>/analysis/etc/smrtpipe.rc to reflect the values of the -D parameters, and editing <smrtanalysis>redist/tomcat/webapps/smrtportal/WEB-INF/web.xml, setting <param-name>jobsAreDistributed</param-name> to 'true'.

[Medhat]

so to make sure that I understand correctly first I will copy the files from this directory;

cp /data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/common/userdata/jobs/016/016452/setting.xml /data/smrtAnalysis/settingCopy/

Note:
I have in

/data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/common/userdata/jobs/016/

four jobs
016452 016451 016450 016438

then after copying those files I will source and run this command

source /data/smrtAnalysis/current/etc/setup.sh

smrtpipe.py --params=settings.xml xml:input.xml --dist -D NPROC=17 -D CLUSTER_MANAGER=BASH

but in the prvious command shall I point to the files in the copied directory or in the job directory? "I mean here the setting and input.xml file "

After success of the previous command I shall navigate to the index file in

/data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/common/userdata/jobs/016/016452/index.html

right?


After all of that
I need to change

/data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/analysis/etc/smrtpipe.rc to reflect the values of the -D parameters
from
NPROC = 11
to
NPROC = 17

right?

and

/data/smrtAnalysis/install/smrtanalysis-2.2.0.133377/redist/apache-tomcat-7.0.23/webapps/smrtportal/WEB-INF/web.xml

<context-param>
<param-name>jobsAreDistributed</param-name>
<param-value>false</param-value>
</context-param>

to

<context-param>
<param-name>jobsAreDistributed</param-name>
<param-value>True</param-value>
</context-param>

[rhall]

Exactly, run the source and smrtpipe.py command in the new directory.