[sagarutturkar]

The [ERROR]
The SMRT cell should show up, it does not need to be approved, my only thought is that maybe you do not have a group assigned to you.

Hi, I am able to import the SMRT cells now. But When I ran the "RS_AHA_scaffolding.1" protocol with my assembly as reference, the job failed.

I have attached log file for the job that failed through SMRTportal (CF80_AHA.zip).

The [ERROR] is a python exception, could you post the entire log so we can get some more context?

The log files for AHA job that failed through command line are also attached (smrtpipe.log.zip).

Many thanks for your help.

-Sagar

[rhall]

The [ERROR] is a python exception, could you post the entire log so we can get some more context?
The SMRT cell should show up, it does not need to be approved, my only thought is that maybe you do not have a group assigned to you. If it is just a personal copy of SMRT portal, log in as administrator and check under 'admin' that you are in group 'all' (note you can place a user in multiple groups by 'ctrl' clicking), also check that your role is 'scientist' (although I'm not sure what the different roles actually mean).
SMRT Cells will only import once, any further attempt will give 'No New SMRT cells found', so it sounds like they are there somewhere.

[sagarutturkar]

Command line AHA including howto to go from filtered subreads and corrected reads:
https://github.com/PacificBioscience...ining/wiki/AHA

I used the same guidelines and got error as:

[ERROR] 2013-02-19 16:21:52,547 [pbpy.smrtpipe.SmrtPipeMain run 648] invalid literal for int() with base 10:

I tried different input files (fasta, bas.h5) and different data, but same error for each of them. Not yet able to figure out whats going wrong.

About SMRT cells importing - kind of weired things happening.

I downloaded the complete data and imported it again. It showed a message "1 SMRT cell imported". However, in design jobs I do not see any SMRT cells available. I tried to use search option but no luck. Does import of SMRT cells need to be approved by administrator? I might be doing some silly mistake or missing some step

When I tried to re-import from same location, I got message "No New SMRT cells found".

Have you noticed something like this before.

Thanks
Sagar

[rhall]

Command line AHA including howto to go from filtered subreads and corrected reads:

https://github.com/PacificBiosciences/Bioinformatics-Training/wiki/AHA

[rhall]

The raw data is not complete, it should look like this:

./
*.metadata.xml
./Analysis_Results
[the files you have]

[rhall]

Sorry,

Yes in 1.4 jobs and raw data can be assigned to a group, then only users in that group can access and see the data.

was in reply to GenoMax.

[rhall]

sagarutturkar,
The basic input type into SMRT portal is PacBio raw data. You should be able to Import SMRT Cells from the 'import and manage' tab, pointing it to the directory structure of the data that comes off the machine.
The filtered_subreads.fastq is useful for using non PacBio software, but you should use raw data for SMRT portal workflows.
Once you have the data imported you should be able to run the RS_AHA_scaffolding workflow, with the reference you have imported, and the raw data.
I'm not aware of a method for using the corrected reads (pacbioToCA) to scaffold an assembly without going to the command line, and outside of the SMRT portal / pipe system.

[sagarutturkar]

Yes in 1.4 jobs and raw data can be assigned to a group, then only users in that group can access and see the data.

Thanks. But how to do this specifically. I tried to import raw data using "import SMRT cells" option. However, after scanning the path it says no SMRT cells found.

my raw data looks lie this:

long_m130123_002504_42153_c100461682550000001523059505101395_s1_p0.bas.h5
m130123_002504_42153_c100461682550000001523059505101395_s1_p0-02.log
m130123_002504_42153_c100461682550000001523059505101395_s1_p0-03.log
m130123_002504_42153_c100461682550000001523059505101395_s1_p0-04.log
m130123_002504_42153_c100461682550000001523059505101395_s1_p0.bas.h5
m130123_002504_42153_c100461682550000001523059505101395_s1_p0.ccs.fasta
m130123_002504_42153_c100461682550000001523059505101395_s1_p0.ccs.fastq
m130123_002504_42153_c100461682550000001523059505101395_s1_p0.fasta
m130123_002504_42153_c100461682550000001523059505101395_s1_p0.fastq
m130123_002504_42153_c100461682550000001523059505101395_s1_p0.sts.csv
m130123_002504_42153_c100461682550000001523059505101395_s1_p0.sts.xml
strobe_m130123_002504_42153_c100461682550000001523059505101395_s1_p0.bas.h5

Apart from this I have

filtered_subreads_CF080.fastq
Corrected_pacbio.fasta

Also, I created a xml file for AHA and tried to run it through command line. I tried this with bas.h5 file as well as corrected_pacbio.fasta files. Each time I got error as

[INFO] 2013-02-19 16:21:52,513 [pbpy.smrtpipe.SmrtPipeScope fitRefLengthToScope 95] Total length of reference is 7024.52 kbp

[INFO] 2013-02-19 16:21:52,514 [pbpy.smrtpipe.SmrtPipeScope fitRefLengthToScope 99] Reference scope is huge

[INFO] 2013-02-19 16:21:52,514 [pbpy.smrtpipe.modules.HybridAssembly run 452] Genome scope is large enough to potentially slow down nucmer repeat detection, so refusing to run. Not running nucmer can increase false positive scaffolds links induced by repeats. To allow nucmer execution increase DENOVO_GENOME_SCOPES in smrtpipe.rcor on command line with e.g. -DDENOVO_GENOME_SCOPES=small:1,large:1,huge:1.

ValueError: invalid literal for int() with base 10: ''
[ERROR] 2013-02-19 16:21:52,550 [pbpy.smrtpipe.SmrtPipeMain exit 760] invalid literal for int() with base 10: ''

Thanks
Sagar

[rhall]

Yes in 1.4 jobs and raw data can be assigned to a group, then only users in that group can access and see the data.

[sagarutturkar]

Dear rhall and GenoMax,

Thanks for the reply. I was able to run the analysis with rhalls minimal settings.xml. Meanwhile our system admin setup the SMRTportal (pretty quick ) and added me as user.

While running SMRT pipe is great for writing complex custom pipelines, it is not the most user friendly. Almost everything you could ever want to do can be achieved via SMRT portal (the web interface), right down to the level of customizing workflows. As someone who works with PacBio data, and has a background in computing / bioinformatics, I still use the SMRT portal web interface for 90% of my analysis.

Now I want to run AHA pipeline to improve my current assembly.

1. I have imported illumina assembly with 48 scaffolds as reference in reference_dropbox folder.
2. I have received pacbio data (filtered_subreads.fastq) file from our collaborator.
3. I also have access to raw pacbio data but collaborator suggested to use filtered_subreads.fastq file.
4. I also have access to corrected pacbio reads (from pacbioToCA pipeline).

In SMRTportal, I selected the "RS_AHA_scaffolding" protocol and "Soap_scaffolds.fasta" as reference. Now how can I input the pacbio data and run the AHA algorithm? I guess using error corrected data would be best. Any suggestions?

Thanks
Sagar

[GenoMax]

I verified that "Rhall's" minimal settings.xml file does indeed work with SMRTpipe v.1.4 on the command line with lambda test data . Thanks Rhall!

PacBio command line is not for the faint of heart.

One problem with previous versions of the web interface was that the "role/account based" restrictions for SMRTcell data did not work (not good in a core environment where everyone could see/access all data). They have supposedly been fixed in SMRTanalysis v.1.4.

Rhall: By chance is that something you have looked at?

[rhall]

The protocols in /opt/smartanalysis/common/protocols/ are not for use with smrtpipe.py, they are templates used with the SMRT portal web interface. The settings.xml file used with smrtpipe.py can be much simpler, but could also include lots of parameters. The simplest settings.xml for filtering, mapping, and calling consensus on lambda data (using all default parameters)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<smrtpipeSettings>
        <protocol id="lambda_resequencing.1">
                <param name="reference">
                        <value>common/references/lambda</value>
                </param>
        </protocol>
    <moduleStage name="fetch" editable="true">
        <module label="Fetch v1" id="P_Fetch">
        </module>
    </moduleStage>
    <moduleStage name="filtering">
        <module label="Filter" id="P_Filter">
        </module>
    </moduleStage>
    <moduleStage name="mapping" editable="true">
        <module label="BLASR v1" id="P_Mapping">
        </module>
    </moduleStage>
    <moduleStage name="consensus" editable="true">
        <module label="Quiver v1" id="P_GenomicConsensus">
        </module>
    </moduleStage>
</smrtpipeSettings>

While running SMRT pipe is great for writing complex custom pipelines, it is not the most user friendly. Almost everything you could ever want to do can be achieved via SMRT portal (the web interface), right down to the level of customizing workflows. As someone who works with PacBio data, and has a background in computing / bioinformatics, I still use the SMRT portal web interface for 90% of my analysis.

P.S. the [WARNING] in the log output is really just a warning, real errors will be tagged [ERROR], and an exit will be tagged [CRITICAL]. The reason that you got zero output but no [ERROR] or [CRITICAL] is that the xml file you used is a template so includes some of the necessary input, without including the tags that do the computation.

[sagarutturkar]

Hello,

Again some problems. I was able to run the smrtpipe.py command without any errors. However when I tried to run the SMRTpipe example as given in http://pacb.com/devnet/files/softwar...ce%20Guide.pdf

The data is located at:

/opt/smrtanalysis/common/test/smrtpipe/lambda_resequencing/*

Created input.xml as :

fofnToSmrtpipeInput.py lambda_resequencing.fofn > input.xml

settings.xml was gathered from:

/opt/smrtanalysis/smartanalysis/common/protocols/lambda_RS_Resequencing.1.xml

However there was no files generated in results and data sub-directories.

Few error lines I see in master.log file are:

[DEBUG] 2013-02-19 11:21:48,983 [pbpy.io.MetaAnalysisXml load 116] No header found in input.xml. Unable to load jobId

[DEBUG] 2013-02-19 11:21:48,984 [pbpy.smrtpipe.InputData loadXml 214] Skipping assignment of JobId. Unable to find header in input.xml

[INFO] 2013-02-19 11:45:30,281 [pbpy.smrtpipe.SmrtPipeContext movieFiles 365] Found /data2/smart/smartanalysis/common/test/primary/lambda/Analysis_Results/m120404_104101_00114_c100318002550000001523015908241265_s1_p0.bas.h5 (81059282 bytes)
[WARNING] 2013-02-19 11:45:30,282 [pbpy.smrtpipe.SmrtPipeMain _getBasVersions 456] Unable to correctly Parse the basH5 versions. Allowing job to proceed, but please fix the compatibility matrix under $SEYMOUR_HOME/common/etc. Error unable to create file (File accessability: Unable to open file)

I have attached log files for reference. Please help with this.

Can anybody post the xml files that worked. Instructions for creating appropriate xml files are way beyond understanding of biologist. More clarification is needed from pacbio

Thanks
Sagar

[sagarutturkar]

Success

Dear rhall and GenoMax,

Thanks you very much for your help and comments regarding installation. We built new ubuntu server and installed SMRTanalysis correctly . Updating the gfortran library helped to resolve errors with the help of system admins.

I want to try AHA pipeline to improve existing assembly with pacbio data.

Thanks
Sagar

[rhall]

AMI howto:

http://files.pacb.com/software/smrtanalysis/1.4/doc/Running%20SMRT%20Analysis%20on%20Amazon.pdf