Error Installing Older Version of SMRT-Analysis (1.4.0)

[OstanNick]

Hello,

I'm trying to install an older version of SMRT analysis (version 1.4.0) on my machine because the raw SMRT Cell data that we receive from a company is in the format of bas.h5, and the newer version (2.2.0) requires 3 bax.h5, 1 bas.h5, and 1 xml metadata file.

In the installation process, which I follow to a tee, upon running the startup script, I get the following errors when I run the following command:

nick@Nick-Bioinformatics:/opt/smrtanalysis/etc/scripts/postinstall$ ./configure_smrtanalysis.sh

gives:

ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type %s' % name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type %s' % name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type %s' % name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type %s' % name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type %s' % name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type %s' % name)
ValueError: unsupported hash type sha512
Traceback (most recent call last):
  File "bin/conf_main.py", line 4, in <module>
    from bin.util.upgradeUtil import *
  File "/opt/smrtanalysis-1.4.0/etc/scripts/postinstall/bin/util/upgradeUtil.py", line 13, in <module>
    from fabric.api import *
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/Fabric-1.2.2-py2.7.egg/fabric/api.py", line 9, in <module>
    from fabric.context_managers import cd, hide, settings, show, path, prefix, lcd
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/Fabric-1.2.2-py2.7.egg/fabric/context_managers.py", line 13, in <module>
    from fabric.state import env, output, win32
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/Fabric-1.2.2-py2.7.egg/fabric/state.py", line 9, in <module>
    from fabric.network import HostConnectionCache
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/Fabric-1.2.2-py2.7.egg/fabric/network.py", line 19, in <module>
    import paramiko as ssh
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/paramiko-1.7.7.1-py2.7.egg/paramiko/__init__.py", line 69, in <module>
    from transport import SecurityOptions, Transport
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/paramiko-1.7.7.1-py2.7.egg/paramiko/transport.py", line 37, in <module>
    from paramiko.dsskey import DSSKey
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/paramiko-1.7.7.1-py2.7.egg/paramiko/dsskey.py", line 23, in <module>
    from Crypto.PublicKey import DSA
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/Crypto/PublicKey/DSA.py", line 33, in <module>
    from Crypto.PublicKey import _DSA, _slowmath, pubkey
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/Crypto/PublicKey/_DSA.py", line 33, in <module>
    from Crypto.Hash import SHA
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/Crypto/Hash/SHA.py", line 32, in <module>
    digest_size = new().digest_size
  File "/opt/smrtanalysis/redist/python2.7/lib/python2.7/site-packages/Crypto/Hash/SHA.py", line 31, in new
    return hashlib.sha1(data)
AttributeError: 'module' object has no attribute 'sha1'

I have edited my setup.sh file for SEYMOUR_HOME to be /opt/smrtanalysis, and I receive this error. I have checked other people with similar issues and have confirmed that it is using the bundled Python.

Any Ideas?

Thanks a bunch!
Nick

[GenoMax]

See if this helps take care of the error: http://stackoverflow.com/questions/2...t-not-with-2-6

[flxlex]

I'm trying to install an older version of SMRT analysis (version 1.4.0) on my machine because the raw SMRT Cell data that we receive from a company is in the format of bas.h5, and the newer version (2.2.0) requires 3 bax.h5, 1 bas.h5, and 1 xml metadata file.

You don't need the xml metadatafile unless you want to run smrtportal. 2.2.0 is backward compatible with older bas.h5 files (no need for a bax.h5 file in that case. If the data is relatively new, the provide should really have given you both the bas.h5 and bax.h5 files.

[OstanNick]

I agree (and understand that it is backwards compatible but I can't import SMRT cell data without an XML file, and the company that did the analysis only provided one bas.h5 file.

If I contact the company do you think they would have the XML file? After all it is the necessary component for import of data.

Thanks,
Nick

[GenoMax]

I agree (and understand that it is backwards compatible but I can't import SMRT cell data without an XML file, and the company that did the analysis only provided one bas.h5 file.

If I contact the company do you think they would have the XML file? After all it is the necessary component for import of data.

Thanks,
Nick

If you are taking the trouble of installing SMRTportal then might as well ask for the entire raw data folder for your sample SMRTcell.

[OstanNick]

If you are taking the trouble of installing SMRTportal then might as well ask for the entire raw data folder for your sample SMRTcell.

I contacted GenomeQuebec and am awaiting their response.

Is there a way for me to generate this XML file?
I.e. is it my responsibility to make it or is it generated when they run the SMRT cell on the PacBio RS II?

It seems like common sense to provide such data.

I re-installed v2.2.0 because I couldn't figure out the error that wasn't letting me install v1.4.0. Now I just need to get my hands on the XML files for each of my reads

Thanks,
Nick

[rhall]

There is no way to generate a metadata.xml file, it includes all the barcode information for the reagents / SMRT Cells, and run conditions.
SMRT Anallysis is backwards compatible, 2.2 runs perfectly with old data. Installing 1.4 wouldn't help, SMRT Portal has always required the metadata.xml file for importing cells.
You don't need the metadata.xml file to run command line smrtpipe.py which is the program underlying SMRT Portal.

https://github.com/PacificBiosciences/SMRT-Analysis/wiki/SMRT-Pipe-Reference-Guide-v2.2.0

[OstanNick]

Thanks for all of your responses.

I went back to v2.2.0 for reasons you have all stated above.

They recommend confirming the installation was successful by doing three things:

1. Running an RS_Resequencing protocol using their lambda reference data
2. Running an HGAP assembly using the same data with a genome size of 40,000bp and a minimum seed read length of 3500bp.

3. Running some of your own data with HGAP.

#2 is where my installation falls apart. Using either my own data or using PacBio's test data, I run into an issue of "runCaHgap failed";

[INFO] 2014-05-01 14:43:10,828 [smrtpipe.status refreshTargets 409] Workflow Completion Status 26/44 in ( ..... 59%) tasks completed.
[INFO] 2014-05-01 14:43:15,833 [smrtpipe.status refreshTargets 419] Starting task://016441/P_CeleraAssembler/runCaHgap
[ERROR] 2014-05-01 14:48:32,924 [smrtpipe.status refreshTargets 413] *** Failed task task://016441/P_CeleraAssembler/runCaHgap
[INFO] 2014-05-01 14:48:32,934 [smrtpipe.status execute 627] Found 1 failed tasks.
[INFO] 2014-05-01 14:48:32,934 [smrtpipe.status execute 629] task runCaHgap FAILED
[ERROR] 2014-05-01 14:48:33,613 [SMRTpipe.SmrtPipeMain run 608] SmrtExit task://016441/P_CeleraAssembler/runCaHgap Failed
Traceback (most recent call last):
File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/SmrtPipeMain.py", line 571, in run
self._runTasks(pModules)
File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/SmrtPipeMain.py", line 300, in _runTasks
workflow.execute()
File "/opt/smrtanalysis/current/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/engine/SmrtPipeWorkflow.py", line 640, in execute
raise SmrtExit(str(e))
SmrtExit: SmrtExit task://016441/P_CeleraAssembler/runCaHgap Failed
[ERROR] 2014-05-01 14:48:33,626 [SMRTpipe.SmrtPipeMain exit 690] SmrtExit task://016441/P_CeleraAssembler/runCaHgap Failed
[INFO] 2014-05-01 14:48:33,627 [SMRTpipe.SmrtPipeMain exit 692] Beginning exit process
[INFO] 2014-05-01 14:48:33,627 [SMRTpipe.SmrtPipeMain shutdown 716] beginning Shutdown Process
[INFO] 2014-05-01 14:48:33,627 [SMRTpipe.SmrtPipeMain _generate_final_report 651] Writing final TOC page with error message : SmrtExit task://016441/P_CeleraAssembler/runCaHgap Failed
[INFO] 2014-05-01 14:48:39,241 [SMRTpipe.SmrtPipeMain shutdown 752] Completed shutdown gracefully
[INFO] 2014-05-01 14:48:39,242 [SMRTpipe.SmrtPipeMain exit 702] Sending message Exiting smrtpipe last error: SmrtExit task://016441/P_CeleraAssembler/runCaHgap Failed
[INFO] 2014-05-01 14:48:39,242 [smrtpipe.status failed 104] Sending Job 016441 failed message 'Exiting smrtpipe last error: SmrtExit task://016441/P_CeleraAssembler/runCaHgap Failed' to url http://localhost:8080/smrtportal/api/jobs/016441/status
[INFO] 2014-05-01 14:48:39,439 [SMRTpipe.SmrtPipeMain run 641] completed SmrtPipeMain.run with exit code -1
[INFO] 2014-05-01 14:48:39,440 [root main 336] Failed smrtpipe version v1.85.133289 with exit code -1 in 558.10 seconds (9.30 minutes)

I haven't been able to find the reason behind this error, even using PacBio-supplied documentation where they address similar issues, here:

https://github.com/PacificBiosciences/SMRT-Analysis/wiki/RS-HGAP-Assembly-protocol-fails-in-SMRT-Portal

Can anyone interpret what may be going wrong?

Thanks,
Nick

[rhall]

My guess would be memory, CA 8.1 has a higher default memory requirement in order to deal with PacBio long reads, 16gb. You can check <job dir>/data/3-overlapcorrection/0001.err to see if a memory allocation error is thrown.
If you do not have a 16gb machine available the only workaround is to recompile CA from source, changing:

#define AS_READ_MAX_NORMAL_LEN_BITS       16

to 15 (4gb memory) in AS_global.H. Note this will limit the length of reads used in CA, any reads longer than 32kb will be silently trimmed.

[OstanNick]

I believe you are correct with the memory issue:

Here is my error log from 0001.err in the directory you specified:

gkpStore = '/opt/smrtanalysis/current/common/jobs/016/016441/data/celera-assembler.gkpStore'
### Starting at  Thu May  1 14:48:26 2014
Could not malloc memory (17180393460 bytes)
correct-frags: AS_UTL_alloc.C:65: void* safe_malloc(size_t): Assertion `p != __null' failed.

Failed with 'Aborted'

Backtrace (mangled):

/opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags(_Z17AS_UTL_catchCrashiP7siginfoPv+0x27)[0x40fe97]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fbc32de6340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7fbc32074f79]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fbc32078388]
/lib/x86_64-linux-gnu/libc.so.6(+0x2fe36)[0x7fbc3206de36]
/lib/x86_64-linux-gnu/libc.so.6(+0x2fee2)[0x7fbc3206dee2]
/opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags[0x40fb17]
/opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags[0x40685d]
/opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags(main+0x106)[0x40c4a6]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fbc3205fec5]
/opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags[0x4058c9]

Backtrace (demangled):

[0] /opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags::AS_UTL_catchCrash(int, siginfo*, void*) + 0x27  [0x40fe97]
[1] /lib/x86_64-linux-gnu/libpthread.so.0::(null) + 0x10340  [0x7fbc32de6340]
[2] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x39  [0x7fbc32074f79]
[3] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x148  [0x7fbc32078388]
[4] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x2fe36  [0x7fbc3206de36]
[5] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x2fee2  [0x7fbc3206dee2]
[6] /opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags() [0x40fb17]
[7] /opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags() [0x40685d]
[8] /opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags::(null) + 0x106  [0x40c4a6]
[9] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0xf5  [0x7fbc3205fec5]
[10] /opt/smrtanalysis/install/smrtanalysis-2.2.0.133377/analysis/bin/wgs-8.1/Linux-amd64/bin/correct-frags() [0x4058c9]

GDB:


Aborted (core dumped)

At the top I see a memory alloc fail.

Should I install this on a 16gb machine (which I do indeed have access to) as opposed to my 4gb standalone I'm currently using?

Regards,
Nick

[rhall]

Installing on a 16gb machine is the easiest option, or is is possible to setup SMRT Analysis on the 16gb machine, then run jobs on the 4gb and only submit CA jobs to the 16gb machine.
This is not really straightforward unless you have some kind of queuing system installed already.

[OstanNick]

Hey, got it installed on a 16gb machine, and I'm still getting the EXACT same error.

I read here that it could be due to the parameters of my Celera Assembler:

http://sourceforge.net/p/wgs-assembler/bugs/234/

But I can't seem to find where I change these parameters.

Any help?
Thanks!

[OstanNick]

Actually, I think I found the appropriate file

--> When I open up one of my previous jobs, I can find runCA.spec, which contains teh following

unitigger = bogart
utgErrorRate = 0.06 
utgErrorLimit = 4.5

cnsErrorRate = 0.25
cgwErrorRate = 0.25
ovlErrorRate = 0.06

frgMinLen = 500
ovlMinLen = 40
  
merSize=14

merylMemory = 4078
merylThreads = 8

ovlStoreMemory = 4078

# grid info
useGrid = 0
scriptOnGrid = 0
frgCorrOnGrid = 0
ovlCorrOnGrid = 0

sge = -S /bin/bash -sync y -V -q None
sgeScript = -pe None 3
sgeConsensus = -pe None 1
sgeOverlap = -pe None 1
sgeFragmentCorrection = -pe None 2
sgeOverlapCorrection = -pe None 1

ovlHashBits = 23
ovlHashBlockLength = 246415360
ovlRefBlockSize =  1

ovlThreads = 1
ovlConcurrency = 3
frgCorrThreads = 2
frgCorrBatchSize = 100000
ovlCorrBatchSize = 100000

sgeName = 016438

However I don't know the following:

a) How to set these parameters PRIOR to running a job from SMRTportal

and

b) Which parameters to alter -- even after consulting this great wiki:

http://sourceforge.net/apps/mediawiki/wgs-assembler/index.php?title=RunCA#OVL_Overlapper

Not sure what much of it means.

Cheers.
Nick

[rhall]

Changing the spec file will not fix the error, the allocation is hard coded. Even on a 16gb machine if something else is running and the memory cannot be allocated the job will fail. The only option to reduce the memory usage is to change
#define AS_READ_MAX_NORMAL_LEN_BITS 15
in AS_global.H and recompile Celera Assembly.

[OstanNick]

Ok awesome, thanks.

How do I go about recompiling only the Celera Assembler?

I installed from their .run file (and I'm a pretty big Linux noob) so I'm not sure how to selectively recompiler.

Additionally, I couldn't find any file called AS_global.H anywhere in my install directory (/opt/...).

Regards,
Nick