problems processing bcl

[nikooki2016]

This seems to be some other problem.

Where did you get this flowcell folder from? Can you verify that the data folder is complete (i.e. all files are there and in full size)?

Download from basespace by using basemount. OK I will double check.

[GenoMax]

This seems to be some other problem.

Where did you get this flowcell folder from? Can you verify that the data folder is complete (i.e. all files are there and in full size)?

[nikooki2016]

I install ubuntu14045 in a new computer, not in virtual box anymore. Then, install bcl2fastq. Now, I solve the file permission problem.

type "ls -l"

drwxr-xr-x 3 matt matt 4096 Mar 24 15:09 Data
drwxrwxr-x 3 matt matt 4096 Mar 27 15:05 fsastqfile
drwxr-xr-x 2 matt matt 4096 Mar 24 15:09 InterOp
drwxr-xr-x 2 matt matt 4096 Mar 24 15:09 Logs
-r--r--r-- 1 matt matt 46 Mar 1 15:52 RTAComplete.txt
drwxr-xr-x 2 matt matt 4096 Mar 24 15:09 RTALogs
-r--r--r-- 1 matt matt 36 Mar 1 15:31 RTARead1Complete.txt
-r--r--r-- 1 matt matt 36 Mar 1 15:52 RTARead2Complete.txt
-r--r--r-- 1 matt matt 922 Mar 1 17:40 RunCompletionStatus.xml
-r--r--r-- 1 matt matt 10078 Mar 1 10:58 RunInfo.xml
-r--r--r-- 1 matt matt 11329 Mar 1 10:58 RunParameters.xml
-rw-rw-r-- 1 matt matt 946 Mar 24 15:21 SampleSheet.csv

And I don't have 'Unable to open...' or 'No such file or directory' anymore.


Then, I run "bcl2fastq --no-lane-splitting", the new error is in the link below.

https://github.com/Jinggg2016/NGS/issues/1

The compute I use is 16GB memory, Intel Core i7. It doesn't meet bcl2fastq's 32GB RAM requirement (from bcl2fastq guideline installation requirements) . So does this cause the "Aborted (core dumped)" issue. If it is, I will run bcl2fastq on AWS.

Thanks!

[GenoMax]

Answer 1: Sure. Once you do that you may find the next file it will complain because of permissions. I am not sure why your file permissions are set that way. Were they like that after you did the install?

Answer 2: Those are the two OS Illumina tests/supports software on. It does not mean you can't get the software to work on other OS's. They won't provide support for problems that arise with other OS.

Why are you not using the sample sheet? This looks like a NextSeq run. Are there multiplexed samples or a single sample? Add --sample-sheet /path_to/SampleSheet.csv to your command. That file seems to be there.

[nikooki2016]

Again I am not 100% sure how/what account your are running this from but the files in that folder don't have read permissions for group "all". Can you do "chmod a+r *" in this folder and see if that helps.

I see what you mean, I did some changes, move run folder from shared folder(/media/sf_Unix) to /desktop and change file permissions. Now user have read/ write permissions to the entire run folder.

nj@nj-VirtualBox:~/Desktop/170301_NS500501_0261_AHJHWKAFXX$ ls -l
total 64
drwxrwxrwx 3 nj nj  4096 Mar 21 09:48 Data
drwxrwxrwx 3 nj nj  4096 Mar 22 15:05 fastq
drwxrwxrwx 2 nj nj  4096 Mar 21 09:48 InterOp
drwxrwxrwx 2 nj nj  4096 Mar 21 09:48 Logs
-rwxrwxrwx 1 nj nj    46 Mar  1 15:52 RTAComplete.txt
drwxrwxrwx 2 nj nj  4096 Mar 21 09:48 RTALogs
-rwxrwxrwx 1 nj nj    36 Mar  1 15:31 RTARead1Complete.txt
-rwxrwxrwx 1 nj nj    36 Mar  1 15:52 RTARead2Complete.txt
-rwxrwxrwx 1 nj nj   922 Mar  1 17:40 RunCompletionStatus.xml
-rwxrwxrwx 1 nj nj 10078 Mar 21 13:27 RunInfo.xml
-rwxrwxrwx 1 nj nj 11329 Mar  1 10:58 RunParameters.xml
-rwxrwxrwx 1 nj nj   946 Mar 21 11:09 SampleSheet.csv

I run same bcl2fastq command again, but still get same error.

nj@nj-VirtualBox:~/Desktop/170301_NS500501_0261_AHJHWKAFXX$ /usr/local/bin/bcl2fastq -R 170301_NS500501_0261_AHJHWKAFXX -o 170301_NS500501_0261_AHJHWKAFXX/fastq --no-lane-splitting
BCL to FASTQ file converter
bcl2fastq v2.19.0.316
Copyright (c) 2007-2017 Illumina, Inc.

2017-03-22 15:10:25 [b5843700] Command-line invocation: /usr/local/bin/bcl2fastq -R 170301_NS500501_0261_AHJHWKAFXX -o 170301_NS500501_0261_AHJHWKAFXX/fastq --no-lane-splitting 
2017-03-22 15:10:25 [b5843700] INFO: Minimum log level: INFO
2017-03-22 15:10:25 [b5843700] INFO: Sample sheet: '170301_NS500501_0261_AHJHWKAFXX/SampleSheet.csv'
2017-03-22 15:10:25 [b5843700] INFO: Sample sheet: NOT FOUND
2017-03-22 15:10:25 [b5843700] INFO: Runfolder path: '170301_NS500501_0261_AHJHWKAFXX'
2017-03-22 15:10:25 [b5843700] INFO: Input path: '170301_NS500501_0261_AHJHWKAFXX/Data/Intensities/BaseCalls/'
2017-03-22 15:10:25 [b5843700] INFO: Intensities path: '170301_NS500501_0261_AHJHWKAFXX/Data/Intensities/'
2017-03-22 15:10:25 [b5843700] INFO: Output path: '170301_NS500501_0261_AHJHWKAFXX/fastq'
2017-03-22 15:10:25 [b5843700] INFO: InterOp path: '170301_NS500501_0261_AHJHWKAFXX/InterOp/'
2017-03-22 15:10:25 [b5843700] INFO: Stats path: '170301_NS500501_0261_AHJHWKAFXX/fastq/Stats/'
2017-03-22 15:10:25 [b5843700] INFO: Reports path: '170301_NS500501_0261_AHJHWKAFXX/fastq/Reports/'
2017-03-22 15:10:25 [b5843700] INFO: Detected CPUs: 1
2017-03-22 15:10:25 [b5843700] INFO: Loading threads: 4
2017-03-22 15:10:25 [b5843700] INFO: Processing threads: 1
2017-03-22 15:10:25 [b5843700] INFO: Writing threads: 4
2017-03-22 15:10:25 [b5843700] INFO: Allowed barcode mismatches: 1 
2017-03-22 15:10:25 [b5843700] INFO: Tiles: <ALL>
2017-03-22 15:10:25 [b5843700] INFO: Minimum trimmed read length: 35
2017-03-22 15:10:25 [b5843700] INFO: Use bases masks: <NONE>
2017-03-22 15:10:25 [b5843700] INFO: Mask short adapter reads: 22
2017-03-22 15:10:25 [b5843700] INFO: Adapter stringency: 0.9
2017-03-22 15:10:25 [b5843700] INFO: Adapter trimming method: Allow matches with indels
2017-03-22 15:10:25 [b5843700] INFO: Ignore missing BCLs: NO
2017-03-22 15:10:25 [b5843700] INFO: Ignore missing filters: NO
2017-03-22 15:10:25 [b5843700] INFO: Ignore missing positions: NO
2017-03-22 15:10:25 [b5843700] INFO: Ignore missing controls: NO
2017-03-22 15:10:25 [b5843700] INFO: Include non-PF clusters: NO
2017-03-22 15:10:25 [b5843700] INFO: Create FASTQs for index reads: NO
2017-03-22 15:10:25 [b5843700] INFO: Use bgzf compression for FASTQ files: YES
2017-03-22 15:10:25 [b5843700] INFO: FASTQ compression level: 4
2017-03-22 15:10:25 [b5843700] INFO: RunInfo.xml: '"170301_NS500501_0261_AHJHWKAFXX/RunInfo.xml"'
2017-03-22 15:10:25 [b5843700] ERROR: bcl2fastq::common::Exception: 2017-Mar-22 15:10:25: No such file or directory (2): /media/sf_Unix/package/bcl2fastq/src/cxx/lib/io/Xml.cpp(201): Throw in function boost::property_tree::ptree bcl2fastq::io::parseXmlFile(const boost::filesystem::path&)
Dynamic exception type: boost::exception_detail::clone_impl<bcl2fastq::common::IoError>
std::exception::what: Unable to open '170301_NS500501_0261_AHJHWKAFXX/RunInfo.xml' file for reading

Question 1:

It said "No such file or directory (2): /media/sf_Unix/package/bcl2fastq/src/cxx/lib/io/Xml.cpp(201)" . If I type "ls -l" this file is also "-rwxrwx--- 1 root vboxsf", cause it is in shared folder. Should I also chance the permission for this file?

Question 2:
Illumina Technical Support told me "The software is not supported on ubuntu14045. We recommend either the CentOS 6 or the RedHat Enterprise Linux 6 platform." Is that true ??

[GenoMax]

Again I am not 100% sure how/what account your are running this from but the files in that folder don't have read permissions for group "all". Can you do "chmod a+r *" in this folder and see if that helps.

[nikooki2016]

Your RunInfo.xml may be correct but is the account you are running bcl2fastq from able to read that directory/file (has read permissions)? I see that you are using virtual box for the software so are the folders mounted correctly on virtual box? Are you able to do "ls -l 170301_NS500501_0261_AHJHWKAFXX" and see a file listing?

Yes, I use virtual box, but I already download the entire run folder to my own computer.

nj@nj-VirtualBox:/media/sf_Unix/Practice/170301_NS500501_0261_AHJHWKAFXX$ ls -l
total 25
drwxrwx--- 1 root vboxsf 0 Mar 21 09:48 Data
drwxrwx--- 1 root vboxsf 0 Mar 21 09:48 InterOp
drwxrwx--- 1 root vboxsf 0 Mar 21 09:48 Logs
-rwxrwx--- 1 root vboxsf 46 Mar 1 15:52 RTAComplete.txt
drwxrwx--- 1 root vboxsf 0 Mar 21 09:48 RTALogs
-rwxrwx--- 1 root vboxsf 36 Mar 1 15:31 RTARead1Complete.txt
-rwxrwx--- 1 root vboxsf 36 Mar 1 15:52 RTARead2Complete.txt
-rwxrwx--- 1 root vboxsf 922 Mar 1 17:40 RunCompletionStatus.xml
-rwxrwx--- 1 root vboxsf 10078 Mar 21 13:27 RunInfo.xml
-rwxrwx--- 1 root vboxsf 11329 Mar 1 10:58 RunParameters.xml
-rwxrwx--- 1 root vboxsf 946 Mar 21 11:09 SampleSheet.csv

[GenoMax]

Your RunInfo.xml may be correct but is the account you are running bcl2fastq from able to read that directory/file (has read permissions)? I see that you are using virtual box for the software so are the folders mounted correctly on virtual box? Are you able to do "ls -l 170301_NS500501_0261_AHJHWKAFXX" and see a file listing?

[nikooki2016]

Can you post the command you are using and the exact error?

Hi thanks for your replay, this is my first time run bcl2fastq, I just install it in ubuntu14045. I follow this https://nhoffman.github.io/borborygm...on-ubuntu.html to install it.

The command I use is:

nj@nj-VirtualBox:/media/sf_Unix/Practice/170301_NS500501_0261_AHJHWKAFXX$ /usr/local/bin/bcl2fastq -R 170301_NS500501_0261_AHJHWKAFXX -o 170301_NS500501_0261_AHJHWKAFXX/fastq --no-lane-splitting


error messages:

BCL to FASTQ file converter
bcl2fastq v2.19.0.316
Copyright (c) 2007-2017 Illumina, Inc.

2017-03-21 15:47:10 [b57d9700] Command-line invocation: /usr/local/bin/bcl2fastq -R 170301_NS500501_0261_AHJHWKAFXX -o 170301_NS500501_0261_AHJHWKAFXX/fastq --no-lane-splitting
2017-03-21 15:47:10 [b57d9700] INFO: Minimum log level: INFO
2017-03-21 15:47:10 [b57d9700] INFO: Sample sheet: '170301_NS500501_0261_AHJHWKAFXX/SampleSheet.csv'
2017-03-21 15:47:10 [b57d9700] INFO: Sample sheet: NOT FOUND
2017-03-21 15:47:10 [b57d9700] INFO: Runfolder path: '170301_NS500501_0261_AHJHWKAFXX'
2017-03-21 15:47:10 [b57d9700] INFO: Input path: '170301_NS500501_0261_AHJHWKAFXX/Data/Intensities/BaseCalls/'
2017-03-21 15:47:10 [b57d9700] INFO: Intensities path: '170301_NS500501_0261_AHJHWKAFXX/Data/Intensities/'
2017-03-21 15:47:10 [b57d9700] INFO: Output path: '170301_NS500501_0261_AHJHWKAFXX/fastq'
2017-03-21 15:47:10 [b57d9700] INFO: InterOp path: '170301_NS500501_0261_AHJHWKAFXX/InterOp/'
2017-03-21 15:47:10 [b57d9700] INFO: Stats path: '170301_NS500501_0261_AHJHWKAFXX/fastq/Stats/'
2017-03-21 15:47:10 [b57d9700] INFO: Reports path: '170301_NS500501_0261_AHJHWKAFXX/fastq/Reports/'
2017-03-21 15:47:10 [b57d9700] INFO: Detected CPUs: 1
2017-03-21 15:47:10 [b57d9700] INFO: Loading threads: 4
2017-03-21 15:47:10 [b57d9700] INFO: Processing threads: 1
2017-03-21 15:47:10 [b57d9700] INFO: Writing threads: 4
2017-03-21 15:47:10 [b57d9700] INFO: Allowed barcode mismatches: 1
2017-03-21 15:47:10 [b57d9700] INFO: Tiles: <ALL>
2017-03-21 15:47:10 [b57d9700] INFO: Minimum trimmed read length: 35
2017-03-21 15:47:10 [b57d9700] INFO: Use bases masks: <NONE>
2017-03-21 15:47:10 [b57d9700] INFO: Mask short adapter reads: 22
2017-03-21 15:47:10 [b57d9700] INFO: Adapter stringency: 0.9
2017-03-21 15:47:10 [b57d9700] INFO: Adapter trimming method: Allow matches with indels
2017-03-21 15:47:10 [b57d9700] INFO: Ignore missing BCLs: NO
2017-03-21 15:47:10 [b57d9700] INFO: Ignore missing filters: NO
2017-03-21 15:47:10 [b57d9700] INFO: Ignore missing positions: NO
2017-03-21 15:47:10 [b57d9700] INFO: Ignore missing controls: NO
2017-03-21 15:47:10 [b57d9700] INFO: Include non-PF clusters: NO
2017-03-21 15:47:10 [b57d9700] INFO: Create FASTQs for index reads: NO
2017-03-21 15:47:10 [b57d9700] INFO: Use bgzf compression for FASTQ files: YES
2017-03-21 15:47:10 [b57d9700] INFO: FASTQ compression level: 4
2017-03-21 15:47:10 [b57d9700] INFO: RunInfo.xml: '"170301_NS500501_0261_AHJHWKAFXX/RunInfo.xml"'
2017-03-21 15:47:10 [b57d9700] ERROR: bcl2fastq::common::Exception: 2017-Mar-21 15:47:10: No such file or directory (2): /media/sf_Unix/package/bcl2fastq/src/cxx/lib/io/Xml.cpp(201): Throw in function boost::property_tree::ptree bcl2fastq::io::parseXmlFile(const boost::filesystem::path&) Dynamic exception type: boost::exception_detail::clone_impl<bcl2fastq::common::IoError>
std::exception::what: Unable to open '170301_NS500501_0261_AHJHWKAFXX/RunInfo.xml' file for reading

[GenoMax]

Hi,
How did you solve the problem? I have same issue, my RunInfo is correct, I double checked.

Can you post the command you are using and the exact error?

[nikooki2016]

How did you solve the problem

Hi,
How did you solve the problem? I have same issue, my RunInfo is correct, I double checked.

[paumarc]

i solved the problem, i don't know why but the files i had were incorrect

[sklages]

If you have the complete runfolder you could use '--runfolder-dir' option to simply point to the runfolder directory.

Can you post the content of /home/pmarcmun/BackupNGS/, e.g.

ls -l /home/pmarcmun/BackupNGS/*

[GenoMax]

Do you have the entire run folder available under /home/pmarcmun/BackupNGS/?

The -i option should point to the /path_to/flowcell_ID/Data/Intensities/BaseCalls directory in that run folder.

[paumarc]

i have installed bcl2fastq in our cluster, i also copied the BackupNGS directroy at the cluster there are files called as duplicati-full-signature.20150312T101755Z.vol1460.zip in it.

Now i would like to run bcl2fastq, to do it i used the following command

/home/pmarcmun/bcl_t/bcl2fastq -i /home/pmarcmun/BackupNGS/ -o /home/pmarcmun/bcl_t/output/bs_1_1.fastq -r 4 -d 4 -p 4 -w 4

and i get those error messages

BCL to FASTQ file converter
bcl2fastq v2.16.0.10
Copyright (c) 2007-2015 Illumina, Inc.

2015-04-16 11:41:50 [2b367952fda0] Command-line invocation: /home/pmarcmun/bcl_t/bcl2fastq -i /home/pmarcmun/BackupNGS/ -o /home/pmarcmun/bcl_t/output/bs_1_1.fastq -r 4 -d 4 -p 4 -w 4
2015-04-16 11:41:50 [2b367952fda0] INFO: Minimum log level: INFO
2015-04-16 11:41:50 [2b367952fda0] INFO: Runfolder path: '/scratch/177420.1.all.q/'
2015-04-16 11:41:50 [2b367952fda0] INFO: Input path: '/home/pmarcmun/BackupNGS/'
2015-04-16 11:41:50 [2b367952fda0] INFO: Intensities path: '/home/pmarcmun/'
2015-04-16 11:41:50 [2b367952fda0] INFO: Output path: '/home/pmarcmun/bcl_t/output/bs_1_1.fastq'
2015-04-16 11:41:50 [2b367952fda0] INFO: InterOp path: '/scratch/177420.1.all.q/InterOp/'
2015-04-16 11:41:50 [2b367952fda0] INFO: Stats path: '/home/pmarcmun/bcl_t/output/bs_1_1.fastq/Stats/'
2015-04-16 11:41:50 [2b367952fda0] INFO: Reports path: '/home/pmarcmun/bcl_t/output/bs_1_1.fastq/Reports/'
2015-04-16 11:41:50 [2b367952fda0] INFO: Detected CPUs: 16
2015-04-16 11:41:50 [2b367952fda0] INFO: Loading threads: 4
2015-04-16 11:41:50 [2b367952fda0] INFO: Processing threads: 4
2015-04-16 11:41:50 [2b367952fda0] INFO: Demultiplexing threads: 4
2015-04-16 11:41:50 [2b367952fda0] INFO: Writing threads: 4
2015-04-16 11:41:50 [2b367952fda0] INFO: Aggregated tiles: AUTO
2015-04-16 11:41:50 [2b367952fda0] INFO: Allowed barcode mismatches: 1
2015-04-16 11:41:50 [2b367952fda0] INFO: Tiles: <ALL>
2015-04-16 11:41:50 [2b367952fda0] INFO: Minimum trimmed read length: 35
2015-04-16 11:41:50 [2b367952fda0] INFO: Use bases masks: <NONE>
2015-04-16 11:41:50 [2b367952fda0] INFO: Mask short adapter reads: 22
2015-04-16 11:41:50 [2b367952fda0] INFO: Adapter stringency: 0.9
2015-04-16 11:41:50 [2b367952fda0] INFO: Ignore missing BCLs: NO
2015-04-16 11:41:50 [2b367952fda0] INFO: Ignore missing filters: NO
2015-04-16 11:41:50 [2b367952fda0] INFO: Ignore missing positions: NO
2015-04-16 11:41:50 [2b367952fda0] INFO: Ignore missing controls: NO
2015-04-16 11:41:50 [2b367952fda0] INFO: Include non-PF clusters: NO
2015-04-16 11:41:50 [2b367952fda0] INFO: Create FASTQs for index reads: NO
2015-04-16 11:41:50 [2b367952fda0] INFO: Use bgzf compression for FASTQ files: YES
2015-04-16 11:41:50 [2b367952fda0] INFO: FASTQ compression level: 4
2015-04-16 11:41:50 [2b367952fda0] INFO: RunInfo.xml: '/scratch/177420.1.all.q/RunInfo.xml'
2015-04-16 11:41:50 [2b367952fda0] ERROR: bcl2fastq::common::Exception: 2015-Apr-16 11:41:50: No such file or directory (2): /home/pmarcmun/bcl2fastq-source/src/cxx/lib/io/Xml.cpp(197): Throw in function boost:roperty_tree:tree bcl2fastq::io:arseXmlFile(boost::filesystem:ath)
Dynamic exception type: boost::exception_detail::clone_impl<bcl2fastq::common::IoError>
std::exception::what: Unable to open '/scratch/177420.1.all.q/RunInfo.xml' file for reading


it looks like if i mess the file RunInfo.xml. Where is it? Do i have to unzip the files in the directory?

thanks