Unconfigured Ad

Collapse
X
 
  • Time
  • Show
Clear All
new posts

  • Brian Bushnell
    replied
    Originally posted by pmiguel View Post
    I think you need to face that DNA, the natural RAWDATA storage form, is superior to your crummy digital methodologies.
    I don't touch that stuff; it's dirty. Data is only real once it's in a computer

    As for how important these considerations are... hmmm, I don't know. I'm just tossing in something to worry about.

    Leave a comment:


  • pmiguel
    replied
    Originally posted by Brian Bushnell View Post
    That approach seems prone to problems for analyses that consider things like technical replicates, batch effects, cross-contamination, and basically anything involving imperfections in the sequencing process. It would be fine if sequencing was perfect and unbiased, and the platforms and chemistry stable and unchanging, but that's not really the case.
    Ah Brian,
    I think you need to face that DNA, the natural RAWDATA storage form, is superior to your crummy digital methodologies. Step out from in front of your computer screen, head down to the lab and take a look at what the real meaning of "high tech" is. Nanotechnology! Pfah! DNA encodes information at a sub-nanometer resolution.

    From the earliest automated sanger machine days there were less processed storage forms for the instruments that could be used to clog up your hard drives for as many years as you might keep them. (How many of you have tried to save the initial TIFF image of an ABI377 gel?)

    Better to let the instruments use their brittle embedded systems to convert that massive data glob into something approaching a durable storage format. For Sanger sequencers that ended up being the .ab1 file. For Illumina sequencers -- fastq. Heave everything else into the dumpster.

    Okay, to be fair, I'm a hypocrite. I still have the autorads from all 100+ 35S sequencing gels that I ran back in the day. I have them labelled and indexed. But I don't see myself going back to re-read them, ever.

    Seriously though, have you actually seen technical replicate differences sufficient to swamp biological replicate differences? I mean in cases that were not just the result of loading errors like over clustering?

    --
    Phillip

    Leave a comment:


  • Brian Bushnell
    replied
    That approach seems prone to problems for analyses that consider things like technical replicates, batch effects, cross-contamination, and basically anything involving imperfections in the sequencing process. It would be fine if sequencing was perfect and unbiased, and the platforms and chemistry stable and unchanging, but that's not really the case.

    Leave a comment:


  • HeinKey
    replied
    We store the library tubes at -20oC.
    Storing all "raw" data consequently is far more expensive than occasionally redo a HiSeq lane.
    We have re-run 2 years old libraries without any problem.

    Leave a comment:


  • sklages
    replied
    Originally posted by GenoMax View Post
    I am not sure why one would want to save the CIF files (perhaps only if the sample is irreplaceable). This may become a moot point as technology moves along.

    @Sven: Does illumina even allow saving CIF files for V4 chemistry runs?
    In the past we used the CIFs for re-basecalling single lanes. We don't do that anymore (there is no need to). It is just the definition of the term "rawdata", some of us are obligated to store the sequencing "rawdata". This defintion varies vastly ...

    v4 chemistry does not allow for saving CIFs using HCS; you can AFAIK tweak the config to do so. But it makes no sense in my eyes (thinking about OLB/RTA development) and is not recommended (supported) by Illumina. One should especially take care with v4 as there is much more data produced in the same time.

    But we haven't upgraded all HiSeqs :-)

    Leave a comment:


  • GenoMax
    replied
    I am not sure why one would want to save the CIF files (perhaps only if the sample is irreplaceable). This may become a moot point as technology moves along.

    @Sven: Does illumina even allow saving CIF files for V4 chemistry runs?

    Leave a comment:


  • dpryan
    replied
    Originally posted by GenoMax View Post
    Devon: There is a subtle but significant difference. Google's nearline storage supposedly offers access with just a 3-5 second delay (so you could compute on it via Google compute, Edit: Not 100% certain about this). Glacier is truly meant for long term storage.
    True, but that's more of an added convenience for us than a requirement. Realistically, we'd keep things locally for 3-6 months and then off-load them elsewhere, so the odds of needing to recompute would be quite small and it might prove more convenient to just move the random mucked up dataset back. Obviously as things grow this might change.

    Leave a comment:


  • dpryan
    replied
    I'd have to double check, but I think we're storing CIF files too.

    Leave a comment:


  • sklages
    replied
    Originally posted by dpryan View Post
    We're also storing the raw hiseq output to tape.
    What is the "raw hiseq output" referring to? Does it include CIF files or is this just the "standard" rundata with BCLs etc?

    Leave a comment:


  • GenoMax
    replied
    Originally posted by dpryan View Post
    We're also storing the raw hiseq output to tape. We're talking internally about moving to some sort of cloud-based storage, at least for the majority of our runs that don't contain datasets that require being kept in-house. I had been thinking of amazon glacier, but google looks a bit cheaper at the moment.

    Has anyone made the jump to cloud-based storage a la google or amazon? I'd love to hear about experiences.
    Devon: There is a subtle but significant difference. Google's nearline storage supposedly offers access with just a 3-5 second delay (so you could compute on it via Google compute, Edit: Not 100% certain about this). Glacier is truly meant for long term storage.
    Last edited by GenoMax; 04-14-2015, 03:00 PM.

    Leave a comment:


  • GenoMax
    replied
    Originally posted by sklages View Post
    We are currently calculating very loosely 2x4TBx4(HiSeqs) = 32TB/two weeks if we'd run WGS PE100. Rawdata, including CIFs. With the 1TB update this amount would be more than doubled (in theory and when saving CIFs).
    That is ~ 6 x 5 TB drives every two weeks?

    Do you get bare drives or full external enclosures? I hope you have a storage room set aside for those drives.

    Leave a comment:


  • dpryan
    replied
    We're also storing the raw hiseq output to tape. We're talking internally about moving to some sort of cloud-based storage, at least for the majority of our runs that don't contain datasets that require being kept in-house. I had been thinking of amazon glacier, but google looks a bit cheaper at the moment.

    Has anyone made the jump to cloud-based storage a la google or amazon? I'd love to hear about experiences.

    Leave a comment:


  • Brian Bushnell
    replied
    We generally don't let customers choose indices. Typically, we prepare the libraries, sequence, quality-control, and assemble or otherwise analyze the data to verify the process was successful before archiving. Most of our customers don't want the raw data anyway.

    It's not entirely clear to me what additional benefit you get from storing anything prior to the fastqs (demultiplexed or not), as long as all the relevant information (lane, bar codes, etc) is in the read headers.

    Leave a comment:


  • sklages
    replied
    Originally posted by Brian Bushnell View Post
    Our long-term storage is tape, and we store the raw demultiplexed gzipped fastqs (as well as some processed files, but I don't think that's necessary). Tape is much cheaper per GB, and since your disks are offline anyway, I don't see there as being a disadvantage. I suspect tape may be more reliable than HDDs, as well.
    We stored our sanger sequencing and project on tape; nowadays it's too much data, so it is easier to use harddisks.

    You do not store any raw run data, e.g. BCLs and/or CIFs etc? What if you/your customer chose a wrong index? No chance of re-processing?

    We are currently calculating very loosely 2x4TBx4(HiSeqs) = 32TB/two weeks if we'd run WGS PE100. Rawdata, including CIFs. With the 1TB update this amount would be more than doubled (in theory and when saving CIFs).

    Leave a comment:


  • Brian Bushnell
    replied
    Our long-term storage is tape, and we store the raw demultiplexed gzipped fastqs (as well as some processed files, but I don't think that's necessary). Tape is much cheaper per GB, and since your disks are offline anyway, I don't see there as being a disadvantage. I suspect tape may be more reliable than HDDs, as well.

    Leave a comment:

Latest Articles

Collapse

  • SEQadmin2
    16S/ITS Profiling vs. Shotgun Metagenomics: What Changes in Your Workflow?
    by SEQadmin2





    NGS has revolutionized clinical microbiology by offering new ways to identify and surveil pathogens without prior knowledge of the target. One popular approach is 16S rRNA and ITS amplicon sequencing, which targets specific microbial markers. But studies are increasingly incorporating shotgun metagenomics, a technique that sequences all DNA from a sample without targeting specific microbial markers. This approach allows more detailed taxonomic classification,...
    10-30-2026, 06:46 AM

ad_right_rmr

Collapse

News

Collapse

Topics Statistics Last Post
Started by SEQadmin2, 11-17-2026, 06:27 AM
0 responses
23 views
0 reactions
Last Post SEQadmin2  
Started by SEQadmin2, 11-12-2026, 07:59 AM
0 responses
39 views
0 reactions
Last Post SEQadmin2  
Started by SEQadmin2, 11-05-2026, 09:25 AM
0 responses
50 views
0 reactions
Last Post SEQadmin2  
Started by SEQadmin2, 11-03-2026, 11:54 AM
0 responses
79 views
0 reactions
Last Post SEQadmin2  
Working...