Tag (string) Compression Techniques

[foolishbrat]

Does anybody know any string compression technique (in R or Perl)?

In particular we usually would like to compress tag of length 35 above,
to store them in data structure for further processing. Note also
that we are talking about ~10million tags to process.

I have such implementation in R to convert tag to numerical value.
But it get overflow error when handling tag of length > 30

tagsequence2tagnum  <- function (tags, length) 
{
    new.tags <- tolower(unlist(strsplit(as.character(tags), "")))
    new.tags[!(new.tags == "a" | new.tags == "g" | new.tags == 
        "c" | new.tags == "t" | new.tags == "s" | new.tags == 
        "y" | new.tags == "b" | new.tags == "k")] <- "n"
    new.tags <- matrix(as.numeric(chartr("acgtnsybk", "012301112", 
        new.tags)), nrow = length)
    colSums(new.tags * 4^((length - 1):0)) + 1
}

tagnum2tagsequence <- function (tags, length) 
{
    new.tags <- t(matrix((rep(tags - 1, each = length)%/%4^((length -  1):0))%%4, nrow = length))
    new.tags <- apply(new.tags, 1, paste, collapse = "")
    chartr("0123", "acgt", new.tags)
}

[lgoff]

Check out nuID

We have used nuID (https://prod.bioinformatics.northwestern.edu/nuID/) in the past for something similar. nuID provides a perl utility and is also part of the 'lumi' R-package available through bioconductor. It is a nice utility because it provides mechanisms for compression, and uncompression with error checking. On a side note, if anyone knows of something similar written in python I would love to hear about it!

[foolishbrat]

Thank you so much, Igoff. This is invaluable.