Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Biomart getBM and qmake parallelization

    Hi there,

    I'm using Biomart in my R script to retrieve gene symbols from ensembl IDs.
    Here is the part of my script making a call to Biomart.

    Code:
    resultTable <- DEXSeqResults(dxd)
    
    ensembl67=useMart(host="may2012.archive.ensembl.org",biomart="ENSEMBL_MART_ENSEMBL")
    mart <- useDataset("mmusculus_gene_ensembl",ensembl67)
    
    ensemblGeneId <- resultTable$groupID
    
    geneNames <-getBM(filters="ensembl_gene_id",
    attributes=c("ensembl_gene_id","external_gene_id"),
    values=ensemblGeneId,
    mart=mart)
    When I execute my script on a single computer, everything goes fine. But when I try to execute it in parallel on a cluster (running on SGE) with a qmake, some jobs work fine and some others produce the following error :

    Code:
    Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  :
      line 1 did not have 3 elements
    Calls: getBM -> read.table -> scan
    I then set the getBM verbose attribute to TRUE and I got the resulting response from the server for failed jobs :

    Code:
    Results from server:
    [1] "\n<html><!-- InstanceBegin template=\"/Templates/biomart_standalone.dwt\" codeOutsideHTMLIsLocked=\"false\" -->\t\n\t<head> \n\t\t<!-- InstanceBeginEditable name=\"doctitle\" -->\n<meta name=\"Description\" content=\"BioMart is a simple and robust data integration system for large scale data querying. It has been designed to provide researchers with an easy and interactive access to both the wealth of data available on the Internet and for in house data integration.\">\n<meta name=\"keywords\" content=\"BioMart, DATABASE, SEARCHES, protein\">\n\n<title>BioMart Webservice</title>\n<script language = \"javascript\">\nvar currentTab=\"Martservice\";\n</script>\n\n<!-- InstanceEndEditable -->\n<link rel=\"stylesheet\" href=\"/biomart/mview/martview.css\" type=\"text/css\" />\n <script language=\"javascript\" type=\"text/javascript\">\n\n\tfunction colourTab(){\n\t\tif((document.getElementById(currentTab))){\n\t\t\t(document.getElementById(currentTab)).className='menuh';\n\t\t}\n\t}\n\n\tfunction rollOver(overTab){\n\t\tif(currentTab!=overTab){\n\t\t\t(document.getElementById(overTab)).className='menuh';\n\t\t}\n\t}\n\tfunction rollOut(overTab){\n\t\tif(currentTab!=overTab){\n\t\t\t(document.getElementById(overTab)).className='menu';\n\t\t}\n\t}\n\n</script>\n\n</head>\n<body style=\"margin: 0px;\" onLoad=\"colourTab()\" >\n\n<div style=\" width:100%; height:15%;\">\n<table height=\"65%\" width=\"100%\" border=\"0\" cellspacing=\"0\" cellpadding=\"0\">\n\t<tr>\n\t\t\n    <td width=\"270\" align=\"right\">\n    <a href=\"index.html\">\n    \t\t<img style = \"margin: 15px 0px 5px 8px;\" height=\"33\" width=\"105\" src=\"/martview/images/biomart-logo.gif\" alt=\"EBI Home Page\" border=\"0\" />\n    </a>\n    </td>\n\t\t <td valign=\"top\" align=\"right\" width=\"100%\">\n\n  \t</td>\n\t</tr>\n</table>\n\n\n<table border=\"0\"  valign=\"bottom\" cellpadding=\"\" cellspacing=\"2\" height=\"35%\" width=\"1000\">\n        <tr>\n        \n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t          <div class=\"menu\" id=\"BioMart Home\" onClick=\"location='/martview/../index.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">HOME</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n          <div   class=\"menu\"  id=\"Documentation\" onClick=\"location='/martview/../install.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">DOCS</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t  <div   class=\"menu\"  id=\"News\" onClick=\"location='/martview/../news.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">NEWS</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n          <div   class=\"menu\"  id=\"Contact\" onClick=\"location='/martview/../contact.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">CONTACT</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div    class=\"menu\"  id=\"Credits\" onClick=\"location='/martview/../credits.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">CREDITS</div> \n\t\t\t</td>\n\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div     class=\"menuh\" id=\"Service\" onClick=\"location='/biomart/martservice'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">MARTSERVICE</div>\n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div    class=\"menu\"  id=\"Query\" onClick=\"location='/biomart/martview'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">MARTVIEW</div>    \n\t\t\t</td>\n\t\t\t\n\t\t</tr>\n</table>\n\n</div>\n\n<div class=\"mart_header\" style=\" width:100%; height:85%;\">\n<table bgcolor=\"black\" width=\"100%\" height=\"5\" border=\"0\" cellspacing=\"0\" cellpadding=\"0\">\n             <tr> \n                <td bgcolor=\"black\" width=\"100%\" height=\"20\" align=\"center\"><!-- InstanceBeginEditable name=\"topnav\" --><nobr></nobr><!-- InstanceEndEditable --></td>\n              </tr> \n</table>\n\n\n<h3>Martservices usage:</h3><br>\n(a) Querying BioMart<br><br>\nTo submit a query using our webservices generate an XML document conforming to our Query XML syntax. This can be achieved simply by building up your query using MartView and hitting the XML button. This XML should be posted to http://may2012.archive.ensembl.org/martservice attached to a single parameter of <b>query</b>. For example you could either:<br><br> \n- save your query as Query.xml and then POST this using the <a href=\"http://cvs.sanger.ac.uk/cgi-bin/viewcvs.cgi/*checkout*/biomart-perl/scripts/webExample.pl?root=BioMart&content-type=text%2Fplain\">webExample.pl</a> script in our biomart-perl/scripts installation.<br>\n- submit using wget: wget -O results.txt 'http://may2012.archive.ensembl.org/biomart/martservice?query=MY_XML' replacing MY_XML with the XML obtained above, first removing any new lines.<br>\n<br>\n(b) Retrieving Meta Data<br>\n<br>\n- to retrieve registry information: <a href=\"/biomart/martservice?type=registry\">http://may2012.archive.ensembl.org/biomart/martservice?type=registry</a><br>\n- to retrieve datasets available for a mart: <a href=\"/biomart/martservice?type=datasets&mart=ENSEMBL_MART_ENSEMBL\">http://may2012.archive.ensembl.org/biomart/martservice?type=datasets&mart=ENSEMBL_MART_ENSEMBL</a><br>\n- to retrieve attributes available for a dataset: <a href=\"/biomart/martservice?type=attributes&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=attributes&datatset=oanatinus_gene_ensembl</a><br>\n- to retrieve filters available for a dataset: <a href=\"/biomart/martservice?type=filters&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=filters&datatset=oanatinus_gene_ensembl</a><br>\n- to retrieve configuration for a dataset: <a href=\"/biomart/martservice?type=configuration&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=configuration&datatset=oanatinus_gene_ensembl</a><br>\n<br>\nFor more information see the webservices section of our <a href=\"http://www.biomart.org/user-docs.pdf\">documentation</a>.\n</div>\n</body>\n</html>\n"
    attr(,"Content-Type")
                      charset 
    "text/plain"      "utf-8"
    whereas in successfull jobs the response looks like that :

    Code:
    Results from server:
    [1] "ENSMUSG00000085299\tGm16627\nENSMUSG00000067978\tVmn2r-ps113\nENSMUSG00000020530\tGgnbp2\nENSMUSG00000074155\tKlk5\nENSMUSG00000087276\tGm11378\nENSMUSG00000048534\tAmica1 ......
    I can also mention that queries in both successfull and failed jobs look the same, and if I re-run a qmake the errors are not necessarly on the same data.

    I guess it's a connection problem but I would like to know if somebody already experienced it and solved it.

    Thanks

    PS : I'm using R-3.1.0 and biomaRt_2.20.0

  • #2
    **may be** the biomart server doesn't want to handle too many requests coming from the same IP at the same time ? As a test I would flag those specific targets with NOTPARALLEL ( http://www.gnu.org/software/make/man...pecial-Targets ) and add a 'sleep' to wait a few seconds before running another target.

    Comment


    • #3
      OHH! BTW, "BONJOUR Erwan" :-)

      Comment


      • #4
        Originally posted by lindenb View Post
        **may be** the biomart server doesn't want to handle too many requests coming from the same IP at the same time ?
        Yep that's what I think too, but I didn't find any informations about that in the documentation or on the web.

        As a test I would flag those specific targets with NOTPARALLEL ( http://www.gnu.org/software/make/man...pecial-Targets ) and add a 'sleep' to wait a few seconds before running another target.
        I will try that.

        Merci Pierre

        Comment

        Latest Articles

        Collapse

        • seqadmin
          New Genomics Tools and Methods Shared at AGBT 2025
          by seqadmin


          This year’s Advances in Genome Biology and Technology (AGBT) General Meeting commemorated the 25th anniversary of the event at its original venue on Marco Island, Florida. While this year’s event didn’t include high-profile musical performances, the industry announcements and cutting-edge research still drew the attention of leading scientists.

          The Headliner
          The biggest announcement was Roche stepping back into the sequencing platform market. In the years since...
          03-03-2025, 01:39 PM
        • seqadmin
          Investigating the Gut Microbiome Through Diet and Spatial Biology
          by seqadmin




          The human gut contains trillions of microorganisms that impact digestion, immune functions, and overall health1. Despite major breakthroughs, we’re only beginning to understand the full extent of the microbiome’s influence on health and disease. Advances in next-generation sequencing and spatial biology have opened new windows into this complex environment, yet many questions remain. This article highlights two recent studies exploring how diet influences microbial...
          02-24-2025, 06:31 AM

        ad_right_rmr

        Collapse

        News

        Collapse

        Topics Statistics Last Post
        Started by seqadmin, 03-03-2025, 01:15 PM
        0 responses
        180 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 02-28-2025, 12:58 PM
        0 responses
        275 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 02-24-2025, 02:48 PM
        0 responses
        663 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 02-21-2025, 02:46 PM
        0 responses
        268 views
        0 likes
        Last Post seqadmin  
        Working...
        X