Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Biomart getBM and qmake parallelization

    Hi there,

    I'm using Biomart in my R script to retrieve gene symbols from ensembl IDs.
    Here is the part of my script making a call to Biomart.

    Code:
    resultTable <- DEXSeqResults(dxd)
    
    ensembl67=useMart(host="may2012.archive.ensembl.org",biomart="ENSEMBL_MART_ENSEMBL")
    mart <- useDataset("mmusculus_gene_ensembl",ensembl67)
    
    ensemblGeneId <- resultTable$groupID
    
    geneNames <-getBM(filters="ensembl_gene_id",
    attributes=c("ensembl_gene_id","external_gene_id"),
    values=ensemblGeneId,
    mart=mart)
    When I execute my script on a single computer, everything goes fine. But when I try to execute it in parallel on a cluster (running on SGE) with a qmake, some jobs work fine and some others produce the following error :

    Code:
    Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  :
      line 1 did not have 3 elements
    Calls: getBM -> read.table -> scan
    I then set the getBM verbose attribute to TRUE and I got the resulting response from the server for failed jobs :

    Code:
    Results from server:
    [1] "\n<html><!-- InstanceBegin template=\"/Templates/biomart_standalone.dwt\" codeOutsideHTMLIsLocked=\"false\" -->\t\n\t<head> \n\t\t<!-- InstanceBeginEditable name=\"doctitle\" -->\n<meta name=\"Description\" content=\"BioMart is a simple and robust data integration system for large scale data querying. It has been designed to provide researchers with an easy and interactive access to both the wealth of data available on the Internet and for in house data integration.\">\n<meta name=\"keywords\" content=\"BioMart, DATABASE, SEARCHES, protein\">\n\n<title>BioMart Webservice</title>\n<script language = \"javascript\">\nvar currentTab=\"Martservice\";\n</script>\n\n<!-- InstanceEndEditable -->\n<link rel=\"stylesheet\" href=\"/biomart/mview/martview.css\" type=\"text/css\" />\n <script language=\"javascript\" type=\"text/javascript\">\n\n\tfunction colourTab(){\n\t\tif((document.getElementById(currentTab))){\n\t\t\t(document.getElementById(currentTab)).className='menuh';\n\t\t}\n\t}\n\n\tfunction rollOver(overTab){\n\t\tif(currentTab!=overTab){\n\t\t\t(document.getElementById(overTab)).className='menuh';\n\t\t}\n\t}\n\tfunction rollOut(overTab){\n\t\tif(currentTab!=overTab){\n\t\t\t(document.getElementById(overTab)).className='menu';\n\t\t}\n\t}\n\n</script>\n\n</head>\n<body style=\"margin: 0px;\" onLoad=\"colourTab()\" >\n\n<div style=\" width:100%; height:15%;\">\n<table height=\"65%\" width=\"100%\" border=\"0\" cellspacing=\"0\" cellpadding=\"0\">\n\t<tr>\n\t\t\n    <td width=\"270\" align=\"right\">\n    <a href=\"index.html\">\n    \t\t<img style = \"margin: 15px 0px 5px 8px;\" height=\"33\" width=\"105\" src=\"/martview/images/biomart-logo.gif\" alt=\"EBI Home Page\" border=\"0\" />\n    </a>\n    </td>\n\t\t <td valign=\"top\" align=\"right\" width=\"100%\">\n\n  \t</td>\n\t</tr>\n</table>\n\n\n<table border=\"0\"  valign=\"bottom\" cellpadding=\"\" cellspacing=\"2\" height=\"35%\" width=\"1000\">\n        <tr>\n        \n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t          <div class=\"menu\" id=\"BioMart Home\" onClick=\"location='/martview/../index.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">HOME</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n          <div   class=\"menu\"  id=\"Documentation\" onClick=\"location='/martview/../install.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">DOCS</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t  <div   class=\"menu\"  id=\"News\" onClick=\"location='/martview/../news.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">NEWS</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n          <div   class=\"menu\"  id=\"Contact\" onClick=\"location='/martview/../contact.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">CONTACT</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div    class=\"menu\"  id=\"Credits\" onClick=\"location='/martview/../credits.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">CREDITS</div> \n\t\t\t</td>\n\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div     class=\"menuh\" id=\"Service\" onClick=\"location='/biomart/martservice'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">MARTSERVICE</div>\n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div    class=\"menu\"  id=\"Query\" onClick=\"location='/biomart/martview'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">MARTVIEW</div>    \n\t\t\t</td>\n\t\t\t\n\t\t</tr>\n</table>\n\n</div>\n\n<div class=\"mart_header\" style=\" width:100%; height:85%;\">\n<table bgcolor=\"black\" width=\"100%\" height=\"5\" border=\"0\" cellspacing=\"0\" cellpadding=\"0\">\n             <tr> \n                <td bgcolor=\"black\" width=\"100%\" height=\"20\" align=\"center\"><!-- InstanceBeginEditable name=\"topnav\" --><nobr></nobr><!-- InstanceEndEditable --></td>\n              </tr> \n</table>\n\n\n<h3>Martservices usage:</h3><br>\n(a) Querying BioMart<br><br>\nTo submit a query using our webservices generate an XML document conforming to our Query XML syntax. This can be achieved simply by building up your query using MartView and hitting the XML button. This XML should be posted to http://may2012.archive.ensembl.org/martservice attached to a single parameter of <b>query</b>. For example you could either:<br><br> \n- save your query as Query.xml and then POST this using the <a href=\"http://cvs.sanger.ac.uk/cgi-bin/viewcvs.cgi/*checkout*/biomart-perl/scripts/webExample.pl?root=BioMart&content-type=text%2Fplain\">webExample.pl</a> script in our biomart-perl/scripts installation.<br>\n- submit using wget: wget -O results.txt 'http://may2012.archive.ensembl.org/biomart/martservice?query=MY_XML' replacing MY_XML with the XML obtained above, first removing any new lines.<br>\n<br>\n(b) Retrieving Meta Data<br>\n<br>\n- to retrieve registry information: <a href=\"/biomart/martservice?type=registry\">http://may2012.archive.ensembl.org/biomart/martservice?type=registry</a><br>\n- to retrieve datasets available for a mart: <a href=\"/biomart/martservice?type=datasets&mart=ENSEMBL_MART_ENSEMBL\">http://may2012.archive.ensembl.org/biomart/martservice?type=datasets&mart=ENSEMBL_MART_ENSEMBL</a><br>\n- to retrieve attributes available for a dataset: <a href=\"/biomart/martservice?type=attributes&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=attributes&datatset=oanatinus_gene_ensembl</a><br>\n- to retrieve filters available for a dataset: <a href=\"/biomart/martservice?type=filters&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=filters&datatset=oanatinus_gene_ensembl</a><br>\n- to retrieve configuration for a dataset: <a href=\"/biomart/martservice?type=configuration&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=configuration&datatset=oanatinus_gene_ensembl</a><br>\n<br>\nFor more information see the webservices section of our <a href=\"http://www.biomart.org/user-docs.pdf\">documentation</a>.\n</div>\n</body>\n</html>\n"
    attr(,"Content-Type")
                      charset 
    "text/plain"      "utf-8"
    whereas in successfull jobs the response looks like that :

    Code:
    Results from server:
    [1] "ENSMUSG00000085299\tGm16627\nENSMUSG00000067978\tVmn2r-ps113\nENSMUSG00000020530\tGgnbp2\nENSMUSG00000074155\tKlk5\nENSMUSG00000087276\tGm11378\nENSMUSG00000048534\tAmica1 ......
    I can also mention that queries in both successfull and failed jobs look the same, and if I re-run a qmake the errors are not necessarly on the same data.

    I guess it's a connection problem but I would like to know if somebody already experienced it and solved it.

    Thanks

    PS : I'm using R-3.1.0 and biomaRt_2.20.0

  • #2
    **may be** the biomart server doesn't want to handle too many requests coming from the same IP at the same time ? As a test I would flag those specific targets with NOTPARALLEL ( http://www.gnu.org/software/make/man...pecial-Targets ) and add a 'sleep' to wait a few seconds before running another target.

    Comment


    • #3
      OHH! BTW, "BONJOUR Erwan" :-)

      Comment


      • #4
        Originally posted by lindenb View Post
        **may be** the biomart server doesn't want to handle too many requests coming from the same IP at the same time ?
        Yep that's what I think too, but I didn't find any informations about that in the documentation or on the web.

        As a test I would flag those specific targets with NOTPARALLEL ( http://www.gnu.org/software/make/man...pecial-Targets ) and add a 'sleep' to wait a few seconds before running another target.
        I will try that.

        Merci Pierre

        Comment

        Latest Articles

        Collapse

        • seqadmin
          Recent Developments in Metagenomics
          by seqadmin





          Metagenomics has improved the way researchers study microorganisms across diverse environments. Historically, studying microorganisms relied on culturing them in the lab, a method that limits the investigation of many species since most are unculturable1. Metagenomics overcomes these issues by allowing the study of microorganisms regardless of their ability to be cultured or the environments they inhabit. Over time, the field has evolved, especially with the advent...
          09-23-2024, 06:35 AM
        • seqadmin
          Understanding Genetic Influence on Infectious Disease
          by seqadmin




          During the COVID-19 pandemic, scientists observed that while some individuals experienced severe illness when infected with SARS-CoV-2, others were barely affected. These disparities left researchers and clinicians wondering what causes the wide variations in response to viral infections and what role genetics plays.

          Jean-Laurent Casanova, M.D., Ph.D., Professor at Rockefeller University, is a leading expert in this crossover between genetics and infectious...
          09-09-2024, 10:59 AM

        ad_right_rmr

        Collapse

        News

        Collapse

        Topics Statistics Last Post
        Started by seqadmin, 10-02-2024, 04:51 AM
        0 responses
        13 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 10-01-2024, 07:10 AM
        0 responses
        21 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 09-30-2024, 08:33 AM
        0 responses
        25 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 09-26-2024, 12:57 PM
        0 responses
        18 views
        0 likes
        Last Post seqadmin  
        Working...
        X