Unconfigured Ad

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • Erwan
    Junior Member
    • Jan 2014
    • 8

    Biomart getBM and qmake parallelization

    Hi there,

    I'm using Biomart in my R script to retrieve gene symbols from ensembl IDs.
    Here is the part of my script making a call to Biomart.

    Code:
    resultTable <- DEXSeqResults(dxd)
    
    ensembl67=useMart(host="may2012.archive.ensembl.org",biomart="ENSEMBL_MART_ENSEMBL")
    mart <- useDataset("mmusculus_gene_ensembl",ensembl67)
    
    ensemblGeneId <- resultTable$groupID
    
    geneNames <-getBM(filters="ensembl_gene_id",
    attributes=c("ensembl_gene_id","external_gene_id"),
    values=ensemblGeneId,
    mart=mart)
    When I execute my script on a single computer, everything goes fine. But when I try to execute it in parallel on a cluster (running on SGE) with a qmake, some jobs work fine and some others produce the following error :

    Code:
    Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  :
      line 1 did not have 3 elements
    Calls: getBM -> read.table -> scan
    I then set the getBM verbose attribute to TRUE and I got the resulting response from the server for failed jobs :

    Code:
    Results from server:
    [1] "\n<html><!-- InstanceBegin template=\"/Templates/biomart_standalone.dwt\" codeOutsideHTMLIsLocked=\"false\" -->\t\n\t<head> \n\t\t<!-- InstanceBeginEditable name=\"doctitle\" -->\n<meta name=\"Description\" content=\"BioMart is a simple and robust data integration system for large scale data querying. It has been designed to provide researchers with an easy and interactive access to both the wealth of data available on the Internet and for in house data integration.\">\n<meta name=\"keywords\" content=\"BioMart, DATABASE, SEARCHES, protein\">\n\n<title>BioMart Webservice</title>\n<script language = \"javascript\">\nvar currentTab=\"Martservice\";\n</script>\n\n<!-- InstanceEndEditable -->\n<link rel=\"stylesheet\" href=\"/biomart/mview/martview.css\" type=\"text/css\" />\n <script language=\"javascript\" type=\"text/javascript\">\n\n\tfunction colourTab(){\n\t\tif((document.getElementById(currentTab))){\n\t\t\t(document.getElementById(currentTab)).className='menuh';\n\t\t}\n\t}\n\n\tfunction rollOver(overTab){\n\t\tif(currentTab!=overTab){\n\t\t\t(document.getElementById(overTab)).className='menuh';\n\t\t}\n\t}\n\tfunction rollOut(overTab){\n\t\tif(currentTab!=overTab){\n\t\t\t(document.getElementById(overTab)).className='menu';\n\t\t}\n\t}\n\n</script>\n\n</head>\n<body style=\"margin: 0px;\" onLoad=\"colourTab()\" >\n\n<div style=\" width:100%; height:15%;\">\n<table height=\"65%\" width=\"100%\" border=\"0\" cellspacing=\"0\" cellpadding=\"0\">\n\t<tr>\n\t\t\n    <td width=\"270\" align=\"right\">\n    <a href=\"index.html\">\n    \t\t<img style = \"margin: 15px 0px 5px 8px;\" height=\"33\" width=\"105\" src=\"/martview/images/biomart-logo.gif\" alt=\"EBI Home Page\" border=\"0\" />\n    </a>\n    </td>\n\t\t <td valign=\"top\" align=\"right\" width=\"100%\">\n\n  \t</td>\n\t</tr>\n</table>\n\n\n<table border=\"0\"  valign=\"bottom\" cellpadding=\"\" cellspacing=\"2\" height=\"35%\" width=\"1000\">\n        <tr>\n        \n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t          <div class=\"menu\" id=\"BioMart Home\" onClick=\"location='/martview/../index.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">HOME</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n          <div   class=\"menu\"  id=\"Documentation\" onClick=\"location='/martview/../install.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">DOCS</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t  <div   class=\"menu\"  id=\"News\" onClick=\"location='/martview/../news.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">NEWS</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n          <div   class=\"menu\"  id=\"Contact\" onClick=\"location='/martview/../contact.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">CONTACT</div> \n\t\t\t</td>\n\t\t\t\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div    class=\"menu\"  id=\"Credits\" onClick=\"location='/martview/../credits.html'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">CREDITS</div> \n\t\t\t</td>\n\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div     class=\"menuh\" id=\"Service\" onClick=\"location='/biomart/martservice'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">MARTSERVICE</div>\n\t\t\t</td>\n\t\t\t\n\t\t\t<td align=\"center\" valign=\"bottom\" width=\"14%\">\n\t\t\t<div    class=\"menu\"  id=\"Query\" onClick=\"location='/biomart/martview'\" onmouseover=\"rollOver(this.id)\" onMouseout=\"rollOut(this.id)\">MARTVIEW</div>    \n\t\t\t</td>\n\t\t\t\n\t\t</tr>\n</table>\n\n</div>\n\n<div class=\"mart_header\" style=\" width:100%; height:85%;\">\n<table bgcolor=\"black\" width=\"100%\" height=\"5\" border=\"0\" cellspacing=\"0\" cellpadding=\"0\">\n             <tr> \n                <td bgcolor=\"black\" width=\"100%\" height=\"20\" align=\"center\"><!-- InstanceBeginEditable name=\"topnav\" --><nobr></nobr><!-- InstanceEndEditable --></td>\n              </tr> \n</table>\n\n\n<h3>Martservices usage:</h3><br>\n(a) Querying BioMart<br><br>\nTo submit a query using our webservices generate an XML document conforming to our Query XML syntax. This can be achieved simply by building up your query using MartView and hitting the XML button. This XML should be posted to http://may2012.archive.ensembl.org/martservice attached to a single parameter of <b>query</b>. For example you could either:<br><br> \n- save your query as Query.xml and then POST this using the <a href=\"http://cvs.sanger.ac.uk/cgi-bin/viewcvs.cgi/*checkout*/biomart-perl/scripts/webExample.pl?root=BioMart&content-type=text%2Fplain\">webExample.pl</a> script in our biomart-perl/scripts installation.<br>\n- submit using wget: wget -O results.txt 'http://may2012.archive.ensembl.org/biomart/martservice?query=MY_XML' replacing MY_XML with the XML obtained above, first removing any new lines.<br>\n<br>\n(b) Retrieving Meta Data<br>\n<br>\n- to retrieve registry information: <a href=\"/biomart/martservice?type=registry\">http://may2012.archive.ensembl.org/biomart/martservice?type=registry</a><br>\n- to retrieve datasets available for a mart: <a href=\"/biomart/martservice?type=datasets&mart=ENSEMBL_MART_ENSEMBL\">http://may2012.archive.ensembl.org/biomart/martservice?type=datasets&mart=ENSEMBL_MART_ENSEMBL</a><br>\n- to retrieve attributes available for a dataset: <a href=\"/biomart/martservice?type=attributes&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=attributes&datatset=oanatinus_gene_ensembl</a><br>\n- to retrieve filters available for a dataset: <a href=\"/biomart/martservice?type=filters&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=filters&datatset=oanatinus_gene_ensembl</a><br>\n- to retrieve configuration for a dataset: <a href=\"/biomart/martservice?type=configuration&dataset=oanatinus_gene_ensembl\">http://may2012.archive.ensembl.org/biomart/martservice?type=configuration&datatset=oanatinus_gene_ensembl</a><br>\n<br>\nFor more information see the webservices section of our <a href=\"http://www.biomart.org/user-docs.pdf\">documentation</a>.\n</div>\n</body>\n</html>\n"
    attr(,"Content-Type")
                      charset 
    "text/plain"      "utf-8"
    whereas in successfull jobs the response looks like that :

    Code:
    Results from server:
    [1] "ENSMUSG00000085299\tGm16627\nENSMUSG00000067978\tVmn2r-ps113\nENSMUSG00000020530\tGgnbp2\nENSMUSG00000074155\tKlk5\nENSMUSG00000087276\tGm11378\nENSMUSG00000048534\tAmica1 ......
    I can also mention that queries in both successfull and failed jobs look the same, and if I re-run a qmake the errors are not necessarly on the same data.

    I guess it's a connection problem but I would like to know if somebody already experienced it and solved it.

    Thanks

    PS : I'm using R-3.1.0 and biomaRt_2.20.0
  • lindenb
    Senior Member
    • Apr 2010
    • 143

    #2
    **may be** the biomart server doesn't want to handle too many requests coming from the same IP at the same time ? As a test I would flag those specific targets with NOTPARALLEL ( http://www.gnu.org/software/make/man...pecial-Targets ) and add a 'sleep' to wait a few seconds before running another target.

    Comment

    • lindenb
      Senior Member
      • Apr 2010
      • 143

      #3
      OHH! BTW, "BONJOUR Erwan" :-)

      Comment

      • Erwan
        Junior Member
        • Jan 2014
        • 8

        #4
        Originally posted by lindenb View Post
        **may be** the biomart server doesn't want to handle too many requests coming from the same IP at the same time ?
        Yep that's what I think too, but I didn't find any informations about that in the documentation or on the web.

        As a test I would flag those specific targets with NOTPARALLEL ( http://www.gnu.org/software/make/man...pecial-Targets ) and add a 'sleep' to wait a few seconds before running another target.
        I will try that.

        Merci Pierre

        Comment

        Latest Articles

        Collapse

        • SEQadmin2
          Nine Things a Sample Prep Scientist Thinks About Before Sequencing
          by SEQadmin2


          I’m not a sequencing expert. I’m a purification scientist who uses NGS to evaluate workflows my group develops. With this perspective, we think about the sample first and the NGS workflow second. The sequencer is an exceptionally honest reporter, but it can only report on what you give it, so whether you get clean, interpretable data from an NGS workflow is largely determined before you begin.


          Here are nine questions we think about, in roughly the order they matter, before...
          Today, 07:11 AM
        • SEQadmin2
          From Collection to Sequencing: Why Sample Preparation and Preservation Define Sequencing Data
          by SEQadmin2


          Data variability is still an issue in sequencing technologies despite the advances in reproducibility and accuracy of these platforms. But the problem does not originate in the sequencing itself, but in the previous steps, before the sample reaches the sequencer.


          The first step is collection, followed by preservation and sample preparation for analysis. Most scientists overlook those steps, but not being careful might just be skewing the experiment’s results.
          ...
          06-02-2026, 10:05 AM
        • SEQadmin2
          Single-Cell Sequencing at an Inflection Point: Early Impacts of New Platforms and Emerging Trends
          by SEQadmin2


          With the launch of new single-cell sequencing platforms in 2026, the field stands at an exciting inflection point. This article surveys the most impactful advances in the field and discusses how they’re reshaping research in cancer, immunology, and beyond.


          Introduction

          Single-cell sequencing technologies have undergone remarkable advances over the past decade, transitioning from low-throughput experimental approaches to highly scalable platforms capable of...
          05-22-2026, 06:42 AM

        ad_right_rmr

        Collapse

        News

        Collapse

        Topics Statistics Last Post
        Started by SEQadmin2, Yesterday, 06:09 AM
        0 responses
        16 views
        0 reactions
        Last Post SEQadmin2  
        Started by SEQadmin2, 06-09-2026, 11:58 AM
        0 responses
        34 views
        0 reactions
        Last Post SEQadmin2  
        Started by SEQadmin2, 06-05-2026, 10:09 AM
        0 responses
        41 views
        0 reactions
        Last Post SEQadmin2  
        Started by SEQadmin2, 06-04-2026, 08:59 AM
        0 responses
        48 views
        0 reactions
        Last Post SEQadmin2  
        Working...