Genomics Data Commons

[Richard Finney]

TCGA data seems to now live at the Geomics Data Commons: https://gdc.nci.nih.gov/

The data is "accessible" via a RESTful interface.

It appears to be a mishmash jammed into some kind of NOSQL database(?). Figuring out, for instance, what rna-seq bam file ids and "legacy" barcodes are associated with TCGA kidney disease (KICH, KIRCH, KIRP), is a real pain.

What I want is just a dump of all the data into a flat file (or a few flat files) ... which I can then parse using jq, sed, awk, grep, custom C and Python programs, etc.

So ... has anybody come up with a script to suck all the "meta" data from GDC via their API?

[Richard Finney]

Quick follow up.
The Genomics Data Commons folks got back with me and say that there is indeed some problems with traciing back to original barcode.

Here's some work arounds until their "graph database" gets fixed.

Theoretically one can identify the "case", find unique clinical features and use an old clinical data file which has the TCGA barcode to figure out who's who.

However, the TCGA barcode is often in the BAM file header.
So, you can bamslice all the samples for example TCGA KICH for your regions of interest using the file_ids. Then you can examine the header to get the TCGA barcode and do whatever ... rename to something useful or use a custom look up table to discombobulate things.

CAVEAT: not all queries finish. Check your results to make sure the REST operation worked. If you are doing large batch queries , you may have to re-run a few. CAVEAT2; official GDC files support hg38, not hg19, not hg18 which cghub used. Use UCSC "liftOver" program to get coordinates converted to hg38.

Here's a script that works for bamslicing ...
export token=YOURTOKENHEREITISLONGANDASCII

function f
{
echo "curl --header "X-Auth-Token: ${token}" 'https://gdc-api.nci.nih.gov/slicing/view/${1}?region=chr11:95992210-96192210&region=chr16:71829584-72029584' --output ${1}.bam ; sleep 1"
}

f 16f3c691-7652-4068-a2ec-f5b6ba219687
f 914e2786-90f8-4ab6-8e7d-8d77e96b9ecf
....
( just call function "f" with your favorite file_id as the argument )
....

Just pipe the output to bash or hack to call curl in-line in the script.

Then you can look in the bam file header using "samtools view -H whatever.bam" and grep for "TCGA"