Download Data Track from UCSC Genome Browser

Did you know you can download tracks you visualize in the UCSC Genome Browser for personal use and analysis?  Here's how to do it.

(1) Click on the grey bar on the far left side of the UCSC data track.  This will bring you to the track settings page.
(2) Click on the link called View table schema.  This will bring up a new page with track information and a description of data fields.
(3) Look for the field called Primary Table and copy the name.
(4) Go to the UCSC FTP site (link) and find the correct genome build you are after.  Usually you will want to select hg19 or hg18.
(5) Click on the database link and then search for the name of the field you copied from the Primary Table field in step 3.
(6) There will usually be a .sql and a .txt.gz for most tracks.  You are interested in the .txt.gz file.  You can click on it to download via your internet web browser or right click on the link to copy the web address and use the wget command to download it.  Here's and example script to download the NHGRI GWAS catalog using the wget command:
(7) Extract the compressed .txt.gz file with the following command, where filename is the name of the file you downloaded

This method should work for downloading the majority of the UCSC data tracks.  Sometimes it takes a bit of digging around the UCSC FTP site to find the dataset you are looking for, but in most cases I have been successful in finding it on the UCSC FTP site.

One final note:  If you are interested in downloading only a small portion of the track (for example, just a region on chromosome 8), you can download this region using the UCSC Table browser.  Here's how to do this:

(1) Follow steps (1) and (2) above.
(2) Once on the Table Schema page for the track of interest go to the link bar on the top of the page and select Tools > Table Browser.  This will take you to the UCSC Table Browser where all the fields will already be filled in with the track you are interested in.
(3) To download your region of interest click on the radio button next to position and type in your desired coordinates (ex: chr8:128362121-129551294).
(4) Make sure all the filters are cleared and give your output a filename.  Select get output and your file will be downloaded.  There is no need to unzip unless you chose the gzip compressed option.

Best wishes and good luck analyzing UCSC data tracks!

Download Region of Indexed VCF File

Tabix is a handy tool to download a region of interest from a .vcf file on an ftp site.  Below is an example of how to extract a region on chromosome 1 from a Complete Genomics .vcf file hosted on the 1000 Genomes ftp site.  The .vcf.gz file must have a matching .vcf.gz.tbi index file for tabix to work and the chromosome region of interest should not include "chr" with the chromosome number.

Download Complete Genomics Reference Files

A reference genome is needed when using cgatools on Complete Genomics data.  Here are links to the ftp sites that contain the reference compact randomly accessible reference (.crr) files.  Just use the wget command from a UNIX cluster to download.

NCBI Build 36:
ftp://ftp.completegenomics.com/ReferenceFiles/build36.crr

NCBI Build 37:
ftp://ftp.completegenomics.com/ReferenceFiles/build37.crr

The next step is to verify that the file downloaded completely.  Run one of the following commands at the command prompt depending on the version of the .crr file you downloaded.


The file output should look like this for build 36.


And this for build 37.

Thousand Genomes Complete Genomics Information

Recently I have been using the Complete Genomics high coverage sequencing data from the 1000 Genomes project.  Its a bit difficult to find information on the populations, samples, and available sequencing data since they are all stored in different places on their ftp server.  I decided to make a post that tried to combine all the useful information into one spot.  Here are links to files that may be of interest.

population file: gives a key of abbreviations used for the 1000 Genomes populations

pedigree file: provides relationship information on 1000 Genomes individuals that are related as well as gender and 1000 Genomes population

sample file: detailed spreadsheet that offers information on sample id, accession number, population, family, gender, relationship, sequencing center, and coverage for each 1000G sample.

Samtools Download BAM Region Only

Often times publicly available sequencing data can serve as a useful reference for a sequencing project. The 1000 Genomes project is a great source, especially with their newly released high coverage Complete Genomics data.  Here is an example UNIX script that shows how BAM files with genomic regions of interest can be created from a whole-genome BAM file that is hosted on an FTP server, without having to download the entire BAM file first.  The bai_file_list.txt file is a file that contains unique identifiers for each BAM file extracted from a previously selected list of BAI files of interest.  Here I am just extracting the BRCA1 and BRCA2 regions of the genome.  The extracted reads are sorted and then saved as a BAM file and an associated BAI index file is also created.  The final step removes excess BAI files that are downloaded and used by Samtools to extract the region of interest from the BAM files on the FTP server.

Thoudand Genomes Complete Genomics Index

As mentioned previously, 1000 Genomes has now made Complete Genomics whole genome sequencing publicly available for download.  They give an index file that mentions some of the individual high coverage .bam files that are available for download, but it seems to be missing a lot of the newer data released this April.  I was trying to find a way to efficiently search through the 1000 Genomes ftp site to get a better index of the available Complete Genomics data.  I am primarily interested in CEU samples and wanted the high coverage evidence support files.  Here is the code I used to search through the ftp. I chose just to search for and download .bai files from the site since they are quick to download and would create a useful index for downloading the bam files.

1000 Genomes Complete Genomics Data

1000 Genomes now has Complete Genomics high coverage sequencing data available for 57 samples online at the ftp sites: ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/data and ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data.  The first ftp is best for US access and the second is best for access from Europe.  Here's a link to the 1000 Genomes page for more details: http://www.1000genomes.org/announcements/complete-genomics-data-available-2012-12-19.