1000 Genome Phase 3 Variant Set Mirrors

The 1000 Genomes consortium has released their final release of the Phase 3 variant set. This is a set of phased called variants in thousand genome individuals. Data is available in compressed .vcf.gz format along with the indexed tabix files. Looking for a way to download the data. See these FTP sites:

US Mirror:
ftp://ftp.ncbi.nlm.nih.gov/1000genomes/ftp/release/20130502/

UK Mirror:
ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/

The US 1000 Genomes mirror is substantially faster than the UK mirror when downloading files in the US.

Generate Random Genomic Positions

Generating random genomic positions or coordinates can be useful in comparing characteristics of a set of genomic loci to that what would be expected from permutations of the underlying genomic distribution.  Below is a Python script to aid in selecting random genomic positions.  The script chooses a chromosome based on probabilities assigned by chromosome length and then chooses a chromosomal position from a uniform distribution of the chromosome's length.  An added gap checking statement is included to ensure the chosen position lies within the accessible genome.  You can choose the number of positions you want, the number of permutations to conduct, the size of the genomic positions, and the genomic build of interest.  A UNIX shell script is included as a wrapper to automatically download needed chromosomal gap and cytoband files as well as run the Python script.  Useage for the UNIX script can be seen by typing ./make_random.sh from the command line after giving the script executable privileges.  An example command would be ./make_random 100 10 1000 hg19.  This command would make 10 .bed files each with 100 random 1Kb genomic regions from the hg19 genome build.  Below are the make_random.sh and make_random.py scripts.

Change Citation Reference Style in EndNote X6

I was quite happy with Mendeley as my reference manager for a while, but recently it has been difficult to produce certain reference styles for lesser known journals.  Added to the difficulty of collaborating with others on manuscripts, I ultimately chose to convert to EndNote (non-open source, paid software...boo).  In all honesty, its not too bad and has plugins for Microsoft Word that work a lot like Mendeley.  When setting things up, I found importing journal citation styles could be a bit tricky, especially for newbies, so here is a quick primer to help out.

First go to Edit>Output Styles and look to see if the style you want is listed in the drop down menu.  If you see it you are in luck.  If not, click on Open Style Manager....

The EndNote Styles dialogue box will pop up.  There should be a list of styles that come pre-loaded with your installation.  If you see the journal style you need in the list, simply click the box next to the style so it is checked, exit out of the dialogue box, and click on that same style in the Edit>Output Styles menu.  If you don't see the reference style you need, keep reading.

To find additional styles, click on the Get More on the Web... button in the EndNote Styles dialogue box.  This will open the EndNote Output Styles Download site in your default web browser.  Use the fields to search for your style and if you find it click to download it.  Once downloaded, double click or drag your style into the EndNote.  A dialogue box with that style should pop up.  If everything looks fine, click File>Save As, delete copy from the style's name and save it.  To use it, check the box next to the style name and select it from the Edit>Output Styles menu.

If you don't find the style you need on the EndNote site, you can create your own style under Edit>Output Sytles>New Style....  A dialogue box named Untitled Style will pop up and you can begin to create a new style that matches your needs.  You are on your own here.  I have little experience with this.  Some tips are to start with a style that is close to what you need and edit it (ie: go to Edit>Output Styles>Edit "StyleNameHere", where StyleNameHere is the name of the style close to what you want).  Just be sure to save your style before closing out of the dialogue box.

Hope this helps.  Feel free to leave tips, helpful suggestions, or comments on anything I left out.

Installing Tabix on UNIX

Tabix is an incredibly useful tool that indexes large .vcf files and makes it very efficient to find a particular genomic region in the file.  It is particularly useful for downloading small portions of files from publicly available ftp data repositories.  Here is a brief tutorial on how to install Tabix on a UNIX operating system.

(1) Go here to download the newest release.
(2) Extract the file:

(3) Compile the program by typing make on the UNIX command line.
(4) Export the path by adding the following line to your .bashrc file, saving your .bashrc file, and typing source on the UNIX command line.  Note: path_to_tabix is the directory where tabix is installed.

Confirm Checksum of a Downloaded File

When copying data from SFTP, FTP, or other websites you may have noticed a lot of these large files and program packages have checksum values posted.  These are essentially "digital fingerprints" to ensure what you have downloaded is truly what you wanted in its complete and full integrity form.  So how exactly do you find the checksum value of the file you downloaded?  This is relatively simple to do in UNIX.  First download the file and go to the directory the file is stored in.  Then see what type of checksum was published for the file.  Here are a few popular checksum types and the UNIX commands to use to calculate them:

Checksum - UNIX command
md5 - md5sum
sha1 - sha1sum
sha256 - sha256sum

Pretty simple command structure.  Usually you just add "sum" after the checksum name and there is an aptly named UNIX command that will perform the function.  As far as usage goes, simply type the command followed by the file name you want to check the checksum for.


Alternatively, to have the checksum program check the sum for you, use the -c (or --check) option with the known checksum value pasted into the command.  See the below example (Note the two spaces between the checksum and the file name and the dash at the end of the command).


If the checksums are equal you will get "filename.txt: OK".  If not, you will get the error "WARNING: 1 of 1 computed checksum did NOT match".  Hope this is helpful for checking file integrity.  Please add suggestions you have in the comments section below.

Download Data Track from UCSC Genome Browser

Did you know you can download tracks you visualize in the UCSC Genome Browser for personal use and analysis?  Here's how to do it.

(1) Click on the grey bar on the far left side of the UCSC data track.  This will bring you to the track settings page.
(2) Click on the link called View table schema.  This will bring up a new page with track information and a description of data fields.
(3) Look for the field called Primary Table and copy the name.
(4) Go to the UCSC FTP site (link) and find the correct genome build you are after.  Usually you will want to select hg19 or hg18.
(5) Click on the database link and then search for the name of the field you copied from the Primary Table field in step 3.
(6) There will usually be a .sql and a .txt.gz for most tracks.  You are interested in the .txt.gz file.  You can click on it to download via your internet web browser or right click on the link to copy the web address and use the wget command to download it.  Here's and example script to download the NHGRI GWAS catalog using the wget command:
(7) Extract the compressed .txt.gz file with the following command, where filename is the name of the file you downloaded

This method should work for downloading the majority of the UCSC data tracks.  Sometimes it takes a bit of digging around the UCSC FTP site to find the dataset you are looking for, but in most cases I have been successful in finding it on the UCSC FTP site.

One final note:  If you are interested in downloading only a small portion of the track (for example, just a region on chromosome 8), you can download this region using the UCSC Table browser.  Here's how to do this:

(1) Follow steps (1) and (2) above.
(2) Once on the Table Schema page for the track of interest go to the link bar on the top of the page and select Tools > Table Browser.  This will take you to the UCSC Table Browser where all the fields will already be filled in with the track you are interested in.
(3) To download your region of interest click on the radio button next to position and type in your desired coordinates (ex: chr8:128362121-129551294).
(4) Make sure all the filters are cleared and give your output a filename.  Select get output and your file will be downloaded.  There is no need to unzip unless you chose the gzip compressed option.

Best wishes and good luck analyzing UCSC data tracks!

Download Nucleotide Sequence for Genomic Region

Sometimes I need the nucleotide sequence for a specific region of the genome to investigate sequence similarity, simple repeats present, or recurring motifs.  I know entire chromosomal .fasta files can be downloaded from the UCSC ftp site, but then I would have to go through the entire file and hopefully extract out the correct sequence I needed.  Today I came across a very easy way to download a nucleotide sequence for a genomic region using the UCSC DAS server.  Simply modify the below web link to include the appropriate genome build and genomic coordinates and you will get a customized XML page generated with the nucleotide sequence for your query.  One word of caution: the DAS server uses an index of +1 for the first base.  Pretty cool and very simple to do.

http://genome.ucsc.edu/cgi-bin/das/hg19/dna?segment=chr1:100000,200000