Add Custom BedGraph Track to UCSC Genome Browser

Previously, I made a post on adding custom tracks to the UCSC Genome Broswer and have even expanded on coloring these tracks.  For most of my applications, I have simply used standard .bed files that plot features of interest in the Genome Browser in relation to other UCSC tracks.  Today I wanted to plot data from overlapping features in a depth plot similar to what one would see after next-generation sequencing.  I found the easiest way to do this was by importaing a BedGraph file into the Genome Browser.  Following steps similar to the custom tracks post, you go to the My Data -> Custom Tracks -> Add Custom Tracks and then upload the BedGraph file.  Below is an example of what the header and the first few lines need to look like.


More details and options on the BedGraph track format can be found here on the UCSC webpage.

Generate Random Genomic Positions

Generating random genomic positions or coordinates can be useful in comparing characteristics of a set of genomic loci to that what would be expected from permutations of the underlying genomic distribution.  Below is a Python script to aid in selecting random genomic positions.  The script chooses a chromosome based on probabilities assigned by chromosome length and then chooses a chromosomal position from a uniform distribution of the chromosome's length.  An added gap checking statement is included to ensure the chosen position lies within the accessible genome.  You can choose the number of positions you want, the number of permutations to conduct, the size of the genomic positions, and the genomic build of interest.  A UNIX shell script is included as a wrapper to automatically download needed chromosomal gap and cytoband files as well as run the Python script.  Useage for the UNIX script can be seen by typing ./make_random.sh from the command line after giving the script executable privileges.  An example command would be ./make_random 100 10 1000 hg19.  This command would make 10 .bed files each with 100 random 1Kb genomic regions from the hg19 genome build.  Below are the make_random.sh and make_random.py scripts.

Visualize CTCF Binding Sites in UCSC Genome Browser

The Encyclopedia of DNA Elements (ENCODE) Consortium has recently added some new and exciting information to the University of California Santa Cruz (UCSC) genome browser relating to CTCF binding sites.  CTCF is a transcriptional repressor encoded by the CTCF gene that regulates the 3 dimensional (3D) structure of chromatin.  CTCF mediated changes in chromatin structure can lead to insulator activity, transcriptional regulation, and changes in RNA splicing.  CTCF binding sites typically have a CCCTC motif used by CTCF to bind together strands of DNA.  Changes in gene expression can result from the bound CTCF sites insulating the activity of an enhancer from a gene's promoter site.

The UCSC genome browser has two tracks that aid in identifying CTCF binding sites in the genome.  The two tracks are the Transcription Factor ChIP-seq track and the Genome Segmentations from ENCODE track.  Both UCSC data tracks use ChIP-seq data and can be filtered to localize CTCF binding sites.  The Transcription Factor ChIP-seq track shows CTCF binding sites from a large collection of ChIP-seq experiments.  The Genome Segmentation track takes things a step further by segmenting the genome into a variety of functionally relevant segments (ex: insulators, enhancers, promoters) using two bioinformatic algorithms (ChromHMM and Segway).

A helpful link to visualize CTCF insulator sites in the UCSC genome browser is here.  This will link to a session where the two above described ENCODE tracks are selected in the browser window.  It should look something like the picture above the post.  If you are aware of other resources or other UCSC data tracks that localize genomic insulators such as CTCF binding sites, please share in the comments below.

Install liftOver Locally on UNIX

Many of the UCSC genome tools are available for download for use locally on your UNIX system.  liftOver is an example of one such tool.  To download, go to their apps download page, select your operating system, and then click on the liftOver link.  Here are links for liftOver_linux.x86_64 and liftOver_linux.x86_64.v287.  For some reason, I could only get the linux.x86_64.v287 version to work on my system.  Once downloaded, make it executable.

The next step is to get the required chain files needed to convert from one genome build to another.  All UCSC genome builds are listed here and you can select the desired "LiftOver files" link under the genome build that you want to convert from.  This will take you to a downloads page with links to the chain files.  Chain files are appropriately named so you know what builds you are converting to and from.  For example,  hg19ToHg18.over.chain.gz is the chain file needed to convert from hg19 to hg18.  Once downloaded, unzip the file for use.

To run liftOver, the useage is:
liftOver oldFile map.chain newFile unMapped

where:
oldFile is the file you want to convert from
map.chain is the chain file used to convert from one build to another
newFile is the converted file you want to create
unMapped is a file that contains all the unmapped positions

For more details on usage, just type liftOver in the command line.

Color Features in UCSC Genome Browser Custom Track with RGB BED File Field

Custom tracks are incredibly useful in visualizing your data in the UCSC genome browser.  All that is needed is for your data to be in a compatible format.  I typically use .bed files.  The first three columns of .bed files are required fields for chromosome, start, and stop position:

(1) chrom - name of the chromosome or scaffold.  Chromosome names can be given with or without the 'chr' prefix.
(2) chromStart - Start position of the feature in standard chromosomal coordinates (i.e. first base is 0).
(3) chromEnd - End position of the feature in standard chromosomal coordinates

The next six fields are optional. Note that columns cannot be empty (ie: lower-numbered fields must always be populated if higher-numbered ones are used).

(4) name - Label to be displayed under the feature.
(5) score - A score between 0 and 1000.
(6) strand - defined as + (forward), - (reverse), or . (not applicable).
(7) thickStart - field used by UCSC drawing code, typically same as chromStart.
(8) thickEnd - field used by UCSC drawing code, typically same as chromEnd.
(9) itemRgb - an RGB color value (e.g. 0,0,255).

Coloring elements is fairly straightforward.  First define RGB values for each track feature in the itemRgb field (9th column) and ensure fields 1-8 have appropriate values.  Then, upload your track as a custom track (see above link) to UCSC.  Once uploaded, click on the track name (usually "User Track") and edit the configuration to include: itemRgb="On".  Other configuration values can be changed as well:

name - unique name to identify this track in the custom tracks list.
description - label to be displayed above the track in Genome Browser.
priority - integer defining the order in which to display tracks, if multiple tracks are defined.
useScore - a value from 1 to 4, which determines how scored data will be displayed. Additional parameters may be needed.
itemRgb - if set to 'on' (case-insensitive), the individual RGB values defined in tracks will be used.

Here's an example configuration:

Encode Features with no Strand Information in UCSC BED Format

The UCSC Genome Browser is a powerful tool that allows you to visualize your own data in custom tracks with respect to scores of other publicly available data tracks.  To use some of the more advanced features for plotting .bed files, strand information is a required field in the .bed files.  Usually, I would just arbitrarily assign a "+" for the strand information to indicate the feature was on the plus strand, when in actuality there is no relevant strand for the feature.  This results in arrows (>>>>>>) being drawn over the feature in Genome Browser, not the end of the world, but still not the polished look I was going for.  To circumvent this, I tried to leave the field blank when uploading a custom track, however, I would get the error: "Error line 1 of custom track: Expecting + or - in strand".  Then I stumbled on a simple fix, that doesn't seem to be documented well.  All Genome Browser needs is a period (.) in the strand field of the .bam file for each feature and all seems to work fine.  The result: a sharp looking feature track in Genome Browser.

UCSC Tools for BigWig Files

Several UCSC data tracks are downloaded in BigWig format (.bw).  This is a compressed file that cannot be read by common text editors.  Rather, you need to use utilities built to handle these files.  Several are available at the ftp website: http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/.  Here are a few UNIX programs I have found useful:

bigWigAverageOverBed

bigWigInfo

bigWigSummary

bigWigToBedGraph

Download Data Track from UCSC Genome Browser

Did you know you can download tracks you visualize in the UCSC Genome Browser for personal use and analysis?  Here's how to do it.

(1) Click on the grey bar on the far left side of the UCSC data track.  This will bring you to the track settings page.
(2) Click on the link called View table schema.  This will bring up a new page with track information and a description of data fields.
(3) Look for the field called Primary Table and copy the name.
(4) Go to the UCSC FTP site (link) and find the correct genome build you are after.  Usually you will want to select hg19 or hg18.
(5) Click on the database link and then search for the name of the field you copied from the Primary Table field in step 3.
(6) There will usually be a .sql and a .txt.gz for most tracks.  You are interested in the .txt.gz file.  You can click on it to download via your internet web browser or right click on the link to copy the web address and use the wget command to download it.  Here's and example script to download the NHGRI GWAS catalog using the wget command:
(7) Extract the compressed .txt.gz file with the following command, where filename is the name of the file you downloaded

This method should work for downloading the majority of the UCSC data tracks.  Sometimes it takes a bit of digging around the UCSC FTP site to find the dataset you are looking for, but in most cases I have been successful in finding it on the UCSC FTP site.

One final note:  If you are interested in downloading only a small portion of the track (for example, just a region on chromosome 8), you can download this region using the UCSC Table browser.  Here's how to do this:

(1) Follow steps (1) and (2) above.
(2) Once on the Table Schema page for the track of interest go to the link bar on the top of the page and select Tools > Table Browser.  This will take you to the UCSC Table Browser where all the fields will already be filled in with the track you are interested in.
(3) To download your region of interest click on the radio button next to position and type in your desired coordinates (ex: chr8:128362121-129551294).
(4) Make sure all the filters are cleared and give your output a filename.  Select get output and your file will be downloaded.  There is no need to unzip unless you chose the gzip compressed option.

Best wishes and good luck analyzing UCSC data tracks!

Download Nucleotide Sequence for Genomic Region

Sometimes I need the nucleotide sequence for a specific region of the genome to investigate sequence similarity, simple repeats present, or recurring motifs.  I know entire chromosomal .fasta files can be downloaded from the UCSC ftp site, but then I would have to go through the entire file and hopefully extract out the correct sequence I needed.  Today I came across a very easy way to download a nucleotide sequence for a genomic region using the UCSC DAS server.  Simply modify the below web link to include the appropriate genome build and genomic coordinates and you will get a customized XML page generated with the nucleotide sequence for your query.  One word of caution: the DAS server uses an index of +1 for the first base.  Pretty cool and very simple to do.

http://genome.ucsc.edu/cgi-bin/das/hg19/dna?segment=chr1:100000,200000