Normal Distribution Functions in R

I always need to look up how to use the distributional functions in R. Rather than it always being a guessing game I made a quick primer with visual example plots of what each command means and the results each command actually returns.

The below examples assume a N(0,1). A link to the R manual is here.

dnorm-returns the height of the normal curve at a specified value along the x-axis

pnorm-the cumulative density function (CDF) that returns the area to the left of a specified value

qnorm-returns quantiles or "critical values"

rnorm-generates random numbers from the normal distribution

These are examples for the normal distribution, but as you could imagine R has commands for numerous other distributions such as the chi-square, beta, uniform, and poisson.

If you are curious, here are the commands I used to plot these figures:

Import Variable into R Script

I always forget the syntax for defining a variable to read into an R script. It is relatively easy to import an external variable into R (such as a Unix variable or user-defined variable). The option to feed R a variable is incredibly useful on a cluster system where you want to farm out a large job into smaller sized pieces. Here is example code for doing so:

Run Unix Commands and Other Programs within Python

The python subprocess module is a part of the standard python library that enables a user to run a variety of subprocess within python and collect their output and error pipes. It is a wonderfully handy tool that really expands the possibilities of what python is capable of doing.  The tool is intended to build upon and essentially replace a lot of the functionality of the os package which I have found cumbersome to use.  The main commands of the subprocess module include:

subprocess.call-runs a command and returns output and the exit code
subprocess.check_call-runs a command; if exit code is 0 returns output, otherwise CalledProcessError

Here are some simple example scripts to help get started using the subprocess inside python code:

Copy and Transfer Files To or From SFTP Site

Transferring files to and from a SFTP site is relatively simple once you have generated a public/private key pair and have your account set up.

To login to the SFTP server, type in sftp username@server at the UNIX command prompt.  Once logged in, you can change directories on the SFTP site similar to how you would change directories in UNIX: with the cd command.  You can also change directories on you local account using the lcd command.  To transfer files from one server to another, you need to first be in the correct local and SFTP directories from which you want the files transferred to/from.

To copy a file from the SFTP server to your local host, use the get command.  For example, if you wanted to get the file ids.txt, you would type get ids.txt at the command prompt.  Conversely, to transfer file to the SFTP site from your local host, use the put command.  The SFTP also uses wildcards (*), so if for example, you wanted to transfer all .jpg files to the SFTP server, you would type in put *.jpeg at the command prompt.  Below are a few other useful commands.

Sftp Command	Description
cd dir	Change directory on the ftp server to dir.
lcd dir	Change directory on your machine to dir.
ls	List files in the current directory on the ftp server.
lls	List files in the current directory on your machine.
pwd	Print the current directory on the ftp server.
lpwd	Print the current directory on your machine.
get file	Download the file from the ftp server to current directory.
put file	Upload the file from your machine to the ftp server.
exit	Exit from the sftp program.

Also, remember it is always a good idea to check the checksums of files you have downloaded with those on the SFTP site to ensure you downloaded the files in their entirety.

List Contents of a .tar.gz File

Here is a quick one liner to see the contents of a .tar.gz file:

This will list all the files in the .tar.gz tarball.  The v command can be left out to not show file information.

Extract Only Desired Files from Compressed File

Lately I have been working with some large tarballs (tar.gz files).  To access the compressed contents, usually I would extract all the files that were compressed in the archive.  This was time consuming to decompress all the files, a pain to filter through looking for the files I wanted, and required more time to delete everything I didn't want.  I knew there was a better way to just get the files I wanted.  Sure enough, the basic tar command has options that allow you just to extract a file of interest or even a set of files using wildcards.  The command looks like this:

where,
big_file.tar.gz is the tarball you are extracting from

path is the path to your file in the tarball

your_file.txt is the file you want to extract

This can also be done using wildcards.  For example, if I wanted all text files in the above path I could substitute *.txt for your_file.txt.

One Line Command to Remove Header from File in UNIX

Sometimes you need to remove a header, footer, or range of lines from a file to better manipulate it in UNIX.  Here are some quick one liners to do so:

Install liftOver Locally on UNIX

Many of the UCSC genome tools are available for download for use locally on your UNIX system.  liftOver is an example of one such tool.  To download, go to their apps download page, select your operating system, and then click on the liftOver link.  Here are links for liftOver_linux.x86_64 and liftOver_linux.x86_64.v287.  For some reason, I could only get the linux.x86_64.v287 version to work on my system.  Once downloaded, make it executable.

The next step is to get the required chain files needed to convert from one genome build to another.  All UCSC genome builds are listed here and you can select the desired "LiftOver files" link under the genome build that you want to convert from.  This will take you to a downloads page with links to the chain files.  Chain files are appropriately named so you know what builds you are converting to and from.  For example,  hg19ToHg18.over.chain.gz is the chain file needed to convert from hg19 to hg18.  Once downloaded, unzip the file for use.

To run liftOver, the useage is:
liftOver oldFile map.chain newFile unMapped

where:
oldFile is the file you want to convert from
map.chain is the chain file used to convert from one build to another
newFile is the converted file you want to create
unMapped is a file that contains all the unmapped positions

For more details on usage, just type liftOver in the command line.

How to Get RGB Values for a Color in R

For continuity in creating presentations and figures it is often necessary to have color schemes that are consistent for certain features, especially when different plots and figures of these features are made using different software.  R includes a nice palette of pre-defined colors for easily implementing rich colors in graphics.  In order to extract the RGB values (ie: red, green, blue) to use these colors in other programs, R has a nice built-in command that extracts the RGB values: col2rgb.  Simply put your color name in quotes and parenthesis and R will return the RGB values.  Here is an example.

Unix Sort by Column Number

I always forget the flags to use when using UNIX sort.  Here is a list of common useful flags.

Flags

-k: selects the column number to sort

-n: indicates you are sorting numeric values so use numeric ordering (ex: 10, 11, not 10, 100)

-r: sort in reverse (descending) order for this column

An example line of code that sorts by numeric values in column 2 in descending order would be as follows.