Search for File Type in UNIX Directory and All Subdirectories

Here's a simple way to find and list all files in a UNIX directory and the containing subdirectories.  This example script shows how to search for all files with the extension ".R":

Run Unix Commands and Other Programs within Python

The python subprocess module is a part of the standard python library that enables a user to run a variety of subprocess within python and collect their output and error pipes. It is a wonderfully handy tool that really expands the possibilities of what python is capable of doing.  The tool is intended to build upon and essentially replace a lot of the functionality of the os package which I have found cumbersome to use.  The main commands of the subprocess module include:

subprocess.call-runs a command and returns output and the exit code
subprocess.check_call-runs a command; if exit code is 0 returns output, otherwise CalledProcessError

Here are some simple example scripts to help get started using the subprocess inside python code:

Install Cufflinks on a UNIX Account

Cufflinks is a useful program for managing and analyzing RNA-seq data.  Cufflinks, along with the included cuffcompare, cuffdiff, cuffmerge, cuffnorm, cuffquant, gffread, and gtf_to_sam programs, can all be downloaded in a pre-compiled .tar file from from their download site.  Just be sure to download the most recent version of the Linux distribution if you want to put it on a UNIX machine.  To install cufflinks, simply follow these steps.

(1) Untar the cufflinks-0.0.0.Linux_x86_64.tar.gz in the desired directory you want to place cufflinks

(2) Add the following line to you .bashrc file and then type source .bashrc.


The above example uses cufflinks-0.0.0 as an example version.  This needs to be changed in the above code to match the current version.

UNIX For Loop through Files in Directory

Sometimes I want to do a particular task to each file in a directory using a UNIX shell script, but for some reason I can never remember the correct syntax to do so.  Here is a quick example bash script to loop through all files of a particular extension in a directory and do an action with the filenames.


The above example loops through all .tar.gz files in the directory and extracts any text file within those files.

Generate Random Genomic Positions

Generating random genomic positions or coordinates can be useful in comparing characteristics of a set of genomic loci to that what would be expected from permutations of the underlying genomic distribution.  Below is a Python script to aid in selecting random genomic positions.  The script chooses a chromosome based on probabilities assigned by chromosome length and then chooses a chromosomal position from a uniform distribution of the chromosome's length.  An added gap checking statement is included to ensure the chosen position lies within the accessible genome.  You can choose the number of positions you want, the number of permutations to conduct, the size of the genomic positions, and the genomic build of interest.  A UNIX shell script is included as a wrapper to automatically download needed chromosomal gap and cytoband files as well as run the Python script.  Useage for the UNIX script can be seen by typing ./make_random.sh from the command line after giving the script executable privileges.  An example command would be ./make_random 100 10 1000 hg19.  This command would make 10 .bed files each with 100 random 1Kb genomic regions from the hg19 genome build.  Below are the make_random.sh and make_random.py scripts.

Copy and Transfer Files To or From SFTP Site

Transferring files to and from a SFTP site is relatively simple once you have generated a public/private key pair and have your account set up.

To login to the SFTP server, type in sftp username@server at the UNIX command prompt.  Once logged in, you can change directories on the SFTP site similar to how you would change directories in UNIX: with the cd command.  You can also change directories on you local account using the lcd command.  To transfer files from one server to another, you need to first be in the correct local and SFTP directories from which you want the files transferred to/from.

To copy a file from the SFTP server to your local host, use the get command.  For example, if you wanted to get the file ids.txt, you would type get ids.txt at the command prompt.  Conversely, to transfer file to the SFTP site from your local host, use the put command.  The SFTP also uses wildcards (*), so if for example, you wanted to transfer all .jpg files to the SFTP server, you would type in put *.jpeg at the command prompt.  Below are a few other useful commands.

Sftp Command	Description
cd dir	Change directory on the ftp server to dir.
lcd dir	Change directory on your machine to dir.
ls	List files in the current directory on the ftp server.
lls	List files in the current directory on your machine.
pwd	Print the current directory on the ftp server.
lpwd	Print the current directory on your machine.
get file	Download the file from the ftp server to current directory.
put file	Upload the file from your machine to the ftp server.
exit	Exit from the sftp program.

Also, remember it is always a good idea to check the checksums of files you have downloaded with those on the SFTP site to ensure you downloaded the files in their entirety.

Generate Public/Private RSA Key Pair for SFTP

To access an SFTP server, an SSH public key must be created and shared with the SFTP host.  To generate a public/private key pair the ssh-keygen utility in UNIX can be used.  This program will calculate a key pair for you by following these steps:

(1) At the UNIX prompt, type ssh-keygen.  A message will appear indicating the application is generating the public/private rsa key pair.

(2) You will next be prompted to enter the location where you want to save the key.  You can specify a filename or just leave the space blank and press Enter.

(3) Afterwards you will be prompted to enter a passphrase to use as a password.  Enter your passphrase here or you can opt to leave the space blank.  It is acceptable to simply hit Enter to proceed without specifying a passphrase.

(4) Finally, you will be prompted to re-enter your passphrase.

That is all there is to creating a public and private key pair.  To access the remote SFTP server the administrator will need you to forward a copy of your public key to set up your account.  The public key will be a file with one line that includes "ssh-rsa", a long key code, followed by your account name and host name (ex: user@host.school.edu).  The file extension will be .pub.

List Contents of a .tar.gz File

Here is a quick one liner to see the contents of a .tar.gz file:

This will list all the files in the .tar.gz tarball.  The v command can be left out to not show file information.

Extract Only Desired Files from Compressed File

Lately I have been working with some large tarballs (tar.gz files).  To access the compressed contents, usually I would extract all the files that were compressed in the archive.  This was time consuming to decompress all the files, a pain to filter through looking for the files I wanted, and required more time to delete everything I didn't want.  I knew there was a better way to just get the files I wanted.  Sure enough, the basic tar command has options that allow you just to extract a file of interest or even a set of files using wildcards.  The command looks like this:

where,
big_file.tar.gz is the tarball you are extracting from

path is the path to your file in the tarball

your_file.txt is the file you want to extract

This can also be done using wildcards.  For example, if I wanted all text files in the above path I could substitute *.txt for your_file.txt.

Installing Tabix on UNIX

Tabix is an incredibly useful tool that indexes large .vcf files and makes it very efficient to find a particular genomic region in the file.  It is particularly useful for downloading small portions of files from publicly available ftp data repositories.  Here is a brief tutorial on how to install Tabix on a UNIX operating system.

(1) Go here to download the newest release.
(2) Extract the file:

(3) Compile the program by typing make on the UNIX command line.
(4) Export the path by adding the following line to your .bashrc file, saving your .bashrc file, and typing source on the UNIX command line.  Note: path_to_tabix is the directory where tabix is installed.

Find SNP Overlap from Different SNP Arrays

If you have .bed files for array manifests it is relatively easy to use UNIX to compare the overlap between SNP positions in the two files.  Here is a quick one line of code to do so:


This essentially finds all rows that are the same in the two files.  It requires all unique rows in each file and will only work for one base pair long items.

One Line Command to Remove Header from File in UNIX

Sometimes you need to remove a header, footer, or range of lines from a file to better manipulate it in UNIX.  Here are some quick one liners to do so:

Install liftOver Locally on UNIX

Many of the UCSC genome tools are available for download for use locally on your UNIX system.  liftOver is an example of one such tool.  To download, go to their apps download page, select your operating system, and then click on the liftOver link.  Here are links for liftOver_linux.x86_64 and liftOver_linux.x86_64.v287.  For some reason, I could only get the linux.x86_64.v287 version to work on my system.  Once downloaded, make it executable.

The next step is to get the required chain files needed to convert from one genome build to another.  All UCSC genome builds are listed here and you can select the desired "LiftOver files" link under the genome build that you want to convert from.  This will take you to a downloads page with links to the chain files.  Chain files are appropriately named so you know what builds you are converting to and from.  For example,  hg19ToHg18.over.chain.gz is the chain file needed to convert from hg19 to hg18.  Once downloaded, unzip the file for use.

To run liftOver, the useage is:
liftOver oldFile map.chain newFile unMapped

where:
oldFile is the file you want to convert from
map.chain is the chain file used to convert from one build to another
newFile is the converted file you want to create
unMapped is a file that contains all the unmapped positions

For more details on usage, just type liftOver in the command line.

Confirm Checksum of a Downloaded File

When copying data from SFTP, FTP, or other websites you may have noticed a lot of these large files and program packages have checksum values posted.  These are essentially "digital fingerprints" to ensure what you have downloaded is truly what you wanted in its complete and full integrity form.  So how exactly do you find the checksum value of the file you downloaded?  This is relatively simple to do in UNIX.  First download the file and go to the directory the file is stored in.  Then see what type of checksum was published for the file.  Here are a few popular checksum types and the UNIX commands to use to calculate them:

Checksum - UNIX command
md5 - md5sum
sha1 - sha1sum
sha256 - sha256sum

Pretty simple command structure.  Usually you just add "sum" after the checksum name and there is an aptly named UNIX command that will perform the function.  As far as usage goes, simply type the command followed by the file name you want to check the checksum for.


Alternatively, to have the checksum program check the sum for you, use the -c (or --check) option with the known checksum value pasted into the command.  See the below example (Note the two spaces between the checksum and the file name and the dash at the end of the command).


If the checksums are equal you will get "filename.txt: OK".  If not, you will get the error "WARNING: 1 of 1 computed checksum did NOT match".  Hope this is helpful for checking file integrity.  Please add suggestions you have in the comments section below.

How to Fully Utilize All Cores of a UNIX Compute Node

Parallelizing tasks can drastically improve computation time of a program.  One way to do this is to ensure your code is utilizing all available cores of a processor.  To do this you need to write your code in such a way that background tasks are being carried out simultaneously.  This is done by inserting the ampersand (&) at the end of a line of code.  If the number of background tasks running equals the number of cores of the computer node, then you are efficiently and fully utilizing the resource.  The final necessary piece of code is to use the wait command.  This tells the computer to wait until all the background tasks are completed before moving on to the next line of code.  The wait command comes in handy to ensure the number of background tasks submitted does not exceed the number of processor cores.  If this happens you will likely overwhelm the processor with too many tasks.  To prevent this the idea is to simultaneously submit a number of tasks equal to the number of cores and then use the wait command to wait for the jobs to finish before submitting more tasks.  Here is some example code I put together to utilize all 8 cores of a node when running 1,000 permutations of a program:


This will run a script with the following commands:

Of course, this could further be parallelized to run permutations simultaneously on different compute nodes as well to further speed up run time. Hope this is a helpful example to help you fully utilize compute nodes and speed up your processing time.

Useful UNIX Aliases

Here are some useful UNIX aliases that help save the fingers from too much typing.  Feel free to borrow as many as you like.

Getting Set Up on a UNIX Cluster

Okay, you have been granted access to a UNIX cluster to work on a project, but have no idea how to get started with things.  No problem.  This post is designed to clue you in on the essentials so you can get up and running in no time.


SSH Clinet
This is the first thing you will need.  SSH clients are programs that enable your computer to connect remotely to the UNIX cluster.  SSH stands for Secure SHell, which is a network protocol for communicating data from one computer to another.  Unquestionably, the most widely used SSH Client for Windows operating system users is PuTTY.  This is a free program and easy to set up.  Download putty.exe and move it to a spot where you can easily access it.  Double clicking on it will bring up a security warning.  Select run and PuTTY will open.  For the Host Name insert the IP address for the cluster (ex: computer.university.edu).  In general, this is all you really need to do before pressing the Open button.  You can choose to save this as a session for easy access in the future, but just accept all the defaults for now.  PuTTY will then connect to the computer at the IP address you specified.  If you are a Mac user, this is much easier to do.  Just go to the pre-loaded Mac terminal and type "ssh username@computer.university.edu", where computer.university.edu is the IP address for the computer you want to connect to.  The first time you connect you will get a warning about a certificate.  Just choose the option to proceed.  Next, you will usually be asked by the remote computer for a username and password.  Input what was given to you by the cluster administrator.  You will not see the password as you type it in.  After your credentials have been accepted, you will have access to the remote computer.  Congratulations, you are now connected!  For Windows users, there are also other flavors of PuTTY that can be used.  I like PuTTYTray for the added options built into it as well as MTPuTTY which allows for multiple tabs to be opened simultaneously.

File Transfer Application
Next, you will need a way to transfer files and scripts from your computer to the remote computer (cluster) and vice versa.  WinSCP is an excellent free program to use from a Windows operating system.  Unfortunately, Macs really don't have an equivalent (if you have any suggestions, let me know).  Once WinSCP is downloaded and setup, open the program and select New.  Fill in the Host name and User name.  I usually leave the password blank (in which case I will be asked to manually provide it later), but this is up to your discretion.  From there you can log in.  After your session has been authenticated, you will be greeted with a window having two major panes.  The left side is your computer and the right side is the remote computer.  Just drag files from one window to the other to transfer from one computer to another.  You now have a way to transfer files to and from the remote computer!

X Server
Some programs require an X server to process output from X11 sessions.  This essentially allows you to open an interactive window on your computer in which you can interact with the remote computer.  Some programs such as R, Python, and Java have modules that can interact in this fashion.  Should you find out you need an X server, I would highly recommend Xming for Windows users.  It is free and just needs to be open in the background while you have the terminal open.  Mac will have this functionality built in.  First though, you need to set this up before logging into the cluster.  On Macs, simply add the -X option when connecting through the terminal (ex: ssh -X username@computer.university.edu).  For Windows, open PuTTY and in the left menu select Connection > SSH > X11 and click on the box enabling X11 forwarding.  Then go back to the top of the menu to Session and log in as before.  To make sure this is working, type the command "xeyes".  Do you see two eyballs looking at you?  If so, it works!

Text Editor
If you like Vi or Emacs, this is not for you.  For everyone else, there are a wide variety of text editors that are helpful in writing your code in a variety of programming languages.  I prefer Notepad++ for Windows.  For Macs, however, I am still searching for a worthy Notepad++ alternative.


Well, I hope this was helpful in getting you up and running on a UNIX computing environment.  From personal experience, I know the learning curve can be steep.  Best wishes as you learn to navigate your way around on a remote cluster.  If you have any helpful suggestions, I highly encourage you to post a comment below.

Install SRA Toolkit on UNIX OS

SRA Toolkit is the set of programatic tools Sequence Read Archive (SRA) provides for accessing information in their .sra files.  The toolkit can be downloaded for a variety of platforms here, but in this post I will focus on installing the toolkit on a UNIX platform.

The first step is to download to the directory you want the program saved in and extract the program:

Next, you need to configure the toolkit by going into the bin directory and opening the java configuration script (make sure you have X11 forwarding enabled):

For the most part you can accept defaults when you walk through the setup, just make note of the directory you specified during configuration.  You should be all setup!  Check out other blog entries for SRA here.

Create 2bit Database for Faster BLAT Searching

Included with the BLAT source code is a useful utility that converts FASTA references to 2bit format.  The utility, faToTwoBit, works simply by providing the original reference .fa file to be converted to a .2bit file.  Here is the syntax to run the program:


A few simple options are also included:
-noMask
-stripVersion
-ignoreDups

For more details on these options, type faToTwoBit at the command line.  Hope this 2bit reference database speeds up your BLAT searches!

How To Import Command Line Arguments into a Python Script

It is often handy to be able to feed an argument from the UNIX command line to a Python script.  This is very simple to do with the sys package installed.  Below is an example of feeding three arguments (chr, start, end) from the command line into Python to be used as variables in a script.

In the script.py file:

At the command line: