Flat files to Databases: For better Speed, Integration and Sharing

In an ordinary dictionary, a word can be sought in two different ways:

1. Use the index and locate your word of choice, or,
2. Start with "A" and keep going, one by one until you get there.

Obviously, the first way is the smart way. But, when it comes to a real-time organised data, most of us prefer the second way by choosing to read (line by line) and write into the flat files; even when the task is repetitive. Relational Database Management System (RDBMS), such as SQL (can be MySQL, OpenSQL, SQLite, PostgreSQL etc) are well suited for such tasks, yet they are under-implemented by many of the bioinformaticians. 

The use of databases can be intimidating without the formal training of database management, but this overall picture has changed to a great extent with the advent of Object Oriented Mapping (ORM) frameworks. ORMs provide language-specific, object-oriented access to databases. It brings the database handling in the comfort zone of object oriented language of user's choice. For example, in order to access a sequence in the database, one can execute,
this will issue an SQL command at the back-end which is,

Another hectic of database handling is the server setup and maintenance issues. This can be reduced to a great extent by adopting a flexible, server-less and fully embed-able RDBMS, such as SQLite or BerkeleyDB. The rest of the operations of creating, modifying and deleting databases, tables and rows are well taken care by ORMs. The most popular ORMs include SQLObject (Python), DBIx::Class (Perl) and Hybernate (Java), which are open source and easily implementable.

In the modern era, the data is integrated from multiple sources and in complex fashions. This vast amount of information needs to be extracted in a reasonable way and channeled into the manageable and biologically meaningful outcomes in respect to medical applications. The database system offers efficient handling of the data and at the same time it delivers easy access via web applications, making it more suitable for scientific data sharing.
    

Python Modules: Expand your reach in Bioinformatics! (Part#2: Hybrid Programming)

A very classic question in bioinformatics is, which programming language is the best for a bioinformatician? Discussions like this never end with a conclusive answer. Interestingly, people find this question as a piece of cake and jump at it with whatever they have in their hands! The result is, you get a nice rainbow of choices, right from "C" to "PHP"!

Each programming language has its own perks and disadvantages. For example, "C" has an incredible speed in execution but it is equally code-intensive in writing even a simple program. Python and Perl on other hand make the same program code-lite but with a mediocre speed of execution. Apart from these performance issues, every language is blessed with a varying degree of third party modules/libraries.

Python has provided interfaces to many system calls and libraries, giving direct access to the shell of an operating system (modules like os, subprocess let you call unix commands directly from the python terminal). Python is also usable as an extension language for applications written in other languages that need easy-to-use scripting or automation interfaces. More than 15 coding projects have started to establish a platform where python can be integrated with other programming languages like C, Java, Perl, PHP, R, Fortran etc.

These hybrid platforms are either available as python modules which can easily be imported, like we do for general (numpy, maths, random etc) modules or accessible from a parent language (i.e. Jython, python implemented in Java)

A detailed list of these hybrid platforms are accessible from here.

Some fascinating platforms I couldn't resist to mention here are:

• elmer: Elmer allows developers to write code in Python and execute it in C or Tcl.
• JPype: JPype allows python programs to fully access java class libraries.
• PyPerlish: Allows the usage of perl idioms in python.
• RPy: Simple and efficient access to R from python.

It is interesting to note that every platform mentioned here was somebody's dream. Since shifting to a new language might deliver new exciting features but at the same time it takes away what you loved the most about the previous one. Following are the words from the creator of PyPerlish,

"I've used perl for several years, and been very impressed with its ease of use. When you need to do something new, chances are there is an idiom which lets you do it in a few keystrokes. I didn't want to lose that in moving to python. Somehow I wanted to get the benefits of perl's idioms with the robust scalability and maintainability of python. So the idea is to emulate perl idioms, no matter how we implement the python code under the covers."       -- Harry George

 

Python Modules: Expand your reach in Bioinformatics! (Part#1: Phyloinformatics)

Python is getting increasingly popular among bioinformaticians, not just due to its simplistic yet powerful structure but also due to the third party modules which are imparting domain specific added advantages. This series is dedicated towards compilation of such modules, specific to each domain.

In this section, the most popular python modules in phyloinformatics are introduced.
  

• E.T.E a python Environment for phylogenetic Tree Exploration

"ETE is a python programming toolkit that assists in the automated manipulation, analysis and visualization of phylogenetic and other type of trees. It provides a wide range of tree handling methods, node annotation features, programmatic access to the phylomeDB database, and automatic orthology and paralogy prediction methods. In addition, an interactive tree visualization program, as well as a highly customizable tree drawing engine, is included."    -- ETE website

ETE examples: Tree with multiple sequence alignment, Bar chart and Pie chart

ETE is very well documented and pretty easy to use. Traversing the tree in different directions (from root to leaves, and leaves to root), manipulating (adding/removing) custom features to an individual node of tree, creating graphics rich plots, integrating multiple sequence alignments, evolutionary hypothesis testing and much more can be easily achieved with this module.

• DendroPy

"DendroPy is a Python library for phylogenetic computing. It provides classes and functions for the simulation, processing, and manipulation of phylogenetic trees and character matrices, and supports the reading and writing of phylogenetic data in a range of formats, such as NEXUS, NEWICK, NeXML, Phylip, FASTA etc. Application scripts for performing some useful phylogenetic operations, such as data conversion and tree posterior distribution summarization, are also distributed and installed as part of the libary. DendroPy can thus function as a stand-alone library, a component of more complex multi-library phyloinformatic pipelines, or as a scripting “glue” that assembles and drives such pipelines."    -- DendroPy Website

Compared to ETE, DendroPy is more focused towards computational aspect of phyloinformatics, which includes simulation of birth-death process trees, population genetic trees, coalescent tress etc. DendroPy also allows calculation of general tree statistics like tree length, node age, probability under the coalescent model, tree distances etc. Unlike ETE, DendroPy also supports variety of character matrices (DNA, RNA, Proteins, any continuous/ discrete-value data), but at the same time DendroPy allows Phylogenetic Independent Contrasts (PIC) analysis (as described by Felsenstein 1985) given a tree and continuous character matrix.
CAUTION: The current release (3.2.0) do not support python 3.0

• Bio.Phylo (BioPython)

Bio.Phylo module was introduced in BioPython 1.54. This module is simplistic but covers all the necessary functionalities including, parsing/writing various tree formats, displaying trees in different color palettes, searching and traversing methods, clade/node specific information extraction/modification etc. Bio.Phylo also allows integration of third-party application like PAML for phylogenetic analysis by maximum likelihood. Likewise, BioPython wrappers are also available for PhyML, RAxML and FastTree.

All the three modules are well documented and irreplaceable given their functional disparity. There are also couple of other modules which are highly function specific and might just fit into your requirement list. These are,

• P4: a python package for phylogenetics
• For maximum likelihood and Bayesian phylogenetic analysis on molecular sequences

• Mavric: a python toolkit for phylogenetics
• Fully interactive editing of phylogenetic trees

Your eLearning Directory of Bioinformatics Essentials!

In the previous section we discussed "What makes you a Good Bioinformatician?" and concluded that one requires to take a slice from the three major pillars of bioinformatics i.e. biology, mathematics and computer sciences. It is required to gain expertise in any of the two, and learn the fundamentals of the third one. We also discussed that there are range of options that one can opt for, from amongst the three. Here, I have hand-picked a list of online resources that can provide a PRIMER on some of these options.

Note: This page will update time to time! You might want to save this page for the current listings or Bookmark it for further updates in this catalogue.


Biology                                                 

• Fundamentals of Biology
• http://ocw.mit.edu/courses/biology/7-01sc-fundamentals-of-biology-fall-2011
• http://webcast.berkeley.edu/playlist#c,d,Biology,CF8E59B3C769FB01 

• Biochemistry
• http://nptel.ac.in/video.php?subjectId=102105034  

• Molecular Biology
• http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewPodcast?id=354820440 

• Principles of Evolution, Ecology and Behavior
• http://oyc.yale.edu/ecology-and-evolutionary-biology/eeb-122 

Mathematics                                        

• Numerical Methods
• http://numericalmethods.eng.usf.edu 

• Linear Algebra
• http://numericalmethods.eng.usf.edu 

• Statistics
• https://www.coursera.org/course/stats1 

• Differential Equations
• http://ocw.mit.edu/courses/mathematics/18-03sc-differential-equations-fall-2011 

• Networks: Theory and Applications
• http://open.umich.edu/education/si/si508/fall2008 

• Dynamic Systems and Chaos 
• http://www.math.tamu.edu/~mpilant/math614

Computer Sciences                                

• Introduction to Computer Science and Programming (Python related)
• http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-00sc-introduction-to-computer-science-and-programming-spring-2011
• https://code.google.com/edu/languages/google-python-class 

•  Data Structures (Java related)
• http://webcast.berkeley.edu/playlist#c,d,Computer_Science,63AE13B304CE443E

• Building Dynamic Websites 
• http://cs75.tv/2010/fall

• Introduction to Databases
• http://www.db-class.org/course 

• Computer Graphics
• http://itunes.apple.com/us/itunes-u/computer-graphics-fall-2009/id457893733 
• http://nptel.iitm.ac.in/video.php?subjectId=106106090 

• Machine Learning
• https://www.coursera.org/course/ml?from_restricted_preview=1&course_id=152

Cross Platform Courses                            

• Computational Molecular biology
• http://biochem218.stanford.ed

• Computational Biology
• http://www.algorithm.cs.sunysb.edu/computationalbiology

• Computational Seminars
• http://videolectures.net/Top/Computer_Science/Bioinformatics

This catalogue is intentionally kept smaller, in spite of dozens of other publicly available courses. The intention is NOT to trigger a "Decision Fatigue", a very common fallacy arose when mind is subjected to multiple options (as very well described by Rolf Debelli), but to facilitate the selection.

If you want to give a try :-) visit:
http://www.coursera.org
http://ocw.mit.edu/courses
http://nptel.iitm.ac.in/
http://videolectures.net/


If you find any alternative and better course for the given topics, please care to use the following comment box. I will update this list whenever possible.

What Makes You a Good Bioinformatician?

Does knowing contemporary data analysis tools and keeping a set of persuasive Unix commands at your fingertips make you a good bioinformatician? Or, is it that your stunning programming skills that transform you into a big tool production house make you a good bioinformatician?

I, recently, performed an extensive literature mining and considered the opinions of pioneering researchers and scientists in the field of bioinformatics on this topic.

Disclaimer:

1. If you are an established researcher and looking from a professional angle then this article is not for you (Although we are going to discuss the foundation here). I would be happy to redirect you to "10 Steps to Success in Bioinformatics"
2. If you are not from the category 1, then you are on the right page.

While mining I came across various raging discussions regarding, what bioinformaticians are meant to do? Some people were adamant on the notion that experts from various fields should collaborate together instead of one person doing all the things poorly! It is crucial to understand that a bioinformatician is different from mathematician/statistician and he/she is also different from a biologist in regards to their ability to establish an interface for research and also channel the requirements and findings from both the directions.

A disparity between a biologist, a mathematician and a bioinformatician!

One thing I observed is that the definition of bioinformatics is not universally agreed upon. Generally speaking, we define it as the development and application of information technology for understanding the problems in biology, most commonly molecular biology (but increasingly in other areas of biology as well). As such, it deals with methods for systematic storage, retrieval and analysis of biological data including but not limited to nucleic acid and protein sequences, their structures, functional roles, regulatory pathways and biophysical interactions.

Some researchers interpret bioinformatics more narrowly and include only those issues dealing with high throughput sequencing data along with the  integration and the analysis of OMICs. A few construe bioinformatics more broadly and include all areas of computational biology, including population modelling and molecular simulations. Others construe bioinformatics as the development of necessary tools and databases for the analysis of biological data to draw meaningful conclusions.

In spite of the diversity in the bioinformatics domains, the foundation of which can be summed up in three major pillars of education:

1. Biology
2. Mathematics and Statistics
3. Computer Science

It is ideal but at the same time ambitious to be able to grasp all the three pillars of bioinformatics. A good bioinformatician would possess a thorough knowledge of any two pillars but AT THE SAME time should also be aware of the fundamentals of the third one.

If you have trouble deciding, then let your INTEREST pick your two pillars, based on which you will develop a bioinformatics flavor, right from sequence analysis to molecular dynamic simulations.

These pillars in themselves are very diverse if we dive into them. For example, when we talk about biology, it comprises of molecular biology, biochemistry, evolution, ecology, behaviour etc. Similarly, in mathematics and statistics, you have a range of options right form probability theory to analytical combinatorics. In soft skills you have options from programming languages to learning systems.

You can browse through some of these streams and learn more about them here at "Your eLearning Directory of Bioinformatics essentials".

One becomes a bioinformatics domain expert based on what slice he/she takes from the stack. You may chose molecular biology, differentials and integrals, and Matlab or equivalent, and be a good bioinformatician in the domain of molecular dynamics and simulations. Some people might argue that this is a very naive theory and one would require to know much more than that. Well naturally there is no upper limit. If you keep your knowledge stagnant for a long, you will soon realise that you have stopped growing. The pace at which advancements are happening in the field of biology and technology, it has become vital to upgrade our knowledge, skills and techniques but at the same time it is absolutely essential to stick to your stream, because success comes from experiences and experiences comes from dedicated efforts.

Your integrity is like a tip of an iceberg. The same tip, from outside may appear smaller than a frozen lake but when it comes to summer, the tip is all that remains since it has a strong foundation which no ordinary summer can melt.

Extracting Specific Fasta record/s from a Multi-fasta File

While dealing with multi-fasta files, it is often required to extract few fasta sequences which contain the keyword/s of interest. One fast way to do this, is by awk.

For example:

Input file: hg19_genome.fa

    >Chr1
    ATCTGCTGCTCGGGCTGCTCTAT...
    >Chr2
    GTACGTCGTAGGACATGCATCG...
    >MT1
    TACGATCGATCAGCTCAGCATC...
    >MT2
    CGCCATGGATCAGCTACATGTA...

We would like to extract the sequence for Chr2 from hg19_genome.fa. Use the following command:


$ awk 'BEGIN {RS=">"} /Chr2/ {print ">"$0}' hg19_genome.fa


Output:

    >Chr2
    GTACGTCGTAGGACATGCATCG...

Note that, the search keyword (here 'Chr2') doesn't have to be an exact match. If you use 'MT' instead, you will get the third and fourth entry, since 'MT' is a sub-string of the third and fourth fasta record.

Now lets break down the command so that we don't have to mug it up or we could mold it and use it in variety of other places.

• awk -- This is the main command (Or more of a very powerful programming language)

• '' -- We write every bit of awk code inside these single quotes

• BEGIN -- This tells the awk to execute the immediately following code in curly brackets at the beginning.
• {RS=">"} -- Record separator  (If we look at the file, we can observe every sequence starts with a ">" sign. This helps us to separate two fasta records)
• /Chr2/ -- keyword to search in the entire record
• {print ">"$0} -- Here $0 is the current record (From "Chr2" to the entire sequence till next ">"). We added ">" at the beginning just to get the standard identitifer which is not included in $0.
• hg19_genome.fa -- This is the input multi-fasta file that we have used.

Now,
Suppose we are interested in more that one keyword then two possibilities arise:


You want BOTH the keywords present,
$ awk 'BEGIN {RS=">"} /Chr2/ && /MT/ {print ">"$0}' hg19_genome.fa

You want EITHER of the keyword present,
$ awk 'BEGIN {RS=">"} /Chr2|MT/ {print ">"$0}' hg19_genome.fa



Note: If you are using Windows, you can download and install 'gwak' and use similar command. In zsh shell you might need to use an escape character for | (pipe).


I am sure many of you might have different flavors to do the same. If you think it is worth sharing then the comment box is all yours.

Happy Coding !!