Results are in for the Second sbv IMPROVER Challenge on Species Translation

Species translation involves searching for biological mechanisms of action (MoA) that are common to different species, in this case, humans and rats. In addition to the interesting scientific question of commonality and its origin, results showing conserved MoA could be very useful in improving the relevancy of animal models in the treatment of diseases and drug development. After all, we know that some drugs work well in many species, and others do not. Why?

During the summer of 2013, 28 teams consisting of 51 scientists responded to the sbv IMPROVER (systems biology verification: Industrial Methodology for PROcess Verification in Research) Species Translation Challenge, a collaborative initiative by IBM Research (Yorktown Heights, NY) and Philip Morris International R&D (Neuchâtel, Switzerland) (funded by the latter).The challenge was designed to use crowd-sourcing for verification of scientific information and results. Results were announced at the symposium held at the Grand Resort Lagonissi in Athens, Greece, October 28–31, 2013.

Data generation

Experimental data sets were obtained by challenging around 50 samples of the same cells from rat and human with around 50 different compounds including cytokines, interleukins, interferons, and several others. The output was of gene expression and phosphorylation activity for each library member as measured by Affymetrix (Santa Clara, CA) microarrays (the human Affy chip was HUMAN GENOME U133 PLUS 2.0 ARRAY (Product Code 900467). The rat Affy chip was Rat Genome 230 2.0 Array (Product Code 900507).

The data files were made available to the 28 teams that signed up for the challenge and expressed interest in making predictions of biological activity (translation and signaling) based upon the Affymetrix patterns. Details on the best-performing teams will be made available at

The first subchallenge was an intraspecies challenge for rat. The second evaluated commonality of response between rat and human. Here the 50-sample rat and human sets were divided in half, with one called the training set and the other the validation set. The third looked for prediction of pathway perturbation in rats, and the fourth for common activation of signaling pathways, as reported by phosphorylation activity in rats and humans.


  1. Intraspecies prediction of protein phosphorylation: The rat data were studied to determine if the change in gene expression induced by the challenge library correlated with protein phosphorylation. Several groups found statistically significant correlations.
    Team AMG, with participants from the University of California at Santa Barbara, University of Groningen (The Netherlands), and Rutgers University (Piscataway, NJ) were joint best performers in this study, and outright best performers in studies 2 and 3 below.
  2. For the interspecies study, half the human set was used as a training set to find apparent correlations between species. Once this was done, the correlations were verified with data from the other half. Again, they found statistically significant correlations between species.
  3. Interspecies pathway perturbation prediction: This study sought to explore whether responsive gene sets and related processes in humans can be predicted based upon the corresponding data in rats. Some predictions were statistically significant, while others were not. Better correlations were observed for signaling at 5 and 25 minutes than in transcription, which was measured after several hours, since it is a longer process.
  4. Species Specific Network Inference searched signaling pathways (phosphorylation) that are similarly affected by the stimulants: Five teams found a few correlates.


The significance of the Second svb IMPROVER Challenge is probably best summarized by the comments of the organizers and participants.

Keynote speakers

Prof. George Kollias, Biomedical Sciences Research Center ‘Alexander Fleming’, Vari, Greece:

“Among the issues in our understanding of species translatability is the need to bridge the gap between informatics and real-world biology. sbv IMPROVER continues to provide valuable insights in this area, bringing together scientists that understand innovative techniques such as machine learning and neural network with molecular and cell biologists. In doing so the project addresses both the technical and biological challenges inherent in making predictions between species.”

Dr. Joaquín Dopazo, Head of the Computational Genomics Department, Principe Felipe Research Center, Valencia, Spain:

“We need new systems to effectively and usefully deal with the complexity of high-throughput data. sbv IMPROVER represents one such system with a number of potential applications, from identifying predictive biomarkers of disease to using network analysis techniques to help understand drug action mechanisms.”


Sahand Hormoz, Postdoctoral Fellow, Kavli Institute for Theoretical Physics, University of California, Santa Barbara (as part of team AMG, joint winner of the Species Translation Challenge subchallenge 1, winner of subchallenge 2, and winner of subchallenge 3):

“It is clearly very difficult to make direct predictions of human biology based on rodent models. However, we can see that relationships that hold within rodent gene sets, identified through our computations, are themselves useful in helping us understand what happens in humans.”

Prof. Michael Biehl, Johann Bernoulli Institute for Mathematics and Computer Science, University of Groningen, The Netherlands (as part of team AMG, joint winner of the Species Translation Challenge subchallenge 1, winner of subchallenge 2, and winner of subchallenge 3):

“The Species Translation Challenge invited us to ask: Are rats and humans closer than we think? Based on our work on the second subchallenge, it seems that there may well be a closer relationship between rat and human than what is currently acknowledged. This is an exciting discovery, with potential implications across a number of fields, and one we look forward to exploring further.”

Prof. Gyan Bhanot, Dept. of Molecular Biology & Biochemistry, Rutgers University, Piscataway, NJ (as part of team AMG, joint winner of the Species Translation Challenge subchallenge 1, winner of subchallenge 2, and winner of subchallenge 3):

“sbv IMPROVER has given us the opportunity to put together an interdisciplinary team and use tailored computational techniques on this unique, exceptionally high-quality data set. We are delighted our approach has proved so successful and that we have been able to contribute to this important initiative on species translatability. It is crucial that we continue to take steps to improve modeling capabilities as a complement to in vivo systems, and we would like to see how we could improve our predictions now that the Challenge is closed and the full data set has been unblinded.”

Dr. Adi Laurentiu Tarca, Director, Bioinformatics and Computational Biology Unit, Perinatology Research Branch (NCIHD/NIH), Wayne State University, Detroit, MI (joint winner of the Species Translation Challenge subchallenge 1, and overall winner of the first sbv IMPROVER Challenge, the Diagnostic Signature Challenge):

“The second sbv IMPROVER Challenge was in many ways more complex than the first, with more data to consider, in rather challenging settings. We were glad to see that the approach we refined for the previous challenge proved once more to be robust. Once again, sbv IMPROVER has proven itself to be unique in allowing us to identify which predictive techniques work best in specific, distinct settings, and at the same time giving us a perspective on the limitations of rodent models in the study of human biology.”


Dr. Manuel Peitsch, VP, Biological Systems Research, Philip Morris International R&D, Neuchâtel, Switzerland:

“The results from each subchallenge may have far-reaching implications for scientists in many fields that use animal models to understand more about human biological systems. It is clear that from a computational biologist’s perspective, rodents and humans are indeed closer than we think. We have been delighted with the quality of submissions to the Species Translation Challenge. If the goal is to make meaningful predictions about human biology from data derived from rodent models, it is extremely important to carefully consider the specific characteristics of the experimental system and the biological mechanisms that may be impacted in order to identify the most adequate model systems for a given question.”

Dr. Pablo Meyer, Computational Biology Center, IBM Research, Yorktown Heights, NY:

“Using Big Data and analytics, the Species Translation Challenge has given us many unique and valuable insights and we are now beginning to see exactly what is and is not translatable between rodent models and humans. True to the spirit of the initiative, a diverse set of approaches have been put forward by participants, so we are able to see without any biases or limitations what computational techniques work best when using different data sets to predict translatable biological processes.”

Robert L. Stevenson, Ph.D., is Editor, American Laboratory; e-mail: [email protected].