Supplementary information: A two-step site and mRNA-level model for predicting microRNA targets

Takay Saito and Pål Sætrom

Table of contents

1. Supplementary Methods
   1. Data partition into positive (down-regulated) and negative genes
   2. Data retrieval for benchmarks
   3. Performance evaluation on independent dataset
2. Supplementary Results
   1. Analysis of target site level training data: Selection of seed types influences the coverage of potential target sites
   2. Prediction of targets with optimal distance sites: Optimal distance is effectively incorporated in our SVM prediction
   3. Site accessibility and sequence conservation: Explicit usage of site accessibility and sequence conservation do not improve SVM prediction
   4. Classification vs. Regression: Support vector classification (SVC) performs better than support vector regression (SVR)
   5. Dataset selection: Different parameters for constructing training or test data do not affect SVM prediction performance
3. Supplementary Tables
   1. Table S1. Target site level features.
   2. Table S2. Analysis of target site level feature influence.
   3. Table S3. mRNA level features.
   4. Table S4. Analysis of mRNA level feature influence.
   5. Table S5. Benchmarks on the Linsley dataset.
   6. Table S6. mRNA/siRNA sequences from five microarray experiments.
   7. Table S7. Parameters used for the Needle software.
   8. Table S8. Benchmarks on the Selbach dataset.
   9. Table S9. Benchmarks on the Baek dataset.
   10. Table S10. Benchmarks on the Linsley dataset – Positive records defined by log ratio -0.1.
   11. Table S11. Benchmarks on the Linsley dataset – Positive records defined by log ratio -0.3.
   12. Table S12. Benchmarks on the Selbach dataset – Positive records defined by log ratio -0.1.
   13. Table S13. Benchmarks on the Selbach dataset – Positive records defined by log ratio -0.3.
   14. Table S14. Benchmarks on the Baek dataset – Positive records defined by log ratio -0.1.
   15. Table S15. Benchmarks on the Baek dataset – Positive records defined by log ratio -0.3.
4. Supplementary Figures
   1. Figure S1. 10-fold cross-validation of target site level classifiers trained on three microarray datasets.
   2. Figure S2. Density plot for the number of potential target sites divided by 3’UTR length.
   3. Figure S3. Comparison between two SVM trainings with or without explicit addition of negative records.
   4. Figure S4. 10-fold cross-validation of mRNA level classifiers trained on three microarray datasets.
   5. Figure S5. Three method-specific benchmarks of 8 different algorithms on the Linsley dataset.
   6. Figure S6. siRNA benchmarks on the Jackson dataset.
   7. Figure S7. miRNA benchmarks on the Lim dataset.
   8. Figure S8. siRNA benchmarks on the Birmingham dataset.
   9. Figure S9. miRNA benchmarks on the Grimson dataset.
   10. Figure S10. Targets with optimal distance sites show higher average discriminant value than other three target types.
   11. Figure S11. Adding site accessibility and sequence conservation features did not affect target prediction accuracy.
   12. Figure S12. SVC performs better than SVR at both target site and mRNA levels.
   13. Figure S13. SVM performance is similar between the training set with up-regulated genes and the one without up-regulated genes.
   14. Figure S14. Two-step SVM retains the performance when trained with proteomics data.
   15. Figure S15. Benchmarks on the Selbach dataset.
   16. Figure S16. Benchmarks on the Baek dataset.
   17. Figure S17. Benchmarks on the Linsley dataset with positive record threshold by log ratio -0.1.
   18. Figure S18. Benchmarks on the Linley dataset with positive record threshold by log ratio -0.3.
   19. Figure S19. Benchmarks on the Selbach dataset with positive record threshold by log ratio -0.1.
   20. Figure S20. Benchmarks on the Selbach dataset with positive record threshold by log ratio -0.3.
   21. Figure S21. Benchmarks on the Baek dataset with positive record threshold by log ratio -0.1.
   22. Figure S22. Benchmarks on the Baek dataset with positive record threshold by log ratio -0.3.
5. References

Supplementary Methods

Data partition into positive (down-regulated) and negative genes

Down-regulated (positive) genes of GEO datasets were defined as records with p-value < 0.001 and log intensity ratio < -0.3 to obtain strongly down-regulated genes. Down-regulated genes for the Birmingham dataset were obtained directly from the published data [1].

Some microarray records contained different results for the same combination of miRNA/siRNA and mRNA; for example, there were five records of a miRNA on the same mRNA, and four of them were regarded as down-regulated but one of them were unaffected. In this case, all records that belonged to this combination were disregarded to enhance the quality.

Negative records of GEO datasets consisted of both unaffected and up-regulated genes. Unaffected genes were defined as records with p-value > 0.3 and log intensity ratio between -0.3 and 0.3, whereas up-regulated genes were defined as records with p-value < 0.001 and log intensity ratio > 0.3. Negative records for the Birmingham dataset were defined as all RefSeq transcripts except the positive records.

As a final filter, all miRNA:mRNA pairs that had either very high (>1.25) or very low (<-1.5) log intensity values in the control experiments were eliminated to reduce potential noise form extreme expression values.

All the positive records were used in the training dataset, whereas negative records were randomly selected to meet a total record number of 20000, unless otherwise specified. A maximum total record number of 20000 was used due to computational time and memory space constraints.

For the test data, positive records were selected as for the training data, whereas all the remaining records were used as negative records.

For the Linsley dataset [2], down-regulated genes were defined as records with p-value < 0.001 and log intensity ratio < -0.2.

For the Selbach and Baek datasets, down-regulated genes (positive) were defined as records with log ratio values < -0.2, whereas unaffected genes (negative) were defined as log ratio values > 0. The records that do not fit in these criteria were not used in this study.

Data retrieval for benchmarks

Prediction data were downloaded from the Segal Lab site (https://genie.weizmann.ac.il/pubs/mir07/) for PITA All and PITA Top, the TargetScanHuman 5.1 site (http://www.targetscan.org) for TargetScan and TargetScan with conserved genes, the MicroCosm Targets Version 5 site (http://www.ebi.ac.uk/enright-srv/microcosm) for miRanda, the miRDB site (http://mirdb.org) for mirTarget2, and the PicTar WEB INTERFACE site (https://pictar.mdc-berlin.de/) for PicTar. Several methods provide only target site level scores for download, and, in that case, summation methods showed in the web sites or in the original publications were applied to obtain scores for the mRNA level. For siRNA predictions, PITA and TargetScan executable programs were downloaded and used to calculate prediction scores.

Performance evaluation on independent dataset

Six dataset – “ROC with All genes”, “ROC10*n”, “ROC with 7mer + Conservation”, and three method-specific datasets for TargetScan, miRanda, and PicTar – were created for evaluation. The “ROC with All genes” dataset was comprised of all the records from the microarray dataset.

“ROC10*n” was based on the standard ROC50 [3] benchmark. The ROC50 benchmark score is the area under the ROC curve until 50 negatives are found. Consequently, the “ROC10*n” benchmark datasets consisted of all the true positives found in the microarray experiments and the 10*n first negatives found by each method benchmarked.

The “ROC with 7mer + Conservation” dataset was comprised of genes with conserved 8mer or 7mer-m8 or 7mer-A1 sites. Sites were defined as conserved if the average phyloP 44 scores in the seed region were > 0.9.

The three method-specific datasets for TargetScan, miRanda, and PicTar were generated by only using the records from the microarray experiments that had corresponding predictions for each method.

Supplementary Results

Analysis of target site level training data: Selection of seed types influences the coverage of potential target sites

Current methods for miRNA target prediction base their analyses on identifying so-called seed sites in 3’ UTR. The definition of seed types varies from one prediction algorithm to another [4,5,6], however, and this heavily influences the coverage of potential target sites. Lower coverage means that predictions may lack many true positive sites, whereas higher coverage may result in poor accuracy because of too many false positive sites.

Based on four different microarray datasets - Birmingham [1], Jackson [7], Lim [8], and Grimson [5], we identified a set of 823 genes that were down-regulated in response to artificial miRNA or siRNA transfection, or up-regulated in response to miRNA inhibition (see Supplementary Methods). This positive set of genes constituted 0.14% of the 571570 potential miRNA:gene interactions assayed in the microarray experiments. Searching for seed sites among the positive genes identified that 347 and 510 genes contained stringent and all types of seed sites. Thus, basing miRNA target predictions on stringent seeds or all seed types would cover 42% or 62% of the positive genes. The corresponding numbers for the negative set of 570747 genes were 128254 and 31277. Consequently, the predictive accuracy, as measured by the positive predictive value (PPV = True positive predictions / All positive predictions), was almost three times greater for stringent seeds (PPV = 1.1%) than for all seed types (PPV = 0.40%). Even so, this showed that predicting miRNA targets purely based on identifying stringent seed sites would give too many false positive predictions to be viable in practice.

Several approaches have shown that other target-related features besides the seed type and additional data such as miRNA and mRNA expression profiles can improve predictive accuracy [9,10]. We therefore chose to include all seed types to achieve high coverage in the training data and then improve the accuracy through two-step SVM training.

Prediction of targets with optimal distance sites: Optimal distance is effectively incorporated in our SVM prediction

We were interested to see how well the features of optimal distances were incorporated into our algorithm. Our model uses two mRNA features that represent optimal distances: (i) the number of sites that have other neighboring sites within 13-46 nt and (ii) the number of sites that have other neighboring sites within 17-25 nt. Furthermore, one target site level feature, “distance to the next target” could be effective to recognize the targets with optimal distance. However, these features showed little contribution when the influence of both target site and mRNA level features was analyzed (Tables S2 and S4). Therefore, we applied a different approach as to see the difference of prediction power between the target genes with optimal distance and other target genes.

First, we categorized all potential targets into four different target types, which were defined by the number of potential target sites, seed types, and existence of optimal distance sites within the gene’s 3’ UTR. “Single non-stringent” and “Single stringent” were targets that had only one “non-stringent” and “stringent” seed site, whereas “Multiple without optimal distance” and “Multiple with optimal distance” were targets that had multiple sites where none or at least one pair of sites were within . Then, we checked the mRNA level classifier’s distributions of discriminant values for these four target types. The resulting box plots (Fig. S10) show that for each target type, the mRNA-level SVM clearly separates between the positive (down-regulated) and negative (non-down-regulated) targets. Moreover, targets with multiple sites had higher discriminant values than targets with single stringent and single non-stringent sites, and the targets with optimally spaced sites had the highest discriminant values. Thus, site distance contributed positively to the SVM’s predictions.

Site accessibility and sequence conservation: Explicit usage of site accessibility and sequence conservation do not improve SVM prediction

Although target site accessibility is important for both miRNA and siRNA targeting [11,12], it is unclear how useful computational predictions of target site accessibility are for genome-wide miRNA target analyses [13,6]. Similarly, even though many target prediction tools rely on conserved sequences, as these indicate conserved function and targeting, sequences with little or no evidence of conservation can still be functional target sites [14]. Moreover, calculating target site accessibility and conservation features demand much higher computational power and resources than calculating the other features. Therefore, we constructed classifiers without using the site accessibility and sequence conservation features to evaluate their importance in our SVM model.

To evaluate the classifier without of site accessibility and sequence conservation features, we compared the ROC scores of three mRNA level classifiers that used the same mRNA level features, but three different target site classifiers; that is, classifiers (i) without site accessibility and conservational information, (ii) with site accessibility, and (iii) with conservational information. The classifiers were then tested on the independent dataset.

All the results from the 10-fold cross-validation and independent tests showed very little difference among the three classifiers (Fig. S11). Although theoretically sound, the computationally intensive accessibility predictions could therefore, in practice, be eliminated from the SVM classifier. In addition, although many miRNA target sites appear to be well conserved, the small influence of the conservation-related features on SVM performance indicate that many non-conserved miRNA targets exist. This is consistent with the results of the benchmarks of the 8 existing miRNA target prediction methods. Consequently, the proportion of non-conserved miRNA target sites in the set of all potential target sites seems too large to be ignored. Therefore, we opted to train our SVM classifiers without using site accessibility and sequence conservation.

Classification vs. Regression: Support vector classification (SVC) performs better than support vector regression (SVR)

The microarray data used in our experiments measured actual changes in gene expression in response to miRNA or siRNA over-expression. Consequently, even though we treated miRNA target prediction as a classification problem, target prediction could also be solved as a regression problem. To test whether treating target prediction as a regression problem would further improve our predictions, we used the PyML library to train SVR models to fit the log ratio values from the Jackson, Lim, and Grimson microarray experiments. We did not use the Birmingham dataset because pre-processed log ratio values were not available. The SVR models used the same two step approach and the same features as our previous SVC models. The same optimization approach as we used for the SVC models showed that a linear kernel with default parameters gave the best predictions at both the target site and mRNA level. This model selection often resulted in over-fitting or no convergence, however, hence requiring more analysis time than SVC to find an optimal kernel. Moreover, the SVR target site and mRNA level models showed lower prediction performance at the 10-fold cross-validation and independent data set than the SVC models did (Fig. S12). Thus, when the input is based on features derived from the target sequence, classification models are as good or better at predicting miRNA or siRNA target gene expression fold changes as are regression models.

Dataset selection: Different parameters for constructing training or test data do not affect SVM prediction performance

It is important to verify that our SVM approach is still effective when we select different positive (down-regulated) and negative (non-down-regulated) datasets for both training and test because it proves that our SVM approach is not optimized only for a certain parameter criteria of data selection. We generated seven different datasets; one for training and six for independent test sets to further investigate SVM prediction power with different data selection parameters.

Two major steps applied to create datasets in our study were; (i) all microarray datasets were separated into three groups; down-regulated, unaffected and up-regulated, depending of the log ratio values, and (ii) positive data were selected from down-regulated genes, whereas negative records were from both unaffected and up-regulated genes. However, some miRNAs are known to potentially up-regulate genes. The mechanism of this up-regulation is unknown, and the features of up-regulated target sites/genes are possibly quite different from those of miRNAs involved in down-regulation. To check the influence of involving these up-regulated genes in our SVM training sets, we constructed a dataset without up-regulated genes, and trained and tested it on the independent datasets. The 10-fold cross-validation ROC plots for both target site and mRNA level classifiers trained without up-regulated genes (Fig. S13 A, C) were almost identical to those trained with up-regulated genes (Fig. S13 E, G). The prediction power was also retained when the classifiers were tested on the Linsley dataset (Fig. S13 B, D) and compared with those trained with up-regulated genes (Fig. S13 F, H).

Moreover, we generated six independent subsets for one transcriptomics (Linsley) and two proteomics (Selbach and Baek) experiments. Two different positive record datasets defined by log-ratio values <-0.1 and <-0.3 were created for each experiment, resulting in six independent datasets altogether. We performed benchmarks on these datasets and compared with the dataset defined by the log-ratio values <-0.2, which had been used as the default value for all the other benchmarks. The number of positive records detected by all of the eight algorithms in the benchmarks was higher in the log ratio <-0.1 sets; Linsley (Table S10), Selback (Table S12), and Baek (Table S14), and lower in the log ratio <-0.3 sets; Linsley (Table S11), Selback (Table S13), and Baek (Table S15), as expected. The ROC scores from the benchmarks show that SVM outperforms other algorithms in most cases regardless of different parameter criteria of data selection. We also analyzed the ROC curves of the benchmarks to check the trends among the log ratio <-0.2 sets; Linsley (Fig. 5, S5), Selback (Fig. 6, S15) and Baek (Fig.6, S16), the log ratio <-0.1 sets; Linsley (Fig. S17), Selback (Fig. S19) and Baek (Fig. S21), and the log ratio <-0.3 sets; Linsley (Fig. S18), Selback (Fig. S20) and Baek (Fig. S22). All ROC curves showed very similar trends among different positive record sets. More importantly, our SVM approach retained its prediction power even when it was tested on the log ratio <-0.3 and <-0.1 sets.

These results indicate that our SVM prediction power is not affected when trained with or without up-regulated genes, or tested on strongly down-regulated pairs (log ratio <-0.3) as well as moderately down-regulated pairs (log ratio <-0.1).

Supplementary Tables

Table S1. Target site level features.

^aVector size used in SVM training. ^bSite accessibility features. ○: Yes, blank: No. ^cSequence conservation features. ○: Yes, Blank: No. ^dFeatures included in the final version of our classifier. ○: included features, blank: features excluded during optimization steps.

---

Table S2. Analysis of target site level feature influence.

^aROC diff shows the difference of ROC scores between the classifier trained with all the features (0.7050) and classifiers trained with one feature removed.

---

Table S3. mRNA level features.

^aVector size used in SVM training. ^bFeatures included in the final version of our classifier. ○: included features, blank: features excluded during optimization steps.

---

Table S4. Analysis of mRNA level feature influence.

^aROC diff shows the difference of ROC scores between the classifier trained with all the features (0.7997) and classifiers trained with one feature removed.

---

Table S5. Benchmarks on the Linsley dataset.

^aPositive and Negative show the total number of positive (down-regulated) and negative (unaffected) genes present within the six benchmarks (Method “All”) and among the predictions of each method on the six benchmarks. ^bROC is the method’s ROC score on the specific benchmark.

---

Table S6. mRNA/siRNA sequences from five microarray experiments.

Jackson (GSE5814), Lim (GSE2075), Grimson (GSE8501), and Birmingham (E-MEXP-668) were used for training, and Linsley (GSE6838) was used for the independent test. Two datasets, Selbach and Baek, from mass spectrometry experiments were used as both training datasets (for the proteomics-based classifiers) and independent test sets (for the mRNA-based classifiers).

^aMAPK14-193 has three different sample groups; (i) GSM133702 and GSM134468, (ii) GSM134488, and (iii) GSM134491. We assigned three different unique IDs to these groups, therefore they were treated as different sequences internally in our model.

---

Table S7. Parameters used for the Needle software.

The scoring matrix was generated based on the matrix used in the miRanda algorithm [15].

---

Table S8. Benchmarks on the Selbach dataset.

^aPositive and Negative show the total number of positive (down-regulated) and negative (unaffected) genes present within the six benchmarks (Method “All”) and among the predictions of each method on the six benchmarks. ^bROC is the method’s ROC score on the specific benchmark.

---

Table S9. Benchmarks on the Baek dataset.

^aPositive and Negative show the total number of positive (down-regulated) and negative (unaffected) genes present within the six benchmarks (Method “All”) and among the predictions of each method on the six benchmarks. ^bROC is the method’s ROC score on the specific benchmark.

---

Table S10. Benchmarks on the Linsley dataset – Positive records defined by log ratio -0.1.

^aPositive and Negative show the total number of positive (down-regulated) and negative (unaffected) genes present within the six benchmarks (Method “All”) and among the predictions of each method on the six benchmarks. ^bROC is the method’s ROC score on the specific benchmark.

---

Table S11. Benchmarks on the Linsley dataset – Positive records defined by log ratio -0.3.

^aPositive and Negative show the total number of positive (down-regulated) and negative (unaffected) genes present within the six benchmarks (Method “All”) and among the predictions of each method on the six benchmarks. ^bROC is the method’s ROC score on the specific benchmark.

---

Table S12. Benchmarks on the Selbach dataset – Positive records defined by log ratio -0.1.

^aPositive and Negative show the total number of positive (down-regulated) and negative (unaffected) genes present within the six benchmarks (Method “All”) and among the predictions of each method on the six benchmarks. ^bROC is the method’s ROC score on the specific benchmark.

---

Table S13. Benchmarks on the Selbach dataset – Positive records defined by log ratio -0.3.

^aPositive and Negative show the total number of positive (down-regulated) and negative (unaffected) genes present within the six benchmarks (Method “All”) and among the predictions of each method on the six benchmarks. ^bROC is the method’s ROC score on the specific benchmark.

---

Table S14. Benchmarks on the Baek dataset – Positive records defined by log ratio -0.1.

^aPositive and Negative show the total number of positive (down-regulated) and negative (unaffected) genes present within the six benchmarks (Method “All”) and among the predictions of each method on the six benchmarks. ^bROC is the method’s ROC score on the specific benchmark.

---

Table S15. Benchmarks on the Baek dataset – Positive records defined by log ratio -0.3.

^aPositive and Negative show the total number of positive (down-regulated) and negative (unaffected) genes present within the six benchmarks (Method “All”) and among the predictions of each method on the six benchmarks. ^bROC is the method’s ROC score on the specific benchmark.

---

Supplementary Figures

 

Figure S1. 10-fold cross-validation of target site level classifiers trained on three microarray datasets.

The ROC graphs show the classification performance of four classifiers trained with three microarray datasets at the target site level. The curve is the average ROC curve of the ten individual cross-validation test results; error bars show standard errors. Dotted lines illustrate random prediction. Avg. AUC is the average of the area under the curve (AUC; ROC score) of the ten individual cross-validation test results.
(A) Trained with the Birmingham, Lim and Grimson datasets.
(B) Trained with the Jackson, Birmingham and Grimson datasets.
(C) Trained with the Jackson, Lim and Grimson datasets.
(D) Trained with the Jackson, Birmingham and Lim datasets.

---

 

Figure S2. Density plot for the number of potential target sites divided by 3’UTR length.

It shows the density distribution of the values calculated by the number of miRNA potential sites divided by the 3’ UTR length. The red line indicates the distribution of down-regulated genes whereas the blue line is for the non-affected genes. The difference between two density distributions is very significant (Kolmogorov-Smirnov test, p-value: 3.862e-11).

---

 

Figure S3. Comparison between two SVM trainings with or without explicit addition of negative records.

Two box plots show the number of potential target sites vs. discriminant values from mRNA level training. The left panel shows the distributions of discriminant values by the number of potential target sites for the classifier trained without any explicit additional of negative records. The right panel shows the classifier trained with data enriched with 1000 randomly selected non-target genes with more than 7 target sites. Records with the number of target sites >25 and/or negative discriminant values were disregarded. Whereas the left box plot shows that discriminant values for negative (non-downregulated) genes generally increase with increasing number of potential target sites, the right box plot indicates that the discriminant values for negative genes are relatively stable.

---

 

Figure S4. 10-fold cross-validation of mRNA level classifiers trained on three microarray datasets.

The ROC graphs show the classification performance of four classifiers trained with three different combinations of microarray datasets at the mRNA level. The curve is the average ROC curve of the ten individual cross-validation test results; error bars show standard errors. Dotted lines illustrate a random prediction. Avg. AUC is the average of the area under the curve (AUC; ROC score) of the ten individual cross-validation test results.
(A) Trained with the Birmingham, Lim and Grimson datasets.
(B) Trained with the Jackson, Birmingham and Grimson datasets.
(C) Trained with the Jackson, Lim and Grimson datasets.
(D) Trained with the Jackson, Birmingham and Lim datasets.

---

 

Figure S5. Three method-specific benchmarks of 8 different algorithms on the Linsley dataset.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms; SVM, PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2, and PicTar, on the Linsley dataset. Dotted lines illustrate random prediction. The values of the area under the ROC curve (AUC) are shown in the legend box. The benchmarks used for the evaluation were (A) ROC with TargetScan dataset, (B) ROC with miRanda dataset, and (C) ROC with PicTar dataset.

---

 

Figure S6. siRNA benchmarks on the Jackson dataset.

Receiver operating characteristic (ROC) graphs show the performance of 3 different target prediction algorithms – SVM not trained on Jackson dataset (SVM), PITA, and TargetScan (TargetS) – on the Jackson dataset. The SVM classifier was trained with the Lim, Birmingham and Grimson datasets. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. The benchmarks used for the evaluation were (A) ROC with All genes, (B) ROC10*n, and (C) ROC with 7mer + Conservation.

---

 

Figure S7. miRNA benchmarks on the Lim dataset.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM not trained on Lim dataset (SVM), PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Lim dataset. The SVM classifier was trained with the Jackon, Birmingham and Grimson datasets. Dotted lines illustrate random prediction. The values of the area under the ROC curve (AUC) are shown in the legend box. The benchmarks used for the evaluation were (A) ROC with TargetScan dataset, (B) ROC with miRanda dataset, (C) ROC with PicTar dataset, (D) ROC with TargetScan dataset, (E) ROC with miRanda dataset, and (F) ROC with PicTar dataset.

---

 

Figure S8. siRNA benchmarks on the Birmingham dataset.

Receiver operating characteristic (ROC) graphs show the performances of 3 different target prediction algorithms – SVM not trained on Birmingham dataset (SVM), PITA, and TargetScan (TargetS) – on the Birmingham dataset. The SVM classifier was trained with the Jackson, Lim and Grimon datasets. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. The benchmarks used for the evaluation were (A) ROC with All genes, (B) ROC10*n, and (C) ROC with 7mer + Conservation.

---

 

Figure S9. miRNA benchmarks on the Grimson dataset.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM not trained on Grimson dataset (SVM), PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Grimson dataset. The SVM classifier was trained with the Jackon, Lim, and Birmingham datasets. Dotted lines illustrate random prediction. The values of the area under the ROC curve (AUC) are shown in the legend box. The benchmarks used for the evaluation were (A) ROC with TargetScan dataset, (B) ROC with miRanda dataset, (C) ROC with PicTar dataset, (D) ROC with TargetScan dataset, (E) ROC with miRanda dataset, and (F) ROC with PicTar dataset.

---

 

Figure S10. Targets with optimal distance sites show higher average discriminant value than other three target types.

The boxplot shows the comparison of mRNA level discriminant values among four different target types. The result shows our SVM approach gives scores as the preference: “multiple with optimal distance” > “multiple without optimal distance” > “single stringent” > “single non-stringent”.

---

 

Figure S11. Adding site accessibility and sequence conservation features did not affect target prediction accuracy.

Receiver operating characteristic (ROC) graphs show the classification performance of three mRNA level classifiers. These classifiers had the same mRNA level features, but their discriminant values were generated by three different target site classifiers: (i) neither site accessibility nor sequence conservation used, (ii) with site accessibility and, (iii) with conservational information. Panels A-C show the three classifiers’ 10-fold cross-validation performance. The curve is the average ROC curve of the ten individual cross-validation test results; error bars show standard errors; Avg. AUC is the average of the area under the curve (AUC; ROC score) of the ten individual cross-validation test results. Panels D-F show the three classifiers’ performance on the independent (Linsley) dataset; ROC graphs were plotted by only considering the genes with prediction scores in the Linsley dataset. Dotted lines illustrate random prediction.
(A) 10-fold cross-validation of the classifier with neither site accessibility nor sequence conservation used
(B) 10-fold cross-validation of the classifier with site accessibility
(C) 10-fold cross-validation of the classifier with sequence conservation
(D) Classifier with neither site accessibility nor conservational information tested on the independent dataset
(E) Classifier with site accessibility tested on the independent dataset
(F) Classifier with sequence conservation tested on the independent dataset

---

 

Figure S12. SVC performs better than SVR at both target site and mRNA levels.

The ROC graphs show the prediction performance of SVM regression (SVR) and SVM classification (SVC). Panels A, C, E, and G show the models’ 10-fold cross-validation performance. The curve is the average ROC curve of the ten individual cross-validation test results; error bars show standard errors; Avg. AUC is the average of the area under the curve (AUC; ROC score) of the ten individual cross-validation test results. Panels B, D, F, and H show the models’ performance on the independent (Linsley) dataset; ROC graphs were plotted by only considering the genes with SVC and SVR prediction scores in the Linsley dataset. Dotted lines illustrate random prediction. SVR and SVC show very similar performance, but SVC shows slightly better performance than SVR at both target site and mRNA levels.
(A) 10-fold cross-validation of SVR target site level
(B) Target site level SVR classifier tested on the independent dataset
(C) 10-fold cross-validation of SVR mRNA level
(D) mRNA level SVR classifier tested on the independent dataset
(E) 10-fold cross-validation of SVC target site level
(F) Target site level SVC classifier tested on the independent dataset
(G) 10-fold cross-validation of SVC mRNA level
(H) mRNA level SVC classifier tested on the independent dataset

---

 

Figure S13. SVM performance is similar between the training set with up-regulated genes and the one without up-regulated genes.

The ROC graphs show the prediction performance of two SVM classifiers trained either with or without up-regulated genes. Panels A, C, E, and G show the models’ 10-fold cross-validation performance. The curve is the average ROC curve of the ten individual cross-validation test results; error bars show standard errors; Avg. AUC is the average of the area under the curve (AUC; ROC score) of the ten individual cross-validation test results. Panels B, D, F, and H show the models’ performance on the independent (Linsley) dataset; ROC graphs were plotted by only considering the genes with SVM prediction scores in the Linsley dataset. Dotted lines illustrate random prediction. Two classifiers show almost identical performances in terms of both ROC sores and ROC curves.
(A) 10-fold cross-validation of SVM without up-regulated genes at target site level
(B) Target site level SVM without up-regulated genes tested on the independent dataset
(C) 10-fold cross-validation of SVM without up-regulated genes at mRNA level
(D) mRNA level SVM without up-regulated genes tested on the independent dataset
(E) 10-fold cross-validation of SVM with up-regulated genes at target site level
(F) Target site level SVM with up-regulated genes tested on the independent dataset
(G) 10-fold cross-validation of SVM with up-regulated genes at mRNA level
(H) mRNA level SVM with up-regulated genes tested on the independent dataset

---

 

Figure S14. Two-step SVM retains the performance when trained with proteomics data.

(A) SVM classifier was trained by the Selbach and Baek datasets. The definitions of the error bars, AUC, true positive and false positive rates were the same as described in Figure 2. (B) Both proteomics and transcriptomics classifiers were tested on the Linsley dataset. Proteomics data contains the Selbach and Baek datasets, whereas Transcriptomics contains the Jackson, Lim, Birmingham, and Grimson datasets. The ROC scores were shown in the legend box.

---

 

Figure S15. Benchmarks on the Selbach dataset.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM, PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Selbach dataset. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. The benchmarks used for the evaluation were, (A) ROC with TargetScan dataset, (B) ROC with miRanda dataset, and (C) ROC with PicTar dataset.

---

 

Figure S16. Benchmarks on the Baek dataset.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM, PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Baek dataset. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. The benchmarks used for the evaluation were (A) ROC with TargetScan dataset, (B) ROC with miRanda dataset, and (C) ROC with PicTar dataset.

---

 

Figure S17. Benchmarks on the Linsley dataset with positive record threshold by log ratio -0.1.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM, PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Linsley dataset. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. Positive records (down-regulated) were selected by log ratio value -0.1 instead of the default value -0.2. The benchmarks used for the evaluation were (A) ROC with All genes, (B) ROC10*n, (C) ROC with 7mer + Conservation, (D) ROC with TargetScan dataset, (E) ROC with miRanda dataset, and (F) ROC with PicTar dataset.

---

 

Figure S18. Benchmarks on the Linley dataset with positive record threshold by log ratio -0.3.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM, PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Linsley dataset. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. Positive records (down-regulated) were selected by log ratio value -0.3 instead of the default value -0.2. The benchmarks used for the evaluation were (A) ROC with All genes, (B) ROC10*n, (C) ROC with 7mer + Conservation, (D) ROC with TargetScan dataset, (E) ROC with miRanda dataset, and (F) ROC with PicTar dataset.

---

 

Figure S19. Benchmarks on the Selbach dataset with positive record threshold by log ratio -0.1.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM, PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Selbach dataset. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. Positive records (down-regulated) were selected by log ratio value -0.1 instead of the default value -0.2. The benchmarks used for the evaluation were (A) ROC with All genes, (B) ROC10*n, (C) ROC with 7mer + Conservation, (D) ROC with TargetScan dataset, (E) ROC with miRanda dataset, and (F) ROC with PicTar dataset.

---

 

Figure S20. Benchmarks on the Selbach dataset with positive record threshold by log ratio -0.3.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM, PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Selbach dataset. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. Positive records (down-regulated) were selected by log ratio value -0.3 instead of the default value -0.2. The benchmarks used for the evaluation were (A) ROC with All genes, (B) ROC10*n, (C) ROC with 7mer + Conservation, (D) ROC with TargetScan dataset, (E) ROC with miRanda dataset, and (F) ROC with PicTar dataset.

---

 

Figure S21. Benchmarks on the Baek dataset with positive record threshold by log ratio -0.1.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM, PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Baek dataset. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. Positive records (down-regulated) were selected by log ratio value -0.1 instead of the default value -0.2. The benchmarks used for the evaluation were (A) ROC with All genes, (B) ROC10*n, (C) ROC with 7mer + Conservation, (D) ROC with TargetScan dataset, (E) ROC with miRanda dataset, and (F) ROC with PicTar dataset.

---

 

Figure S22. Benchmarks on the Baek dataset with positive record threshold by log ratio -0.3.

Receiver operating characteristic (ROC) graphs show the performances of 8 different target prediction algorithms – SVM, PITA All (PITA), PITA Top (PITA_top), TargetScan (TargetS), TargetScan with conserved genes (TargetS_c), MicroCosm miRanda (miRanda), mirTarget2 and PicTar – on the Baek dataset. Dotted lines illustrate random prediction. The ROC scores are shown in the legend box. Positive records (down-regulated) were selected by log ratio value -0.3 instead of the default value -0.2. The benchmarks used for the evaluation were (A) ROC with All genes, (B) ROC10*n, (C) ROC with 7mer + Conservation, (D) ROC with TargetScan dataset, (E) ROC with miRanda dataset, and (F) ROC with PicTar dataset.

---

References

1. Birmingham A, Anderson EM, Reynolds A, Ilsley-Tyree D, Leake D, Fedorov Y, et al. 3′ UTR seed matches, but not overall identity, are associated with RNAi off-targets. Nature Methods 2006;3:199–204. https://doi.org/10.1038/nmeth854.
2. Linsley PS, Schelter J, Burchard J, Kibukawa M, Martin MM, Bartz SR, et al. Transcripts targeted by the MicroRNA-16 family cooperatively regulate cell cycle progression. Molecular and Cellular Biology 2007;27:2240–52. https://doi.org/10.1128/mcb.02005-06.
3. Gribskov M, Robinson NL. Use of receiver operating characteristic (ROC) analysis to evaluate sequence matching. Computers & Chemistry 1996;20:25–33. https://doi.org/10.1016/s0097-8485(96)80004-0.
4. Gaidatzis D, Nimwegen E van, Hausser J, Zavolan M. Inference of miRNA targets using evolutionary conservation and pathway analysis. BMC Bioinformatics 2007;8:69. https://doi.org/10.1186/1471-2105-8-69.
5. Grimson A, Farh KK-H, Johnston WK, Garrett-Engele P, Lim LP, Bartel DP. MicroRNA targeting specificity in mammals: determinants beyond seed pairing. Molecular Cell 2007;27:91–105. https://doi.org/10.1016/j.molcel.2007.06.017.
6. Hammell M, Long D, Zhang L, Lee A, Carmack CS, Han M, et al. mirWIP: microRNA target prediction based on microRNA-containing ribonucleoprotein-enriched transcripts. Nature Methods 2008;5:813–9. https://doi.org/10.1038/nmeth.1247.
7. Jackson AL, Burchard J, Schelter J, Chau BN, Cleary M, Lim L, et al. Widespread siRNA "off-target" transcript silencing mediated by seed region sequence complementarity. RNA 2006;12:1179–87. https://doi.org/10.1261/rna.25706.
8. Lim LP, Lau NC, Garrett-Engele P, Grimson A, Schelter JM, Castle J, et al. Microarray analysis shows that some microRNAs downregulate large numbers of target mRNAs. Nature 2005;433:769–73. https://doi.org/10.1038/nature03315.
9. Huang JC, Babak T, Corson TW, Chua G, Khan S, Gallie BL, et al. Using expression profiling data to identify human microRNA targets. Nature Methods 2007;4:1045–9. https://doi.org/10.1038/nmeth1130.
10. Sales G, Coppe A, Bicciato S, Bortoluzzi S, Romualdi C. Impact of probe annotation on the integration of miRNA-mRNA expression profiles for miRNA target detection. Nucleic Acids Research 2010;38:e97–e97. https://doi.org/10.1093/nar/gkp1239.
11. Kertesz M, Iovino N, Unnerstall U, Gaul U, Segal E. The role of site accessibility in microRNA target recognition. Nature Genetics 2007;39:1278–84. https://doi.org/10.1038/ng2135.
12. Long D, Lee R, Williams P, Chan CY, Ambros V, Ding Y. Potent effect of target structure on microRNA function. Nature Structural & Molecular Biology 2007;14:287–94. https://doi.org/10.1038/nsmb1226.
13. Baek D, Villén J, Shin C, Camargo FD, Gygi SP, Bartel DP. The impact of microRNAs on protein output. Nature 2008;455:64–71. https://doi.org/10.1038/nature07242.
14. Sethupathy P, Megraw M, Hatzigeorgiou AG. A guide through present computational approaches for the identification of mammalian microRNA targets. Nature Methods 2006;3:881–6. https://doi.org/10.1038/nmeth954.
15. Enright AJ, John B, Gaul U, Tuschl T, Sander C, Marks DS. MicroRNA targets in Drosophila. Genome Biology 2003;5:R1. https://doi.org/10.1186/gb-2003-5-1-r1.

---

Leave a comment

---