Using deep learning and similarity-based transfer to better predict CRISPR-Cas9 off-target effects

We conducted our off-target prediction experiments on seven well-known public CRISPR-Cas9 datasets:

The dataset's name, the CRISPR-Cas9 technique used, the number of gRNAs, the number of samples in both the minority and majority classes, and the class imbalance ratio for each of these datasets are summarized in Table 1. The datasets are publicly available in our GitHub repository at: https://github.com/dagrate/transferlearning_offtargets↗.

To encode sgRNA-DNA sequences, we adopted the encoded scheme introduced by Lin et al. [37] that integrates mismatches, insertions, deletions, and matches to preserve the mutual information between on-target and off-target sites. This scheme represents each sgRNA-DNA sequence pair using seven-bit one-hot encoding: a five-bit channel (A, C, G, T, _) and a two-bit direction channel used to indicate the insertion/indel or mismatch directions. Consequently, a 7×23 matrix (where 23 represents the sequence length, including the 3-bp PAM adjacent to the 20 bases) allows for considering three types of base mismatches, missing bases (RNA bulge or insertion), and extra bases (DNA bulge or deletion) in off-target sites. An overview of this encoding technique, with examples including an insertion (RNA bulge), a mismatch, and a deletion (DNA bulge) is presented in Fig 2.

We specifically selected three datasets, CD33, CIRCLE, and SITE, as potential source datasets due to a large number of positive samples in their minority class (i.e. active off-targets) and the lowest class imbalance among the seven datasets considered, as indicated in Table 1. This selection enhances the robustness of our analyses during the training process.

We used a standard train-test split from the scikit-learn [44] implementation with shuffling, a ratio of 0.3, and equal stratification of the classes. The stratification was employed to ensure that the class distribution within the training and testing datasets accurately reflects the original class proportions before the train-test split. By maintaining relative class ratios, stratification mitigates biases and enhances the reliability of model evaluation to address the issue of data imbalance [45].

An overview of the classification models used in our experiments is provided in Supporting Information. The two following Python libraries were used for model implementation:

We present hereinafter the methodology used for hypertuning the traditional ML classifiers and DL network models considered in our study.

Both traditional ML classifiers and DL networks require overfit monitoring during their training [46]. In our experiments, we employed two essential callbacks to mitigate overfitting in our deep learning models. First, we used the Reduce Learning Rate on Plateau callback dynamic approach that automatically adjusts the learning rate during training based on the model's performance. If the validation loss reaches a plateau (i.e., stops improving), the learning rate is reduced, allowing the model to converge more effectively. Second, we used the Early Stopping callbacks with a patience of 8 epochs and a minimum delta parameter of 0.02 on the validation loss. This callback function monitors the model's performance during training. If the validation loss fails to improve significantly (i.e. it is less than the specified delta) for a certain number of consecutive epochs (determined by the patience value), the training is halted early to prevent overfitting. These combined strategies help ensure that our DL models generalize well to unseen data [48].

In this section, we delve into the background and detailed explanation of the proposed approach for similarity-based transfer learning off-target prediction in CRISPR-Cas9. A notable concern in transfer learning is the risk of negative transfer [49], which arises when the source dataset is inappropriately selected. In this case, a model pre-trained on a larger but dissimilar dataset may perform worse than a model trained from scratch with randomly initialized weights. This issue emphasizes the importance of quantifying the similarity between source and target datasets to ensure the success of transfer learning. To address this challenge for off-target CRISPR-Cas9 data, we evaluate the similarity between the two involved datasets (i.e. a potential source and the given target datasets) using three different metrics: cosine similarity (here, we use its distance form), and Euclidean and Manhattan distances:

Here, 퐚 and 퐛 are vectors of length K = 7L (i.e. L is the sequence length, which is equal to 23 in our study) representing the encoded sgRNA-DNA sequence pairs in the source and target datasets, respectively. Each encoded matrix of size 7×L (see Fig 2) is flattened into a vector of length 7L in order to calculate the distance between sgRNA-DNA sequence pairs from the source and target datasets. It is worth noting that the number of rows, i.e. 7, in the matrix corresponds to the number of bits used to encode a given sgRNA-DNA sequence pair, including a five-bit character channel (i.e. A, G, C, T, and _) and a two-bit direction channel (to indicate if they appear in the sgRNA or DNA sequence). The five-bit channel encodes the presence-absence of the four nucleotides and insertions/deletions, whereas the direction channel identifies the location of the mismatched nucleotides or insertions/deletions, if any (see Fig 2 for three examples of such a sequence encoding). We should highlight that 7 is the minimum number of rows one can use to encode the presence-absence of the A, G, C, T, and _ characters in two sequences, including their location information, without any information loss.

It is important to note that cosine, Euclidean, and Manhattan distances compared in our study have different strengths and disadvantages. For example, cosine distance is sensitive to direction in high-dimensional sparse spaces. It is used when the magnitude of the vectors is not important. This is the case of binary data such as our sgRNA-DNA sequence pairs encoded as 7L binary vectors. With cosine distance the difference in values is not fully taken into account, but this is not a disadvantage in our settings since this difference can be either 0 or 1 with our sequence encoding. Theoretically, cosine distance should work better in cases when the encoded sgRNA-DNA sequence pairs have more matching nucleotides, than in cases with frequent insertions, deletions, and mismatches, since better matching sequences would lead to more sparse binary spaces. Euclidean distance is certainly the most natural distance choice as it is directly computed from the cartesian coordinates of the points using the Pythagorean theorem. Euclidean distance is sensitive to magnitude but, as specified above, this is not of importance in our binary settings. This distance usually works well with low-dimensional data. Manhattan distance is known for its robustness to outliers. It is less intuitive than Euclidean distance but works well with discrete and binary components as it considers the veritable path that can be taken within values of those components. Thus, in our settings, we could expect that Manhattan distance treats equally well binary encoded matching nucleotides, insertions, deletions, and mismatches.

Clearly, the most intuitive way of computing the distance between the source and the target datasets is the following:

Such an exhaustive approach would work in practice when both the source and target datasets are small (<50,0 elements each). However, real-world CRISPR-Cas 9 datasets often contain hundreds of thousands elements (see Table 1) each of which must be encoded in a numerical vector format beforehand in order to perform machine learning experiments. For example, the execution of the above-mentioned exhaustive approach applied to the CIRCLE (used as source data) and SITE (used as target data) datasets would require several weeks of intensive computation on a modern PC computer. Moreover, in many practical situations the target datasets is so small that deep learning experiments, which usually necessitate a huge amount of data, cannot be performed on it (e.g. see [25,28] for examples of such off-target datasets used in CRISPR-Cas9). Finally, to perform our Monte Carlo simulations to determine the most appropriate distance measure as well as the most suitable ML and DL models in the context of CRISPR-Cas9 off-target transfer learning, we need several hundred real-world datasets of realistic size.

Thus, we decided to perform our Monte Carlo simulations with bootstrap replicates of the considered benchmark target datasets. A bootstrap replicate of a given target dataset per similarity experiment was generated and the average simulation results were then reported. The size of each bootstrapped target dataset was 250, while the number of iterations (i.e. comparisons of each target element with the source elements used to assess the distance between the source and target datasets) was set to 5,000. As we determined experimentally, with this number of iterations (denoted by n_itr in Algorithm 1 below), the average distance between the source and target datasets found by the random search converges towards the distance provided by an exhaustive search algorithm. In the large majority of cases, this number of iterations was sufficient to achieve two-digit precision after the decimal point during the distance calculation. This allowed us to obtain reliable results without having to run the computations for several days. Regarding the size of 250 of the bootstrapped targets, it was selected to see how the proposed methodology would work with some high-quality CRISPR-Cas9 datasets of realistically small size. For example, in a recent Nature Communication paper, Ham et al. [50] used the TevSpCas9 dataset with 279 samples as well as the SpCas9 dataset with 303 samples to conduct their transfer learning experiments with a novel machine learning architecture (crisprHAL) meant to improve sgRNA activity prediction.

Furthermore, we made sure that the sample ratios between the majority and minority classes in each bootstrapped target datasets were equivalent to those in the complete target dataset.

Algorithm 1 outlines the key steps of our similarity-based transfer learning approach. The algorithm takes as input N potential source datasets (representing labeled data), denoted as DS:={D1,....,DN}, a given bootstrapped target dataset (representing unlabeled data), denoted as D풯, and a distance measure d (cosine, Euclidean, or Manhattan distance, in our case).

The set of the encoded vector representations of the source dataset D_i (i=1,...,N) is denoted as {퐗퐢}, and of the bootstrapped target dataset D풯b as {퐗퐭}, each of them of dimension 7L.

Algorithm 1 Similarity-Based Transfer Learning for CRISPR-Cas9 Off-Target Prediction

During the first phase of Algorithm 1, refereed to as Similarity Analysis phase, we systematically evaluate the distance between each potential source dataset Di∈DN and the bootstrapped target dataset D풯b to determine the most suitable source dataset for a given target. Cosine, Euclidean, or Manhattan distance between the following vectors is then computed: (1) 퐚m - a vector of length 7L representing the m^th encoded sgRNA-DNA sequence pair in the bootstrapped target dataset D풯b, where m∈{1,…,|D풯b|} and (2) 퐛n - a randomly selected vector of length 7L representing the n^th encoded sgRNA-DNA sequence pair in the i^th source dataset 풟i, where n∈{1,…,|풟i|}. For each dataset 풟i∈DS, we iterate over every element in D풯b computing the current distance value, d_current, between each element in D풯b (i.e. 퐚m vectors) and a random subset of elements in 풟i (i.e. 퐛n vectors). The number of elements in this random subset equals n_itr. Thus, a unique subset of the current source dataset 풟i is generated through random sampling with replacement. If the computed current distance d_current is smaller than the previously stored minimum distance dist_min, we update dist_min to d_current. This process is repeated over n_itr iterations, ensuring that dist_min consistently represents a close match between the target element and the source dataset 풟i. We then construct the vector 퐝 of dimension |D풯b| that includes the minimum distance values for all samples in D풯b. The mean of this vector is used to determine the optimal source candidate dataset for transfer learning (i.e. the set that exhibits the highest similarity with a given target).

The second phase of Algorithm 1, referred to as Transfer Learning phase, implements the transfer learning process using the optimal source dataset(selected at Phase 1). D S O p t

Similarity analysis evaluates the closeness of a given target dataset to a potential source dataset. This analysis is crucial for determining the appropriateness of employing transfer learning for the data at hand. In our study, cosine, Euclidean, and Manhattan distances were used to quantify the degree of similarity between datasets. Each of these metrics has its own strengths and weaknesses. Thus, cosine distance is preferable for high-dimensional and text data, Euclidean distance provides an intuitive measure of similarity for normalized data, whereas Manhattan distance is beneficial for datasets encompassing outliers or non-linear relationships [51].

To conduct our simulation study, we selected the CD33, CIRCLE, and SITE datasets as candidate sources datasets, as they offer a sufficient number of minority class samples (i.e. off-targets), thereby increasing the robustness of our approach. In our simulations, the size of the subset of the source dataset compared to the given target dataset was set to 5,000 (i.e. n_itr = 5,000 in Algorithm 1), whereas the size of the bootstrapped target datasets was set to 250 (the class imbalance ratio in the bootstrapped datasets was equivalent to that of the corresponding complete dataset; see Table 2). We observed that the distance estimations usually converged when the number of iterations, i.e. n_itr, was between 4000 and 5000.

Table 3 reports the average estimated similarities between the three source datasets (CD33, CIRCLE, and SITE) and the seven bootstrapped target datasets (CD33_BS, CIRCLE_BS, SITE_BS, Tasi_GUIDE_BS, Listgarten_GUIDE_BS, Kleinstiver_GUIDE_BS, and Hmg_BS) calculated using cosine, Euclidean, and Manhattan distances. Each similarity estimate appearing in Table 3 was computed as 1–NormalizedAverageDistance between the selected source and bootstrapped target dataset using cosine, Euclidean, or Manhattan distance. The exact procedure used in our experiments to compute the similarity values is as follows:

In addition, Fig 4 presents the corresponding bar plot diagrams for each of the three considered distance measures.

The results presented in Table 3 and Fig 4 demonstrate that the cosine metric provides the highest overall similarity values between source and target datasets, compared to Manhattan and Euclidean distances, whereas Euclidean distance corresponds to the lowest similarities. However, Manhattan and Euclidean metrics provide the largest differences between the similarities corresponding to the recommended and non-recommended source datasets. This means that Euclidean and Manhattan distances, being sensitive to absolute differences, better highlight stark dissimilarity between datasets. However, our findings suggest that overall magnitude is less important for transferability than feature direction, which is captured by cosine distance (see Sect 2.3). Such a result should be related to the binary nature of the encoded sgRNA-DNA sequence pairs since in our settings even the vectors with different locations of matches, mismatches, and indels are sparse enough to have at least 50% of matching 0 values, thus leading to the lowest cosine similarity values that are slightly higher than 0.5. In the case of Euclidean and Manhattan distances, the lowest normalized distance values can be close to 1, leading to the corresponding similarity values that are slightly higher than 0. Obviously, the maximum similarity value is limited by 1 (in case of a perfect sequence match) for all three metrics considered.

As expected, the most appropriate source datasets for the bootstrapped target datasets CD33_BS, CIRCLE_BS, and SITE_BS were their complete source counterparts CD33, CIRCLE, and SITE, respectively. The corresponding cosine similarity values for these source-target pairs were 0.9585, 0.8650, and 0.8841, respectively.

Interestingly, for the other target datasets, i.e. Tasi_GUIDE_BS, Listgarten_GUIDE_BS, Kleinstiver_GUIDE_BS, and Hmg_BS, the choice of the most suitable source dataset depends on the selected distance/similarity measure. For example, for Tasi_GUIDE_BS, the CIRCLE dataset stands out as the most suitable source, achieving the highest similarity across all three distance metrics (cosine, Euclidean, and Manhattan). However, for Listgarten_GUIDE_BS, the CIRCLE dataset is the most suitable source according to cosine similarity, but both Manhattan and Euclidean metrics indicate CD33 as the most suitable source dataset for it. In the case of Kleinstiver_GUIDE_BS, the CIRCLE and SITE datasets provide a slightly better performance compared to CD33 according to cosine similarity. However, according to both Euclidean and Manhattan metrics, CD33 shows the highest similarity with this target dataset. For Hmg_BS, the obtained results reveal that the SITE dataset demonstrates the highest similarity with it using cosine similarity, whereas the CD33 dataset is designated as the most suitable source for it according to Euclidean and Manhattan metrics. Clearly, the choice of the most appropriate source dataset depends on the specific distance/similarity measure being employed as the results provided by Euclidean and Manhattan metrics are usually well aligned, but don't always correspond to those yielded by the cosine metric.

The performance of the proposed model was assessed using several standard evaluation metrics, as detailed below:

In this section, we evaluate the impact of similarity analysis in transfer learning with CRISPR-Cas9 off-target data. Thus, we assess the reliability of the similarity scores reported in Table 3 for ML and DL-based off-target predictions. This evaluation is structured around three distinct scenarios, where machine learning models are trained on one of the three source datasets (CD33, CIRCLE, and SITE) and applied, via transfer learning, to different variants of seven bootstrapped datasets constructed as outlined in Table 2. For each scenario, we report the average results for 10 DL models: FNN models with 3, 5, and 10 layers; CNN models with 3, 5, and 10 layers; LSTM and GRU models with 4 layers; MLP models with 1 and 2 layers; as well as for traditional RF and LR classifiers (see Supplementary Information for further details on the models considered).

In the first scenario, the CD33 dataset served as the source dataset for transfer learning. The Receiver Operating Characteristic (ROC) curves (see Fig 5A) and Precision-Recall (PR) curves (see Fig A(A) in S1 Fig) are presented for various models trained on the CD33 dataset and tested on its bootstrapped counterpart, CD33_BS. To help evaluate the models performance, the AUC-ROC values are displayed in descending order within these figures. Furthermore, Fig 6 and Fig B in S1 Fig present the ROC and PR curves, respectively, for all considered ML and DL models using the CD33 dataset as source and six other bootstrapped datasets as targets. Additionally, Table 4 reports the values of the six selected evaluation metrics, including AUC ROC, Precision, Recall, F1-score, Brier score, and Accuracy, obtained using the CD33 dataset as source for all bootstrapped targets. Target datasets exhibiting the highest similarity to the CD33 dataset are marked with an asterisk (based on the similarity scores reported in Table 3). When the CD33_BS dataset served as the transfer learning target, GRU and MLP1 achieved a superior AUC-ROC performance compared to other models, with MLP1 providing the highest AUC-ROC score of 0.9863, closely followed by GRU at 0.9839. Both GRU and MLP1 consistently outperformed the other competing models across all evaluation metrics. Among the bootstrapped datasets, Listgarten_GUIDE_BS, Kleinstiver_GUIDE_BS, and Hmg_BS showed the highest similarity with CD33 according to Euclidean and Manhattan metrics (see Table 3). When Listgarten_GUIDE_BS was used as target, MLP1 achieved the highest AUC ROC (0.9629) and Precision (0.6828) results. When Kleinstiver_GUIDE_BS was used as target, MLP1 and MLP2 outperformed all other models across all metrics. Moreover, MLP2 consistently provided the best results across all metrics for the Hmg_BS target dataset, with the highest AUC-ROC, precision, and F1-score values, and the lowest Brier score.

In the second scenario, the CIRCLE dataset was used as the source dataset in our transfer learning experiments. The corresponding ROC curves (see Fig 5B) and PR curves (see Fig A(B) in S1 Fig) are presented for the considered ML and DL models trained on the entire CIRCLE dataset and evaluated on the CIRCLE_BS dataset. Additionally, Fig 7 and Fig C in S1 Fig show the ROC and PR curves for all considered models obtained using the CIRCLE dataset as source and the six other bootstrapped datasets as targets. The detailed quantitative results are reported in Table 5. When the CIRCLE_BS dataset was used as target, MLP2 performed exceptionally well with an AUC ROC score of 0.9959, a precision of 0.9564, a recall of 0.9231, an F1-score of 0.9600, a Brier score of 0.0021, and an accuracy of 0.9980. When the Tasi_GUIDE_BS dataset was used as target, MLP1 provided the best overal performance across all metrics. Similarly, for the Listgarten_GUIDE_BS target dataset, MLP2 showcased a robust performance, maintained consistency across all evaluation metrics. When the Kleinstiver_BS dataset was used as target, the best overall results were achieved once again using the MLP1 and MLP2 models.

In the third scenario, the SITE dataset served as the source dataset in our transfer learning experiments. The obtained ROC curves (see Fig 5C) and PR curves (see Fig A(C) in S1 Fig) are presented for all considered ML and DL models trained on the SITE dataset and evaluated on its bootstrapped counterpart, SITE_BS. Additionally, Fig 8 compares the ROC curves for all considered models, using the complete SITE dataset as source and the bootstrapped variants of the remaining datasets as targets (for PR curves see Fig D in S1 Fig). Further quantitative results are provided in Table 6. For the SITE_BS target dataset, FNN3 and MLP2 emerged as the best-performing models across all metrics. When the Kleinstiver_BS and Hmg_BS datasets were used as targets, the FNN5 model demonstrated notable results across diverse evaluation metrics.

Based on the evaluation results summarized in Tables 4, 5, and 6 across the three scenarios, the CD33, CIRCLE, and SITE datasets were found to be the most suitable sources for their respective bootstrapped counterparts: CD33_BS, CIRCLE_BS, and SITE_BS. This result was rather expected given the highest similarity scores between the complete datasets and their bootstrapped counterparts observed for all the three similarity measures considered (see Table 3). Furthermore, among the three source datasets (CD33, SITE, and CIRCLE), CIRCLE was identified as the optimal source for the Tasi_GUIDE_BS target dataset across all metrics, when using the MLP1 model (this corresponds to the highest similarity scores between Tasi_GUIDE_BS and CIRCLE provided by cosine, Euclidean, and Manhattan metrics; see Table 3), and for the Listgarten_GUIDE_BS target dataset, also across all metrics, when using the MLP2 model (this corresponds to the highest similarity score between Listgarten_GUIDE_BS and CIRCLE provided by cosine similarity; see Table 3). When the Kleinstiver_GUIDE_BS dataset was used as target, the CIRCLE (with the MLP1 and MLP2 models) and SITE (with the FNN5 model) datasets emerged as the optimal sources (see Tables 5 and 6). This corresponds to the highest similarity score between Kleinstiver_GUIDE_BS and both CIRCLE and SITE provided by cosine similarity (see Table 3). When the Hmg_BS dataset was used as target, the SITE dataset (using the FNN5 model) was identified as the optimal source (see Table 6); once again, this reflects the highest similarity score between Hmg_BS and SITE provided by cosine similarity (see Table 3).

These results validate two critical points of our study: First, similarity score results are reliable and trustworthy indices for determining the most appropriate source dataset for a given target dataset prior to performing transfer learning experiments in CRISPR-Cas9. They reinforce the effectiveness of our methodology as a robust pre-selection tool for transfer learning, providing a systematic approach for identifying efficiently suitable source data. Second, cosine distance (or cosine similarity) emerges as the most dependable metric, among the three metrics considered, for selecting the most appropriate source dataset for transfer learning.

Moreover, our results clearly demonstrate that similarity-based source data pre-selection is necessary to mitigate negative knowledge transfers. If a source dataset is chosen solely by size, availability, or even class imbalance ratio, but without similarity assessment, this could eventually lead to a suboptimal or negative transfer. For example, the CIRCLE and SITE datasets considered in our study have comparable sizes and almost identical class imbalance ratios (0.0128 and 0.0176, respectively - see Table 1), but a knowledge transfer from CIRCLE to SITE as well as that from SITE to CIRCLE are clearly negative with the highest F1-score values of 0.1345 (see Table 5) and 0.1008 (see Table 6), respectively, over all competing ML and DL models. Our similarity-based analysis suggests that such transfers should be avoided (see Table 3).

It is worth noting that in some cases a potential source dataset with a slightly lower cosine similarity with the target might still yield competitive or even superior performance for specific models - MLP-based models in our case. This could be due to such factors as a richer representation of specific rare patterns important for the target task or some MLP inductive biases aligning better with the source data distribution. For example, the recommended transfers with lower cosine similarity scores of 0.5701 from CIRCLE to Listgarten_GUIDE_BS and of 0.5672 from CIRCLE to Kleinstiver_GUIDE_BS led to competitive knowledge transfers with the corresponding best F1-score score values of 0.5646 and 0.7048, obtained, respectively, with MLP1 and MLP2 (see Table 5). It was not so for Euclidean and Manhattan similarities whose highest values in this case led to transfers from CD33 (instead of CIRCLE) with much lower best F1-score score values of 0.2500 for Listgarten_GUIDE_BS and of 0.1946 for Kleinstiver_GUIDE_BS, both obtained with MLP1 (see Table 4).