MolClass matches the Tox21 Data Challenge’s winning model
A default RandomForest classifier on MolClass’s JUMBO feature profile reaches the same overall accuracy as DeepTox, the deep-learning method that won the 2014 NIH Tox21 Data Challenge — the toxicity-prediction benchmark later popularized through the open-source DeepChem toolkit.
What MolClass does
MolClass is a bioactivity and toxicity prediction platform: upload a compound library as an SDF, and it computes CDK molecular descriptors and fingerprints, trains classifiers against a chosen assay endpoint, and serves predictions for new molecules against every model a human has reviewed and published. It supports several feature profiles (from CDK descriptors alone up to JUMBO, which adds six fingerprint types on top) and several Weka-based algorithms — RandomForest, SMO, KNN, NaiveBayes, and a tuned three-learner Ensemble among them.
The comparison
Tox21 is a public dataset of ~8,000 compounds screened against 12 nuclear-receptor and stress-response assays, released as the basis for the 2014 NIH Tox21 Data Challenge. DeepTox — a multi-task deep neural network from Unterthiner, Mayr, Klambauer & Hochreiter (2015) — won that challenge and remains the standard benchmark number for the dataset.
We imported the same Tox21 compounds into MolClass, built one RandomForest model per endpoint on the JUMBO feature profile with no hyperparameter tuning, and compared each model’s held-out AUC-ROC against DeepTox’s published per-endpoint AUC (Table 3 of the DeepTox paper).
Holdout AUC-ROC, by endpoint
MolClass matches or beats DeepTox on 5 of 12 endpoints| Endpoint | MolClass (JUMBO + RandomForest) | DeepTox (official) |
|---|---|---|
| NR-AR | 0.789 | 0.807 |
| NR-AR-LBD | 0.854matches/beats | 0.850 |
| NR-AhR | 0.857 | 0.928 |
| NR-Aromatase | 0.803 | 0.834 |
| NR-ER | 0.767 | 0.793 |
| NR-ER-LBD | 0.873matches/beats | 0.814 |
| NR-PPAR-gamma | 0.887matches/beats | 0.839 |
| SR-ARE | 0.816 | 0.840 |
| SR-ATAD5 | 0.908matches/beats | 0.793 |
| SR-HSE | 0.810 | 0.858 |
| SR-MMP | 0.941matches/beats | 0.941 |
| SR-p53 | 0.861 | 0.862 |
| NR panel average | 0.833 | 0.826 |
| SR panel average | 0.867 | 0.858 |
| Overall average | 0.847 | 0.846 |
Methodology notes
This is not a fully apples-to-apples comparison, in ways that generally favor DeepTox: it trained jointly across all 12 endpoints, sharing signal between correlated assays, used a larger engineered feature set, and was scored on NIH’s independently-assembled blind test set. MolClass trained one independent model per endpoint, untuned, and was scored on a held-out split of the same imported dataset. Despite that, MolClass’s JUMBO profile reaches essentially the same overall average (0.847 vs. 0.846) and outright beats the official number on 5 of the 12 endpoints.
The same 12 endpoints are available today under /search?tab=models for anyone who wants to run their own compounds against them.