Predicting Molecular Tumor Biomarkers from H&E:

A number of recent research papers have demonstrated that molecular tumor biomarkers can be predicted from H&E using deep learning. This page lists all known work in this area. Please email if you know of a paper that should be added to this list.

Reference Cancer Type Biomarker
Woerl, Ann-Christin, et al. "Deep Learning Predicts Molecular Subtype of Muscle-invasive Bladder Cancer from Conventional Histopathological Slides." European Urology (2020). bladder molecular subtype
Xu, Hongming, et al. "Using transfer learning on whole slide images to predict tumor mutational burden in bladder cancer patients." bioRxiv (2019). bladder TMB
Xu, Hongming, et al. "Deep transfer learning approach to predict tumor mutation burden (TMB) and delineate spatial heterogeneity of TMB within tumors from whole slide images." bioRxiv (2020). bladder, lung TMB
Couture, Heather, et al. “Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype.” NPJ Breast Cancer (2018). breast hormone receptor status: ER
genomic subtype: PAM50
Couture, Heather, et al. “Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology.” International Conference on Medical Image Computing and Computer-Assisted Intervention (2018). breast hormone receptor status: ER
genomic subtype: PAM50
Jaber, Mustafa I., et al. "A deep learning image-based intrinsic molecular subtype classifier of breast tumors reveals tumor heterogeneity that may affect survival." Breast Cancer Research (2020). breast genomic subtype: PAM50
Bychkov, Dmitrii, et al. "Deep learning identifies morphological features in breast cancer predictive of cancer ERBB2 status and trastuzumab treatment efficacy." Scientific reports (2021). breast hormone receptor status: HER2
La Barbera, David, Kevin Roitero, and Vincenzo Della Mea. "Detection of HER2 from Haematoxylin-Eosin Slides Through a Cascade of Deep Learning Classifiers via Multi-Instance Learning." Journal of Imaging (2020). breast hormone receptor status: HER2
Naik, Nikhil, et al. "Deep learning-enabled breast cancer hormonal receptor status determination from base-level H&E stains." Nature communications (2020). breast hormone receptor status: ER
Oliveira, Sara P., et al. "Weakly-Supervised Classification of HER2 Expression in Breast Cancer Haematoxylin and Eosin Stained Slides." Applied Sciences (2020). breast hormone receptor status: HER2
Rawat, Rishi R., et al. "Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images." Scientific Reports (2020). breast hormone receptor status: ER, PR, HER2
Shamai, Gil, et al. "Artificial intelligence algorithms to assess hormonal status from tissue microarrays in patients with breast cancer." JAMA (2019). breast hormone receptor status: ER, PR, HER2
Chauhan, Ruchi, et al. "Exploring Genetic-histologic Relationships in Breast Cancer." ISBI (2021). breast mutations: TP53, PIK3CA
hormone receptor status: ER, PR, HER2
intrinsic subtypes
Xu, Zhuoran, et al. "Deep learning predicts chromosomal instability from histopathology images." iScience (2021). breast chromosomal instability
Valieris, Renan, et al. "Deep Learning Predicts Underlying Features on Pathology Images with Therapeutic Relevance for Breast and Gastric Cancer." Cancers (2020). breast and gastric DNA repair deficiency
Echle, Amelie, et al. "Clinical-grade detection of microsatellite instability in colorectal tumors by deep learning." Gastroenterology (2020). colorectal MSI
Krause, Jeremias, et al. "Deep learning detects genetic alterations in cancer histology generated by adversarial networks ." The Journal of Pathology (2021). colorectal MSI
Jang, Hyun-Jong, et al. "Prediction of clinically actionable genetic alterations from colorectal cancer histopathology images using deep learning." World Journal of Gastroenterology (2020). colorectal mutations: APC, KRAS, PIK3CA, SMAD4, TP53
Hildebrand, Lindsey A., et al. "Artificial Intelligence for Histology-Based Detection of Microsatellite Instability and Prediction of Response to Immunotherapy in Colorectal Cancer." Cancers (2021). colorectal MSI
Kather, Jakob Nikolas, et al. "Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer." Nature Medicine (2019). colorectal MSI
Yamashita, Rikiya, et al. "Deep learning model for the prediction of microsatellite instability in colorectal cancer: a diagnostic study." The Lancet Oncology (2021). colorectal MSI
Popovici, Vlad, et al. "Image-based surrogate biomarkers for molecular subtypes of colorectal cancer." Bioinformatics (2017). colorectal genomic subtypes
Sirinukunwattana, Korsuk, et al. "Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning." Gut (2020). colorectal genomic subtypes
Zhang, Hongrun, et al. "Piloting a deep learning model for predicting nuclear BAP1 immunohistochemical expression of uveal melanoma from hematoxylin-and-eosin sections." Translational Vision Science & Technology (2020). eye protein: BAP1
Kather, Jakob Nikolas, et al. "Deep learning detects virus presence in cancer histology." bioRxiv (2019). gastric
head and neck
virus: EBV
virus: HPV
Klein, Sebastian, et al. "Deep Learning Predicts HPV Association in Oropharyngeal Squamous Cell Carcinomas and Identifies Patients with a Favorable Prognosis Using Regular H&E Stains." Clinical Cancer Research (2021). oropharyngeal virus: HPV
Chen, Mingyu, et al. "Classification and mutation prediction based on histopathology H&E images in liver cancer using deep learning." npj Precision Oncology(2020). liver mutations: CTNNB1, FMN2, TP53, ZFX4
Coudray, Nicolas, et al. "Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning." Nature Medicine (2018) lung mutations: STK11, EGFR, FAT1, SETBP1, KRAS, TP53
Yu, Kun-Hsing, et al. "Classifying non-small cell lung cancer histopathology types and transcriptomic subtypes using convolutional neural networks." bioRxiv (2019). lung genomic subtypes
Jain, Mika S., and Tarik F. Massoud. "Predicting tumour mutational burden from histopathological images using multiscale deep learning." Nature Machine Intelligence (2020). lung TMB
Schaumberg, Andrew J., Mark A. Rubin, and Thomas J. Fuchs. "H&E-stained whole slide image deep learning predicts SPOP mutation state in prostate cancer." BioRxiv (2018). prostate mutation: SPOP
Shao, Yanan, et al. "Improving Prostate Cancer Classification in H&E Tissue Micro Arrays Using Ki67 and P63 Histopathology." Computers in Biology and Medicine (2020). prostate protein: Ki-67, p63
Kim, Randie H., et al. "A deep learning approach for rapid mutational screening in melanoma." bioRxiv (2019). skin mutation: BRAF
Dammak, Salma, et al. "Prediction of tumour mutational burden of squamous cell carcinoma using histopathology images of surgical specimens." Medical Imaging 2020: Digital Pathology Photonics. skin TMB
Kather, Jakob Nikolas, et al. "Pan-cancer image-based detection of clinically actionable genetic alterations." Nature Cancer (2020). multiple mutations
genomic subtypes
hormone receptor status
Fu, Yu, et al. "Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis." Nature Cancer (2020). multiple mutations
gene expression
Schmauch, Benoît, et al. "A deep learning model to predict RNA-Seq expression of tumours from whole slide images." Nature communications (2020). multiple gene expression
Liu, Yiqing, et al. "Predict Ki-67 positive cells in H&E-stained images using deep learning independently from IHC-stained images." Frontiers in Molecular Biosciences (2020). multiple protein: Ki-67