Breast cancer is a clear example of the effectiveness of precision medicine. Tumors that test positive for the HER2 protein can be targeted with immunotherapy, radically improving the prognosis. Treatments for many cancer types may be effective only for specific mutations or genomic profiles. Understanding the subset of patients who may benefit is the key to personalized cancer treatments.
However, the methods to assess such molecular properties of tumors are expensive and time-consuming. Each new test also requires additional tissue. Recent advances in computational pathology may have found an alternative: artificial intelligence algorithms applied to H&E histology. Twenty different studies - all published in the last two years - have demonstrated that one or more molecular properties can be predicted from H&E alone using the advancements of deep learning.
The Limitations of Molecular Testing
Molecular and genomic properties can be used to stratify patients into subtypes. Each subtype responds to some courses of treatment but not others and tends to have a distinct prognosis. Advances in precision medicine over the last decade have focused on these molecular biomarkers, which have helped identify the significant proportion of patients who don’t respond to immunotherapy.
The technologies used in assessing these molecular properties are expensive and time-consuming to perform. They may involve DNA sequencing to detect mutations, RNA sequencing to assess gene expression, or immunohistochemical staining to identify protein biomarkers. Most are not routinely performed on all patients who could benefit and are not done at all in labs with limited resources. To complicate things further, in many cases only a small amount of tissue is excised from a tumor, and there is not enough for additional analyses beyond what a pathologist examines through the microscope.
New studies keep identifying more molecular properties of potential clinical value, each requiring its own tissue sample and processing procedure. Current workflows are not designed to incorporate this many tests. While comprehensive molecular testing will be difficult to implement at scale, histological staining of tissue is common practice and imaging of such samples is becoming increasingly available with the transition to digital pathology.
Deciphering Histological Signatures with Deep Learning
Digital pathology alone is not sufficient to assess molecular biomarkers as their histological signatures are often too complex to be recognizable by pathologists. But recent advances in artificial intelligence can now decipher patterns that are beyond the limits of human perception.
Four types of molecular biomarkers have been successfully predicted from H&E in studies of more than ten different types of cancer:
- point mutations, tumor mutational burden, and microsatellite instability
- genomic subtypes and expression of individual genes
- protein biomarkers
- virus
Most of these studies used whole slide images and either had a pathologist annotate each slide with the tumor region or trained a machine learning model to identify it. Some studies used tissue microarray images instead.
Using a training set of images and their associated biomarker label (for example, the presence or absence of a particular mutation), they trained a model that can predict the label for a previously unseen image.
The most powerful machine learning model available today - and the type used in each one of these studies - is deep learning. Deep learning is a technique to learn patterns in images. The model consists of multiple layers of features where the higher-level concepts are built upon the lower level ones. Going up the hierarchy, the features increase in both scale and complexity. Similar to human visual processing, the low levels detect small structures such as edges. Intermediate layers capture increasingly complex properties like texture and shape. The top layers of the network are able to represent objects and more complex properties like tissue architecture.
Deep learning has previously shown success for finding mitoses, segmenting tissue types, and detecting tissue structures. Until recently, the focus had remained on detecting image properties that pathologists can see. Applying this powerful technique to higher level (often tumor level) properties, for which a pathologist cannot provide detailed annotations, is a more recent innovation.
Each study demonstrated the ability to predict one or more molecular properties from H&E alone, relying only on patient-level labels for anywhere from a few hundred to a couple thousand samples for training the model.
Most recently, two independent publications have demonstrated this technology across multiple types of cancer and multiple biomarkers. Further research has even shown that some of the deep learning-based biomarker predictions are associated with survival.
Next Steps and New Possibilities
As with all image analysis algorithms for digital pathology, a number of challenges are still present in turning this new technology into a robust solution.
More rigorous validation is needed on larger data sets with a wider variety of patients. When applied to multiple cohorts of patients, the domain shift due to processing in different labs with different populations of patients also becomes a challenge. Tissue staining methods can vary from lab to lab and can change over time, and tumors from different populations of patients may have different characteristics.
Pathologists will need to be trained in interpreting the output from these algorithms and in making decisions with this new information. Deep learning algorithms still present the challenge of limited interpretability - something that will need to be overcome to best guide pathologists in using the information and for manufacturers in obtaining regulatory approval.
Deep learning research has been advancing rapidly, with numerous research groups tackling these challenges for pathology and in other application areas. Explainability methods currently focus on highlighting the regions of an image that most contributed to a prediction. Perhaps future developments will guide us towards histological features that are more human interpretable. These and other research topics are being explored across many application areas of deep learning. The answers likely won’t come from pathology alone but from cross-disciplinary efforts.
Deep learning-based biomarkers from H&E could provide an additional tool for pathologists and new insights for companion diagnostics and drug discovery. Based on only H&E images and no longer limited by the amount of tissue excised or the processing time for individual molecular tests, deep learning-based methods can provide easier access to vital information about a tumor. They may yield a screening method to determine when to run molecular tests or an alternative when molecular tests are not possible.
What will be the first clinical use of this technology? How will it be translated and approved for clinical use? My expertise is in algorithm development, so I pose the question to those working closer to clinical applications. If you’re a DPA member, please comment here. Otherwise, I’d love to hear your thoughts via email.