New AI technology using H&E images creates an alternative path for molecular diagnostics

Molecular properties are critical in selecting cancer treatments to target weaknesses of individual tumors. However, the methods to assess such properties are expensive and often not routinely performed. Applying artificial intelligence to microscopic images of tumor tissue can provide a more cost-effective screening method.

The gold standard for diagnosing cancer is with histopathology. A tumor sample is prepared on a glass slide and viewed under a microscope by a pathologist who decides whether there are cancer cells present. While pathologists are experts in diagnosing cancer, the complexity of tumor properties that they can assess is still limited. At the University of North Carolina we investigated whether a computer can find features to predict molecular biomarkers that are too complex for pathologists to assess from images. We found this to be true for breast cancer and other researchers have recently proven the same for other cancer types.

Molecular Subtypes for Tumor Stratification

One of the goals of cancer research is to identify smaller groups that a type of cancer can be divided into based on characteristics of tumor cells. These groups are called subtypes, and they help doctors to find common causes and tailored treatments for cancer. Each subtype responds to some courses of treatment but not to others and tends to have a distinct prognosis.

Stratifying tumors into subtypes using molecular properties has recently shown great success. For example, breast cancer is divided into five different subtypes that add prognostic information and treatment efficacy insights. The technologies used in assessing these molecular properties are expensive and time-consuming to perform. Most are not routinely performed on all patients who could benefit and are not done at all in labs with limited resources. To complicate things further, in many cases only a small amount of tissue is excised from a tumor and there is not enough for additional analyses beyond what a pathologist examines through the microscope.

New studies keep identifying more molecular properties of potential clinical value, each requiring its own tissue sample and processing procedure. Current workflows are not designed to incorporate this many tests. While comprehensive molecular testing will be difficult to implement at scale, histological staining of tissue is common practice and imaging of such samples will become increasingly available with the transition to digital pathology.

Histological Imaging for Diagnosis

Pathologists use histological staining in studying tissue and looking for cancer cells. H&E consists of a pair of stains that turn the tissue blue and pink. H&E slides are inexpensive to produce with a short turnaround time. Tumor tissue is already studied in this manner for the initial diagnosis of cancer so this workflow is well-established. Pathologists find cancer cells and assess the aggressiveness of tumors.

Pathologists cannot, however, assess the molecular biomarkers from H&E that are important for guiding treatment. Image analysis methods can provide a screening mechanism to identify patients who would benefit from further molecular testing or act as an alternative when molecular analysis is not possible.

In studying H&E histology, pathologists look at the size and shape of cell nuclei, their arrangement, and the presence of other tissue structures like glands. They also look for small structures such as dividing nuclei called mitoses and immune cells called lymphocytes. But in order to go beyond the complexity of what human experts can assess visually, we must create new image features. We must learn features from the images themselves. This is known as feature or representation learning and, currently, the most powerful tool in this set is deep learning.

The Rise of Deep Learning in Histopathology

Deep learning is a technique to learn patterns in images that enable the computer to make predictions. After training on a data set of images and labels (such as the biomarker status), the model can predict these labels on new data that it was not trained with. The model consists of multiple layers of features where the higher-level concepts are built upon the lower level ones. Going up the hierarchy, the features increase in both scale and complexity. Similar to human visual processing, the low levels detect small structures such as edges. Intermediate layers capture increasingly complex properties like texture and shape. The top layers of the network are able to represent objects and more complex properties like tissue architecture.

This method has previously shown success for finding mitoses, segmenting tissue types, and detecting tissue structures. However, the focus had remained on detecting image properties that pathologists can see. Applying this powerful technique to higher level (often tumor level) properties, for which a pathologist cannot provide detailed annotations, is a more recent innovation.

Deep learning performs very well when given a lot of training data – often millions of examples. However, it struggles when training data is limited, as is typically the case with medical images. A common solution for small datasets is transfer learning: a model is trained on one data set and then applied to a different one. A large benchmark data set of photographs called ImageNet is typically used for this initial training step. In my work with breast cancer, I used the pre-trained model to compute features on H&E images and then trained a classifier to predict the biomarkers. Subsequently, I fine-tuned the model for improved performance on breast cancer.

Prior to working on breast cancer, my first experience with image analysis for pathology was in trying to predict melanoma mutations from H&E. We used hand-crafted features – the size, shape, and texture of nuclei and their spatial arrangement – properties that a pathologist looks for in diagnosing and grading cancer. We were not able to distinguish tissue samples with the mutations from those without. This was 2012 and deep learning was in its infancy. The research work that popularized deep learning was published in 2012 and open source tool kits started to become available over the next few years. It was not until 2015 that today’s more easily accessible libraries were first released. These developments in machine learning laid the groundwork for many pathology applications, including biomarker prediction from H&E.

Molecular Biomarker Prediction with Deep Learning

Recent work at NYU has shown that the exact mutations we looked at in melanoma are predictable from H&E – using deep learning. Similar results have been found for point mutations in breast, colorectal, gastric, lung, and prostate cancer. A team at the Cleveland Clinic found that tumor mutational burden – a measurement of mutations carried by tumor cells – is predictable for bladder cancer. Detecting mutations is typically done with DNA sequencing, which is costly and time-consuming, but can guide targeted therapies.

Our work at UNC on breast cancer addressed genomic subtypes. Gastric, lung, and colorectal genomic subtypes have also shown success at other institutions. Genomic subtypes stratify patients based on the activity level of specific genes, providing prognostic information and also helping to guide treatment decisions. While assays are available for some genomic subtypes, RNA sequencing is still the most accurate.

Going beyond mutations and genomic subtypes, we predicted estrogen receptor status on breast cancer. Knowing the receptor status helps doctors decide how to treat cancer as certain hormone therapy drugs may be used. Other researchers have since tested a simpler method on a larger data set and further expanded to include an additional eighteen protein biomarkers. The standard method for assessing protein biomarkers is immunohistochemical staining – an alternative tissue staining methodology that requires a pathologist’s subjective interpretation. It is also time-consuming and costly.

Even the presence of virus can be detected from H&E. Human papilloma virus was identified in head and neck cancer and Epstein Barr virus in gastric cancer. Both viruses are major causes of human cancer and knowing about their presence impacts treatment decisions. Molecular tests for these viruses are expensive and not ubiquitously available.

Deep Learning Creates New Possibilities

The first step in each of these approaches is to identify the tumor regions in each tissue slide. Some groups relied upon pathologists to delineate the tumor – something that pathologists are already experts at. Others have automated this step of distinguishing tumor tissue from non-tumor. All groups rely upon deep learning’s abstract features to predict the molecular biomarkers. Traditional hand-crafted features are just not powerful enough for these tasks. It is only through more complex properties – even beyond the capability of expert pathologists – that we can now screen for some molecular biomarkers from H&E alone.

Deep learning is opening up a new world of possibilities in capturing properties that are too complex for even the best trained pathologists. It is a possible screening opportunity for any marker that can guide further testing and selection of treatments. While pathologists are, without a doubt, the experts in their craft, these additional insights can assist in targeting therapies for each patient.

Would you like help implementing deep learning-based histology biomarkers?

I’ve worked with a variety of teams to develop advanced machine learning algorithms to extract new insights from pathology images. Schedule a free Machine Learning Strategy Session to get started.