Resources for Computer Vision Teams:

LinkedIn – Connect with Heather.
Computer Vision Insights Newsletter – A biweekly newsletter to help bring the latest machine learning and computer vision research to applications in people and planetary health.
Computer Vision Strategy Session – Not sure how to advance your computer vision project? Get unstuck with a clear set of next steps. Schedule a 1 hour strategy session now to advance your project.

Transcript:

[INTRODUCTION]

[00:00:03] HC: Welcome to Impact AI, brought to you by Pixel Scientia Labs. I’m your host, Heather Couture. On this podcast, I interview innovators and entrepreneurs about building a mission-driven, machine-learning-powered company. If you like what you hear, please subscribe to my newsletter to be notified about new episodes. Plus, follow the latest research in computer vision for people in planetary health. You can sign up at pixelscientia.com/newsletter.

[EPISODE]

[0:00:34] HC: Today, I’m joined by guest, Max Alber, co-founder and CTO of Aignostics to talk about pathology for precision medicine. Max, welcome to the show.

[0:00:43] MA: Hi, Heather. Thank you for having me and Aignostics.

[0:00:46] HC: Max, could you share a bit about your background and how that led you to create Aignostics?

[0:00:49] MA: Yes, happy to. I’ve studied computer science and attended a PhD in machine learning at the Technical University in Berlin, focusing on scale of machine learning and explainable AI. For other time, I’ve worked as a software engineer at a few startups, Fraunhofer, and Google. After that, I was happy to join my co-founders to spin Aignostics out of Charité, which is Europe’s biggest university hospital. It was unique opportunity to work for good purpose and an exceptionally interdisciplinary environment, and it’s still a lot of fun.

[0:01:18] HC: What do you do Aignostics and why is it important for healthcare?

[0:01:22] MA: Our mission is to turn biomedical data into insights. Basically, this means, we do data analysis on data from pathology labs, we focus on image data. Aignostic clients are top 20 pharma companies, and we landed first in translational research. Next month, now, into clinical studies and clinical applications. Additionally, over the last year, we also naturally moved into biomarker discovery and target identification. We recently started the strategic partnership with Bayer, for finding novel targets for cancer therapies.

Good example for our work is a collaboration between Aignostics and The Charité, where we analyze a cohort of over 1000 NSCC patients across key modalities such as H&E, IHC, multiplex immunofluorescence, proteomics, transcriptomics, with a goal to find novel biomarkers. Why is this important for patient care? We believe AI-powered data analysis will be, on one hand, the basis for personalized medicine, and the discovery of novel biomarkers, and drug targets. And on the other hand, it will increase the effectiveness of pathologists in order to cope with increased workload, medical complexity, also, lack of doctors.

[0:02:28] HC: What role does machine learning play in this technology?

[0:02:31] MA: Yes. We define ourselves as an algorithm company. On the clinical side, this means, we want to build machine learning models and algorithms, and distribute them by third-party platforms such as Proscia and Paige. Thus, compared to competitors, we don’t want to build a clinical platform. On discovery side, such as biomarker search and targeting, we believe platform and algorithm development are closely tied together until we build our own in-house platform. Thus, everything we do is driven by the belief that machine learning and data science will help us improve healthcare and different machines that ultimately drives our company.

[0:03:02] HC: What are some examples of models that you create? Not looking for details in architecture, just more maybe examples of what the input is, and what the output is, and how you put all that together.

[0:03:12] MA: Yes. Application-wise, we started off with having a focus on single cell and tissue-based prediction models. For example, in H&E, differentiating between the most common cell types like carcinoma cells, epithelial cells, immune cells like lymphocytes, plasma cells, macrophages. And detecting stroma regions, carcinoma regions, necrosis, and so on, and so forth. Similarly, the same in IT for IT markers, where we also do the intensity scoring, and so on, and so forth. Bringing this all together to characterizing the tumor microenvironment of the 4G sites.

This is ultimately used by us and our clients to connect us to answer questions, such as, “Will this patient survive longer?” or “Does this therapy work or not?” Next thing we ought to do also prediction task, like mutation predictions with multiple instance learning setup. On that side yet, many supporting models like models for tissue QC, and for artifact detection, other things interesting. I forgot to mention, I think we also have strong foothold in multiplex immunofluorescence, we got models to do aforementioned tasks.

[0:04:20] HC: You mentioned at the end there, multiplex immunofluorescence. Then, you’ve also mentioned H&E and immunohistochemistry. Are these the main three types of modalities that you’re working with? Do you have any other types of data that go into your models?

[0:04:33] MA: This is our focus on image data set, on the – as I mentioned before, biomarker discovery and targeted site. We also work a lot with proteomics, and ingested, and transcriptomic data by now. But this is more of a new avenue. It’s pretty broad, but I said, I think we want to specialize in being able to analyze all the data that comes out of a pathology lab.

[0:04:52] HC: Pretty much any kind of spatial imaging histopathology type data. There’s a variety of modalities there, but those are the ones you stick with.

[0:05:01] MA: Yes, that’s right.

[0:05:01] HC: You recently published a paper about a foundation model. Could you describe what a foundation model is for those in the audience that aren’t familiar and when it is useful?

[0:05:11] MA: Happy to. A foundation model is a model that can be used as a starting point for building a machine learning application, with the promise that the foundation model already has a great understanding of the domain, say, pathology in our case. And thereby, A, improves generalizability, and B, enables applications to otherwise not be possible. The prime example in another domain would be the foundation model GPT that powers Chat GPT. Typically, this is achieved by unsupervised or self-supervised featuring, where a foundation sees a lot of unlabeled data, and is therefore able to understand intrinsic patterns of the domain.

This understanding can then be used in typically supervised downstream task, where there is often not enough data available to train a generalizable model. An extreme example in pathology, the pathology chat assistant, which was a program showcased by the Mahmood Lab at Harvard. This is an application that has not been possible without the foundation model. At Aignostics, foundation models are exactly use for those two use cases: A, improving generalizability on all existing tasks, such as segmentation, solid detections, classification. And B, to enable novel applications, such as case or image retrieval, or anomaly in rare disease detection.

[0:06:19] HC: The foundation model that you built, how did you go about this? Are there ways that differs from other foundation models out there for histopathology?

[0:06:27] MA: That’s a great question. In a previous podcast, with Paige wanted a lot of data, a lot of compute, and a great team to build a foundation model. I think if you’re given an example on how data diversity, and creation, and domain knowledge is another important factor for building foundation models, and can be an alternative to pure data and compute scaling. If you did so by having a pathologist, and experts in the loop for creating a diverse data set, and developing sampling strategies on slide and tissue level.

Our dataset is most diverse, but it’s the most diverse used so far pathology, containing data from our 50 tissue types, from our 15 labs from the US and EU, and from six different scan type types from non-oncology disease cases. And most importantly, from over 100 different staining types, and the staining comprise H&E, a variety of IHC stains, and so-called special stains.

On top of that, we, including our pathologist ask ourselves, “What part of the data does really matter, and how much does it matter?” In that, we build clusters and groups on slide and tissue level, and use them to send the data for training. This improvement showed I think the best results on public benchmarks. Our model is on par with more strength, with magnitude, more data, and much more compute. As I was saying, in our paper, due to our company’s focus, as previously mentioned, features different benchmarks and other papers from some cell or tissue-based predictions as well as ICC-based tasks.

Compared to pure cancer detection, those benchmarks for distinctions for cell types such as carcinoma, and epithelial cells, or different immune cells are a bigger challenge, and allow for an additional differentiation for foundation models. Then, those benchmarks, we said our foundation model, would also perform best compared to available foundational models, including UNI from Mahmood Lab, and Phikon from Owkin. Nearly all use cases. Especially, the comparison to UNI, it’s interesting because they use a similar amount of data, and the same pre-training method and code. But our set of features a more diverse data set, hinting on the importance of data creation.

[0:08:28] HC: Diversity is the key to the data set of this. Sounds like you put a ton of efforts into identify what diversity means in pathology and how to curate your data so that is diverse. How are you using this foundation model or is it already used in some of the some of your products?

[0:08:48] MA: Exactly. We migrate all our downstream tasks in [inaudible 0:08:51] foundation model, and definition model made a significant difference for each application. For example, we respectfully tried our foundation model on a [inaudible 0:09:00] project with the US pharma company. So, jumped on average of 10 percentage points in balance accuracy across all setups without changing anything else in the training setup, except the foundational lens or the modeling transition moving into the finished model.

Similarly, we did the test in label efficiency agency, and discovered that our foundation model reached maximum performance with potentials, the labels or annotation from the pathologists, compared to the old setup. Overall, this means, the foundation model allows us, Aignostics to build more robust models in a fraction of the time that we needed before without the foundation model. Basically, it’s already used in each service and product that Aignostic is offering. On top of that, we are in active discussions for licensing our foundation model to other companies in order to enable their development as well. Important here is that we develop our foundation model along regulatory requirements, which will allow it to be used in medical products.

[0:09:53] HC: You mentioned that the use of this foundation model enables you to develop models much quicker than you had previously. Do you have any rough numbers on what that difference is and maybe how long it took to develop a particular feature without the foundation model versus with it.

[0:10:09] MA: An example before that I mentioned is really good use case. There was a lot of the project, up to 500,000, slides, 500 cases from clinical trial. We wanted to get the best performance out of that, because the downstream analysis is very important. We had several rounds of improvements with additional annotations from pathologists with the old setup. With plugging in the foundation model, this would have taken a tenth of the time. Literally, a tenth of situations and annotations to get at the same performance, which is another sufficient performance. This is, for us, a tremendous impact on how we build, and how quickly we can ship our models.

[0:10:46] HC: That’s a huge difference, one-tenth of the time, and that’s one project, and you have other features, other projects, other things you’re developing. So I suspect, it would have a similar impact on many of them.

[0:10:57] MA: Yes, definitely.

[0:10:58] HC: Are there any lessons you’ve learned and developing foundation models that can be applied more broadly to other types of imagery?

[0:11:04] MA: Yes, really good question. I think data diversity, the right benchmarks, and feedback signal, and the ability to iterate quickly are key. From our experience, as noted before, data diversity and creation has a big impact, and can be multiplied compared to pure data scaling and compute scaling. Then, additionally, from our experience, the losses in metrics for image-based pre-training don’t correlate well with downstream performance. This makes it very difficult. Here, some methods around like rank computation or smaller benchmarks that run for the training on the site can help, but it nevertheless remains a problem. Ultimately, given that there are so many unknowns with the engineering and the good setup in order to scale and iterate quickly is also very important. That one can figure out what you don’t know.

[0:11:48] HC: Do you have any suggestions for others, maybe even applications beyond pathology on how they could develop better benchmarks and iterate more quickly with their foundation models?

[0:11:57] MA: That’s an even better question. I think it’s very much a tradeoff. Ultimately, your best benchmark is the real application. But of course, often, it’s just too expensive or too tedious run that. I think this is really domain-specific, where you need to think about what are really the benchmarks and problems that matter the most, and how you can make them smaller so that you can run them quicker. I don’t think there’s a one-size-fits-all solution in reading requests, domain knowledge, and benchmark knowledge.

[0:12:26] HC: Yes. I think what you said there about domain knowledge is probably the key point there. Understanding the data so you can figure out what those benchmarks are, and how it should be applied in that domain. You’re doing it in pathology, but others need to understand the data they’re working with. One thing that’s mentioned on your website that I’d like to dig a bit deeper on is model explainability. This is certainly a challenging concept for deep learning. It’s not impossible, but as challenging. Is it something that’s important for the models that you’re working with and how do you make them explainable?

[0:12:58] MA: Another good question. I think there are three main use cases for explainability, and they are trusted and novel insights. For trust, explainability is important for understanding what the model based its decision on, especially in the medical context. The question is, does the model use a biologically meaningful signal or a confounder? This can be on a technical level be used just for debugging a model, or stakeholders and pathologists to do a plausibility checks, and get a general understanding what signal the model uses.

One project example that we had was with the tower company, again, that we explained and investigated the model before it went to the next development phase for them. Leveraging our [inaudible 0:13:37] explainable AI approach [inaudible 0:13:39]. It really helped them to get insights on to get the go or no-go decision if the model would work. Because just on performance, it was not sufficient. They weren’t confident enough to just move forward.

For the insights use case here, the scenario is that the model can predict something but the human cannot. In this case, it’s important to understand if there’s a confounding factor, or there might be another insight that one can leverage. A confounding factor can be for example, that most of the positive cases where it can just scan A, and most of the negative cases can just scan B, and the model only is able to differentiate between scanners, which people typically really are not able to know if you would not suspect if that what is the model uses.

If you want a larger model in the clinical trial or in the clinical environment, you want to make sure that this is not the case. Notice the prime example to figure out another insight is compute identification, where you want to figure out, you want to predict for example, survival or response to therapy, and you want to understand why there’s a response or not based on, for example, multimodal data.

How to make models explainable, as [inaudible 0:14:47] at the moment, there’s certainly a tradeoff between machine model performance and explainability. Extending larger models and deep learning models is very hard, and therefore our strategy is twofold. So, depending on data type, we rely on structuring and making all the inputs explainable, and/or, also, it can be combined. Use [inaudible 0:15:08] abilities such as [inaudible 0:15:09] experiments, propagation or PatternNet, which we are very familiar with. They scale very well model size, and data size, and there are several [inaudible 0:15:18] experience that make us confident here.

[0:15:21] HC: Does the use of foundation models affect explainability, either in a positive or negative way?

[0:15:27] MA: Positive way, the transformer structures give you some form of explainability, such as attention maps. On the other side, they are in a broader form of explainability. It’s harder to explain because there’s such big models, and they also have a much longer runtime. I think, overall, compared to other deep learning models, it doesn’t change much compared to models that work with more structured inputs. They, of course, much more challenging to explain.

[0:15:53] HC: Machine learning is advancing quite rapidly now. There are new advancements in the headlines more frequently than ever. Are there any new developments in computer vision or AI more broadly that you’re particularly excited about and can see a potential use case for Aignostics?

[0:16:07] MA: Yes. I guess next, the foundation model is the most exciting development or methods and models that are increasing the model context, such [inaudible 0:16:12]. On one hand, this will be interesting to see how they integrate with vision foundation models. On the other hand, they might enable competition pathology, to process much larger fractions of the whole site image, which is composed of up to several hundred thousands of image patches at once, which was not possible before or just via multiple instance learning, and so on, and so forth. Those methods might enable us to have a more principled approach to working on such data input.

Next, I think the most interesting developments will be at the intersection between computer vision and other modalities. On this, the language between texts [inaudible 0:16:50] chat base pathology system is very interesting if you’re facing most of the work the biological side and omics technologies, which will be very interesting on how. First, people will be able to build the foundation models on technologies, or data that they have much less samples, like proteomics, or any other omics modality. Secondly, it will be very interesting on how researchers will be able to combine those modalities, ideally into multimodal foundation model.

[0:17:20] HC: Is there any advice you could offer to other leaders of AI-powered startups?

[0:17:24] MA: That’s a difficult question. There is so many moving pieces in a startup. Nevertheless, I think the key ingredients for an AI-powered set or team, the data and the technology, which one should make sure to be strong in each of those pillars. Next, I think it’s important to consider market and technology dynamics. Many things are important today, but will not be important tomorrow, or even be commoditized. Meaning, that one needs to build a technology that either makes a difference in the long run, or one must be able to innovate at a very fast pace.

[0:17:54] HC: Finally, where do you see the impact of Aignostics in three to five years,

[0:17:57] MA: I think the next three to five years will be decisive in the computational pathology space. A lot of groundwork has been done, especially on the image and computing division side. There, the field will start to consolidate over the next three to five years. Hopefully, every pathology lab will routinely run digital pathology solutions for each slide and case. On the target identification, and biomarker search side, or even multimodal learning side is going to be more broad. We hopefully will see many proof points how integrating different data modalities such as images help record omics data. This machine learning will lead to new targets or biomarkers and subsequently lead to new drugs and better patient care. We at Aignostics, we do our best to contribute to those developments.

[0:18:43] HC: This has been great, Max. I appreciate your insights today. I think this will be valuable to many listeners. Where can people find out more about you online?

[0:18:51] MA: I think it’s best on LinkedIn, or otherwise, follow Aignostics.

[0:18:54] HC: I’ll link both of those in the show notes, then. Thanks for joining me today.

[0:18:57] MA: Yes. Thanks a lot for having me.

[0:18:59] HC: All right, everyone. Thanks for listening. I’m Heather Couture, and I hope you join me again next time for Impact AI.

[OUTRO]

[0:19:09] HC: Thank you for listening to Impact AI. If you enjoyed this episode, please subscribe, and share with a friend. If you’d like to learn more about computer vision applications for people in planetary health, you can sign up for my newsletter at pixelscientia.com/newsletter.

[END]