Resources for Computer Vision Teams:

LinkedIn – Connect with Heather.
Computer Vision Insights Newsletter – A biweekly newsletter to help bring the latest machine learning and computer vision research to applications in people and planetary health.
Computer Vision Strategy Session – Not sure how to advance your computer vision project? Get unstuck with a clear set of next steps. Schedule a 1 hour strategy session now to advance your project.
Foundation Model Assessment – Foundation models are popping up everywhere – do you need one for your proprietary image dataset? Get a clear perspective on whether you can benefit from a domain-specific foundation model.

Transcript:

[INTRODUCTION]

[00:00:03] HC: Welcome to Impact AI, brought to you by Pixel Scientia Labs. I’m your host, Heather Couture. On this podcast, I interview new innovators and entrepreneurs about building a mission-driven, machine-learning-powered company. If you like what you hear, please subscribe to my newsletter to be notified about new episodes. Plus, follow the latest research in computer vision for people in planetary health. You can sign up at pixelscientia.com/newsletter.

[INTERVIEW]

[0:00:34.1] HC: Today I’m joined by guest, Yair Rivenson, co-founder and CEO of PictorLabs, to talk about virtual tissue staining. Yair, welcome to the show.

[0:00:43.2] YR: Thank you for having me, Heather.

[0:00:45.4] HC: Yair, could you share a bit about your background and how that led you to create PictorLabs?

[0:00:50.2] YR: PictorLabs started in UCLA and so if I go back, you know, on my background specifically, I did the online education in Israel, focusing on optics and then, you know, starting to go more into how to basically build optical systems that use downstream algorithms and then kind of optimize the optical system to make the best use of current algorithms. So, it’s kind of thinking of things in reverse I would say, that’s what I did during my Ph.D. and the end of my master’s.

So really, you know, kind of cutting-edge stuff with those algorithms that were the best available back then, and then I moved to LA, to Professor Ozcan’s lab, the other co-founder of Pictor, and very early, at some point, we started working on deep learning microscopy and it was a very interesting journey. If you go back and see our earliest papers, we basically worked through a lot of resistance, in the beginning, to try to get those papers in and show that deep learning is really the next frontier in enhancing the capabilities of microscopes, something that sounds, you know, maybe trivial these days but back then, in 2016, 2017, 2018, it was still something that people did not readily accept. I am not a pathologist so probably, we understand now and all my background was, you know, standard electrical engineering.

We actually had – we were working a lot with tissue samples in UCLA for multiple microscopy projects and something just connected for us. You know, thinking of, “Okay, so at the end of the day, the way we analyze those.” Not us but like the pathologists analyze those tissues, it’s really looking at all these stains, all these different contrasts that those things are creating and we really asked ourselves a very simple question after like, a few of those deep learning papers, “Can we learn to simulate this chemistry that creates those things?”

So, you know, that moment, I still, you know, kind of remember that, and why not? And you know, we started then and never look back, created the first paper that was published in Nature Biomedical Engineering. It garnered a lot of attention, you know, we were able – you know, at some point, to secure funding for PictorLabs and we’re off to the races after exclusively licensing the IP from UCLA, from that lab that this whole thing originated at.

[0:03:58.3] HC: So, what does PictorLabs do today and why is it important for healthcare?

[0:04:02.9] YR: So, what Pictor is doing is really all about painting the microstructures of tissue sections. So, Pictor in Latin, means painter and this is really what we do. We learn how to paint those microstructures of tissues, giving the autofluorescence of a non-stained tissue section. So, we kind of reverse – maybe not reverse but so, again, maybe I should kind of go back.

Our process is there’s still a biopsy, there’s still a tissue sectioning, and there’s still a glass slide but this is where we kind of depart from the standard process. What we do at that point, once there is a slide, we image it using a fluorescence microscope. The cool thing is that – even if you look at the first paper we did, we’re using off-the-shelf available microscopes. So, for example, you know, that the usual suspects, the ZEISS, Olympus, Hamamatsu, et cetera.

So, we can use their fluorescence microscopes. Today, there are quite a few of those high throughput, high volume scanners that are available for us to use. Then we scan the unstained tissue section and it appears that there is a very rich contrast in the autofluorescence of the tissue. Of course, the pathologists cannot do diagnostics on that auto-fluorescent signal.

So, what we do after that, we take this scanned image, digitized image, and using the machine learning algorithm, we basically transform it into its stained version. So, in other words, now that the pathologist can get multiple stains on the same tissue section. It’s very quick, I mean, it’s GPU speed and you know, there is a huge market right now for GPUs. So, this market is pushing the manufacturers to create better, faster, more cost-effective GPUs per year.

So, we’re actually gaining a lot from that type of amazing acceleration in the performance of those GPUs and this will enable the scaling and the speed of our algorithms. The other cool thing is, you end up, basically, cutting slides and all these things are being performed digitally. So, you start you know, thinking, “Okay, do I need to have all these chemicals in my lab? Do I need to have all the supply chain that is related to those chemicals?”

And then, again, I think the most important factor for the healthcare system, for the patient is the fact that you can get all the results, all the work, and all the different stains from a single tissue section very, very fast. What is fast? What is very fast? So, we did like a mockup experiment or a breast tissue, a breast cancer workup. You know, if you look at it, it typically takes two to three days and this is, you know, from a very good lab.

This will, we include, you know, the HNE, P63, ERPR, Virtual Ki-67-ish, PD01, et cetera. This really will take you two to three days, practically it’s probably going to take you way more than that and our technology has the potential to give you all the workup in about 45 minutes. The entire work of the entire stain work and the other cool thing is that you are left with a lot of tissue to run molecular studies.

One of the complaints we keep on hearing is that about 30% of the tissues or the biopsies are just being depleted before they have a chance to go to molecular and that’s something that you know, our technology can also alleviate by the fact that you only need one tissue to multiplex your entire stain work.

[0:08:49.0] HC: So, this is an important process for reducing some of the bottlenecks in the current pathology lab.

[0:08:55.9] YR: Indeed, indeed. It’s again, reagents, its chemistry, and the cool thing again is also, it doesn’t really change the workflow in a sense that, you know, there’s still the slides, there’s still everything that’s needed and also the pathologists, they don’t need to retrain themselves on the downstream side of it. There’s no retraining on their end, it’s like they’re reading their stains in a standard away.

I mean, other than the fact that it’s not under a microscope, luckily. It’s all on the computer screen or in the other screens that they like to use but there’s no training for them to use this thing. It takes them, you know, a few minutes to basically get used to the fact that “Oh, suddenly, you can do multiple stains on the same tissue section.” That’s it.

[0:09:52.8] HC: So, how do you use machine learning to do this virtual staining?

[0:09:56.3] YR: Great question. So, the machine learning is really the engine behind this whole, the virtual staining. In a sense, that’s what takes those images, coming from autofluorescence of the unstained tissue section, and converts it into a stain that pathologists can use for their diagnostics, and the way they do it really is supervised deep learning. We put a lot of efforts in registration of images to the pixel, sometimes sub-pixel level.

Basically, we take our machine learning architecture and again, make it learn, as supervised learning prone of mapping, autofluorescence on stained tissue section to their stained version and the stained version could always do H&E. So, still the most important stain but then, we have shown it, those type of framework can work for, you know, if we go in the order of complexity, special morphological stains, and then more structural, IDC stains that is, and then you’re going to you know, more – even more complex, more of the immunophenotyping, et cetera. So, that’s you know, how we would use machine learning with our technology.

[0:11:22.4] HC: One of the largest challenges, more broadly with generative models is hallucinations. Is this something that you’ve seen and if so, how do you mitigate it with – for virtual staining?

[0:11:33.4] YR: So, the first thing is, yes, it is something that is known in the industry. I think we’ve mitigated in multiple ways, I think the first way is the one I mentioned earlier. So, we pose the problem as a supervised learning prompt and you know, we put so much effort on doing the registration as accurately as possible and that’s, you know, one of the major ways for us to avoid hallucination or anything like that.

I think, also, you know, a typical hallucination that we see could be artifacts on the actual slide that we scan, meaning the physical slide. For example, dust. Another thing that is a major issue with machine learning for pathology is out of focus. So, when the sample is being scanned, it’s out of focus. Again, any other artifact of that nature, there are a few known ones but again, this is not a problem specifically for Pictor.

I think it’s a problem for any machine learning algorithm. The results when that happens, results, you know, a little blurry or artifactual. It gets – a professional can very quickly realize that that is something that is artifactual but we’re also developing all sorts of mechanisms and I think, not just us, probably most of the industry are doing similar things, basically creating all sorts of alerts for the users that there might be a problem with the region of the image.

For example, it was out of focus. These things are very easy to detect, for example. The other thing is, I like to kind of think of this as you know, if I kind of zoom out on the problem and not just go from the perspective of generative models or AI, I will say that you know, one of the major things is the chemistry itself, and again, maybe it’s not always, we don’t always think about, but the chemistry itself is not perfect, let’s put it this way.

So, part of what we do is, I like to think about it as you know, as the next step maybe in the evolution of those things because what we do, we only – the best things to go into our algorithm. So, we’re trying to eliminate, we will eliminate any regents or slides that have you know, off-target stains or maybe low-intensity stains. We’re trying to get those things out of the models because, at the end of the day, the network is as good as the data that it learns from.

So, that’s something we’re trying to standardize. Again, as pathologists, you know, they learn to live with those types of artifacts but we’re trying to really go to the next step in the evolution of stains and basically, avoid those types of things moving forward and maybe get rid of those artifacts that are also present in chemical stains.

[0:14:56.4] HC: I guess, the next component of this is not just, you know, making sure you can not have hallucinations and properly handle artifacts but to validate that your results are what you expect them to be, whether it’s for pathologists or whether it’s for another algorithm, however those images are going to be processed. How do you go about validating your virtual staining models?

[0:15:18.9] YR: So, we are doing it in the following way, that the way we really like doing this is blinded reviews. So, it’s sort of, if you think about it in the way that they validated digital pathology in some senses. So, they let pathologists do their diagnostics on tissues that were stained and then you know, they look at it through the standard microscope, and then there was a washout period of two to three weeks.

And then they ask them to look at the digitized versions and render their diagnostics and there was a – they call it, non-major discordance between the digitized version and physical versions that the pathologist uses to look at on standard microscope. So, we’re following the same path but of course, in our case, it’s comparing to digital versions of a slide. One digital version is you have a standard process of staining and then scanning with usually a bright field scanner, right?

Let’s put it that way to make it simple and that’s you know, cohort one. The second cohort is coming from the same group of slides but they’re unstained and then virtually stained version, and basically, we’re doing those types of concordant studies and this is you know, how we validate our models and we also have the ability to look at those things in a matter that is maybe not a side by side or you know, from different cuts.

So, we can look at particular areas and solve a specific challenge if there are any for those types of areas but really, the way we’re doing it is really following the path of what they’ve done to validate to standard, the digital pathology scanners.

[0:17:26.7] HC: So, we’ve talked about a couple different challenges that you encounter and dealing with histological samples. Are there others or are there other major ones that we have to overcome in making this type of model work?

[0:17:38.2] YR: Yes. So, certainly and this is again, something that everyone works and that works in digital pathology knows, the slides are pretty big in terms of they’re not – their gigabyte weight on the hard drives and you know, we are working on a cloud platform. So again, we’re bandwith dependent. There is a lot of good movement on that region, on that field. I mean, the bandwidth is luckily keep some increasing for potential clients.

But again, as we’re cloud-based, you know that’s always something that we need to be mindful of. There are all sorts of solutions for that as well, impression algorithms, etcetera, but I think really the problem is that those are quite a big slides, and if you could end up with a few tens of gigabytes, first time when you look at the data, the autofluorescence data. It also again, I just said that the slides are big and this is a disadvantage.

But also an advantage because when we train our algorithms because we train on basic learning and transformation that mimics chemistry, every slide contains many fields of view for us to learn from, and because we’re trying to learn the pixel level, it gives the algorithm a lot of opportunities to learn from every one of those slides. So, you know, we can basically create models from 80 to 100 slides.

We can create very robust models, which is you know a big advantage for us but again, as I said, they are pretty big so we are – that the way to kind of resolve it is going with a high bandwidth and you know, compression technologies that we are incorporating.

[0:19:36.8] HC: So, this process of virtual staining, it will change the workflow on a pathology lab. It will simplify it but it will change nonetheless. How do ensure that the way you deploy this technology will fit smoothly into the clinical workflow and provide the doctors with the right kind of assistance?

[0:19:55.1] YR: Right, so as we’ve said, this is more of an upstream application, so most of it will affect the laboratory and laboratory economics in terms of time that they’re going to save, in terms of free agents, in terms of the first result being available to review for the pathologist especially as those are gaining more and more complexity, in terms of the stain work that needs to be done. This is where pathologists will see the highest gain.

I would say for the labs, it’s a sense, you know, we will still have the same type of workflow with cutting the slides. I think you know, it’s still compatible in some senses with the workflow. It’s just that instead of let’s say, putting things into one of those stainers. For example, those automatic stainers or doing it manually, you will have basically a one-stop-shop for all your stain work. You’ll just need to put it into a microscope.

Again, we don’t sell or market those microscopes, they are already out there, which I think is a big advantage. There are a few thousands of those already deployed locally and you just put those samples in there, you set up imaging, and sometimes, you kind of forget about it at that point. It’s just going to run and it’s going to perform all the stain work that you need. It’s all going to be from one instrument.

The lab manager doesn’t need to worry anymore about things such as, “Do I have that reagent or do I have this reagent? Did I validate those probes?” or anything of that nature. So, it kind of simplifies and flattens the process because there won’t be any difference even if you think of –let’s go with the most simple thing again for the sake of argument, up until you know, the most complicated stain work out there. Labs don’t even want to try in some sense because it’s a mess of doing those. Some of those special stains, for example, that no lab wants to do those. So, this is where I think there’ll be a major advantage for the labs and again, downstream for the pathologists that will review it in such a short time and get all their stain work done immediately and they don’t need to wait or think about it anymore. I think it will be a great advantage for them.

[0:22:54.7] HC: You mentioned earlier that a number of the publications related to this virtual staining were done at UCLA before establishing PictorLabs. I think I believe you’ve continued to publish after starting the company and some of those publications are in collaboration with UCLA and perhaps other universities. How do you collaborate with economic partners while staying focused on business objectives?

[0:23:19.7] YR: Great question. So, you said we’re a university-based startup, so we have deep roots in everything that has to do with academic collaborations. You know, we put a few posters and papers, as you mentioned, in the last three years with UCLA, with USC. So, one of the – basically, the first pathologist who ever laid eyes on virtual staining, Dr. Dean Wallace, now in USC, that we work with him also extensively, and we have some – a lot of work right now with University of Maryland.

Again, for us, it’s – also, by the way, we’re working on academia. You know, we always think of evidence generation. We also work on evidence generation with some commercial entities and you know, we believe that they’re as powerful and deep as some of our academic collaborations. Again, if you kind of look at it the way we kind of see it is that I should say, is you know academics is really our key to establish capability.

And what’s good about working with academics is they provide you real-time feedback in some senses on performance and stuff that they want to see improved. They’re not shy about it and it’s a great experience for us. If I maybe want to put some more color into this, you know our goal really with those academic collaborations is to validate the efficacy of virtual stains for specific use cases.

So, in many cases, the way we’re doing it is trying to use a virtual stain models that we already developed in-house and then fine-tuned or then fine-tuned those models for their specific study use case. The way we do a typical study would look like the one I think that we recently performed with University of Maryland and was presented a couple of weeks ago in the US.

We’ve shown that, for example, virtual H&E appears to be fit for purpose for or non-inferior to chemically-stained H&E in a lymph node evaluation and then included quality checks, initial impressions by the pathologists, and establishing a differential diagnostics. In lymph node, there is a lot of immunostains involved. So, it also – you have the same result in guiding the immunostains ordering pattern.

So, in other words, you see in those studies no major discordances in between virtual and chemical H&Es. You know, if you look at all sorts of things such as architectural evaluation necrosis and order of immunostains, and actually something that repeats itself right now is that many of those studies you see that the quality that pathologists assign to the virtual H&E is sometimes higher than the chemical H&E.

And again, just because the, as I mentioned, we are trying to select the best things that goes into the training of the algorithm, so the algorithm comes out very consistent in that sense, unlike chemistry-based. Basically again, kind of wrapping this up, we’re really looking at the academic partners in order to establish our evidence generation. Of course, the priorities are always for us is the commercial aspect but the credibility is still academia is the king for that.

[0:26:57.8] HC: Is there any advice you could offer to other leaders of AI-powered startups?

[0:27:01.9] YR: Oh, yeah, quite a few. The most important thing is to get a product. Again, I’m probably not saying anything that is groundbreaking here, but really, it’s all about getting your product fast into the market, into academic partners. Really get it fast, get feedback, you know, and improve the product based on that type of feedback. That’s super important. Of course, you know there should be a good plan off the runway.

So, you need to kind of deeply analyze you know, your business in a sense of, “Okay, do I have proprietary data or is it mostly based off public data and some clever algorithm that we created?” So, basically, what is the source of the data because data, forget about you know, buying it, it’s also how do you get the data. So, in our example, you have to purchase microscopes, okay?

And because you know we started from scratch, so we have to buy two microscopes in order to capture autofluorescence from unstained slides. So, okay, you start saying, “Okay, I am going to invest this amount of money into those microscopes and then I am going to invest this amount of money to buy slides, and then I need to think of the storage. I need to think of the computing power and resources.” And then those things start scaling up.

So, the data and the data cost needs to be top of mind and really thought through when you start your AI-powered business. I think the third advice I have in mind right now is to really trying to think very early on, once you establish this data, how do I get the data and how do I store it and etcetera, and how do I train those models. It’s very important for you to understand how to create – How to basically automate the data in just then as fast as possible, automate as much as you can.

That’s another key part, the more you automate, the better off you’ll be in the long run. So, maybe it’s an investment you know, that you need to do early on as well but it will be – it will yield a lot of dividends for you moving forward and maybe I have one more thing, you know, it depends again on your product.

And so we’re, in some senses very lucky we fall into because it is all around, you know, every facet of medical sciences. You know, going from safety studies, toxicology, research, drug development, lab work, clinical, you name it. So, it’s kind of all around and you have a lot of market opportunities but also you know, you want to – if you have a product that is extremely very specific and it will only go to free patients, of course, get your grip on the regulatory part, especially with the regulatory AI is forming right now. You know, get some good grip on that early so you can start putting the right controls early as well.

[0:30:59.2] HC: And finally, where do you see the impact of Pictor in three to five years?

[0:31:03.6] YR: I would like to see Pictor replaces many stains in as many service lines as possible. That’s where I want to see Pictor at three to five years from now. I think this technology has great advantages and I think in many, again, facets of pathology that I mentioned, starting from let’s say, pharma work and drug development, there’s so much value there by the fact that you can create right now a virtual block out of the single slide.

And then you don’t need to run through all these very precious blocks that were purchased by these pharma companies for the drug development program evolves and you have – you can take one cut, run your entire stain work on it. So, it’s a huge, huge advantage for those. It could speed up the process of drug development. It can save tremendous amounts of costs and so then I think I would like to see Pictor being used a lot in those frameworks.

Everything that has to do with pharma and drug development and then of course, you know we’d like to – Pictor will probably be a part of lab work for a few of the service lines. I predict that those will probably be the ones where pathologists are used to work with special stains or used to work with quite a few immunostains. Those type of service lines, this is where I think it will be adopted and used and this is where I want to see Pictor in three to five, for sure.

[0:32:54.2] HC: This has been great. Yair, I appreciate your insights today. I think this will be valuable to many listeners. Where can people find out more about you online?

[0:33:02.9] YR: So, we have a website, PictorLabs, one word, pictorlabs.ai. So, we have you know, our products there and our publications. You can find additional information there. We have our LinkedIn page, which is getting more and more active right now. So, these are the channels you can find us, yeah, find us at.

[0:33:28.3] HC: Perfect, thanks for joining me today.

[0:33:31.0] YR: Thank you. Thank you for having me.

[0:33:32.8] HC: All right everyone, thanks for listening. I’m Heather Couture and I hope you join me again next time for Impact AI.

[END OF INTERVIEW]

[0:33:44.1] HC: Thank you for listening to Impact AI. If you enjoyed this episode, please subscribe and share with a friend, and if you’d like to learn more about computer vision applications for people in planetary health, you can sign up for my newsletter at pixelscientia.com/newsletter.

[END]