Machine Learning Domain Shift Assessment

Are you confident that your machine learning models will generalize to images from different labs and scanners?

Have you thoroughly validated your models? What variations should you even be testing for?

Sometimes the distribution change causing degraded performance is obvious, like a different scanner, lab, or patient population. But it can also be hidden or caused by multiple factors.

Gathering a large, diverse training set may be the ideal solution but is often not feasible – especially for medical images.

What do you do?

Perhaps you have a few techniques in your toolkit like color augmentation or stain normalization to handle the obvious changes in image color.

But have you ever taken a step back to fully understand the batch effects in your data?

Or what types of domain shifts you might encounter in the future?

Generalizable models are important for multiple reasons:

  • So that they will make correct predictions on previously unseen images

  • To ensure they work with images from a variety of scanners, labs, and patient populations

  • To get regulatory approval for your product

  • So that they are unbiased and provide the most valuable information to researchers, clinicians, and patients

Get a clear strategy for validating your models and improving their generalizability

This assessment helps pathology startups handle challenging datasets by:

  • Identifying sources of variation, batch effects, and domain shifts

  • Properly validating models to reveal how they might fail

  • Applying data-, image-, and model-centric techniques to make models more generalizable

We start by understanding the variations between your training and test sets, as well as the batch effects within your training data, what physical processes or differences in patient distributions may have caused them, and how these variations impact your model performance. Then I’ll devise a plan for you to improve model generalizability with adjustments to your training data and your algorithms.

Here’s how it works:

I’ll interview some of your leadership, pathology, and machine learning team members to discover your overall goals, the variations present in your training and inference data, and your current approach to validation and generalizability. With this knowledge, I’ll deliver a report with recommendations for the following three areas:

Part 1:
Sources of variation
Par 2:
Analytical validation of models
Part 3:
Improve generalizability
  • Sample preparation variability
  • Image acquisition batch effects
  • Patient subgroups
  • Cross-validation
  • Out-of-cohort predictions
  • Metrics
  • Stratification
  • Results analysis
  • Approaches for each type of variation identified
  • Data-centric
  • Image-centric
  • Model-centric

A follow up call two weeks after report delivery is also included to ensure that you have all the information you need to move forward.

Results you can expect from this assessment:

  • Save time by targeting the variations and batch effects in your data instead of wasting time on unsuccessful approaches

  • Discover challenges early in product development so that they don’t become surprises later

  • Get your product to market faster

Ready to get started?

Are you ready to get your project on the path to success? Apply for an ML Domain Shift Assessment by clicking the button below. The price is $6,000, which could save you many thousands down the road.

100% money-back guarantee!

If you don’t have a clearer perspective on your domain shift challenges after receiving the report, just request a refund.

Don’t just take my word for it…

Heather has accelerated the development of our machine learning projects by challenging the status quo and introducing the team to new approaches. She helped us get out of the echo chamber and brought an outside perspective with very relevant alternative approaches that would have taken a lot longer to do on our own.

-- Joe Sturonas, Ancera, VP of Systems Development

Heather has deep understanding of digital pathology and machine learning and their application to whole slide images. Her in-depth review of the current state of the art research has enabled Gestalt to rapidly focus our machine learning efforts on approaches that are yielding value for our pathologists and their patients.

-- Brian Napora, Gestalt Diagnostics, VP of Sales & Product Management

Heather provided practical feedback and guidance, which allowed us to quickly take steps in the right direction and improve the performance of our models.

-- Darragh Maguire, Deciphex, Artificial Intelligence Engineer

Still have questions?

What happens after I apply?

We’ll schedule a short call to confirm that this package is suitable for your goals. Then I’ll send over a contract. Once you submit your payment, you’ll receive an email with a link to a questionnaire that I’ll use to begin scheduling interviews with some of your team members.

Will you sign an NDA before we start?

The contract for this ML Domain Shift Assessment will include confidentiality and non-disclosure provisions.

Can’t we just gather a larger training set?

If you’re able to do this, go for it. But the key is not only a larger dataset – diversity is critical. You need to understand the variations in your data and what subpopulations are underrepresented in order to focus your data collection efforts. For many medical applications, it’s not possible to cover all sample preparation techniques, scanners, and patient populations. The alternative is to narrow the scope of your product. But there will still be variations, even with samples imaged on a single scanner. This assessment will help you understand those variations and guide how you handle them.

We haven’t even started training models yet. Is it too early to think about generalizability?

No, it’s not too early to think about it. If you’re getting ready to start acquiring data, you want to be thinking about ways to reduce the sample processing and image acquisition variations. If you’re scanning your own slides, you might be able to include a color calibration step in your process to reduce variations from different scanners. You should also be thinking about the patients in your training set and how you might obtain one or more external cohorts for testing. These are all topics covered in the assessment.

We’ve already analyzed our datasets and understand the domain shift challenges. We just need help making our models more generalizable.

If you’ve already completed this analysis, you might consider starting with my ML Domain Generalizer instead. The focus of that service is on identifying and applying advanced data-, image-, and model-centric approaches to improve model generalizability.

Our project is different from other applications. It’s not just H&E histology. Is this a problem?

No. If you’re using something other than H&E or you’re developing a novel medical imaging device, it’s even more important to analyze the variations in your data. Less common modalities are less understood, so we might need to dig a little deeper to identify the sources of variation.

What’s next after the assessment? Can you help us with implementation?

After completing this assessment, we have a few options. Your team could go off and implement the recommendations outlined independently. However, my clients have found that their projects are completed more efficiently with continuous guidance. You can sign up for my Advisor service and receive advice as your team progresses. If, after implementing the first line solutions to improve generalizability, you find the need for more advanced approaches, my Domain Generalizer is a good choice.

Still not sure if this is the right package for you?

Schedule a free Strategy Session, and I’ll help you determine if this is a good fit.

Who are you, anyway?

I’m Heather Couture, Machine Learning Consultant and Founder of Pixel Scientia Labs. I have 17 years of experience in computer vision and machine learning, 11 of those with applications to pathology. I completed an MS at Carnegie Mellon University and a PhD in Computer Science at the University of North Carolina, where my research focused on developing deep learning methods to predict breast cancer molecular biomarkers from H&E. I have published in top-tier computer vision and medical imaging venues. I write regularly on LinkedIn, for my newsletter Pathology ML Insights, and for a variety of trade publications. You may have heard me on my podcast Impact AI or at conferences.

I help my clients get to market faster by building more generalizable computer vision models. I make use of the latest ML research to amplify their results and support their in-house team for the long term. My mission is to fight cancer with AI – and I do that by strengthening the ML component of my clients’ most impactful projects.

Availability is limited

I only offer two Domain Shift Assessments per month. Scheduling is first come, first served. The sooner you apply, the sooner you will have a clear path to more generalizable models.