Distribution Shift Diagnostic

Are you confident that your computer vision models will generalize to the variations and batch effects in your inference images?

Have you thoroughly validated your models? What variations should you even be testing for?

Sometimes the distribution change causing degraded performance is obvious, like a different imaging device or geographic location. But it can also be hidden or caused by multiple factors.

Gathering a large, diverse training set may be the ideal solution but is often not feasible – especially for medical images.

What do you do?

Perhaps you have a few techniques in your toolkit like color augmentation or normalization to handle the obvious changes in image color.

But have you ever taken a step back to fully understand the batch effects in your data?

Or what types of distribution shifts you might encounter in the future?

Or searched for spurious correlations that may be biasing your results?

Generalizable models are important for multiple reasons:

So that they will make correct predictions on previously unseen images
To ensure they work with images from a variety of scanners, labs, patient populations, geographic regions, vegetation types, etc.
To get regulatory approval for your product
So that they are unbiased and provide the most valuable information

Get a clear strategy for validating your models and improving their generalizability

This diagnostic helps organizations using computer vision handle challenging datasets by:

Identifying sources of variation, batch effects, spurious correlations, and distribution shifts
Properly validating models to reveal how they might fail
Applying data-, image-, and model-centric techniques to make models more generalizable

We start by understanding the variations between your training and test sets, as well as the batch effects within your training data, what physical processes or differences in data distributions may have caused them, and how these variations impact your model performance. Then I’ll devise a plan for you to improve model generalizability with adjustments to your training data and your algorithms.

Here’s how it works:

I’ll interview some of your leadership, domain expert, and machine learning team members to discover your overall goals, the variations present in your training and inference data, and your current approach to validation and generalizability. With this knowledge, I’ll deliver a report with recommendations for the following three areas:

Part 1: Sources of variation	Part 2: Analytical validation of models	Part 3: Improving generalizability
Pathology: Sample preparation variability Image acquisition batch effects Patient subgroups Remote sensing: Variations across geographic regions Seasonal and lighting changes Weather conditions	Cross-validation Out-of-sample predictions Metrics Stratification Error analysis	Approaches for each type of challenge identified Data-centric Image-centric Model-centric

Part 1:
Sources of variation

Part 2:
Analytical validation of models

Part 3:
Improving generalizability

Pathology:

Sample preparation variability
Image acquisition batch effects
Patient subgroups

Remote sensing:

Variations across geographic regions
Seasonal and lighting changes
Weather conditions

Cross-validation
Out-of-sample predictions
Metrics
Stratification
Error analysis

Approaches for each type of challenge identified
Data-centric
Image-centric
Model-centric

A follow up call two weeks after report delivery is also included to ensure that you have all the information you need to move forward.

Results you can expect from this diagnostic:

Save time by targeting the variations and batch effects in your data instead of wasting time on unsuccessful approaches
Discover challenges early in product development so that they don’t become surprises later
Get your product to market faster

Ready to get started?

Are you ready to get your project on the path to success with a clear strategy for handling distribution shifts? Apply for a Distribution Shift Diagnostic by clicking the button below. The price is $10,000, which could save your team many iterations of experimentation – and hundreds of thousands of dollars.

Apply Now

Don’t just take my word for it…

Heather has accelerated the development of our machine learning projects by challenging the status quo and introducing the team to new approaches. She helped us get out of the echo chamber and brought an outside perspective with very relevant alternative approaches that would have taken a lot longer to do on our own.

-- Joe Sturonas, Ancera, VP of Systems Development

Heather has deep understanding of digital pathology and machine learning and their application to whole slide images. Her in-depth review of the current state of the art research has enabled Gestalt to rapidly focus our machine learning efforts on approaches that are yielding value for our pathologists and their patients.

-- Brian Napora, Gestalt Diagnostics, VP of Sales & Product Management

Heather provided practical feedback and guidance, which allowed us to quickly take steps in the right direction and improve the performance of our models.

-- Darragh Maguire, Deciphex, Artificial Intelligence Engineer

Still have questions?

What happens after I apply?

If needed, we’ll schedule a short call to confirm that this package is suitable for your goals. Then I’ll send over a contract. Once you submit your payment, you’ll receive an email with a link to a questionnaire that I’ll use to begin scheduling interviews with some of your team members.

Will you sign an NDA before we start?

The contract for this Distribution Shift Diagnostic will include confidentiality and non-disclosure provisions.

Can’t we just gather a larger training set?

If you’re able to do this, go for it. But the key is not only a larger dataset – diversity is critical. You need to understand the variations in your data and what subpopulations are underrepresented in order to focus your data collection efforts. For many medical applications, it’s not possible to cover all sample preparation techniques, scanners, and patient populations. The alternative is to narrow the scope of your product. But there will still be variations, even with samples imaged on a single scanner. This diagnostic will help you understand those variations and guide how you handle them.

We haven’t even started training models yet. Is it too early to think about generalizability?

No, it’s not too early to think about it. If you’re getting ready to start acquiring data, you want to be thinking about ways to reduce the sample processing and image acquisition variations. If you’re scanning your own slides, you might be able to include a color calibration step in your process to reduce variations from different scanners. You should also be thinking about the patients in your training set and how you might obtain one or more external cohorts for testing. These are all topics covered in the diagnostic.

We’ve already analyzed our datasets and understand the distribution shift challenges. We just need help making our models more generalizable.

If you’ve already completed this analysis, you might consider starting with my Monthly Advisory Services instead. The focus of that service is on a clear set of priorities to accelerate your model development. Guidance is customized to address the specific challenges that you’re facing – one of which might be generalizability.

What’s next after the diagnostic? Can you help us with implementation?

After completing this diagnostic, we have a few options. Your team could go off and implement the recommendations outlined independently. However, my clients have found that their projects are completed more efficiently with continuous guidance. You can sign up for my Monthly Advisory Services and receive advice as your team progresses, even diving deeper into more advanced approaches to identify and mitigate challenges with distribution shift, batches effects, and bias.

Still not sure if this is the right package for you?

Schedule a free Discovery Call, and I’ll help you determine if this is a good fit.

Availability is limited

I only offer one Distribution Shift Diagnostic per month. Scheduling is first come, first served. The sooner you apply, the sooner you will have a clear path to more generalizable models.