Resources for Computer Vision Teams:

LinkedIn – Connect with Heather.
Computer Vision Insights Newsletter – A biweekly newsletter to help bring the latest machine learning and computer vision research to applications in people and planetary health.
Computer Vision Strategy Session – Not sure how to advance your computer vision project? Get unstuck with a clear set of next steps. Schedule a 1 hour strategy session now to advance your project.

Transcript:

[INTRODUCTION]

[0:00:03] HC: Welcome to Impact AI, brought to you by Pixel Scientia Labs. I’m your host, Heather Couture. On this podcast, I interview innovators and entrepreneurs about building a mission-driven, machine-learning-powered company. If you like what you hear, please subscribe to my newsletter to be notified about new episodes. Plus, follow the latest research in computer vision for people in planetary health. You can sign up at pixelscientia.com/newsletter.

[EPISODE]

[0:00:34] HC: Today, I’m joined by guest Marshall Moutenot, co-founder, and CEO of Upstream Tech, to talk about environmental monitoring. Marshall, welcome to the show.

[0:00:43] MM: Thanks for having me.

[0:00:44] HC: Marshall, could you share a bit about your background and how that led you to create Upstream Tech?

[0:00:48] MM: Yes, sure. I will start at the beginning, because I think it’s helpful to understand some of the decisions I’ve made. Growing up, both of my parents were self-employed. My dad’s a recording engineer and music producer, and my mom became a real estate agent when the music industry started to struggle. I’ve never really been exposed to the nine-to-five quintessential lifestyle. I think that planted in me early on a desire for the kind of life that entrepreneurship can afford. So, hard work, but on your own terms.

I was a total nerd growing up. I had a knack for technology. I started building and selling computers in websites in middle school and grew it into a tiny little business. From there, I also ran a computer camp in high school, where I coordinated sessions for sometimes hundreds of kids spending Lego robotics to web programming. That was my first taste of a mini business and my love for spreadsheets really formed there. I think that was the first time I really realized like, “Okay. Doing your own thing can be really fun.”

I went to school in Boston for computer science. While I was there, got to experience the big tech side of the equation, Microsoft and Mozilla, followed by a startup called Crashlytics that became part of Twitter. That was awesome, because it taught me quickly that I do best under pressure. Again, back to my parents, on my own terms, and startups where I wanted to be. I left Twitter to start a company with then HubSpot executives, David Cancel and Elias Torres, company called Drift. Talk about pressure, it was hypergrowth, shooting for the stars. We iterated on many early product ideas, and ultimately landed on a MarTech company that went on to become quite successful.

That level of intensity was awesome. What I learned from that, similar to what I took away from the big tech companies was that intensity, great. But the substance of the work was as important as the environment. If I was going to put in long days to hire all my pals, a lot of emotional energy, I was going to do my best work working on something that mattered to me on a personal level. So, don’t get me wrong. MarTech can be really cool, but it’s not my personal raison d’etre.

I started something with Alden, who I haven’t mentioned yet, but worked with me at every one of those jobs since graduating college. We’re basically inseparable co-worker, co-founders. We looked at two areas that we cared about; education and the environment. This was before, I think, the latest climate tech boom. It was a little less structured as we did our investigating. The latter immediately clicked as a place where the skills we had could build something that didn’t exist yet, and so, we started Upstream.

[0:03:56] HC: What does Upstream?

[0:03:58] MM: Upstream develops software solutions. We have two products today. One that is forecasts to power a renewable grid, and the other Lens to expand access to satellite imagery.

[0:04:11] HC: Could you tell me a little bit more about each of those?

[0:04:13] MM: Yes. We started Lens once we noticed that there was a real challenge as nature-based climate solutions, and environmental conservation projects we’re scaling. Which is that, monitoring was a real challenge at scale. Traditionally, a lot of these projects would be monitored in person. So, someone would hop in the car, drive to a project, get out of the car, look around, make sure that either nothing has changed or kind of growth or restoration that was expected was occurring, get back in the car, and drive to the next project. It was just super inefficient, super costly, and not something that scaled.

We set out to build something that allowed that kind of monitoring process remotely, using satellite imagery, and an application that really supported, and stewarded the remote monitoring process. That’s what we created and became now Lens, which is an awesome product, as hundreds of organizations using it to monitor projects around the world. As we were doing that, and this is always a real challenge for a startup, having two products. But we started hearing more and more that water was a challenging resource to monitor. And when you think about climate change, really, is at the end of the day, water change. Change in precipitation, an increase in extreme weather and drought.

We started doing some research on the side to see, “Okay. Can we use some of the same data that’s in Lens to understand the past, present, and future of water?” That R&D was very successful and was the birth of our second product, HydroForecast, which leverages machine learning, as well as those satellite images and weather forecasts, to forecast how much water will flow through rivers and streams in the future. That’s really important in the climate equation for a couple of reasons, obviously, adaptation, because water is an invaluable natural resource. But also, from the perspective of energy. Hydropower continues to be an essential asset class for our grid.

The better forecasts we can provide to hydropower, the better hydropower can produce electricity, complement the kind of intermittent generation from wind and solar as it grows. So, that’s been really exciting for us, and HydroForecast has found immense amount of success, working with energy utilities that own and operate hydropower. In addition to organizations that are dealing with water as a natural resource and trying to understand the risks associated with that in the future.

[0:07:04] HC: How do you use machine learning in this technology?

[0:07:06] MM: Yes. HydroForecast is the primary application or product that uses machine learning. Breakthroughs in time series forecasting, long, short-term memory networks, if you want to get into the details are really what have enabled the kind of breakthroughs in forecasting accuracy that we’ve been able to bring to market with HydroForecast. Basically, you can kind of categorize forecasts into two different camps. You have the physics and conceptual modeling, which as humans, we’re going to try to describe the environment through equations. We’re going to try to describe physical interactions with mathematical equations, and then, use data to calibrate the rest. That’s how, in the past, forecasts have been made, whether they’re miracle weather predictions or hydrology forecasts. Hydrology is the science of water.

What these new machine learning models that we’re employing allow us to do is to provide enough data to the model to actually create those same equations to describe physical interactions. But do so on a scale that was previously impossible, and in a way that is incredibly rapid and fast evolving. So, we’re able to integrate new inputs like satellite imagery, or a new weather forecast in a way that is super efficient and drives accuracy.

Just to illustrate kind of the result of that, we competed a few years ago in a yearlong competition held by the Bureau of Reclamation, along with a number of energy utilities in North America. Every morning, us, government forecasts, and utilities in-house forecasting teams would submit their forecasts for the day. A third party crunch the numbers and figured out who was most accurate. Everyone was super surprised that these new AI models that we were using were outperforming everything, including government forecasts had been developed and iterated on for decades. Adaptability of these models is just something that is really exciting for the field overall.

[0:09:26] HC: You mentioned the satellite imagery and weather forecasts. Are these the main inputs that go into these models?

[0:09:32] MM: Yes. Those are definitely the main inputs. The weather forecasts are essential, and the satellite imagery is something that is a bit new. In the conceptual model case, that can be very difficult to incorporate. What we’re able to use that for is to get almost real-time. Every day, we integrate an image that gives us information about the percent of the basin that’s covered by snow and where within that basin the snow is. How much of the basin is vegetated and how that’s changing over time. Those are just examples of ways that we’re able to incorporate this spatial information that’s fast updating in a way that really helps contribute skill to these forecasts.

[0:10:16] HC: What kinds of challenges you encounter in working with these different types of geospatial data and training models based off of them?

[0:10:23] MM: Those data are only challenges. I think, there’s – I guess you could call it a meme in the Earth observation space, which is, just try to buy a satellite image. It’s incredibly challenging. And, almost as if the companies that produce the images do not want to take your money. That was a core thesis that went into the development of the first product that I talked about, Lens. How do we make the acquisition of satellite imagery easy for non-technical organizations, a tiny land trust with just a couple of employees with no GIS background? How can we make it so that they can easily acquire a recent Airbus image, so they conduct remote monitoring?

The same is true for HydroForecast. To produce your own hydrology model, you have to go and collect all of the necessary weather data, as well as the satellite imagery as inputs to your model. Even if there are open-source models, that process of collecting the inputs is incredibly challenging. To get the latest weather forecast for your location, that’s trivial. Anyone can go and look that up. But what’s hard is to get last year’s forecast across an entire state. That kind of archival access across space and across time, that’s where weather forecasts get gnarly. It’s a kind of dimensionality, I think that people don’t really think about when they think about weather forecast.

You have space, you have a new forecast issued, sometimes many times a day. Then, each of those forecasts go forward in time. So, it is this multi-dimensional problem that becomes very cumbersome and very challenging. That’s why I think right now, we’re kind of in a weather startup renaissance, where organizations that can develop weather derivative products like our forecast are really succeeding. Because just dealing with those kinds of data is an immense challenge for organizations.

[0:12:40] HC: On your website, you talked about the HydroForecast modeling as being theory-guided. What does it mean for a model to be theory-guided? How do you accomplish this?

[0:12:49] MM: Yes. I think a lot of people think of machine learning as a panacea, you can kind of slap it together, and it’s going to produce something better. That’s a real risk. First, the misconception, of course. But it’s a real risk as we’re talking about solutions for a changing climate, for building resiliency in the face of extreme weather, or sea level rise. Say, I had data for a single hydropower reservoir, the inflow data, so how much water was coming into the reservoir and the level of that reservoir. I could slap together some off-the-shelf machine learning models, trained on that data, and correlated with, say, rainfall in the area, and come up with something that looks okay on the surface. But the issue there is that, as soon as a storm came that wasn’t represented in that historical data, the model becomes untrustworthy. It doesn’t know how to make that correlation.

What’s really exciting, I think, in the field of machine learning, is this concept of integrating and incorporating domain knowledge and scientific principles into the design and development of machine learning algorithms. It means that you build upon decades of the scientific field. So, for us, that’s hydrology, and use that to figure out, “Okay. How are we going to pick what inputs are important?” I mentioned the satellite-derived snow and vegetation. That’s because we know from the scientific literature that those are things that are important to hydrology. It may seem very obvious, but that selection is important.

Furthermore, it allows us to design, actually choose the kind of model architecture that is the best fit for those kinds of relationships. Again, just using hydrology as the example, we know that snowpack from the previous winter that was accumulated is going to impact what’s called the fresh out, at the spring melt, the spring runoff. It’s really important that we chose a model that has memory and is able to accumulate that memory.

These are all examples of how you can take the scientific field, and build upon it, and use it to really engineer your input, select your inputs, and choose your model architecture, and then also evaluate that model’s performance, right? It’s not just one metric, but are you hitting the peaks. Peaks are really important in hydrology, because when you have an extreme event, that peak and the peak timing is how dam operators are going to make their decisions. It’s kind of this holistic view. We want to make sure that we’re incorporating everything we know about this domain into the beginning, middle, and ongoing operation of the machine learning model.

[0:15:53] HC: How do you ensure that your models continue to perform well over time, particularly as things change, and particularly, the climate changing? Is this theory-guided nature part of the solution there or are there other techniques?

[0:16:06] MM: Yes, absolutely. That’s one of the most common objections we hear. You’ve trained it on this data, how will it ever deal with non-stationarity, which is the fancy term for things changing. The most important facet to that is they’re right. If those objections are valid, if you take that approach of, I have five years record, some rain data, one site going to train a simple, quick regression. Where the way you combat that is by training something that’s more foundational. Our approach, and again, pulling from the theory is that. we train our model; we train the foundational model on every basin where we have really high-quality ground truth for. These are made up of, for example, places where there are USGS gauges is one type that we look for.

What we do by training a single model on all of those basins, which exhibit a wide diversity of different basin characteristics. Like aridity, how tropical they are, their patterns of seasonal rainfall, if they’re snow driven, or very flashy, and based on thunderstorms. So, we train a single model on that wide diversity, which forces the model to learn the common rules, cross all of them. That creates the theory-guided machine learning model that looks a lot like the conceptual physics-based models that came before the machine learning.

That means that we aren’t just training a regression that falls on its face when you have a storm that you haven’t seen. You have something that, at its core, understands hydrology. We exhibited back to that competition I mentioned, there are two things. First, it was actually drier than any year in our training record, in one of the regions of the competition. And yet, the machine learning models outperformed all of these conceptual models. So, it did a better job predicting a totally unrepresented event in the training data than the conceptual models did.

Similarly, a very snow-driven basin, there was a huge storm, a huge peak that, again, was way outside of anything that had happened in the historical record for that location. Our model, again, performed the best. This approach, this foundational approach is really core to making sure that you can deal with non-stationarity.

[0:19:01] HC: does your team plan and develop a new machine-learning product or feature? In particular, what are some of the early things you should do on the process?

[0:19:10] MM: That’s one of the harder things running a machine learning startup that’s really based in science. Because, when we scope a feature that add this kind of visualization or build an alerting feature, we can say, okay, that’s going to take a week or a month. When we’re talking about how do we improve the accuracy of our seasonal forecast, it’s an unknowable roadmap. We have ideas and we break it into a set of experiments, but it’s a much more challenging problem for managing a product team, because you don’t know upfront which experiments are going to work and which aren’t. It’s definitely challenging, but again, there are some things that make it a lot easier, which is, we treat it just like we would a non-machine learning product. We identify our customer’s needs, how accurate do they need it before there are diminishing returns, how can we tie it to how they’re going to use it and build out the model so that its output affords different kinds of downstream features.

I think it’s like every product development, just it has this added complexity of R&D and experimentation, where you really don’t know what’s going to result in something you can use or not. Some of our hardest features we just built out, we’re calling it ST3, it’s the third version of our short-term forecast model. It’s incredibly cool. It allows us to incorporate new weather forecasts that are regional, or higher resolution to drive better accuracy. It’s resilient to different weather forecasts going offline. Like just yesterday, the European Center, which is one of the two major weather forecast providers, they had an error that made their forecast unavailable. ST3 is able to produce forecasts even when one or more of the underlying weather forecasts is missing.

We embarked on that journey, because we knew what we wanted to do. We wanted to make sure that we were able to produce forecasts, regardless of whether the weather forecasts were available or not. And we wanted to produce a number of different features that would be afforded by this. But it took a long time, a lot of iteration, because that kind of problem in machine learning, dealing with missing inputs is evergreen. It’s an evergreen challenge, and one that doesn’t have a right answer. So, it took a lot of R&D and research to build something that we were really happy with.

[0:22:04] HC: Is there any advice you could offer to other leaders of AI-powered startups?

[0:22:09] MM: I think that the most important thing is to not get too attached to the tools. AI is just a tool. I view myself as an AI pragmatist. It’s just another hammer, your toolbox. It’s very tempting in the funding climate where in, if you have .ai in your startup’s domain name, you’re a golden goose, you’re set. But I think it’s really important to not lose sight of the range of options you have. I’ve referred to this in a talk I gave recently at a big conference.

You have what I call narrow AI and you have broad AI. Narrow AI is AI that is trained for a very specific task. I want to classify images for this purpose. I want to say if this email is spam or not. I want to forecast this one specific signal, solar energy, or hydropower. Broad AI is the new wave of generative, which has getting a lot of the buzz. But I really urge founders to look at that spectrum. every time you’re trying to solve a problem, and choose the simplest one, and not the one that looks best, or just leverages technology for technology’s sake.

[0:23:39] HC: Finally, where do you see the impact of upstream in three to five years?

[0:23:43] MM: Yes. I think we are really excited about the impact we’ve had in two categories. One, HydroForecast has been enabling energy utilities operating hydropower to be the balancing energy producer, amidst the growth of intermittent renewables. As an organization, one of our purposes is to see the 100% renewable grid become a reality. So, we want to continue to contribute to that and to build forecasts that enable that future.

On the other side with Lens, we’re seeing millions of acres being monitored from conservation land, to forest carbon projects, to forest restoration. We are really proud of the rigor and transparency that we’re bringing to those projects, so we want to continue to deepen. In three to five years, I think we want to see more of what we’re doing, of course. I think a lot of our focus from product perspective is going to be, how do we really drive that transition to a 100% renewable grid. We’ve had so much success with HydroForecast. What other forecasts can we make that a reality? That’s something that we as a team are very excited about.

[0:25:16] HC: This has been great, Marshall. I appreciate your insights today. I think this will be valuable to many listeners. Where can people find out more about you online?

[0:25:24] MM: Yes. I have an old-fashioned blog with RSS feed. It’s my name, Marshall with no vowels .com. I’m sure folks can figure that out. My company’s Upstream Tech, upstream.tech is our website. I’m happy to answer any questions. My email is on my website. My social media stuff is on my website as well. I love when founders or people early in their startup journey reach out. I would love to chat.

[0:25:53] HC: Perfect. I’ll link to all of that in the show notes. Thanks for joining me today.

[0:25:56] MM: Thank you, Heather. I appreciate it.

[0:25:58] HC: All right, everyone. Thanks for listening. I’m Heather Couture, and I hope you join me again next time for Impact AI.

[OUTRO]

[0:26:08] HC: Thank you for listening to Impact AI. If you enjoyed this episode, please subscribe, and share with a friend. If you’d like to learn more about computer vision applications for people in planetary health, you can sign up for my newsletter at pixelscientia.com/newsletter.

[END]