Resources for Computer Vision Teams:

LinkedIn – Connect with Heather.
Computer Vision Insights Newsletter – A biweekly newsletter to help bring the latest machine learning and computer vision research to applications in people and planetary health.
Computer Vision Strategy Session – Not sure how to advance your computer vision project? Get unstuck with a clear set of next steps. Schedule a 1 hour strategy session now to advance your project.
Foundation Model Assessment – Foundation models are popping up everywhere – do you need one for your proprietary image dataset? Get a clear perspective on whether you can benefit from a domain-specific foundation model.

Transcript:

[INTRODUCTION]

[00:00:03] HC: Welcome to Impact AI, brought to you by Pixel Scientia Labs. I’m your host, Heather Couture. On this podcast, I interview innovators and entrepreneurs about building a mission-driven machine-learning-powered company. If you like what you hear, please subscribe to my newsletter to be notified about new episodes. Plus, follow the latest research in computer vision for people in planetary health. You can sign up at pixelscientia.com/newsletter.

[EPISODE]

[00:00:33] HC: Today, I’m joined by guest, Benji Meltzer, Co-founder and CTO of Aerobotics, to talk about optimizing crop yields. Benji, welcome to the show.

[00:00:42] BM: Thank you, Heather. Thanks for having me.

[00:00:44] HC: Benji, could you share a bit about your background and how that led you to create Aerobotics?

[00:00:48] BM: Sure. I’m originally from Cape Town in South Africa, which is where I’m based at the moment. I’m an engineer. My undergraduate degree was in what’s called Mechatronics Engineering here. It’s kind of a mix of electrical and mechanical engineering. In my undergrad, I met James Paterson, who’s now my co-founder and the CEO of Aerobotics. My personal interests are more in computer vision and, I guess, software engineering space. Post my undergrad, I worked as a consultant initially in the mining industry, looking at process optimization and modeling operations in the mine. Then, studied a postgraduate degree in the UK in Biomedical Engineering, specifically looking at computer vision, machine learning, and trying to understand the brain through imagery and to diagnose traumatic brain injuries, as an example.

I mean, other than that, I’ve also got experience working at Uber. I was running analytics for Sub-Saharan Africa. I guess, in parallel to my personal journey, James, who I mentioned earlier, who had met at the University of Cape Town, he also went on to study a postgraduate degree at MIT in Astro and Aeronautical Engineering. He was particularly interested in autonomous flight, drones, and flight planning, and building that technology out. I guess just one other piece of context on his side is he grew up on a farm in the Western Cape in South Africa. The two of us were both overseas, we had always spoken about starting a business together. And a few things came together with a starting Aerobotics. I think, more than anything, honestly, it was just a combination of our personal interests.

So, his in agriculture, drones, et cetera, and mine in computer vision, and building complex products that model complex systems in ways that decisions can be made. Yes, I think combining that with the fact that James in the US and me in the UK had identified that we saw a bit of an arbitrage opportunity almost to come back to South Africa and build something locally. We thought, we’ve got access to really great talent, access to market in the agriculture space to build this technology out, and I mean, we were also quite naive at the time, I think. And, yes, just decided to come back, combine our technical and personal interests, and start the business. I guess from there, there’s been a lot of learning and growth along the way. I mean, that was all nine years ago.

[00:03:29] HC: What does Aerorobotics do? Why is it important for sustainable agriculture?

[00:03:32] BM: Aerobotics is a precision agriculture business. We use predominantly aerial imagery from different types of sources, and well, technology that helps farmers make decisions throughout the season to optimize their yields through analytics derived from this imagery. We focus on perennial crops. So, that’s tree crops, fruit, and nuts in particular, and the types of things that we help farmers with are things like irrigation and nutrition optimizations, understanding how the trees are taking up water or fertilizer, where problems are occurring, giving them near real-time insights as to where the problems are, helping diagnose what’s causing the problems, and enabling them to make some intervention in the season and hopefully improve the yields and the outcome.

One of the other pieces of technology that we’ve built recently, which is gaining quite a lot of traction is yield estimation. We also help farmers through the growing season. So, while the fruits we’re growing and the trees, understand what that crop load looks like, and off of that make more informed decisions around, again, interventions that they could take in the season to change those outcomes.

In terms of why it’s important, I think, I mean, a lot of it speaks for itself, to be honest. But food security is obviously super important. Growing food and producing food isn’t going anywhere. The climate and environment is only becoming more difficult. I think there are obviously things getting more extreme. The weather environment, a lot of the macro effects like fertilizer costs going through the roof, water becoming less common labor, becoming more expensive and difficult to come by given things like COVID, and what that’s done, and the need to farm more efficiently and farm in a way where we’re using the performance of the crop to inform how we farm and becoming more responsive and reactive rather than farming completely preventatively, which is more of a traditional way of blanket-applying treatments and interventions. I think it’s definitely the way that things are going.

Yes, the role that Aerobotics is playing is really building that layer of insight and understanding into how the crop is actually performing to enable people to make these decisions. I think something else worth noting is the farmer is very much the main beneficiary and customer that we’re building for, but there are a number of players throughout the supply chain who derive value from this data as well. It’s the likes of crop insurance companies, input providers and irrigation companies, fertilizer companies, ag retailers, and all other kinds of consultants that play in the space can use this information to help offer better and more targeted services to growth.

[00:06:25] HC: What role does machine learning play in this technology?

[00:06:27] BM: I mean, the technology wouldn’t exist without machine learning. I think it’s one of those interesting cases where, just the amount of data that we’re dealing with, it’s impossible to consume it in its raw form. I mean, the main platform we use to collect data still is drone imagery, super high-resolution data, it’s multispectral data often so it’s outside of your standard red, green, and blue visual data that you’re getting. We’re collecting different parts of the infrared spectrum and millions of pixels of information that we’re collecting. Take into account that we’re also scanning given trees and orchards multiple times throughout the season. That complexity compounds and it’s pretty much impossible for any human to engage with this information in its raw form. I guess at its core, machine learning plays a role of almost helping downsample this information into a format that can be consumed.

I mean, some specific examples of what that looks like in our stack. There’s the whole kind of computer vision side of things, which is all of machine learning models, where we effectively talk about digitizing pixelated information that we’re getting. So, what that means practically is things like identifying each tree on the farm, giving it a location, GPS coordinates, and a number of metrics associated with that tree, and tracking over time how that individual tree is performing.

The next level of, I guess, machine learning broadly that we’re looking at is now interpreting this data and trying to understand, is there a problem or not? What pattern is developing both in space and time and is it anomalous? Yes or no? I guess, are there models that we run after that on, okay, this looks like a problem. Can we diagnose what the problem is? Again, that’s made machine-learning-based models where in agriculture, there are hundreds of variables that could affect your crop to look a certain way. We’ve been able to collect substantial datasets. We’ve got over 300 million trees that we’ve analyzed across the world now. Off the back of that data, we can become better at attributing the signal that we’re seeing in the data to specific root causes, and how it works in agriculture. If you can do the diagnosis, coming up with the recommendation and some corrective action is a pretty well-understood step a lot of the time.

In summary, machine learning plays a role throughout our value chain. There’s a number of other examples, things like understanding customer engagement and analytics and different ways of benchmarking and aggregating data. But at its core, honestly, this product wouldn’t exist without machine learning.

[00:09:12] HC: So, it sounds like there’s a variety of types of models that would be involved in this as well. Everything from detecting individual trees, maybe you’re even segmenting the moats, different ways to characterize how each tree is growing, diagnosing whether there’s a problem or not, what the problem is. Each of those, I suspect is an individual model that feeds in data from some of the previous models. Is that accurate?

[00:09:37] BM: Exactly. It’s exactly the way that it works. So, they’re kind of chained together, the outputs of one model will form the inputs for another. Yes, I guess as we go, we’re developing more models as different crop types that we’re adding, different cultivars, and just given the way that this industry works. I mean, we’re dealing with the natural world. It’s very much a long-tail problem. Each tree is different, farm is different, kinds of buildings these models in ways that generalize in itself is very challenging. As you start chaining them together, again, that kind of complexity just increases.

[00:10:07] HC: How do you gather and annotate the data that’s needed to train these different types of models?

[00:10:12] BM: Yes, that’s one of the biggest challenges, and I think it’s something that we’ve done quite uniquely. The bulk of the raw data that we’re collecting is in the form of imagery. The biggest challenge in the space that we found is often building ground truth data sets of labeled data to train these models off of. Some of these can be done remotely. For things like identifying the canopy of individual trees, I mean, you collect the imagery, and you could have someone sitting in the other corner of the world. For farming in California, we could have someone in Cape Town, looking at a screen and drawing a circle around the canopy of the tree, and that could be used to train a computer vision model on what a tree does and doesn’t look like, and we’ve done a lot of that thing.

We’ve got our own data annotation teams internally that are helping kind of train and improve these models. In some cases, we collect the data, so the drone imagery in this case, at our own cost. But often, we’re using real-world data the customers are using to collect, oftentimes for other use cases. Where the challenge really comes in is when the ground truth data that we need can’t be collected remotely. An example would be the yield estimation product that I spoke about earlier. One of the things we’re trying to predict is how many fruits there are on a tree. So, we want to take imagery of a tree. You could see certain fruit on the canopy of the tree. There’s a whole lot of fruit you can’t see because they’re occluded inside the tree, and the model needs to predict how many fruits there are.

The only way of developing the model and training it, taking a machine learning approach is having the truth for how many fruits that were on that tree, and the only way to do that is to go and manually count the fruits on the tree. So, there we’ve come up with different methods to collect that data from, again, deploying our own teams in the field to go and do that. I think, more interestingly, and the kind of core strategy that we’ve taken here is to leverage the customers that we’ve got to almost crowdsource collecting this data for us at scale. It’s through developing products that incentivize a collection of this data for their own use.

For example, with the yield estimation product in its current form for a new crop type, where we haven’t seen enough example data to have confidence in our model results, we position it as calibration that customers need to do to collect this data manually on the ground, which we use to calibrate the models and give them more accurate results. Then outside, that data is used to inform and train these models.

We also have a team of agronomists internally who are kind of vetting and adding heuristics to the models based on the science of how agriculture actually works. But a substantial amount of that data is actually coming from customers on the ground, and we’ve built a number of data labeling and annotation tools ourselves to help our customers help us collect that data.

[00:13:07] HC: So, once you’ve collected and annotated the data, what kinds of challenges do you encounter in working with and training models on this drone imagery? You mentioned, the quantity of data is perhaps one, but I suspect there’s others?

[00:13:18] BM: Yes. The quantity is obviously one, and the environments that we’re working in are super challenging in themselves. Connectivity is tough, just transferring this amount of data from the field to the cloud, where a lot of the processing happens, and vice versa, is tough. Similarly, the real-world environments can be really variable, whether it’s weather conditions, or what a tree looks like, it can be quite challenging. I imagine we’ll talk about generalization at some point. But outside of that, just knowing in building a product when the case that you’re dealing with is different to what you’ve seen before, and kind of warrants a different model, or an exception to be raised can be really difficult to know, as well. It’s just a function of that long-tail problem that I mentioned before, it’s very, very difficult to build the kind of one-size-fits-all approach in the space.

Over and above that, like just in our industry, there’s complexity throughout the stack, was the complexity in the environment, and just the fact that we deal with nature, and trees, and fruit, and things like that. The types of technology that we’re working with, imagery, computer vision, and machine learning itself, obviously, has noise associated with it. To the same thread as we were talking about earlier, as those things chain together, it becomes very, very difficult to build a scalable generalizable product. That’s where we’ve put a huge amount of our time and effort. I think, in summary, in agriculture, there’s a lot of exceptions and being able to firstly identify when you’re dealing with an exception or not, in itself, is hard.

Secondly, to handle that at scale, just given also that things are changing each day. I mean, you collect this static data, an imagery of a crop. I mean, the next day your fruit has grown a few millimeters in diameter, and that information becomes stale very quickly. So, trying to build these products that can return this data quickly enough, accurately enough is also pretty challenging.

[00:15:19] HC: Let’s talk a bit more about generalizability. You mentioned that one of the variations that you see is different weather conditions. Well, I imagine there might also be different lighting conditions, things might look different in different geographic locations. How do you ensure that your models can generalize to these many different types of variations?

[00:15:38] BM: There’s quite a lot of metadata that we collect, that we use to tag specific data sets, and those would be things like the geography. Obviously, we’ve got the coordinates of where we’re collecting this data, the crop type. More than just crop type, there’s actually the variety of the crop. When it was planted, the age of the trees, there is time of season information. Basically, we’ve structured all of that, which our agronomists have helped us to do in a way that we would have a subset or segments of data that are comparable against each other, and we just make sure that we’ve got enough data in each of those segments to build meaningful models. We’ve kind of got bespoke models, a lot of the time for each unique case.

That metadata is generally collected a mix of remotely by us with things like the coordinates of where we’re operating, we could measure remotely, but we also built a platform where customers tell us a lot of their data, like which variety they are farming.

A lot of it’s done in a relatively structured way. We have developed quite a lot of proprietary technology that allows us to transfer learnings across different contexts. I mean, this is almost the main IP and development that the team is working on is, what defines a context? What is different enough about this crop type of this environment that warrants a new model? And how can we leverage what we’ve built before?

So, an example is our citrus yield estimation product. We needed about 10,000 datasets to get accurate enough models. We then shifted to apples, and despite it being a completely different crop type, we could basically take that pre-trained model that we built in citrus and tune it and calibrate it to apples. All of a sudden, we only needed a thousand data points.

There’s a lot of that thing we’re doing as well, which is just a function of the fact that these are ultimately both sets of fruit that are growing on trees. There is some similarity in how they grow and how they look. But there’s enough differences to mean that you need to kind of fine-tune and calibrate the models to these additional contexts. We’ve built pretty granular and bespoke approaches to handling this. It’s quite difficult to kind of explain in more detail how we’re doing that other than there’s a whole lot of engineering that’s been done to pick up when it’s a different context, what the context is, and how we develop those models, leveraging what else we’d built before.

[00:18:03] HC: Sounds like there’s a lot of transfer learning and fine-tuning from one context to the next, and then maybe to the next and the next set as you go. But there’s a lot of thought and planning as you go into what you’re transferring from, and when it’s time to transfer and tune a new model. Are there any scenarios where it makes sense to train a broader model across different contexts? Or for your application, does that not work? Or does it not make sense for what you’re doing?

[00:18:28] BM: It does. Obviously, the value of having a broader model across contexts is there are less models to maintain. You could leverage more data, et cetera. Actually, the way that the pipelines work a lot of the time is – so, on the yield estimation side as an example. There’s one model that everything runs through, and then there’s sub-modules or models that will run depending on the context and the crop type. I think it’s got a hybrid approach in context. But generally, we found that these cases are so nuanced, and just fundamentally, the way that the crops themselves work are so different that it warrants individual models. I think, from a commercial perspective, also, like each of these markets in themselves are enormous.

So, if you look at a specific variety of citrus, from an ROI perspective, you could quite easily argue for having an individual model. It comes down to the definition of, again, what the context is, and how broad or narrow those are. We’re constantly trying to understand how can we make those as broad as possible while getting good enough results across the cases. Yes, there just isn’t a one-size-fits-all approach.

[00:19:34] HC: So, you need to understand the particular scenario, perhaps see how your existing models perform, and then decide from there, whether you’re in the territory of a new context.

[00:19:44] DM: Exactly. Exactly.

[00:19:46] HC: How does your team plan and develop a new machine learning product or feature? What kind of steps do you go through particularly at the beginning of the process?

[00:19:55] DM: There are pretty much two types of products or features, I guess. One is an external customer-facing product. And there, we do a whole lot of customer discovery. I mean, an example now, as an example is on the yield estimation side, we look at things like the count and estimating the number of fruit that we can see. We also look at the size of the fruit. So, can we estimate the distribution of the size of the fruit in the trees? One of the other features that we think could be interesting is measuring, say, the color, or the quality in terms of, are there blemishes on the fruit or bruises? Things like that.

I mean, it’s obviously all the product discovery work that we do in talking to customers and understanding if these things are actually valuable, and could we create more value and capture that value through charging more for these features? But obviously, there’s a technical feasibility question of, can we actually do this thing to begin with? We’ve built a culture internally of doing quite rapid prototypes off of dummy data sets a lot of the time, just to validate if we can or can’t do this thing to begin with, and then the hypothesis that would come out of that would be okay, if we want to build a generalizable product here that we can actually sell commercially, this is how much data we’re going to need to get this level of accuracy. This is how much it’s going to cost to collect that data, and then the product discovery work would come back with kind of what the return on that investment would be that we could expect to get. Off the back of that, we would make was purely financial decision on whether we do or don’t go with it.

I mentioned that there’s two types of products or features. Those are the customer-facing products, which are the main things that we’re working on. There’s a huge amount of internal products that we’re building as well, and they would affect things like how quickly we can turn around data internally. Do we use a different convolutional neural network to do image segmentation? Or do we look at using transformer architectures, et cetera? I guess there, it’s almost a little bit easier for us to assess, because the outcomes are a bit more quantitative and easy to predict in terms of accuracy, turnaround time, compute costs, and things like that, and we’ll make those decisions often almost entirely within the technical team.

But it’s always fitting into the same framework of trying to get an estimate on, is this thing feasible? What would the cost be of getting it to a production-ready level cost in terms of time, compute cost, data collection, et cetera? And what is the expected return on that? In many cases, we say, “Look, we’re willing to take a bet because the market is big enough and we think that, if we could build this out, the return would be substantial.”

[00:22:40] HC: How does the seasonal nature of agriculture affect your machine learning development? For example, do you focus on certain things activities during the growing season and others in the offseason? Or how does that affect your planning?

[00:22:52] DM: Yes, it’s a really good question, and it’s another one of the variables that we were challenged with. The way that we accounted for this is by building geographical diversity. The company is currently operating in about 20 countries in both the northern and southern hemispheres. What’s quite interesting about the sector is that, while there’s obviously all the variability and differences between farms, crop types, and geographies, ultimately, the problems that farmers and crops face are quite uniform. A soft citrus tree in South Africa would struggle with a lot of the same things as a soft citrus tree in California, just because, by definition, they’re grown in similar climates, similar farming practices, et cetera.

That’s always been the main way that we’ve been able to get around this is through building that geographical diversity. A lot of what we do does look at specific seasonal effects and understanding where in the season we are. There’s a lot of work we’ve done on understanding the phenology of how a plant develops through the season, and we account for that through a mix of heuristic agronomic modeling and just controlling if and when data gets measured. I think, to the second part of your question, we do focus on different activities during the growing season and others in the offseason. So, depending on even where you are in the growing season, there’s different interventions that you might take.

An example there is talking to the yield estimation product, again, early season, yield estimates could be used to make decisions like do you apply growth regulator to your crop? Or do you thin the crop? It’s one of the interventions the farmers might take where, later stage, the product use case changes a bit, where now it’s more but can you use this estimate to understand how many fruit you’re going to harvest, and use that information to plan for labor, on how many people you need in the farm? The product itself almost accounts for where you are in the growing season. Typically, in the offseason. I mean, there’s one or two products that we offer there, but that would generally be the time where we’d go to literally the other hemisphere, and start pushing those core product propositions.

[00:25:03] HM: Is there any advice you could offer to other leaders of AI-powered startups?

[00:25:07] BM: I’m always wary of offering advice. To be honest, also, my sample size in building this type of business is one, and everything we’ve learned has been on the job. I think I can share experiences and things that we’ve learned, and I guess something that’s been really – one of the mistakes or learnings that we’ve had is we’re dealing in a really, really complex space. I’ve spoken about all of the examples as to why that is in the domain, and it’s very easy to turn that into building a product itself that is really complex.

I think, at least in our domain, the role that AI, where AI can add the most value is in using this technology to reduce that complexity, and to, like I said, right at the beginning, downsample and simplify this information into patterns and decisions that people can actually consume. It’s almost too easy to just compound that complexity and not actually solve the underlying problems.

I think the problems themselves are often a lot simpler than what you think, and it’s quite easy to get caught up in all the latest, greatest AI technology without solving the core fundamental problem. I also think there are so many real-world problems out there at the moment where huge amounts of data exist or is starting to exist with the introduction of sensors and remote sensing, in our case, in different ways to collect this data. AI just gives you a really kind of necessary set of tooling to synthesize that information and expose it in ways that make sense.

I think, we’re seeing now, especially in the generative AI space, there’s a lot of noise and a lot of hype and a lot of amazing technology. But a lot of the time it’s actually solving a real-world problem. I mean, the reason I really enjoyed building Aerobotics and working in the space, is it very clearly can be used to create real environmental impact.

[00:27:07] HC: Finally, where do you see the impact of Aerobotics in three to five years?

[00:27:10] BM: I think, we’ve been able to build on a strong presence around the world and integrate with a number of growers. I think, what we’re starting to see happen is our data being used, like I said, rather than just by the farmers themselves, more as a platform-type way where this data is used to make decisions throughout the supply chain. Really, we’d like to be positioned as the underlying technology, which at its core, can give you a quantitative objective view of what the performance of your crop looks like, at multiple points in the season, and that objectively can be used throughout the supply chain to make better decisions and improve efficiencies.

I think how we would get there is through, one, just integrating and working with more customers and getting more acres on the platform, and really neatly understanding what performance means and looks like in different parts of the world in the different crop types that we work with. But also, going vertically through the supply chain and working with all of these other sorts of players. Crop insurance is one space in the US where we’re very actively working at the moment, and we’re starting to see that this data can really be used to better understand risk and offer better products to farmers using that information. I mean, that’s just one example. We’d like to kind of keep expanding within the niche that we operate to other verticals in the space.

[00:28:34] HC: This has been great, Benji. Your team at Aerobotics is doing some really interesting work for precision agriculture. I expect that the insights you shared will be valuable to other AI companies. Where can people find out more about you online?

[00:28:45] BM: Everything about robotics should be on the website, which is www.aerobotics.com. Yes, for me personally, I think LinkedIn is probably the best way to keep track. But our marketing team is pretty active on the website and keeping the blog up to date and sharing learnings as we go.

[00:29:00] HC: Perfect. Thanks for joining me today.

[00:29:03] BM: Thank you, Heather.

[00:29:03] HC: All right, everyone. Thanks for listening. I’m Heather Couture and I hope you’ll join me again next time for Impact AI.

[OUTRO]

[00:29:13] HC: Thank you for listening to Impact AI. If you enjoyed this episode, please subscribe and share with a friend, and if you’d like to learn more about computer vision applications for people in planetary health, you can sign up for my newsletter at pixelscientia.com/newsletter.

[END]