Resources for Computer Vision Teams:

LinkedIn – Connect with Heather.
Computer Vision Insights Newsletter – A biweekly newsletter to help bring the latest machine learning and computer vision research to applications in people and planetary health.
Computer Vision Strategy Session – Not sure how to advance your computer vision project? Get unstuck with a clear set of next steps. Schedule a 1 hour strategy session now to advance your project.
Foundation Model Assessment – Foundation models are popping up everywhere – do you need one for your proprietary image dataset? Get a clear perspective on whether you can benefit from a domain-specific foundation model.

Transcript:

[INTRODUCTION]

[00:00:03] HC: Welcome to Impact AI, brought to you by Pixel Scientia Labs. I’m your host, Heather Couture. On this podcast, I interview innovators and entrepreneurs about building a mission-driven, machine-learning company. If you like what you hear, please subscribe to my newsletter to be notified about new episodes. Plus, follow the latest research in computer vision for people in planetary health. You can sign up at pixelscientia.com/newsletter.

[EPISODE]

[0:00:33] HC: Today, I’m joined by guest, Ranveer Chandra, managing director of research for industry and CTO of Agri-Food at Microsoft to talk about agriculture. Ranveer, welcome to the show.

[0:00:44] RC: Yes. Excited to be here, Heather.

[0:00:45] HC: Ranveer, could you share a bit about your background and how that led you to Microsoft Research?

[0:00:50] RC: Yes, Heather. My background, I grew up in India. I grew up in the eastern part of India, in a city called Jamshedpur. It’s close to Calcutta, for people who know India. I spent a lot of time growing up with my grandparents. They live in Bihar in India. I did my undergrad in India, came in from IIT Kharagpur. I came to the US. I did my Ph.D. from Cornell University in computer science. Right after my Ph.D., I joined Microsoft Research. Since then, I have been here because Microsoft Research is one of the few places, which allows you to innovate, to do fundamental research in computer science, and any of the research that you do. You can scale it to billions of people through Microsoft products. That’s what got me here.

That would have become an academic, by the way, that was – Microsoft Research was the only industry place I interviewed in. But I joined here because I was able to ship my Ph.D. thesis as part of a Microsoft product.

[0:01:50] HC: Some of your more recent projects are focused on agriculture, which is not exactly something that Microsoft is typically known for. How did Microsoft get involved in agriculture?

[0:02:00] RC: Yes. So I grew up in India, as I said, but I did spend a lot of time growing up with my grandparents in Northern Bihar. Back then, as it happens in India, and any of your listeners who are from India would relate to it. Growing up, my parents would work during the summer and winter vacation, that is about four months, every year. They would drop me off at my grandparents’ place, which was in a village in Bihar. Back then, I did not like anything to do with agriculture. These villages, they did not have electricity, they did not have toilets. That’s where I would spend four months every year. But even though I did not like agriculture, I did get exposed to a lot of primitive forms of agriculture, a lot of poverty.

Even though my Ph.D. is in computer science, when I joined Microsoft, I always have a project that is longer term, and that has – if it gets successful, it could change the lives of a lot of people, especially in the emerging markets in places like rural India. That’s what got me thinking about this. Then in 2014, I had written a memo saying, what Microsoft should be doing in this space, and how technology could have a deep impact on agriculture. It could address the world’s food problem; it could help improve livelihoods of a lot of smallholder farmers. That’s how my memo was received. Well, they got initial funding, and that’s what got us started on this path of bringing technology. When I talk about technology. It’s bringing data, cloud, edge computing, robotics, artificial intelligence, computer vision. All of that to benefit agriculture to benefit farmers to help make the climate better. So that’s where the entire effort started.

[0:03:57] HC: I’ve read about a few different agriculture initiatives that you’re involved in. FarmBeats, FarmVibes. As your data manager for agriculture, could you share a bit about these different projects that you’re involved in? Specifically, what the goals are and why they’re important?

[0:04:13] RC: Yes. The reason we started working in agriculture is, the world has a food problem. We need to grow more food to feed the growing population of the planet. It’s not just about growing more food. We need to grow good nutritious food, and we need to grow this nutritious food without harming the planet. The soils are not getting any richer. There’s not a lot more arable land, the water levels are receding, there’s climate change. The key question is, how do you sustainably nourish the planet? How do you sustainably nourish the people in this world? In order to do that, our approach, one of the things we think that can help us get there is this concept of a data-driven agri-foods system. What we mean by a data-driven agri-food system is, if every entity involved agriculture and food supply chain, all the way from input companies to equipment companies, farmers, warehouses, logistics companies, consumers, and retailers.

If every entity started to use data and AI, it would significantly improve efficiencies through the entire agriculture and food supply chain. But if you could start sharing the data across the agriculture and food supply chain, it could lead to new business models around sustainability, nutrition, track, and trace. Towards this future of a data-driven agri-food system, we have been working on various parts of the entire agriculture and food supply chain. Those are some of the projects that you mentioned all the way from – what we started working on initially was this concept of data-driven agriculture. What’s the vision with that?

Farmers, they know a lot about the farm that they work in. They’ve been farming many times. These farmers have been farming there for several years, decades. In fact, even generations, there was a farmer that we work within eastern Washing – actually, this farmer is close to Microsoft campus. He could feel the soil, and give ideas of what’s happening in the farm. There’s a farmer in upstate New York who could tease the soil, and see what’s going on. They know a lot about the farm. A lot of decisions they took, was based on guesswork, like when to plant, when to harvest, when to irrigate, where to irrigate, and so on.

Our vision with data-driven agriculture is not to replace a farmer, but to augment a farmer’s knowledge, given that they know so much about the farm to augment their knowledge with data and AI. We want to replace guesswork with data-driven insights, with AI-driven insights. Towards that, that’s the vision of where we want to get to with all the projects that you mentioned. Our holy grail is to get to this vision of data-driven agriculture, and empowering every farmer augmenting their knowledge with data and AI.

[Inaudible 0:07:07], the very first step is, how do you even get the data in the farm or any farm in the world? Suppose we draw a polygon. A polygon could be a farm, a field, a plot. For that polygon, how do you bring all the data for that polygon in one place? Now, that’s a hard problem. Because, you know, many of the farms are disconnected, you can look at satellite data, but not on the satellite. Data is cloudy. How do you start bringing all of that data in one place? That was what project FarmBeats was about. As part of project FarmBeats, we created a lot of the capabilities of – you define a polygon, and we bring all the data for that polygon in one place. That became, now, we announced a product called Azure Data Manager for agriculture, which is the evolution of project boundaries. The first part is, of course, getting the data in one place.

The second piece is, once you get the data, how do you translate that data into insights? How do you take that data and come up with insights based on that data? That’s the project we started, which we called project FarmVibes.AI. We started building individual models, these AI models. We also started building workflows. Like for example, a workflow could be what is the expected yield in my farm. If that is the workflow, that’s what you want to do, then the individual models could be things like, get the satellite images, detect clouds, remove clouds, compute vegetation indices, make a prediction model of what the [inaudible 0:08:37] these individual models, and then you combine them into these workflows.

That’s what we did with Project FarmVibes.AI, which we released in open source. Right now, it has over 65 workflows that we’ve made available for the community to start building on top. It’s on GitHub. You could go and start playing with it, and it’s very well maintained by my team. That’s what we did as the second part. The first part was bringing the data, and the second piece is turning that data into insights.

The third part, which we are working on right now is, how do you make that data accessible to the different stakeholders of the agricultural industry. Like for example, farmers, agronomists, and so on. That is where we are actively working on generative AI for agriculture. How do we take all of these insights and make it consumable by the farmer so they could ask the question in natural language or take a picture? Then, how do we then turn that data, or whatever questions they have, and provide answers to them, using natural language as well, using generative AI? That’s the third piece of it.

[0:09:41] HC: What types of data do you collect from the farm in order to do all of this?

[0:09:46] RC: Yes. To do this, right now, at the base level when we talk of FarmBeats, or as a data manager for agriculture, our goal is to be able to map a farm, or any polygon, our goal is to be able to see what’s happening in the farm right now, not just on the surface, or above the surface, but also below the surface of soil, like where the roots are. What’s happening right now, what happened in the past, and what will happen in the future? What will happen, I mean, what will happen with respect to the crops, to the climate, and to soils? If you think of that’s what we need to predict, we need to build an accurate map for that farm.

What that means is, we need data about soils, we need data about plants, we need data about roots, we need data about climate. In order to get all of that data, we need to look at different sources. There is no single data stream that can give us all the data. So we take data from satellites, we take data from drones, from sensors, from tractors, from sensors, from soil samples. The goal is to bring in as much data as possible.

One of the realizations we had was that, there is no single data stream, or farms that can provide us the absolute truth of what’s going on in the farm. Because of which, a lot of our innovations. If you look at what we did with Project FarmVibes.AI, nearly all our innovations, they are about multimodal AI. We have to take data from multiple data streams. and fuse them to come up with this detailed knowledge, this ground truth about the farm. For example, I’ll give a few examples about multimodal AI. Like for example, if we wanted to see below the clouds in satellite imagery, over 70% of the images that satellite takes have clouds in them, and thick clouds. How do you see what’s happening below the clouds and satellite imagery if all you’re getting is optical images?

There, what we had to do was, we had to take data from different types of satellites, radar satellite, and an optical satellite. We used initially a [inaudible 0:11:58] based scheme. Now, we’re using another generative AI scheme to see what’s happening below the clouds and satellite imagery. Radar signals, they go through the clouds, and they reflect differently from different surfaces, like if the soil is moist, if it’s dry, if the crops are wilting, or the crops are doing well. We take that information to then predict what’s happening below the clouds and satellite imagery. For three of the four seasons, we get to a 95% accuracy in predicting what’s happening.

This concept of multimodal AI is also useful for many other schemes. Like for example, suppose you asked me a question, “Hey, what is the soil moisture level throughout my farm?” If we had to make an accurate map of soil moisture, say six inches below the surface, you will need a soil sample every few meters. Because soil moisture, for example varies row to row. But putting a sensor, getting measurements every few meters is expensive to deploy, to manage. It will come in the way of the farmer as the farmer does the day-to-day job. The key idea we came up with was to use very few sensors. Again, using multimodal AI, we use ground samples, and then use aerial imagery to interpolate that data, and build heat maps of farms.

Like for example, what is a carbon map? What is a fertilizer map? What is a soil moisture map and so on. Similarly, we had to use multimodal AI for microclimate prediction. We take sensor data and combine that with weather station data. These are just a few examples. We’ve come up with a suite of models around multimodal AI, where we combine different modalities of data. I talked about radar plus optical, or weather data, weather station data versus sensor data, sensor data versus optical data. We’ve done more work on drone data, plus satellite imagery, and so on.

We have a suite of models that are multimodal AI, which was critical for us to start building those models for farms. Going back to your question, we basically take data from a variety of data streams, any data stream that you could get any information about the farm. It could even be a farmer’s notes, a farmer taking pictures of the farm. But the key technique that we have to build was this concept of multimodal AI to be able to build better models for months.

[0:14:17] HC: It sounds like, one of the challenges of working with farm data is that, no single modality alone solves your problems. That multi-modality is the solution that you’ve just talked about there. But with these multimodal data, and the models you build, what kinds of challenges you still encounter while you’re training models based on all of these modalities together.

[0:14:37] RC: Yes. While multimodal AI is one such technique, preprocessing of this data becomes important. Because a lot of the data, especially data which you’re getting from say, soil labs, or manually collected data often has errors. That is something that – error sometimes, it’s in data collection. And at other times, the error could be because of error in the way the tests are done. Like, for example, soil labs in different places might have different ways in which the same experiment is done. So the data might be consistent. How do you normalize that data?

That’s a challenge that we had to come across when we’re normalizing the data, making sure that data – especially for data that is either manually collected, or has variations because of the way the tests are done. That was something we had to – the other challenge when working and training with models on agricultural data is to look at the economic viability of doing that work. For example, I could go and put huge amounts of sensors in the farm to build a very accurate model of farm. Is that economically viable? [Inaudible 0:15:52].

Once we start thinking of the economic models around this, we run into interesting challenges around, how do you make the entire system, what’s the business model? That drives a lot of challenges as well. It is not just about instrumentation in the farm. It’s also about, even data in the cloud. How much of cloud resources do you put in for certain types of crops? That’s where we need to be smart and intelligent to make sure that the things aren’t – which is why a lot of optimizations come in.

Like for example, if the satellite images are not changing on a daily basis, do you really need to keep processing high-risk satellite data? Or do you only process high resolution satellite data when we know there is a change that is happening? That, again, leads to optimizations that we need to make in the machine learning pipelines to make sure that the system is not being too resource hungry. This becomes increasingly important for agriculture, where the farmers operate on thin margins. You have to build a business model for them. You need to make sure that it’s something that is economically viable.

[0:17:08] HC: How have these technologies been deployed so far?

[0:17:11] RC: Yes. With respect to the Project FarmBeats, we do some data collection, and it’s become a product right now. It’s a Microsoft product called Azure Data Manager for Agriculture. We’ve announced partnerships with Land O’Lakes, with their population and other organizations, who are then using it to manage their farmlands. Their farms, like for example, Land O’Lakes is a cooperative with over 100,000 farmers in managing 25 million acres of land, and their data platforms are running on top of Azure Data Manager for agriculture.

Similarly, with Bayer Corporation is building their agri-powered services on top of Azure Data Management. What we built is the building blocks. But this is where – Microsoft is not an agriculture company. Wo we are not sending anything to farmers, but we are providing the tools on top of which you could build solutions for farmers, or partners, or customers build solutions and take the solutions to farmers. We are continuing to work on, so that’s with Project FarmBeats and Project FarmVibes. We’ve made it in open source, we have partners who are deploying the technologies in different regions around the world.

We’ve announced special partnerships. For example, Accenture is building solutions using Project FarmVibes.AI. We’ve also been partnering with, like for example in India, with – this is in Parramatta in India, where we actually deploy it. We are working with Agriculture Development Trust in Oxford University, where we established the FarmVibes.AI Center of Excellence in India. This is available, as I said in open source, where different partners are building solutions using Project FarmVibes.AI. The generative AI pieces in early days. We are still working with different partners, and making this available, making the copilot capabilities available to them.

Recently at Agritechnica, where a corporation announced their bare agri-copilot building on top of Azure Data Manager for agriculture or LLMAPIs. Overall, holistically, our go-to-market here is with key partners and customers in the agriculture industry. We are continuing to expand and build on that.

[0:19:31] HC: Machine learning is advancing quite rapidly now. There are new advancements hitting the headlines more frequently than ever before. Are there any new developments in AI that you’re particularly excited about?

[0:19:42] RC: Yes. Heather, of course, I need to be. I’m so excited about what’s happening there. Generative AI is not just what’s happening with language models. That’s of course, important. But also, with multimodal generative AI. I’m very, very bullish on what things we could be doing. For example, in agriculture, farmers, a lot of times when they’re out in the middle of the field, they see something, which they don’t know what it is. They want to take an image and they want to know what it is.

There, the ability of some of these generative models to not just say what it is, but to why it thinks what it is, could lead to very compelling scenarios for agriculture, very useful tool for farmers. The other reason I’m so excited about generative AI in large language models is that, farmers. when they are out in the middle of the field, their hands are soiled. They might not want to use a phone, they might just want to talk to the device. Also, a lot of farmers worldwide, and even in the US, where a lot of farming population, especially the immigrant farmers, they are not the most tech savvy or educated.

Being able to bring technology to them through generative AI could really help drive data driven agriculture. This mission that we have of empowering every farmer augmenting their knowledge with data and AI. We need to make data consumable, and generative AI has the suitability to make that data more consumable. Of course, if you’d asked about where are things headed, what am I most excited about here? I think the place where I’m even more excited about is, what if we start applying these generative AI models on new agricultural data, applying it on satellite data, applying it on agricultural data, and then using some of these big tree trained models so that you don’t necessarily need a farmer to start compromising with their personal data. But even then, we can start making recommendations for farmers. These benefits of pre-K models for regenerative agriculture is huge.

[0:21:50] HC: Is there any advice you could offer to other leaders of AI-based projects?

[0:21:55] RC: Agriculture is one industry, which I think stretches technology. It really brings, takes technology to the next level to help address some of these problems in agriculture. I will definitely, especially the researchers and academics. I would definitely encourage them to look at agriculture as an industry, and start helping solve some of these really hard problems in agriculture. Because even if you could solve something in this space, it not only helps address the world’s food problem, because the world needs more food, we can help feed the growing population of the world.

It also helps improve the livelihoods of farmers. There are over 500 million smallholder farmers worldwide whose lives would benefit with artificial intelligence. In addition to that, it could also help address the problems around climate change. Agriculture is one of the biggest emitters of greenhouse gases. It’s the most impacted by climate change as well, farmers, because of unpredictable climate. The other ones who are least prepared to address it. I would definitely open this up as a challenge for machine learning researchers and in academia to really go after some of the problems in the space.

For the industry, people – I think we need more tools in this space. As people are building tools, they should start thinking of this as an industry that is not as much touched with artificial intelligence and has a huge potential of impact. This is a new market for a lot of startups in the space, market for food market, also around sustainability.

[0:23:41] HC: Finally, where do you see the impact of these data driven and AI technologies for agriculture in three to five years?

[0:23:48] RC: Yes. I see this – three to five years is a short time, but I do see more and more farmers starting to benefit from this. More farmers worldwide starting to use data and artificial intelligence. The one word of caution that I have, which also relates to the previous question you asked, Heather, around advice for two other leaders is that – one of the things we’re seeing with agriculture is broadly applicable, is that we should be mindful that AI, that we don’t accidentally create an AI divide. The same way there has been a digital divide. The world has, with respect to connectivity, the haves, and the have-nots. There are people who had Internet connectivity like you and me, and we are benefiting from that. The people who are still are not connected to the internet, and there are over two billion people in the world who are not connected to the Internet. We need to address a digital divide.

With AI, if we don’t take the right steps, and it’s not just technical, it’s technical, it’s polity and business. We might end up creating an AI divide, where there are the haves and the have nots. One place, for example, where it could show up in a very stark fashion is with agriculture. There are farmers who might be tech savvy, the big farmers who might benefit from AI and the others, farmers who are not as tech savvy are left behind. To start benefiting, the people who already have some of the benefits of technology. But we need to make sure that this kind of polarization doesn’t happen, and we truly democratize data-driven agriculture. We need to make sure that every farmer anywhere in the world, be the small farmer or the big farmer, rich or the poor farmer. Everyone benefits from technology and from AI.

Farming is just one-use case, but a very important one., and we need to make sure that we don’t end up creating this AI divide. As there are entrepreneurs, researchers, policymakers, as they keep working in this space, it will be useful if they start thinking, and kind of important for all of us to start thinking of ways in which we don’t end up feeling such, anyway, we are divided.

But to conclude on a positive note, I’m very excited that this AI will start touching so many more people, so many more farmers around the world. The benefits of this are not just going to be with farmers, it’s going to be in the food that we eat, it will be healthier, it will be fresher, it will be – it will have less of a climate impact. I’m very excited, very bullish on the future of food and the future of agriculture because of the advancements in data and artificial intelligence.

[0:26:43] HC: This has been great, Ranveer. I appreciate your insights today. I think this will be valuable to many listeners. Where can people find out more about you online?

[0:26:52] RC: Yes. I would love for people to connect with me on LinkedIn and all of the projects that we have LinkedIn or Twitter. They keep posting the updates there. If anyone has any ideas, we’d love to hear from them.

[0:27:03] HC: Perfect. Thanks for joining me today.

[0:27:06] RC: Yes. Thank you, Heather.

[0:27:07] HC: All right, everyone. Thanks for listening. I’m Heather Couture, and I hope you join me again next time for Impact AI.

[OUTRO]

[0:27:17] HC: Thank you for listening to Impact AI. If you enjoyed this episode, please subscribe, and share with a friend. If you’d like to learn more about computer vision applications for people in planetary health, you can sign up for my newsletter at pixelscientia.com/newsletter.

[END]