Resources for Computer Vision Teams:

LinkedIn – Connect with Heather.
Computer Vision Insights Newsletter – A biweekly newsletter to help bring the latest machine learning and computer vision research to applications in people and planetary health.
Computer Vision Strategy Session – Not sure how to advance your computer vision project? Get unstuck with a clear set of next steps. Schedule a 1 hour strategy session now to advance your project.

Transcript:

[INTRO]

[00:00:03] HC: Welcome to Impact AI, brought to you by Pixel Scientia Labs. I’m your host, Heather Couture. On this podcast, I interview innovators and entrepreneurs about building a mission-driven machine learning-powered company. If you like what you hear, please subscribe to my newsletter to be notified about new episodes. Plus, follow the latest research in computer vision for people in planetary health. You can sign up at pixelscientia.com/newsletter.

[INTERVIEW]

[00:00:34] HC: Today, I’m joined by guest Kit Merker, CEO of Plainsight Technologies, to talk about vision intelligence filters. Kit, welcome to the show.

[00:00:42] KM: Hey, great to be here.

[00:00:42] HC: Kit, could you share a bit about your background and how that led you to Plainsight?

[00:00:46] KM: Yes, of course. Well, I’ve been in the software business for feels like too long, 20-plus years. I spent a bunch of time at Microsoft and at Google and worked on building software for a variety of different things, cloud computing and other things like that. I’ve always been interested in artificial intelligence and machine learning. In fact, when I was a teenager, I built a neural net that used backpropagation, and supervised learning for image recognition. But I was doing it on a 486, and I didn’t have Internet access, except through the AOL disc I got every month. It was very hard to actually build anything that was useful.

Fast-forward to today, computer vision and machine learning is now the hot topic and everything. But what’s been very cool with Plainsight, which I joined a few months ago as CEO, is I’ve been able to bring cloud computing, DevOps, and other kinds of experiences from my other domains and software that I’ve worked in. We really try to focus on building a product that brings that together. It’s been a cool experience to pick computer vision focus but then add this business-oriented, data-oriented software update, DevOps orientation to it. It’s really been a cool thing.

[00:01:58] HC: What does Plainsight do, and why is it important?

[00:02:01] KM: We like to say we make your cameras count. We basically work with companies that are trying to extract meaningful data for their enterprise out of the cameras. We build this software concept we call a filter, which is inspired by Instagram or Snapchat filter, the idea of an app that helps solve sort of the attention problem in machine learning. This is one of the big problems in artificial intelligence is like, “What should I pay attention to?” We solve that by using a filter that knows how to find a particular type of object and pull out interesting data about it. That could be cattle. It could be mushrooms. It could be pallets coming off the back of a truck.

All these different industrial, manufacturing, supply chain-type use cases that eventually ends up in data analytics system or an enterprise resource planning or ERP system like SAP or NetSuite. What we do is we basically have the software life cycle to build and train these filters. They are AI-powered apps. Deploy them at the edge, run and process camera data, and pull that data into data pipelines that support these different business processes. That’s really what Plainsight does.

[00:03:04] HC: How do you go about building these filters? How do you set up the machine learning problem?

[00:03:08] KM: Yes. There’s two parts to it. One is the application development part, and the other is the data-to-model part of it. In the application development side of it, filter runtime is this tool called filter box, which is a dockerized platform and allows us to build the common components of computer vision. Things like access to the video feeds and access to PLC controllers and MQTT, which is the data format we use to stream the data. This gives us a foundation to rapidly build and develop these apps that can be deployed into Kubernetes in the cloud or at the edge. That becomes like just any other app. The fact that it’s AI-based is sort of almost coincidental. Any IT department can deploy that and connect it to their existing data lake. They can get it running very quickly and connect it to their IT infrastructures. That’s one part of it is making that side of it not seem like a foreign ML part of it.

The other side of it is the sort of data-to-model problem. Our goal is to give the customers very high accuracy on their models and to do that at the edge, which means that we can’t have it connected to cloud computing necessarily. Or we can’t use LLMs. We do use smaller-scale hardware. What that looks like is we take a model that we source from wherever models come from but academic and Hugging Face, et cetera, YOLO models. We take these different models, and then we fine-tune them using customer data.

Because of our software update infrastructure, the idea is that we can collect more video and image data from customers. We run it through whatever labeling annotation we need to. Then that is trained into a model and then wrapped up in the filter and deployed. We sort of treat the model almost like a configuration file, I mean, to make it overly simplified. But the output from that training process is essentially a file that then gets wrapped into the application development [inaudible 00:04:58] life cycle, and then deploy that into the customer environment. That’s a very iterative process.

Then as we’re getting better and better data, our goal is to continuously improve the models we’re using from customers or find other third-party models. Our expectation is over time there will be more and more off-the-shelf models available that we can use as a starting point for fine-tuning. Then that will give customers the ability to quickly adopt good enough models to start getting value out of the data. Then they can improve over time. At a high level, at least that’s kind the process.

[00:05:30] HC: These are object detection models I assume.

[00:05:32] KM: Yes. Generally speaking, object detection is the primary method. There are other scenarios like face detection and other types of features and regression models for visually weighing and sizing things, et cetera, a number of different types of models that can be used. Object detection is I’d say probably the primary one.

[00:05:51] HC: Are any of these models shared across different customers? Or is it a specific model for specific customer’s data?

[00:05:58] KM: Yes. That is a great question. One of the fundamental things I believe anyway is think about sort of data privacy as a hot issue. If I’m a business trying to adopt AI, as many are right now, the motivation behind it is to do that to gain a competitive advantage. I’m trying to build a business. I want to gain an edge. I’m not adopting AI for fun, right? I’m doing it because I want to build my business and run a better enterprise. It only stands the reason that the data that’s used to train those models should be kept within my own domain.

This is how we structure our enterprise agreements. We segment the data so that – not that these are customers with like – the Pizza Hut data doesn’t end up with Domino’s, so to speak. Just using those competitors as illustrations, not that they’re customers of ours, but the idea that you would keep the data trained for one customer environment separate and then use that for fine-tuning. Oftentimes, it is based on a foundational model that we’re getting from a third party anyway. But for the fine-tuning part of it, we’re using – we’re keeping it segmented.

Now, in other cases, we do have customers that are coming in at a lower price point and are less concerned with data privacy. We’re seeing this particularly in some of the cattle and agricultural space. There’s a lot of different farms and ranches and things like that where they have less concerns about data privacy, and they have smaller budgets. In those cases, we’re able to come up with an arrangement where the data is shared. But they get the advantage of a lower price point, and they get – the model has higher accuracy than they can possibly get from just their data alone. There’s a variety of approaches that we can take, but the enterprises that we sell to care a lot about owning and maintaining that data for themselves.

[00:07:37] HC: Your process for training these models, is it the same across the different types of imagery and different types of filters that you’re trying to create? Or does it ever change based on something unique about a type of imagery or maybe domain knowledge, anything like that?

[00:07:53] KM: The biggest variables in running this type of business is scale. We have to constantly change and improve the process. We’ve looked at different services, vendors, labeling teams. We offshore a lot of that, and we also are using different techniques to sort of optimize the quantity of images we have to tag. There is a high degree of domain expertise required in some situations. To the extent that we can, we try to simplify that for the end customers and make it easy for them to describe and explain the different examples of the features. Defects are really – this is one of the ones that requires the most domain expertise is defect detection. Anytime you’re looking for exceptions, anomalies, and damage. Creating the source data for that, as well as labeling it properly, there’s a high degree of variation there. That, I think, is one of the hardest ones.

I’d say probably where we’re investing the most in terms of our secret sauce is how we can optimize that data-to-model life cycle and make this something that’s not happening. One time to create the model, but we’re continuously collecting more data, labeling, annotating it, improving it, and sending it back to the customer, and then comparing that to various benchmarks. This really is, I think, one of our core areas of investment. That’s something that, I think, over time gaining that consistency but also sort of the clever tricks that you can do to reduce the effort required while also getting a high degree of throughput and output from these different model training exercises.

[00:09:26] HC: Does the domain expertise mainly come in in the data curation and labeling phase, like you mentioned? Or does it also affect how you train a model, what type of architecture you choose, different things like that?

[00:09:39] KM: The way I conceptualize this is that as we already discussed sort of the training time or labeling time issue of making sure that you understand what you’re looking at, right? It’s like, “Oh, I’m looking at –” Not that we’re doing this like, “I’m looking at these X-rays of lungs. Let’s make sure that’s –” We really know what we’re looking at here because that’s a high degree of sophistication. There’s a lot of scenarios like that we run into where you’re looking at some sort of object in the business that is very particular to that business. They have a defect or a situation that visually, you just make sure you know where that is. That is a very clear area for expertise and translating that into labeling test.

The other side of it, to your point, is that the techniques for machine learning and artificial intelligence are even changing all the time. The state-of-the-art is moving at an incredibly rapid pace. The concept would be because we have these filters as apps, we have a set of strong interfaces around that in terms of how we collect the data from video and image to output the data into MQTT streams. But we almost get to treat the filter as a black box. You can chain them together into pipelines that support a variety of detections on one stream.

Or conversely, you can parallelize them and say, “I have one stream that I’m going to now have a variety of detections that run independently.” This might be useful maybe for mapping departmental process, where you have one stream of detections that are being used for perhaps inventory management and another being used for surveillance, for example, just as a way to think about it. You might have one camera that’s now running multiple filter pipelines.

This application construct where you really have one piece of data output from each filter, it allows you to innovate within those. The way I think about that is you have different approaches for ML or versioning. We can actually create these different apps that use and encapsulate these different forms of machine learning or detection that are optimized for different use cases or different video feeds or different ML techniques. That can all run together, just like Docker is used to run apps side by side on Compute. We can do the same thing. Then behind the scenes, we can do some optimizations around use of the GPUs and things like that for inference.

This kind of app-centric way of thinking about it, where the models are wrapped by application logic and chained together and feel much more like traditional IT applications, this has really simplified our ability to use a variety of ML tools, while still having a consistent software paradigm for software updates and continuous improvement.

[00:12:18] HC: How do you plan and develop a new filter? Maybe there’s something you need to detect. Or whatever the purpose is, it’s different than you’ve done before. What are some of the steps you take in the process?

[00:12:30] KM: The concept, I like to explain it as like an app store, right? You have this app store where we have all these different filters, and those filters have some configuration that are specific to a customer. Think of the model, again, as that configuration, right? Oh, I want to take this filter that knows how to measure the size of a pile of gravel, and it’s for this very particular mining site. Therefore, we’re using the model for that customer that’s trained on the data that for them, needs some fine-tuning. That’s like their version [inaudible 00:12:57].

Well, if I turn around and I get a customer tomorrow that has a very similar problem, I’m not starting from scratch, right? I can come in to them with base off-the-shelf model that doesn’t use any other customer’s data, et cetera. If it’s in the app store already, it becomes very straightforward to take a customer through that journey and say, “We have a starting point. If it’s accurate enough, you can get started with it. If you want to fine-tune it, you can. Now, it’s just managing a different version in the app store for them. Everybody gets their own private app store is a good way to think about this, too. It’s like an enterprise app store where we have the base versions. Then those can be customized and [inaudible 00:13:33].

Now, we do run into this case. In the beginning, you run into it much more frequently. Then over time, it happens less often but where you run into a novel application structure where maybe they need a different data integration, or they need a different model format, et cetera. Yes, when we have one of those, we basically just cover the R&D cost ourselves. We don’t do it as a custom development. We do R&D to support creating that new class of filter. Then we license it to them as if we’d already created it.

That way, it’s really given this consistent licensing model, which I think is one of our exciting innovations it’s like we’re really just letting people subscribe to these filter instances that they can run the edge on this annual subscription. What they’re really getting is access to updates for that software. Those updates are really the value. This can cover security updates, functional updates, and model updates, all through the same up mechanism which feels, again, very IT-centric where you have Docker and Kubernetes and these very standard IT ways of deploying applications.

The new thing begins with taking a request that is novel, reusing and repackaging a new filter from the base runtime, creating the approach. That’s where the developers get involved. Then from there, we just have this ever-expanding pool of filter apps that are in our app store. That’s kind of the idea.

[00:14:56] HC: One of the larger concerns with AI right now is bias. How might bias manifest with models trained on images? Are there some things your team is doing to mitigate it?

[00:15:06] KM: Yes. Bias is a huge problem. I think the bias issue comes at a couple of levels. One is when foundational models are trained on vast swaths of the Internet, there’s consumer-oriented or even almost a political level to this where it’s racial bias or gender bias or other kinds of bias that shows up. Of course, we’re reusing those foundation models. We inherit those issues, and it has a tremendous impact, especially from a societal level.

In the cases that we’re targeting in the – what I say between the camera and the spreadsheet. I mean, turning cameras into counting machines that can put data into your ERP and analytics. There are some situations where that form of bias can show up in crowd control-type scenarios that are more I almost say like surveillance or customer-type issues. That actually is less of what we focus on. We do have those use cases we can support in particular for crowds and in things like stadiums or public spaces.

Primarily, what we’re looking at is counting anything from livestock to inventory to manufacturing parts to supply chain or looking at shelves and stores, those kinds of things where we’re actually looking a lot more at inanimate objects. The type of bias that is our issue is really more about encountering novel situations or not having enough representative data to train a model that can be accurate to these things. Sometimes, it can take a long time to encounter enough situations.

A couple of ways that we’re counteracting this, one is we think that model accuracy is forever projects. You have to constantly improve and maintain, and you want to constantly look for drift and bias and other things that could lead to less-than-perfect results. We know perfection is not ever going to be achieved, but we want to constantly manage to maintain that. That, I think, is the first thing is rather than thinking like, “Okay. Once I hit some arbitrary target, then the job is done.” You want to build that maintenance process into it, so you’re not constantly having to come back manually and say, “Oh, well. I guess it’s time to update the model.” No. It’s like, “We’re just going to update the model on a regular basis continuously, whether you like it or not.”

Then the second part of it is using generative AI to fill data gaps. This is basically a synthetic data strategy. We can take an incomplete set of data. There are certain situations that are pretty hard to recreate physically, and so we’re using generative AI for that. I will say that it’s not a huge part of the data. It’s really more of an enrichment, so you can’t use it to replace that. But the scenario would be like maybe you have some endangered fish that you want to make sure aren’t getting caught with the tuna. Using generative AI to create images and then feed that into a traditional training model, training process with annotated images. Generate images, annotate those images, feed them into the set, and then use that for training.

That can help give you an accuracy boost on rare situations. It’s usually better if you have the actual data, but it can help and round out a data set. That’s what we would do to address the bias issue. I think over time, people are becoming more and more aware and conscious of the bias in AI. I see a lot of efforts around the ethical use of AI, so it’s a really good thing. As an industry, I think we’re still early in fully tackling it.

[00:18:23] HC: Could you share some details about a couple of the more interesting or impactful applications that you’ve worked on?

[00:18:28] KM: Well, I will share one that I’m quite proud of. It’s still pretty early, but we launched recently called wildfirewatch.org. Wildfire Watch is a pre-public website that focus on the State of Washington right now, but it has potential implications further out. I live in Washington State, and we have a big wildfire problem here in the Pacific Northwest. We took the 1,547 public cameras that Washington State Department of Transportation runs, and we created a filter to do wildfire detection. We are every five minutes, applying that filter to every camera in Washington State and giving it a fire likelihood rating. Then we put it onto a map, and we have a histogram of all the images. We’re working with the Department of Natural Resources and some firefighters to improve the models.

We still have detection issues around headlights, sunsets, and some other smokestacks. There’s things like that that as they’re occurring, we’re working on eliminating them. We just launched it a couple of months ago, and we haven’t quite gone through fire season yet, so we haven’t had a chance to do a live test, fortunately. It’s actually good that we haven’t had to do that yet.

But as fire season approaches, we will be working closely with Department of Natural Resources to make sure that we’re correctly and accurately mapping, and we see wildfires. We’re looking if anybody wants to help to contribute to that. Obviously, we’re always looking for volunteers. It’s a totally free site, and our plan is to open it up to the world because it makes no sense to make money off of wildfire detection. To me, that is something that the whole world should benefit from. Any researchers or fire departments that want to collaborate on that, we’re open to do it

In terms of some of the other use cases, I think they are quite exciting. One customer of ours is a company called Paypixl, and they’re in Florida. What they do is they actually connect drone pilots to make – they help to make money, and the way that they’re doing that is by basically sending their own pilots off on missions and collecting their data about roofs. Then they use that data to help inform both the insurance, as well as disaster prevention and roofing companies, to get data about what’s happening. It’s very cool business, and they’re using our technology for that.

We’re also working with marine biologists, and Washington State spun out of the University of Washington, that uses our technology to count fish at dams and bridges for various required biological testing, marine biology checking. That’s another cool one. Then we’re working on – I mean it’s like the number of interesting things are crazy, but it’s like everything from counting cattle to defects in manufacturing processes for solar panels, looking at the size and shape of mushrooms. Really, it’s many different inventory and supply chain-type use cases.

The cool part is holding this quantifiable data that is giving higher accuracy to counts and to yield estimation and resource planning and then tying it into operational processes that exist in their existing ERP systems or data analytics systems. They can tie it into existing business processes and not have this AI thing be off to the side in a science or a project. It really is becoming part of the core operational aspect of the business.

Then being able to start small because with our model, we’re really not selling a platform which implies that you have a lot of data scientists, and you have a lot of different AI going on in these companies. A lot of times, these traditional enterprises, they’re looking for a solution or an app. The filter is like an app, and so customers can start really small with us, get an app that they trust the data, and then expand from there. They don’t have any machine learning expertise required in order to run this, and their IT department can easily deploy it, as I mentioned.

To me, that’s the more inspiring part. Some of the use cases are somewhat mundane. I’m not trying to chase the most fancy, exciting, descriptive LLM type of AI. Actually, I’ll leave that to someone else. We’re trying to do the basic boring stuff like counting and take that off people’s plate so that they can focus on more interesting things. Then give these businesses higher accuracy and visibility into what’s going on. Giving them ground truth that’s really hard to get actually and requires a lot of spreadsheets and barcode scanning. We just replace that with a camera, put the data directly where they need it, put it into their existing process.

[00:22:48] HC: Yes. These are all things that can be done much more efficiently by a computer.

[00:22:51] KM: That’s right.

[00:22:52] HC: It’s great to see the diversity of applications that it can cover.

[00:22:55] KM: Yes, that’s right. Yes.

[00:22:56] HC: Is there any advice you could offer to other leaders of AI-powered startups?

[00:23:00] KM: Don’t fake your demos I think is probably my number one piece of advice. I think it hurts the industry as a whole whenever we have any question of the credibility or veracity of demos. I understand that sometimes you got to put one lasagna in the oven and then pull the other one out for time. That’s not what I’m talking about here. I won’t call anybody out by name, but we see this time and time again that the AI is overpromised, overhyped. That would be my number one.

Then second is really just understand that customers are trying to solve problems. There’s a lot of budget or there has been a lot of budget slashing around for innovation and things like that. That’s just not the case anymore. I think the big thing with AI is making any venture, and this is starting a new company, doing a startup. You want to figure out what the path is to create customers, how customers are going to pay you, how you’re ultimately going to produce profitability, so you can reward your employees and shareholders.

I think it’s really easy to get caught in the hype and excitement around technology or even see some of these eye-watering valuations and getting excited by that. I just always have to remind myself that anytime you raise money and announce a big number, you’re really just setting a higher expectation for what the business has to produce. I think it’s just really important for people in this economic environment to stay grounded. They still have to build a business. First of all, don’t be discouraged if you’re not raising money to high valuation. You probably won’t regret it. The bigger thing is just creating customers and focusing on what’s going to really move the needle for creating the business.

Anyway, that maybe is – I’m getting older, so I can give this kind of advice, young startup executives. To me, this is always the thing. It’s managing expectations and growth and not getting caught in a hype cycle and what people expect from you, but instead charting a path that’s going to be right for your business, right for your team, right for your shareholders. It’s a hard thing to do, but it all starts from having a really clear picture of where you can fit in the market and what your technology and your product or your service can solidly do. That comes from listening and not having just an idea or a belief but really listening and learning from customers in the market and seeing what’s come before you.

[00:25:21] HC: Finally, where do you see the impact of Plainsight in three to five years?

[00:25:26] KM: I hope – If I can dream of what we’re trying to do, what we’re trying to build here is I can see us becoming a business that many customers rely on as owning this point in the software stack for them, this really particular type of data that is coming from their cameras that are mapping the physical world into their digital systems. By giving a lot of value in that, I think we can create something that, as you mentioned, covers many different use cases. I think we can tie ourselves directly to the buyers of ERP and data systems that are highly concerned with data accuracy and will see the value of this, but don’t have the AI expertise to oversee a roll out.

I think we can build a business quite large. I mean, if I look at the ERP market, it’s measured in the almost 100 billion level, growing very rapidly, huge services market around it. These platforms are right for innovation in AI and computer vision. I think we solve a very concrete problem. I’m not trying to do everything. We’re trying to do one thing super, super well. You might argue that that’s limitation on our business. I see it the other way. I see it as something that if we can do it super well and we can execute flawlessly, we can build a business that has global reach and a massive impact. It’s going to be a lot of work. But I think if we focus, the only thing left from my point of view is really about execution and creating the best product we can, the best team we can. I believe that we’re at the start of that.

Anyway, hopefully, we’ll be talking again and before three to five years can check on my progress against this ambitious journey we’re setting out to do.

[00:27:01] HC: This has been great. Kit, I appreciate your insights today. I think this will be valuable to many listeners. Where can people find out more about you online?

[00:27:09] KM: Yes. We’re at Plainsight.ai, and I’m on Twitter @KitMerker. I guess it’s called X now. I’m constantly reminded, so X @KitMerker. I’m on LinkedIn, and I’m pretty easy to find. I’m going to be at the Kubernetes birthday party on June 6th in San Francisco. If this goes live before then, you should come see me there. If not, I had a great time.

[00:27:33] HC: Perfect. Thanks for joining me today.

[00:27:35] KM: Thank you, Heather.

[00:27:36] HC: All right, everyone. Thanks for listening. I’m Heather Couture, and I hope you join me again next time for Impact AI.

[OUTRO]

[00:27:46] HC: Thank you for listening to Impact AI. If you enjoyed this episode, please subscribe and share with a friend. If you’d like to learn more about computer vision applications for people and planetary health, you can sign up for my newsletter at pixelscientia.com/newsletter.

[END]