Using computer vision and an ever-growing dataset to perfectly detect situations where personnel is needed is no small feat, and answers the staffing issues often associated with care facilities during the night. In our chat, Harro explains some of the technical aspects of the software and the major improvements he has overseen recently before going into some connected topics such as privacy concerns, hiring practices, and validating the accuracy of the models. Harro is also kind enough to offer some general comments and advice regarding AI startups, and the areas he believes are most vital for founders to attend to.
So if you would like to hear about a great practical application of AI in the healthcare space, and some thoughts from a leader making waves in some uncharted waters, be sure to listen in with us!
- Harro talks about his academic and professional background and his companies before Kepler Vision.
- The specific problems that Kepler Vision is solving.
- Understanding the role of machine learning in Kepler Vision's service.
- Harro shares the biggest challenges that he and his company have faced.
- The task of building trust and the hiring practices that contribute to this.
- Validating the accuracy of models; Harro unpacks the labor-intensive process.
- The improvements that have been made to the software through iterative updates.
- Measuring the impact of the software; Harro talks about customer satisfaction.
- Advice from Harro to AI startups about hiring and focus.
- Harro shares his vision for the next five years at Kepler Vision and where to find them online.
LinkedIn – Connect with Heather.
Computer Vision Insights Newsletter – A biweekly newsletter to help bring the latest machine learning and computer vision research to applications in people and planetary health.
Computer Vision Strategy Session – Not sure how to advance your computer vision project? Get unstuck with a clear set of next steps. Schedule a 1 hour strategy session now to advance your project.
Foundation Model Assessment – Foundation models are popping up everywhere – do you need one for your proprietary image dataset? Get a clear perspective on whether you can benefit from a domain-specific foundation model.
[INTRODUCTION]
[00:00:03] HC: Welcome to Impact AI, brought to you by Pixel Scientia Labs. I’m your host, Heather Couture. On this podcast, I interview innovators and entrepreneurs about building a mission-driven, machine-learning-powered company. If you like what you hear, please subscribe to my newsletter to be notified about new episodes. Plus, follow the latest research in computer vision for people in planetary health. You can sign up at pixelscientia.com/newsletter.
[INTERVIEW]
[0:00:33.7] HC: Today, I’m joined by guest, Harro Stokman, CEO and founder of Kepler Vision, to talk about activity recognition for healthcare. Harro, welcome to the show.
[0:00:42.3] HS: Thank you so much, Heather.
[0:00:43.8] HC: Harro, could you share a bit about your background and how that led you to create Kepler Vision?
[0:00:48.6] HS: Yeah, I have a Ph.D. in computer vision and from the University of Amsterdam, and in the past, I had another startup. They were the very first ones I think in the world, which was 2009, that could recognize in photos and videos the occurrence of cats and dogs and sunsets and we were also the first ones that could manage to run those algorithms, deep confusion neural networks on a smartphone, and the company got very successful.
It was acquired by QUALCOMM, the American semiconductor company and after that, I saw the world of machine learning progressing and new topics in machine learning and computer vision, where human activity recognition and human activity recognition is a person standing up or bending forward on his knees and what is the person doing, is he changing the tires of a car or is he smoking a cigarette or eating or drinking?
And I thought, “Well, if these new technologies going to hit the market, where can it – where should it be?” and I saw so many opportunities that I decided to start up a new company and that became Kepler Vision. So we bring a new field of machine learning to the market, which is human activity recognition and we apply it to the care vertical, because healthcare has a bunch of problems.
In the Netherlands alone, there are over a hundred thousand job vacancies today because of aging population, the amount for care increases year over year between three to 6%, and on top of that, 25% of the caregivers are going to retire in the next five years. So there is this massive problem coming towards us, lack of care staff and I think that computer vision, machine learning, human activity recognition provides a piece of the puzzle too to solve this problem and that’s the background of Kepler Vision.
[0:02:50.2] HC: So what specific problems are you solving with Kepler Vision right now?
[0:02:54.7] HS: I think the biggest problem in care facilities is the lack of staff and then especially the lack of staff during the night. Can you imagine that if you have to start your work at 11:00 at night and then continue to work until 7:00 in the morning, that is the biggest problem and therefore, we created the Kepler Night Nurse, and the Kepler Night Nurse is a solution that it used a sensor.
The sensor is placed in the room of a client at the elderly care facility or a hospital or mental care facility and the sensor looks after the well-being of the client so that care staff doesn’t have to do it. So the productivity of care staff increases significantly during the night and what we can do is our software can sensor, it can recognize if a client is in bed. It can recognize if the client is sitting at the edge of the bed and often, they need help to get out of bed or to get to the bathroom.
Our software can recognize if a client has fallen if he’s in the bathroom and then he is there too long because clients may slip there or if he’s wandering in the hallways of a care facility and because we can do that, the care staff doesn’t need to worry, doesn’t need to do routine jobs and the same work can be done with less staff and in addition, there’s also a big advantage for the clients or for the residents in the care facilities.
If for instance they fall, our software recognizes the fall within 10 seconds, and then our software sends in an alarm to the phone of the caregiver so that the caregiver can instantly run to the client and get him on his feet whereas the alternative is that the client, yeah, remains on a cold floor, gets on the cold, maybe the client has fallen and he’s bleeding and often, they are – they take medication for block thinners.
So once they bleed, they can continue to bleed. So it has many advantages not only for the care staff but also for the residents.
[0:05:08.7] HC: So how does machine learning work within this? How does it recognize these different activities, how do you set that up?
[0:05:14.8] HS: By providing training examples. So we have collected, I think in the past few years, over a million training examples of the real thing or for real care facilities or real hospitals. We started out with basic machine learning models and discovered they did not work on the very well. We added many of the missed falls and missed detections to the training set so that now, we have over a million memory in updated samples. We train the algorithms from that. Yeah, that’s how we do it.
[0:05:52.4] HC: Are you working with still images or video in order to recognize these activities?
[0:05:57.9] HS: Both. So, you run it on still images but hen they seek to combine evidence so if one the single image you are softening text that a person has fallen, it’s due to be, before it sends a message and make sure that also in the next subsequent, 10 to 20 frames the person has fallen.
And anly if that is the case, it sends out an alarm then by averaging over time, this ensures that the false alarm rate and the missed detection rate is incredibly low.
[0:06:30.8] HC: What kinds of challenges do you encounter in working with this type of imagery from healthcare facilities and what kind of challenges does it create and trying to train a machine learning model?
[0:06:39.8] HS: Yeah, I think the main challenge is the privacy concerns that customers often have. They’re completely right. So, we use a camera to look after clients that cannot look after themselves, because they have physical problems, because they have mental problems and we film them in their most private moments.
And the way we resolve it, is three-fold. First of all, we stick to all the rules regulations so we sign with care facilities, we sign a data processing agreement and the data processing agreements is defined how long we can store the data that the data needs to be encrypted. What type of people can look at the data so that is one. The second one is that although, the use of camera essential, the output of our algorithm is not an image but it is a text string so we covert video to text which is very privacy-preserving and a server clean, we do use, we do store images now and then to calibrate the models and to further improve the accuracy of our algorithms.
For those images, similarly to Google street view, we blur the faces so that even if you want to look at those images, the patients or the residents are not recognizable. And to give you an example for what that means, a few weeks ago, we had a customer, a big care facility that had an intruder, a burglar. The police saw that there were cameras in the room, so the police asked for the imagery. We provided those images, however, and the intruder was indeed visible but our software had blurred the face, so the images were not useable.
So, that is how we solve the privacy concern. And I think, that is the main hurdle that we often need to take. After that also, staff needs to be trained to use our software, they need to build up the trust that they don’t need to do the nightly rounds anymore, that they can trust our system, and that takes lots of explanation. But I think those two are the main things that we need to resolve, and that we have resolved.
[0:08:57.9] HC: How do you go about building trust then? You mentioned the explanations, is it by helping them understand how the system works or there are other means that you can use to build trust?
[0:09:08.0] HS: Yeah, I think in healthcare there is a division between the people that provides the care themselves, the nurses on the one hand and there is also what I refer to as the IT guys and very clearly, I am an IT guy and lots of the Ph.D.’s that work for me are also IT guys. People that work in care do not trust or have lots of affinity with people from IT, they much rather talk to people that also had a past of providing care.
So actually what we have done and what we have learned that, we have hired care consultants, former nurses, people that were nurses in the past. We have trained them on how to use our software and they are the best representatives of our business. They are trusted by caregivers, they are the ones that provide the training courses.
[0:10:03.6] HC: How do you go about validating your models to be sure they’re performing as accurately as you need them to?
[0:10:10.0] HS: That is lots of manual labor. So we do the first month or so after we connect a new client, we do quality assurance. So the first five or ten signals that are software generates our manually inspected by quality assurance professionals that work for us and they check whether a generated alarm like a person sent when they get to the bed, person leaving the room, a person being on the floor whether those alarms are correct and if they are not, they are flagged.
Added to the training set, a new neural network is trained and deployed again to the customer and that is now through an incredibly low false alarm rate and I think we have one false alarm per room for three months, which is way better than any other sensor that we encounter in the field but that is the strength of machine learning of today’s machine learning.
[0:11:11.0] HC: So that feedback loop of identifying errors and feeding them back into that model that definitely helps to improve your model over time. Are there any other things you’re doing to ensure that your model will perform maybe in different facilities on different types of people on different care environments? Is there anything special you’re doing there?
[0:11:32.2] HS: So I think in professional care, our software now pretty much works out of the box. It took some time to get to where we are. So in the first customer and the first client’s rooms, our software would make mistakes. There would be a hat hanging on the wall, there would be a coat underneath, and a pair of boots, and our software would mistake that for a person and it would send an alarm.
Around Christmas, lots of small little statues of Joseph and Maria would be placed on closets that tricked our software. We also had a case of an elderly client who used to have pigeons, who have stuffed pigeons, which confused our software but over time, we added more and more examples to our training sets and we are now at a phase where our software pretty much works out of the box actually.
[0:12:26.4] HC: So it is really that iterative updates to the training sets that has got you to this level of performance.
[0:12:33.6] HS: Yes, that is correct.
[0:12:35.0] HC: Are there any modeling aspects perhaps to make your model more robust in different lighting conditions? Are there any modeling list aspects like that that’s played a part in getting to this level of performance or was it really just to focus on data?
[0:12:48.3] HS: Yeah, I think it was on data because we have now so many thousands, our software has now seen so many thousands of rooms that it has seen pretty much all of the different lighting conditions. So if a camera is misconfigured, if all of the pixels or 50% or 20% of the pixels are overexposed, then our software doesn’t work properly, and we had also cases that cameras were put up but the installers had forgotten to take off their plastic, the covers, that protects the lens.
We have also cases that a very big fly or a mosquito is on the lens, while in those cases our software doesn’t work, but we can detect that automatically that there is something blocking the lens, there is something wrong and we can then warn a nurse but in all other cases, I think it is a matter of lots of training data and automatically the software works out of the box.
[0:13:46.2] HC: How do you measure the impact of your technology?
[0:13:48.9] HS: I think the answer there is customer satisfaction. So often we have new customer, they want to sort of try out or they’re careful, so they want to try it out on one facility with 16 beds or 20 beds and try it for a year and then after two months, after three months, the nurses get more and more excited by our software because it helps them a lot. It saves them lots of time and it improves the lives of the residents and then afterwards, they scale up.
So a customer that scales up from 20 beds to 200 beds or to 2,000 beds, I think that is the biggest indicator of the positive impact that our software has on the life of the nurses and the life of the residents.
[0:14:38.5] HC: Is there any advice you could offer to other leaders of AI-powered startups?
[0:14:42.6] HS: Yeah, I thought about it. One is to — what helped Kepler a lot is to hire people that have worked in healthcare that do not have IT backgrounds but they have a healthcare background, that have been former nurses. I think that was a game changer and I think your answer is you need to focus. So Kepler focuses on, I often say, we are one inch wide and you know, one mile deep.
So on the field of elderly care and hospital care, our software can look after the wellbeing of elderly clients and that is all we do and we do nothing more but what we do, we do incredibly good. Our false alarm rate is way beyond what anyone else does and this gives so much evidence to potential customers that our software is really so much better than anything else out in the market.
I think that focus has helped us a lot. The alternative would be to go much broader, so we’ve been asked to provide detection in industrial settings or in the cleaning of oil tankers and so on but we have always stayed focused on healthcare because you know, with limited resources you can handle all the – so many verticals and we stayed faithful to the healthcare vertical. So that focus, yeah, my advice would be to focus.
[0:16:14.9] HC: That definitely is very good advice. It is much better to solve a very specific problem very well than trying to do a whole bunch of things and being mediocre about them. So I definitely hear what you say there and finally, where do you see the impact of Kepler in three to five years?
[0:16:30.7] HS: The goal of Kepler, the mission is to that our software looks after the wellbeing of one million clients, residents, and patients by 2030. So three to four years from that, we should be at say, a quarter of a million and that is yeah, that is the impact. That is still under the growth trajectory, maybe not a quarter of a million but I expect three years from now, our software looks after 100,000 patients, and if we achieve that, I will be incredibly proud of what the team has achieved.
[0:17:06.6] HC: I look forward to following you and hearing more about that journey. This has been great. Harro, your team at Kepler Vision is solving a really important problem with activity recognition. I inspect that the insights you’ve shared will be valuable to other AI companies. Where can people find out more about you online?
[0:17:23.1] HS: Our website is www.keplervision.eu.
[0:17:28.5] HC: Perfect. Thanks for joining me today.
[0:17:30.7] HS: Thank you, Heather, for inviting me and giving me the opportunity to speak about my company.
[0:17:35.9] HC: All right everyone, thanks for listening. I’m Heather Couture and I hope you join me again next time for Impact AI.
[END OF INTERVIEW]
[0:17:46.5] HC: Thank you for listening to Impact AI. If you enjoyed this episode, please subscribe and share it with a friend and if you’d like to learn more about computer vision applications for people in planetary health, you can sign up for my newsletter at pixelscientia.com/newsletter.
[END]