Machine Vision—Coming to a Kiosk Near You

August 12, 2021

Kenton Williston

Machine vision in kiosks, retail kiosks and machine vision

Self-service kiosks have changed the way consumers interact with retailers, banks, and the hospitality industry. Now, the way consumers interact with kiosks is undergoing a revolution. But it’s going to demand a lot in terms of new hardware, new software, and new connectivity to back-end systems.

We look at the changing role of self-service kiosks across multiple industries with Stephen Borg, Group Chief Executive Officer for AI technology company meldCX. We explore the role of machine vision in creating a seamless and connected experience, and how businesses can get the most out of their kiosks.

How do you see the ecosystem behind kiosks changing?

It’s quite interesting, in that we started from a software platform helping customers or partners execute more quickly. What we found, especially around machine vision, was that the integration between software and physical-device design needed to be more tightly integrated. What we do is help design that service, or work with partners to do that.

But most of all we’re finding that our customers are trying to replicate or create a better customer experience. It’s not just about pulling people through quicker, or line busting, or those traditional use cases. Usually the checklist we get is: “I want to create a richer, more engaging experience while minimizing the number of touches. But I want it to be more personal.”

Our mission is a seamless experience—not only for the customers using the #kiosk, but for the businesses deploying it. via @insightdottech

How is the industry responding to demands for new technologies like machine vision?

We’re seeing machine vision playing a really big part in those applications that make the whole process seamless. A great example is in postal services, where it can be quite complex—sending a parcel and making sure you fill out your forms right.

Machine vision does handwriting recognition and automatically detects the destination and what else we need to know, or verifies the address. It cleans the data on the way through to ensure your parcel gets there.

And it’s being used to connect an experience. We’re working with a retail bank right now, where through tokenization it distinguishes your skill level in using that kiosk. So it can go straight past any instructional content and get you right to the point. Because that’s your expectation—when you’ve used the kiosk once or twice, you want that interaction to be quite seamless.

Hotel check-in is another example of machine vision and AI applications being brought into new environments, such as the work we’re doing with the Marriott Group. They want to create a universal premium experience down to a kiosk device. For check-in, valet, any service that you typically require.

These are some of the use cases driving the need to tightly integrate AI or machine-vision solutions back into kiosk applications.

What are people doing to make sure high-traffic kiosks stay clean and safe to use?

The first thing we did was to open our platform to a lot of different options, such as eye and finger tracking—all these things where you don’t have to physically touch the device. What we found was that the end user wasn’t quite adapted to that. And so it didn’t create the best experience.

We found two main trends. Antimicrobial is one of them, but there’s another area, which is quite interesting. We created a piece of AI that allows you to heatmap touched areas on a kiosk. It uses a combination of the pressure sensor and touchscreen—and if there are any physical cameras in the screen—it allows you to create a complete digital manifest of areas that were touched.

Research we did with customers found that they were concerned about cleaning—that the kiosks were cleaned, and for long enough in the correct areas.

We created another AI tool that sits in the background and keeps a manifest of everything that’s touched or interacted with. You can set thresholds at a corporate level, and it messages a local attendant, or it can even stop the kiosk being used if it hits a threshold. And then it goes ahead and creates a complete digital manifest of who cleaned it and when.

Once you put it in that mode, it shows all the heavy-usage areas, and if there’s one area in a touchscreen that’s heavily used, it would literally make you rub that out. It’s like you’re rubbing an eraser. And we found that to be hugely popular, because it gave our customers confidence that their staff on-site were cleaning appropriately. And it gave them a full audit of their activity for cleaning.

This idea was started by a customer that had an outbreak in Australia even though it was regularly cleaning its kiosk. But they weren’t paying attention to the other devices that are on the kiosk. They were cleaning the screen, but they weren’t cleaning, say, the PIN pad. So this system would create a process flow and say, “Clean PIN pad now” on the screen.

What are some of the broader ways your customers are using machine vision for kiosk applications?

We have a customer that wants to detect the type of handbags females are holding when they’re interacting with their devices. So they know what their spending capacity is—which is really interesting. So if that’s in a mall, they’ll know: “Do I arrange to have a Gucci in this shopping center? Or is it a Coach?”

And, more interestingly, there’s product recognition. We’re seeing more and more customers, especially in grocery, that want to reduce waste. They want to be more conscious, not only about wasted packaging, but people taking only the portions they need.

So we use deep learning to detect the device or object—even through bags—and let the kiosk know what that is. The customer has a very seamless experience; they don’t need to enter a PLU code or a barcode or scan anything; they just put the object on there, and it’s remarkably quick.

Recognizing labels as well, to make sure there’s certain compliance—we’re finding that on deli and meat products. We’ve got a customer that will make sure a client doesn’t leave with something that’s out of date, or very close to out of date. All those type of things you’re starting to see to make that shopping experience more convenient. But they also have other goals—being environmentally conscious, reducing waste, and food safety.

What do you see as being the critical considerations of integration with the larger software and services universe?

We’ve been heavily focused on that because our mission is a seamless experience—not only for the customers using the kiosk but for the businesses deploying it.

And Intel^® has been fantastic, giving us access to tools and getting our kits ready-to-market so others can use them. And further, we’ve created some universal APIs—not only to common peripherals but over 3,000 integrations to things like Salesforce and ServiceNow. So customers can easily take their API token, apply it, and they’re ready to go.

We have worked with Intel on multiple solutions for other kiosk manufacturers.

I’ll give you an example. There are a few legacy-payment terminal types that might have a situation where the PIN pad gets out of sync with the kiosk software itself. That situation might require a hard reboot or PIN pad timeout—which ultimately creates a bad customer experience. They’re in a state where they don’t know if their payment’s being processed; they don’t know how to move on.

In this case, we have different layers of what we look at. There are AI timers, and those look at various operations at a kiosk and automatically intervene. They might cut power to a PIN pad and re-engage that power while you’re in that transaction so you can continue. Or they might cycle a card reader. Or any of those things that would typically be done when you’re calling a support desk—it does that automatically, and sometimes seamlessly.

Is there anything that we didn’t get to that you’d like to add?

The one thought I’d leave you with is that when we talk to customers now, especially about kiosks, it’s so advanced, and there’s this perception that they’re very transactional. We really start the journey by asking our them: “What would you like the kiosk to hear, to see, and to do?” Because that’s what it’s really about.

And when customers think more broadly, you see some really interesting use cases, and those manifest into great experiences.