SMART CITIES

Jaume Portell: We think that next big wave, is making sure that everything that we see that can improve that customer experience is seen, is detected and then transform that into action, action to the customers and action to the managers of the stores.

Kenton Williston: That was Jaume Portell, Co-Founder & CEO at Beabloo. And I’m Kenton Williston, the editor-in-chief of insight dot tech. Every episode on the IoT Chat, I talk to industry experts about the technology and business trends that matter for developers, systems integrators, and end users. Today I’m talking to Jaume about the reopening of public spaces such as transportation hubs, retail establishments, hospitality venues, and schools. Beabloo has done some great work in the last year using multi-sensor Edge AI and intelligent signage to create environments that are both safe and inviting. I want to know what’s next for these spaces, and how to deploy the next generation of technology while protecting their budgets. So Jaume, welcome to the show.

Jaume Portell: Thank you very much, Kenton.

Kenton Williston: So, I know that you are a co-founder of Beabloo, and first thing I wanted to ask you was what exactly led you to start Beabloo, and how the company has evolved since you began it in 2008?

Jaume Portell: Well, actually, we started thinking that we could bring some of the intelligence that was by then deployed in the e-commerce sites, where e-commerce was controlling the message and measuring the impact of every single step in the process to brick-and-mortar stores. And the challenge was very interesting, because we had to bring the analytics first, and then also control the digital-message delivery in those physical locations. So we do this by using computer vision and other sensing technologies to understand how customers move, what they want, what they’re touching. And then we adapt the communication, the value proposition of the store, using signage, using electronic shelf labels, and also in sending hints to staff in the store to serve customers better.

Kenton Williston: Yeah, those are all really, really hot areas right now, so it’s great to see. You’ve been around since 2008, definitely a leader in that space. I am curious, though—I know there is a company called Metriplica that’s part of the Beabloo group. What is that, and what do they do?

Jaume Portell: Metriplica is the consulting arm of Beabloo. We realize that when you’re doing digital transformation of retailers, or airports, or banks, or retail banking, you are touching beasts that sell billions of dollars, right? So if you want to do a proper job, you need to prove the value you’re creating with your digital-transformation process. So it’s not about applying technology and that’s it; it’s applying technology with sense. And that we do measuring the before and after, creating control groups. And we do this with consulting firms, and we are collaborating nowadays with companies like Deloitte or Accenture. And Metriplica is our consulting firm, and we acquired them to bring artificial intelligence know-how to the company—to bring very brilliant data science to the company.

And we use their know-how to improve our algorithms, and at the same time, to deliver these direct success cases to the local market in Spain. And those success cases have help us to be known, to prove the outcome we generate with those digital-transformation processes, that they by themselves generate a positive ROI on the first year, and then help other systems integrators, or other large systems integrators, to replicate those success cases in other markets.

Kenton Williston: Got it. That totally makes sense, and leads me to the topic of today’s conversation, which is, of course, that the world was totally turned upside down last year with the pandemic. And there’s been a complete rethink about how public spaces—whether those are retail establishments, or airports, or whatever the case might be—need to run their operations. And so all these technologies you’ve been talking about—in terms of being able to observe the behavior of people in public spaces and create intelligent analysis of that—has become valuable for completely new reasons. And I’m wondering, from your perspective, what you see as some of the biggest challenges in public spaces as we move more towards reopening.

Jaume Portell: Well, first of all, we’ve been creating technology to improve customer experience. If you want to improve the life of someone or the experience of someone, you need to take care of that someone. That is what our technology does, is look at what people need and react to it. In times of pandemic, you want to deliver messages to customers to help them understand that they are working in a space that’s protected and properly clean and where measures of protections are properly taken. And that is what digital signage does in this context. We have analytics in that digital signage that senses how those messages are being understood by customers—if they are paying attention to it, if they are actually reacting positively to it. And we’ve seen that messages related to COVID protection, related to the new rules of the game—wearing masks or social distancing—this type of stuff is 300% more interesting to the audience than the value proposition of the retail space. So, extremely relevant: communicate the new rules of the game.

The next one is, are you wearing the mask if you’re walking inside? So we have trained our systems to detect masks on the faces and create real-time reaction to that, and our digital signage systems warn anyone trying to walk in a physical space where it’s regulated by mask protection, and in the case they are not protected, it warns them to not walk in. At the same time, physical distance is critical. We’ve trained the systems to sense physical distance between human beings, and we trigger alarms in the system if they are too close to each other. And that is also a real-time warning system that helps everyone to be aware that that’s a little dangerous. And on top of that, we can sense temperature using specialized cameras that are—on one feet of the camera detecting if you’re wearing a mask, on the other one detecting if you have fever potentially, and it triggers alarms to the staff in the store or in the bank branch or in the hospital or in the airport—to the staff to take care of that situation.

So technology that senses risk and communicates that to whoever can help to protect the staff, to protect the other users of that physical space. And we do this with computer vision. We also add some additional layers of artificial intelligence to clear noise from that data, to create heat maps that represent for the owners of that space where the interactions of risk are happening in that physical store, so that we can sense how to recreate the flows in that store to improve the security of people visiting. So there’s many things you can do. We’ve seen that the hardware in most of the cases that’s already there, it lacked some additional intelligence for the situation where we’ve been. We created that intelligence, and that same hardware now helps you sell more, but at the same time also protects your staff and your customers.

Kenton Williston: Yeah, so you’ve touched on a ton of interesting points here. And some of the things that really stand out to me—you’re talking about the idea of creating a space that, is it comfortable and inviting for customers? And of course, again, even what that means has changed a lot. But I think the implications around computer vision and AI—there’s a lot of sensitivities around that. So, here in the States, for example, there’s been a lot of, I might say, backlash—a lot of concern around how these technologies might be used, especially when it comes to sensitive things like racial profiling. And then, even globally there’s plenty of places where people have different sorts of opinions about mask mandates and things of this nature. So it seems to me like a double-edged sword. But I do know that you’ve done some really great case studies that really showcased the power of these technologies, and proved out—people really are having a positive reaction to this technology. So can you walk me through what you did there, and the results you saw?

Jaume Portell: So, basically, , we deployed our intelligent digital signage network with audience analytics. So we sense who looks at the content. When we say “who,” we mean the computer-vision systems see faces and take note of: this looks like a face of a man of this age. And it’s been looking at the content for 15 seconds. That is what we take note of. There’s no ID; there’s no way we can re-create an ID from that if the system sees the same face again 20 seconds later. It thinks it’s again, another face, different face, probably same age, same gender, but it has no clue that that is the same person. So no data-privacy issues at all. The system doesn’t record images; the system has been trained to count faces. It’s like you’re working in a physical space and someone is taking notes of 1, 2, 3, 4. So that’s what our system does. It’s very specialized to measure certain demographic characteristics, but they are anonymous, 100%.

We are GDPR compliant. It’s one of the toughest standards in the world for data privacy in Europe. And we apply it everywhere, with the goal of protecting the privacy of anyone that has been seen from any of those cameras, which doesn’t mean that the system cannot understand things that can be of certain risk. And this can detect two bodies too close to each other, or it can detect someone not wearing a mask, and it can trigger automatically a message saying, “Please remember that you have to wear a mask to be in this physical space.” But it’s not pointing to anyone, and it’s not recording any ID at all. So it’s a marketing system that makes physical spaces smarter, but it doesn’t trigger any concern about data privacy in customers at all.

Kenton Williston: Got it. That totally makes sense. In other words, it’s not like, “Hey, you—Jaume, you’re not wearing your mask. Demerits for you.” It’s more of a matter of, there’s a number of messages you could be displaying on these screens, anything from—let’s take the airport example again—directional information or flight updates or whatever, to all kinds of things that we’re specifically responding to, what was happening as people pass by—whether it’s people being too crowded together, or not wearing masks, or even if they’re revenue-driving things you wanted to do on that same display, like, “Hey, I see you’re of a certain demographic that might be interested in a new pair of headphones,” or whatever it could be. These are all things you could do, and all of these would be non-intrusive and not personalized in the sense of to a specific individual, but very meaningful to the audience, because they’re very responsive to what’s actually happening in real time.

Jaume Portell: Totally correct. We’ve done cases in airports where we’ve used the artificial intelligence technology to adapt the content based on—where is the next flight departing to? And that improved sales in duty free stores in a relevant way. And, at the same time, detecting if you have some staff in certain areas of the retail stores in the airport, or not, explains very much the conversion on those areas of products. And that is, again, another very valuable message, because it tells us how that customer experience is built up. If you went to a store in an airport and you end up buying a suit, and the reason why you did that was because you had half an hour waiting time.

That is very relevant information for the brand, very relevant information for the retailer, and very relevant information to improve the customer experience of the next guy like you—that if we detect someone walking in, probably it makes sense to have someone in the suit section to try to convert and create a better customer experience. So that is how the artificial intelligence that we are deploying in airports is enhancing, improving that customer experience—is taking care of the magic of what’s happening, and it’s controlling the messaging, and adapting it to the situation. The situation is the genders that are walking in, the families that might be walking in, the type of customers that might be walking in—still completely anonymous, we don’t know who that is, but we know that in thousands of times before, that type of customer wanted certain types of services or products.

So offering it improves the conversion, because it improve the customer experience. So that’s the magic of the systems. Right now, if you walk in to these spaces and I tell you how safe is the space, and how much we’re taking care of—cleaning and everything else—you will feel confident in walking in. If we tell you that we are counting every single individual that’s walking in and out, and we tell you: It’s now 10% full. So you know you’re free to walk in, it’s super safe—you will walk in way happier about that customer experience and your experience with that space will be more positive, and you will be more likely to purchase something from that location.

Kenton Williston: So, I think one of the most interesting things about everything you just described here is, it really highlights how you’re not only implementing AI technologies on the spot at the Edge to say, Who’s coming in? What kind of person or group of people is this? What are they doing in terms of masking, social distancing, etc., but also that added layer of intelligence, like, okay, if there’s a flight coming soon, where is it going to? What kind of sales history have we had in the past for these kinds of demographics? And I think that gets to the point you were making about that higher-level AI. It’s not just the on-the-spot recognizing the demographics and the social conditions, but also taking that to the next level and saying, “Okay, we have history now that we’ve developed, and we can start bringing some even higher-level intelligence to what actions we should take for this particular scenario.” And that’s really powerful. I love that.

I do want to talk, though, about some of the other sectors. So, everything we’ve talked about so far has been in the transportation sector, and really great to hear how you’re enabling transportation hubs to reopen, because I think that’s really critical to getting our world back on solid footing. But then there is the—a world more broadly of retail, and similar customer-facing establishments like banks and food service establishments. And what do you see as some of the big challenges there, and how are you addressing those?

Jaume Portell: So, retail banking is in serious transformation all over the world. They’re trying to help us all use the banking systems online, and they keep a certain level of branches open for higher-level face-to-face services. Their goal is to improve the customer experience in those locations, and help us in the journey of financing our dreams while we’re in those locations, right? So they want to explain to us that we can finance the university of our children, that we can get a system or a loan to pay for our next trip to Maldives, or whatever. So they want to help us on that front. And that requires communication, and it requires customer experience. If our visit to the bank is a pain, then we don’t want to come back, and we might end up doing that by ourselves somewhere else.

So they want to be our partner to make our life better and easier in the area of purchasing those dreams of our life—our wedding or the university or something else. So if you want to do that, you have to have a great experience in the point of sale. And that requires communication, it requires music, it requires sensing waiting times. It requires understanding the quality of service you’re delivering by asking your customers with digital customer-satisfaction surveys, and also measuring with analytics their waiting times, and when and how they are enjoying the experience. And that is a lot about sensing, and it’s also a lot about understanding what happened when you implemented certain campaigns to flood the traffic in branches. So queuing systems or scheduling visits to your bank branch is having an impact on that.

Measuring the waiting times in banks can confirm that computer vision is key to understand waiting times. You need to connect one camera to the next one and the next one, and follow customers when they walk in, so that you know who is getting in, who’s getting out. Not who with an ID, but who as to count how many seconds each of them had in the branch. And that is very relevant to measure the customer satisfaction, and that is very relevant to build loyalty in the customer. So banking needs sensing, needs digital transformation of the stores with computer vision, with those screens that explain the value proposition of the bank, the values of the bank, and it needs to be well connected with a queue-management system so that everyone feels informed of when they will be served.

So all this is perfectly possible with technologies like the one Beabloo is doing, and it enhance that customer experience a great deal. With all those technologies understanding where are the zones of risk of COVID-19. Or which are messages that are getting higher impact in your customers when they are in the store is another usage, very relevant in these COVID-19 times. But, actually, we sense and we see how valuable it is for customers, and how valuable it is also for the staff of the banks, because they also enjoy that level of information. They like to know that the occupation of banking is within the limits, that the distance is most of the time followed properly, and the measures of protection are working well.

Kenton Williston: So, Jaume, I have to tell you, this is a very relatable set of examples. I’m right now just about to close on buying a house. And I’ve been working with different financial institutions in order to do this. And it’s funny—some of the folks I’m working with have been fully remote, and some people I have worked with in person. And it’s wonderful to be able to do things remotely, but I have to say, the experience of working with people face-to-face, or mask-to-mask as the case might be, has really been so much better. And I think there’s clearly a need for institutions like banks to be able to provide an inviting environment for people to come into—to provide an environment where they feel like they’re getting great service, where it’s very enjoyable. Because, I have to say, the people that I’ve worked with in person I feel much more connected to and attached to, and I can definitely say from my own experience that I’m much more likely to go back there and use their services again, versus the people who are remote and are just an email address.

Jaume Portell: Yeah, human contact happened to exist before even the language existed, right? So we are rooted to looking at our eyes and trusting each other. And that is extremely valuable in retail, and it is extremely valuable in banking. Sometimes you need someone to trust, and that’s the value of local services. You can look at their eyes, and you can understand that they are doing the right thing. And that is unique, and it’s not transferable to an e-commerce website. And that is the value of human beings serving human beings in a direct mode. And that is measurable with computer vision. We can measure the staff context with customers, and how much value is that creating for brands in physical locations? And that is revealing the value of those interactions, of that local service that cannot be changed by an e-commerce transaction.

Kenton Williston: Yeah, absolutely. Totally agree. So, another thing that I have to say I have a very personal connection to is the education sector. So, my daughter right now is just in the other room, in our very small apartment, which is why I’m currently looking at buying a house—we’re too close together. So, again, I’m very grateful that she’s been able to continue her education remotely. But I’m very excited for her to get back to school and for us not to be so close together all the time. So, what do you see in the education sector as being some of the ways this technology can help schools reopen?

Jaume Portell: Yeah, well, digital communication in schools has been there for a while—the intranets, the local content delivery—mix hybrid systems, where part of it is online, part of it is face-to-face. I think there’s a lot of value to be offered in the interaction between the community and the organization. So, the professors talking to the organization, the students talking to the university, and doing all transactions by interacting with kiosks—by interacting with physical, digital displays that can talk to them and can understand them way better than their traditional web interfaces. And I think there’s a big, big opportunity there in voice and in natural language processing systems that understand what the customer is saying, what their intent is, and navigate them through the complicated systems of the schools or the universities in order to get information on how you’ve done, and which is the next step you need to do in order to graduate, or get to the new course, or whatever.

And navigating through those information systems is a very, very complex challenge that, when articulated through natural language, is way easier. And that is where universities will save a lot of time, create a better customer experience—in that case, customers are professors, and the whole community and students as well—and serve them better with information they need to do the work, or to just complete their tasks.

Kenton Williston: Yeah, I think this is a great example—what you were saying earlier about different kinds of sensors that can be incorporated into these systems. It’s the vision, it’s the thermal sensors, it’s audio sensors—probably other things as well. And I think we put these different technologies together, along with the higher-level intelligence—whether it’s knowledge about students’ needs, or travelers’ needs, or people in banks—you can begin to come up with some very, very interesting use cases. One last area that I want to touch on, though, is, I think, probably the most challenging if I’m thinking from a health and safety perspective. And that is in the space of things like hospitality and event venues and entertainment complexes—where the whole point of these facilities is to get people together and have them stay together for some period of time. And of course, this is very counter to how the world has worked for the last little bit. So what do you see as being some of the most important trends in those sorts of applications?

Jaume Portell: In computing, occupation is extremely relevant. Understanding when these spaces are full of people or not, is critical to cleaning them when they are not, and getting them ready for the next set of users. So, you with your family are in a resort. You want to go to—you want to go to the swimming pool. You would love to have your hamacasand stay there to have a drink. There’s a table—you don’t know if that’s clean or not. So you want to know if that’s clean. Having computer-vision systems observing the space to make sure that after someone uses it, someone went from the staff to clean it up, will give you the confidence that that place is right. So that’s a simple use case, but it is very relevant. You want to know that the space is available. Interaction Care, our system of understanding if a physical space has too many people or not will tell you that.

And you want to know if it has been cleaned up for your usage and the safe usage of your family. And that can be told by a system that’s observing the occupation, and, later on, it’s observing if the cleaning staff went there to take care of it. So you can see the tables—all of them empty, and you can see that nine out of ten are actually clean. But there’s one that is not yet cleaned. So that is, in form, in a digital signage system you know where to go, because there is a computer-vision system taking care of that. So it’s taking care of the invisible enemy of COVID-19. Because we human beings bring it here or there, but we human beings can make it clear for the next usage. So that is how, in hotels or resorts, you could use this type of technology.

Kenton Williston: So I’m going to take just a second here—I want to do a time check because we’ve got about eight minutes left in the hour. I can go a little bit longer, and I think if you’re available for maybe another 10, 15 minutes past the hour, we could get all the way to the end of our questions. Let me just real quickly send my colleagues a note, because I’ll be a little bit late to the next meeting. Okay, perfect. So, yeah, that totally makes sense. And I’m curious—do you foresee that there might be government regulations in the future around this sort of cleaning? I’m sorry, just a second here—just getting a poke from my colleagues.

Okay. So I’ll start my question over again. So, yeah, that totally makes sense to me. I can see how that would bring a lot of comfort to guests. And I’m also curious—do you foresee the possibility of government regulations around some of these cleaning protocols that you could help organizations meet? Or do you think this will be more of an informal, based-on-customer-needs scenario?

Jaume Portell: I think there’s two areas, two folds. One is what governments are enforcing you to do—or to restaurants, to hotels. This is happening in Europe—there are regulations, you need to follow them. But the most important one is the signature of quality of that space. So, actually, hotels and restaurants want to do it even better and communicate it like that—proving that they are doing it like that, so that everyone feels safe. And I think this is the point—it’s not about doing what the government says, it’s about doing what your customers will appreciate you doing, and communicating it, and making sure everyone knows that you’re doing the right thing, because that makes everyone feel safe. “Oh, no, please don’t sit there—it hasn’t been cleaned yet. We will clean it, then you can use it safely.” And that is the point, is you want to feel that they are taking care of you. And the best regulations are the ones self-imposed by those bars and restaurants and hotels to take care of their customers in their own way.

Kenton Williston: Got it. And the other big question I have is the cost question. So, it’s obvious to me how beneficial these technologies are. But I think there’s always a question of—especially given how tight budgets have been for a lot of the industries we’ve been talking about—how affordable these technologies are.

Jaume Portell: Well, actually it is surprisingly affordable, because how many retail spaces do you know that have security cameras, CCTV cameras? Many, right? Or banking—more of them, or hotels, many of them too. So those are the devices that they need to take care of their customers. How many of them have digital screens? Many of them have. And the thing is, the only issue is not the hardware—it’s the usage of the value that hardware is generating to actually make the places smarter and take care of customers. And that means connecting those streams from the cameras to a computer—an Intel computer that can get the information out of those streams, make sense out of it, and react in real time using the digital signage that anyway was available in the front door of the location. So what we’re talking about here is injecting intelligence in the hardware that is already in those physical spaces.

So we are talking about software deployment. It’s easy, it doesn’t require much installation, and it’s creating value by itself immediately, and, actually, it usually pays off for the hardware as well. Even if it was already there, right? So these deployments can be started from scratch, they pay for themselves, and they bring a positive return of investment, even taking care of the hardware cost. But if you already have digital screens, CCTV cameras, networking devices—you can use all of them to create more value with Beabloo Active Customer Intelligence Suite, that will sense with the cameras, sense with the Wi-Fi access points, understand what’s going on, and trigger the right answer to your digital signage systems, and actually analyze also the sellout of your stores of your cafeteria of your retail space to understand what’s working, what’s not working in your digital campaigns, and improve the selection of them based on the hour of the day, based on the day of the week, to increase your customer service perception and their conversion to sales.

Kenton Williston: Yeah, so, in other words, what you’re saying is most of the infrastructure is already there, and it’s really just a matter of making sure you have somewhere in your system—some Intel-based hardware that can run the Beabloo software, and then you’re good to go. And, like you said, yes, there’s some cost to installing these things, but they very quickly pay for themselves both in the ephemeral, hard-to-measure, customer experience, and very, very directly in terms of helping you optimize the digital advertising that you were doing anyways, and making that much more effective.

Jaume Portell: That’s absolutely right. I couldn’t explain it better. I will record this one, then I’ll make sure I can use it like that. Very well phrased—thank you.

Kenton Williston: Perfect. Well, so I want to think a little bit about where we’re going next, and what organizations who manage all these different kinds of public spaces should be thinking about as we move forward into the next, post-pandemic era.

Jaume Portell: The first thing is, when you start thinking about improving customer experience or taking care of your customers in a situation like the one we are in right now, you are sensing customers and giving them what they want as soon as possible. This is exactly what you want to do when the pandemic is over. You want to keep serving them and taking care of your customers as much as possible. So part of it is real-time reaction to it. Part of it is making sure that the stores and the physical spaces give the information the audience needs at any given point of time. So, real-time preparation for the scene and for the context—every hour of the day, every store, every space differently. But then the next one is to prepare the staff in the store so that they are ready to serve customers better when the day starts. So all these systems that we’ve been discussing are sensing people passing by, people walking in, people buying stuff—they know what the customers are buying, when they are buying it, so they can feed in an artificial intelligence engine.

In this case, it doesn’t need to be run at the Edge. Beabloo runs that on Intel computers, but in the cloud—can train a machine learning system to understand what will happen tomorrow. Today, this morning, was a sunny day in Barcelona. A sunny day has a completely different behavior for retail environment than a rainy day. And considering that today is Thursday, the purchase pattern of customers in that particular store is different than tomorrow, which is Friday, and it’s the day before the weekend. So these things are seen, perceived, and understood by machine learning algorithms that can explain it, when the day starts, to the staff in the retail store. And they can tell them, “Look, today’s Friday, and this weekend there’s going to be good weather.” So most of our customers will buy barbecue stuff from the supermarket.

“So make sure that we are replenishing that area as often as possible, because otherwise we might not serve them as well as we would like to.” That simple. I mean—and this can be sent as a video message to the staff in the store in the early morning, so that they understand what will happen and how to serve their customers better. So the intelligence is there also to empower staff to provide a better customer experience. And that is the next big thing—is sharing that intelligence with everyone, sharing that intelligence with the customer, sharing that intelligence with the staff. So if we’re counting how many people are in a given store at a given point of time, we can warn someone walking by that it’s not safe to walk in, or it actually is super safe to walk in.

We’re sharing the intelligence with the customers. If we tell the store manager that, today, they will be selling a lot of ice cream, because there will be a peak of temperature at noon, and please make sure that you have the right level of stock because we want to serve our customers better. This is using the intelligence of the store to help the staff to do their work better. So it’s about sharing the intelligence with everyone that can increase the customer experience. And that’s the point. And we think that the next big wave is making sure that the intelligence is collected from the hardware as much as possible, so that everything that we see that can improve that customer experience is seen, is the tech that is analyzed, and then transform that into action—action to the customers and action to the managers of the stores.

Kenton Williston: Perfect. Well, I look forward to eating ice cream together on a sunny day soon.

Jaume Portell: Me too.

Kenton Williston: So with that, I’d just like to say, thank you so much for joining us today.

Jaume Portell: Thank you for this wonderful time. Your questions have been very interesting, very enlightening for me too. It’s always very nice to have these types of discussions.

Kenton Williston: And thanks to our listeners for joining us. To keep up with the latest from Beabloo, follow them on Twitter at beabloo, that’s b e a b l o o. And, if you enjoyed listening, please support us by subscribing and rating us on your favorite podcast app. This has been the IoT Chat. We’ll be back next time with more ideas from industry leaders at the forefront of IoT design.

The preceding transcript is provided to ensure accessibility and is intended to accurately capture an informal conversation. The transcript may contain improper uses of trademarked terms and as such should not be used for any other purposes. For more information, please see the Intel^® trademark information.