Your robot can now chat to you: a little too well! (science video)

Greetings, all you Nestizen sci-fi enthusiasts. Today, we’ve got a little something that will tickle your circuits and fuel your imagination. Picture this: a robot tour guide, strolling around, engaging with its audience, and providing delightful commentary on its surroundings. It’s not a scene from your favorite sci-fi movie. It’s real, it’s here, and it’s brought to you by the wizards of technology and artificial intelligence. Let’s dive into the world of robots that can chat, shall we?

Our journey begins with a quirky creation, a robot tour guide, to be precise. But oh, it’s not just any robot tour guide. It’s Spot, the infamous robotic dog, but with a twist. Spot has been integrated with ChatGPT and a plethora of other AI models to create a proof-of-concept that’s bound to leave you in awe.

Now, if you’ve been keeping up with the tech world, you’d know that generative AI has been all the rage. Chatbots impersonating humans, image generators creating masterpieces, and speech generators that sound eerily lifelike. All of these marvels are now at the fingertips of the average Joe and Jane. What’s fueling this rapid expansion, you ask? Large Foundation Models (FMs), massive AI systems trained on ginormous datasets, that’s what!

Curiosity killed the cat, but not our intrepid team of innovators. Led by Matt Klingensmith, Principal Software Engineer, and his trusty crew of Michael McDonald, Radhika Agrawal, Chris Allum, and Rosalind Shinkle, they embarked on an adventure to explore the capabilities of these FMs in the realm of robotics. And oh, what an adventure it has been!

The star of the show, Spot, showcased its ability to make real-time decisions with the help of FMs. Think of it as a highly sophisticated autocomplete algorithm on steroids. It’s like having a robot that can roleplay, understand culture and nuances, make plans, and maintain a coherent conversation over time.

For the proof-of-concept, the team chose to create a robot tour guide. Simple enough, right? Spot would wander around, observe its surroundings, describe them using a Visual Question Answering (VQA) or captioning model, and then add a sprinkle of commentary using ChatGPT. The robot could also answer questions from the audience and decide on its next action, all in real-time. It’s like having an improv actor in the form of a robot!

Of course, creating a chatty robot is no small feat. It required a symphony of hardware and software integrations. From 3D printed mounts for speakers and microphones to offboard computers communicating with Spot over its SDK, every piece of the puzzle had to fit perfectly. And let’s not forget the heart of the operation: the large language models, the visual question answering software, and the text-to-speech tools that breathed life into our robotic friend.

But it wasn’t all smooth sailing. The team encountered a fair share of surprises, and even a few gotchas along the way. From the robot spontaneously deciding to ask for help at an IT help desk, to convincing its audience to go on a bigfoot hunt around the office, Spot certainly kept things interesting. Of course, there were limitations, such as the occasional fabrication of facts (yes, robots can be quite the storytellers!) and a bit of latency here and there. But hey, no one’s perfect, right?

So, what’s next on the horizon for our chatty robotic companions? The possibilities are endless. The combination of artificial intelligence and robotics is proving to be a match made in tech heaven. Robots are finding their footing in the real world, while AI models are lending a helping hand (or paw) by providing cultural context and common-sense knowledge. It’s only a matter of time before robots become our companions, entertainers, guides, and so much more.