When a human tells a robot, “Don’t spill on the laptop,” they aren’t just issuing a warning about a single piece of hardware; they are communicating a broader intent to protect electronics.
Now, a new AI system called FLEX allows robots to grasp that unspoken logic.
Developed by engineers at MIT and New York University, the system uses a large language model (LLM) to infer unstated intentions from natural language commands. The research team presented their findings this month at the annual Conference on Robot Learning, according to a report from Mit News.
FLEX Uses a Three-Step Process to Bridge Intuition and Execution
FLEX operates through a specific three-step process designed to bridge the gap between human intuition and machine execution. It begins with “Add,” a standard procedure where the robot identifies a specific object from a command—such as a mug—and adds it to an avoidance list.
The system then moves to “Generalize.” During this stage, FLEX feeds the identified object into GPT-4o, a large language model. By drawing on vast training data, the LLM identifies patterns: it reasons that liquids combined with devices result in damage.
Consequently, the robot expands its avoidance list to include all fragile or electronic objects in the vicinity. Finally, a “Mask” step uses computer vision to clean the feature list, removing objects that are no longer present in the environment. The entire sequence takes approximately five seconds.
“The idea is that there is some higher-level intention behind the words that you say. People naturally get this. But you have to carefully explain that to a robot, which takes time and effort. FLEX gives you an interface of communication with a robot to say, ‘When people say this, they mean that.’” Nathan Dennler, postdoc in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL)
Simulating Common Sense in the Home
To test the system, the team utilized simulated household scenarios involving a robotic arm performing three distinct tasks: moving a candle, serving coffee, and carrying a bowling ball. Volunteers provided natural language commands to guide the machine.
The results showed the LLM’s ability to grasp context. When commanded “Don’t spill on my laptop,” the robot avoided both the laptop and a nearby phone. When told to “Stay away from the stack of paper,” it avoided the paper and a nearby candle. Similarly, the command “Don’t drop on the mug” prompted the robot to avoid the mug and other fragile items.
“LLMs have seen so many texts from the internet,” Dennler explains. “People might have written about, ‘Oh, my phone dropped in the water and now it doesn’t work,’ or ‘I spilled coffee on my computer and now it doesn’t work.’ So LLMs pick up on the idea that liquids plus devices leads to things not working.”
Solving the Customization Problem
Current robotics often rely on rigid, pre-programmed rules, but FLEX aims to sidestep that manual labor by allowing users to teach a robot with just a few sentences.
“Everyone’s homes are different, and everyone cares about different things, so no robot comes out of the box knowing what matters to you,” says Andreea Bobu, assistant professor of aeronautics and astronautics at MIT. “What excites me about FLEX is that it can pick that up from a sentence or two instead of asking you to spell out every rule.”
From Simulation to the Real World
While the current simulations serve as a proof of concept, the next phase of development requires testing in dynamic, unpredictable, real-world settings. This research was supported in part by the MIT Generative AI Impact Consortium.
