Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a novel system called Feature Fields for Robotic Manipulation (F3RM) that integrates 2D imagery with foundational model features to create 3D representations of environments. This approach allows robots to understand and interact with objects using open-ended language cues, making them more adaptable in complex settings such as warehouses and homes.
Inspired by how humans intuitively manage unfamiliar items—much like discerning the contents of a friend’s fridge in a foreign country—the F3RM system enables robots to recognize and handle objects even when they appear in unexpected forms or packaging. By combining traditional 2D images with advanced semantic features, the method transforms visual information into detailed 3D scenes, allowing the robots to identify and pick up items based on natural language instructions.
For example, if a person instructs the robot to “pick up a tall mug,” F3RM uses its understanding of geometry and semantics to locate and grasp the object that best fits the description. This ability to interpret vague or broadly defined commands marks a significant step toward robots that can generalize their behavior to new, untrained objects.
According to Ge Yang, a postdoctoral researcher at the National Science Foundation AI Institute for Artificial Intelligence and Fundamental Interactions and MIT CSAIL, achieving real-world generalization is a considerable challenge. Yang emphasized the project’s aim to extend robotic capabilities from handling a few objects to dealing with the vast array of items found in places like MIT’s Stata Center, ultimately making robots as flexible as humans in adapting to unseen objects.
Enhancing Robotic Perception Through “Seeing”
F3RM’s methodology can be particularly beneficial in environments like large fulfillment centers, where robots are tasked with identifying and retrieving items from cluttered spaces. In these settings, robots are often given textual descriptions of items, and they must accurately match those descriptions to objects among thousands of possibilities. With F3RM’s robust spatial and semantic understanding, robots can better locate objects, place them correctly, and streamline the shipping process, thereby enhancing operational efficiency.
Yang also noted that the system’s versatility extends beyond small-scale applications; F3RM is capable of functioning on a room- or even building-scale. This scalability opens up possibilities for creating simulation environments for robot learning and mapping large areas. However, the current focus is on optimizing the system for speed to enable real-time robotic control, which is crucial for dynamic tasks.
Building a “Digital Twin” of the Environment
The F3RM system starts by capturing a series of images from multiple angles using a camera mounted on a selfie stick. By taking about 50 images from different positions, it constructs a neural radiance field (NeRF) that forms a 360-degree digital representation of its surroundings. This “digital twin” not only reflects the physical layout but also incorporates semantic details through the use of CLIP, a vision model trained on vast amounts of image data.
By lifting 2D features into a 3D context, F3RM enriches its understanding of both the spatial arrangement and the identity of objects, thereby facilitating more effective manipulation by the robot.
Open-Ended Interaction Through Language
Once the system has learned the spatial and semantic layout of its environment, it can respond to open-ended text queries. When a user provides a command, the robot evaluates potential actions based on how well they align with the prompt, their similarity to past demonstrations, and their safety regarding potential collisions. The option that scores highest is then executed.
In one demonstration, the system was prompted to pick up Baymax—a character from Disney’s “Big Hero 6”—even though the robot had not been explicitly trained with that specific object. The advanced spatial and language features allowed the robot to successfully interpret and execute the command.
Furthermore, the system allows users to specify objects at varying levels of detail. For instance, if two glass mugs are present—one containing coffee and the other juice—a user can specify which one to pick up by mentioning the desired contents. This nuanced level of understanding is made possible by the semantic features embedded in the 3D representation.
MIT PhD student William Shen highlighted that, unlike humans who can easily transfer the knowledge of handling similar objects, enabling robots to achieve such adaptability is challenging. F3RM’s blend of geometric insight and extensive semantic training allows robots to generalize from a small set of demonstrations to a wide range of objects.
The research team, including Ge Yang, William Shen, MIT professor Phillip Isola, CSAIL principal investigator Leslie Pack Kaelbling, and undergraduate students Alan Yu and Jansen Wong, developed F3RM with support from organizations such as Amazon.com Services, the National Science Foundation, the Air Force Office of Scientific Research, and several others. Their findings will be presented at the 2023 Conference on Robot Learning.
