.jpg) |
| As the robot moves across an environment, it takes multiple photos with onboard cameras. On left are three views of a dark object on a rooftop. The system initially states it can’t give an accurate description and then identifies the object incorrectly as a squirrel. As it continues moving, the robot integrates uncertainty quantification and it refines the view and states the object is “a dark gray, cylindrical chimney with a slightly tapered top and a smooth surface.” Image Credit: Courtesy of the researchers (CC BY-NC-ND 3.0) |
Scientific Frontline: Extended "At a Glance" Summary: UQ-DAAAM (Uncertainty Quantification for Describe Anything, Anywhere, at Any Moment)
The Core Concept: UQ-DAAAM is a spatial memory system for robots that identifies, quantifies, and corrects unreliable object descriptions within its stored memory in real time.
Key Distinction/Mechanism: Rather than accepting vision-language model (VLM) captions as absolute facts, the system fuses the model's confidence levels with consistency metrics across multiple viewpoints. When uncertainty is high, it utilizes a computationally efficient heuristic to select optimal camera frames and employs a large language model (LLM) to synthesize a highly accurate, unified description.
Origin/History: Developed by MIT researchers Luca Carlone, Harry Zhang, and Nicolas Gorlo, the system was introduced in November 2026 at the Conference on Neural Information Processing Systems as an advancement of their prior DAAAM framework.
Major Frameworks/Components:
- Vision-language models (VLMs) for initial scene description.
- 3D map-based spatial memory representations.
- Uncertainty quantification algorithms.
- Large language models (LLMs) for caption fusion and ambiguity resolution.
- Heuristic algorithms for optimal, low-compute frame selection.
Branches of Science: Robotics, Artificial Intelligence, Machine Learning, Computer Vision, Autonomous Systems.
Future Application: The framework will allow autonomous robotic manipulators to safely execute complex, multistep tasks, such as assisting human workers with machinery assembly, clearing tables, and autonomously navigating highly dynamic industrial environments.
Why It Matters: Endowing physical AI with the ability to measure its own uncertainty prevents critical downstream errors, such as misidentifying emergency equipment, which is essential for the safe and reliable integration of autonomous robots into society.
Autonomous robotic systems deployed in dynamic environments require highly accurate spatial memory; misclassification of critical objects, such as emergency medical equipment, poses significant operational and safety risks.
To address these limitations, MIT researchers have developed a novel spatial memory framework. The system identifies entries in a robot’s memory that might be wrong and corrects the descriptions of mislabeled objects.
In the future, this method could make household or industrial robots better able to perform complex, multistep tasks, such as clearing a dining room table or working alongside human collaborators to assemble a piece of machinery.
Called UQ-DAAAM (Uncertainty Quantification for Describe Anything, Anywhere, at Any Moment), the system compares multiple descriptions of the same object, seen from different viewpoints, to quantify the robot’s uncertainty about what it saw.
It then selects additional views to reduce the uncertainty and combines these captions into a more accurate description of the object, which it saves to memory.
The researchers implemented a computationally efficient heuristic algorithm to streamline the process, enabling mobile platforms to continually update memory entries in real time.
When compared with baseline methods, UQ-DAAAM boosted the robot’s confidence and improved the accuracy of its answers to questions about objects, their locations, and when it saw them.
“Contrary to ChatGPT, when we think about physical AI, it is not enough to simply put a disclaimer cautioning that sometimes the robot might be wrong. Allowing a robot to know what it doesn’t know is going to be essential for this technology to be deployed safely and have a positive impact on society,” says Luca Carlone, an associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro), principal investigator in the Laboratory for Information and Decision Systems (LIDS), director of the MIT SPARK Laboratory, and the senior author of the paper presenting UQ-DAAAM.
The paper was co-authored by MIT graduate students Harry Zhang and Nicolas Gorlo. The research will be presented at the Conference on Neural Information Processing Systems.
Amending Memory
This new method builds on the researchers’ prior work, DAAAM, a robot memory system that uses a vision-language model (VLM) to richly describe objects in scenes a robot captures with its onboard cameras. These descriptions are stored as detailed captions in a 3D map-based representation of the robot’s environment that it can call upon later to answer questions.
However, optical sensors frequently capture partial, distant, or degraded images due to suboptimal lighting or motion blur. In these cases, DAAAM might describe these objects incorrectly, such as by mistaking a gray chimney for a squirrel perched on a rooftop.
“With no way to correct itself, this wrong memory would persist as the robot proceeds in its environment, which could impact downstream applications,” Zhang says.
UQ-DAAAM integrates uncertainty quantification into the DAAAM framework. It allows the robot to measure its confidence in the accuracy of object descriptions and fix incorrect captions as it travels through an environment.
While on the move, the robot constantly collects and stores images of the world around it. Many of these images show the same objects captured from different angles. UQ-DAAAM uses these additional viewpoints to identify and correct unreliable descriptions.
First, it calculates the VLM’s confidence level in the words it generated to describe a potentially mislabeled object. Then, it measures consistency by comparing this description to captions of images that describe the same object from different vantage points.
“We fuse both sources of uncertainty into a single number that captures how certain the VLM is in its overall understanding of the object,” Carlone explains.
If that number exceeds a certain threshold, the uncertainty is too high, and UQ-DAAAM will refine the caption by using additional images of the object.
This is achieved using a heuristic algorithm designed to identify which additional viewpoints optimally reduce spatial uncertainty. For instance, it might choose an image captured a few frames later that clearly shows smoke emanating from the chimney.
The researchers mathematically proved that this heuristic can identify the best frames to select without rerunning the VLM, dramatically reducing the amount of computation.
“The key challenge Nico and Harry had to overcome is how to combine VLMs with uncertainty quantification into something that is running in a fraction of a second on a robot that is moving,” Carlone says.
Remembering with Confidence
Once it has identified additional captions that will reduce its uncertainty about the object, the system uses a large language model (LLM) to fuse them into a single object description. To boost accuracy, the LLM puts more weight on new captions and those with less ambiguity.
This refinement step is able to resolve ambiguities for about 92% of the objects in the robot’s environment, while only requiring two additional captions per object. It increased the accuracy of object descriptions by 5% compared with the state of the art.
Compared with baseline methods, UQ-DAAAM also provided more accurate answers to questions about the objects it observed, where it saw them, and when it last encountered them.
“Performing this uncertainty quantification procedure across the entire map the robot generates allows it to improve its understanding of what it has observed,” Gorlo says.
Next, the researchers want to use UQ-DAAAM to guide different types of robots, such as mobile robotic manipulators, and apply their uncertainty quantification techniques to other domains.
In addition, they are studying the use of UQ-DAAAM to identify areas of high uncertainty the robot could revisit to improve its understanding of the environment. They are also exploring uncertainty quantification in systems that predict the future state of a robot’s environment.
Funding: This research is supported, in part, by the U.S. Army Research Laboratory and the U.S. Office of Naval Research.
Disclaimer: Carlone is currently on sabbatical as an Amazon Scholar; this article describes work performed at MIT and is not associated with Amazon.
Published in journal: arXiv (preprint server)
Title: Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guarantees (HTML) (PDF)
Authors: Harry Zhang, Nicolas Gorlo, and Luca Carlone
Source/Credit: Massachusetts Institute of Technology | Adam Zewe
Edited by: Scientific Frontline
Reference Number: rob110526_01
