A security robot patrolling a backyard glances up at a dark shape on the roofline and logs it as a chimney. Months and several camera angles later, it turns out to be a squirrel feeder bolted to the gutter — but the robot's memory still insists "chimney," and every downstream task that depends on that label inherits the mistake. Researchers at MIT's Laboratory for Information and Decision Systems have built a system called UQ-DAAAM that catches exactly this kind of error, flagging uncertain object descriptions in a robot's memory and fixing roughly 92 percent of them using just two extra glances at footage the robot already recorded.
The work, led by graduate students Harry Zhang and Nicolas Gorlo under associate professor Luca Carlone, director of the MIT SPARK Lab, appears in a paper titled "Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guarantees," accepted to the Conference on Neural Information Processing Systems. MIT News detailed the system on November 5.
Robot memory has become a bottleneck as embodied AI systems move from single tasks to open-ended operation in homes, warehouses and hospitals, where a robot might not revisit the same spot for hours or days. Unlike a chatbot that can simply admit uncertainty in a sentence, a robot that misremembers an object's identity, location or last-seen time can act on that error physically — reaching for the wrong tool, misdelivering a package, or reporting incorrect information to a human supervisor.
Why robots misremember things
UQ-DAAAM builds on the team's earlier memory framework, DAAAM, in which a robot uses an onboard camera and a vision-language model to generate text descriptions of objects it sees, then pins those descriptions to locations on a 3D map. The approach works well until an object is distant, partially blocked, poorly lit, or simply ambiguous from a single angle — the exact conditions that produce the chimney-or-squirrel-feeder problem.
The new system scores every stored description for uncertainty by combining two signals: how confident the vision-language model was when it generated the caption, and how sharply that caption disagrees with other captions of the same object taken from different viewpoints. When the combined score crosses a threshold, UQ-DAAAM doesn't send the robot back to look again. Instead, it mines footage the robot already captured, selects frames with better angles or less occlusion, and feeds them to the vision-language model for a second opinion. A large language model then merges the competing descriptions, weighting newer and less ambiguous captions more heavily, and writes a corrected entry back into memory.
What the numbers show
Across tested robot environments, the method cleared up ambiguity for about 92 percent of flagged objects while needing only two supplemental captions per object — a small computational cost for a meaningful accuracy gain. On a spatiotemporal question-answering benchmark that asks a robot what an object is, where it is, and when it was last seen, UQ-DAAAM lifted descriptive accuracy roughly 5 percentage points over the prior state of the art, while also reducing errors in both position and timing estimates.
Carlone frames the result as a safety question as much as an accuracy one. "Allowing a robot to know what it doesn't know is going to be essential for this technology to be deployed safely," he said. For robots that increasingly make decisions based on memory rather than fresh perception alone, that distinction between confident and uncertain knowledge may matter more than raw description accuracy.
The team is now studying how UQ-DAAAM's uncertainty scores could point a robot toward specific locations worth re-examining in person, rather than relying solely on footage already in hand — turning the system from a passive auditor of memory into an active planner that decides where to look next.