Developing and refining robotic systems, especially those that interact with constantly changing objects in Amazon’s facilities, is a time-intensive process. To speed up development, engineers are increasingly turning to virtual environments where robots can be designed and tested before being physically built, though achieving accurate simulations is challenging.
For decades, digital models and virtual simulations have helped drive product design. However, the conventional tools available haven’t met Amazon’s demands for creating and scaling a fleet of complex robots. To illustrate, consider modern video games that create visually convincing worlds but often rely on approximations when it comes to the underlying physics. For example, while a racing game may show realistic car acceleration, the forces driving that acceleration might not reflect real-world dynamics. This means that even if a car appears to have enough power to perform unrealistic feats, the simulation isn’t based on true physical principles.
Many industrial simulations use similar approximations. At Amazon, visual simulators help plan facility layouts and estimate how robots move safely among employees. Yet, for advanced robotic manipulation, it is essential to combine visual realism with precise physics. Failing to do so can result in a “sim2real gap,” where minor discrepancies between the simulated and real world become significant issues. Testing code on physical systems is common, but given the vast variety of robots Amazon is developing, this approach isn’t practical. The goal is to develop software first in a virtual setting, then verify and fine-tune on actual hardware—a process that is beginning to show promising progress.
Achieving this vision requires modeling not only the robots but also the objects they handle. For instance, a robotic arm might feature a pneumatic gripper with several suction cups. Accurately simulating such a system involves calculating air flow through the gripper’s tubes and valves, assessing the forces at play when the suction cups contact a package, and accounting for any deformations that occur during the process. These complex models help engineers experiment with both typical and unusual scenarios, such as the risk of dropping a package.
Additionally, simulations must replicate how a robot’s vision system distinguishes individual items in a mixed pile and how the arm determines the best angle and force for lifting. The complexity increases when considering the variety of packages, ranging from rigid boxes to flexible bubble-wrap mailers that can shift weight unexpectedly. Instead of creating individual models based on trial and error, Amazon is striving to model the fundamental physics of these interactions, which includes detailed aspects such as airflow, friction, inertia, and aerodynamics.
A significant part of this effort is supported by a tool known as Drake—a collaborative, open-source project originally developed at MIT. Drake provides a robust framework for simulating robot dynamics and control systems. Its open-source nature is vital because it allows Amazon to examine and modify the underlying equations if issues arise. Drake combines a powerful dynamics engine, a systems framework for integrating custom models, and a suite of solvers to handle complex numerical optimizations, including the critical task of calculating contact forces between interacting bodies.
Despite the inherent challenges of bridging the gap between simulation and reality, Amazon continues to validate its virtual models against real-world data. Over time, as the models become more accurate, any errors diminish. However, handling deformable objects—like soft toys or flexible packages—remains a particularly demanding area. Researchers are exploring whether items with both rigid and flexible components could serve as a manageable starting point for broader simulations.
Early successes are already evident. One project team was able to validate a new robotic manipulation concept in just a month using simulation, a process that would have taken significantly longer with physical prototypes. This demonstrates the powerful impact of high-fidelity simulation on speeding up development and innovation.
Ultimately, the vision is to have all robotics research and development initiated in a virtual environment. By doing so, engineers can rapidly test new ideas, explore rare conditions, and collaborate on different aspects of a project simultaneously—all without waiting for physical hardware. The ongoing progress in simulation technology not only promises faster development cycles but also improved safety in Amazon’s facilities and enhanced service for its customers.
