Ashok Elluswamy is Tesla’s VP for Autopilot Software. He delivered a keynote titled, “Building foundational models for Robotics at Tesla”, at the International Conference on Computer Vision.

Tesla FSD cameras feed into a large-scale network of neural networks (with increasing parameter counts and soon to be scaled 10x by new hardware).

Future Outlook.
Scale roboticaxi to unsupervised nationwide service. Cybercab: Low-cost, two-seat autonomous vehicle for robotaxis. Beats public transit economics.
Extend robotics: Same technology transfers to Optimus (e.g. action-conditioned video generation to navigate).
Tesla’s approach is scalable across vehicles, locations and weather conditions; it emphasizes safety, comfort and speed.

Tesla has access a “Niagara Falls” of data — hundreds of years of collective fleet driving.
* Smart data triggers are used to capture rare corner cases, such as complex intersections and unpredictable behavior.

Efficiency and Quality:
* Extracts only the necessary data to train models efficiently.

Interpretability and Debugging:
Even though the system runs end-to-end Tesla can still instruct the model to output interpretable information:
3-D occupancy, road boundary, objects, traffic lights, signs, etc.
Natural language querying – ask the model to explain why it made certain decisions. These auxiliary predictions do not drive the car, but they help engineers debug it and ensure safety.

Tesla’s Advanced Gaussian Splatting (D Scene Modeling).
Tesla developed a custom ultra-fast Gaussian Splatting system for reconstructing 3D scenes using limited camera views.
* Produces crisp and accurate 3D renderings, even with limited camera angles – far superior to standard NeRF/splatting methods.
* Allows rapid visual debugging in 3D of the driving environment.

Evaluation and World Models: * Evaluation is the biggest challenge: models that perform well in a virtual environment may not work in a real-world setting.
Tesla builds balanced and diverse evaluation datasets that focus on edge cases, not just highway driving.

Introduced the learned world simulator (neural-network-generated video engine).
Can simulate 8 Tesla cameras simultaneously — fully synthetic.
Used for training, testing, and reinforcement learning.
Allows adversarial events to be injected (e.g. adding a pedestrian or vehicle cutting into).
Allows replaying of past failures in order to verify new model improvement.
Can run in real-time and let testers “drive” within a simulated environment.

Next Steps:
* Expand robotaxi service worldwide.
Unlock full autonomy for the entire Tesla fleet. Cybercab is a next-generation 2-seater vehicle specifically designed for robotaxi, aiming to achieve the lowest transportation costs (cheaper that public transit).
The same neural networks will power Optimus, a humanoid robot. The same video generation system will be applied to Optimus. The system can simulate and plan robot movements, and adapt easily to new forms.

via International Conference on Computer Vision.