Autonomous super-vehicles will have maneuvering and navigation capabilities that go far beyond what human-piloted or human-driven vehicles have today. They will be endowed with high-rate, high-resolution sensors. They will pack custom-built special-purpose computers. They will run software that is specifically designed for these sensors and computers. MIT's high-throughput autonomous drone racers pack Graphics Processing Units, Field Programmable Gate Arrays (FGPAs), high-rate cameras and inertial measurement units.
FlightGoggles is envisioned to be development environment that allows the design, implementation, testing and validation of autonomous super-vehicles. FlightGoggles currently provides exteroceptive sensor simulation based on the Unity3D engine as well as vehicle dynamics and inertial sensor simulation capabilities.
Photogrammetry powers the two unique virtual environments that comes with FlightGoggles. Photogrammetry is a technique that creates virtual environments through their photographs. To create both of the two FlightGoggles virtual environments, we took countless photographs of real-world assets. We created 3D graphics assets from these photographs.
The Abondoned Factory Virtual Environment is created by creating virtual replicas of eighty different objects, such as columns, walls, rubble, barrels, and more. The environment is then put together by creating around a thousand instances of these eighty different objects, hence replicating some of the objects many times.
The environment features visually-challenging elements, such as water puddles, rubble, broken objects, and various lighting effects. The lighting can be modified to reflect different times of day.
More information and visuals are here.
The Stata Center Virtual Environment is a virtual replica of two floors of MIT's Stata Center - a Frank-Gehry-designed building. Stata Center is one of the landmarks of Boston area. It is the home to our parent laboratory, the Laboratory for Information and Decision Systems (LIDS).
The Stata Center environment is carefully crafted from tens of thousands of photographs. The environment is hand-cleaned in order to perform rendering in real time.
The Stata Center environment is truly unique as being one of the most visually-compelling virtual replicas available today. In most places, it is an excellent match for the real Stata Center. A formal comparison is difficult to make. But, we provide some examples of visual benchmarks. More information and visuals are here.
FlightGoggles has been used for selecting the best nine teams that will compete in the AlphaPilot challenge. AlphaPilot is a competition co-organized by Lockheed Martin, NVIDIA, and the Drone Racing League (DRL). FlightGoggles simulation challenge is the main test that the teams are asked to complete in order to make it to the race.
The video shows the top twenty teams that submitted solutions for the simulation challenge. We overlaid their trajectories, so that you can watch the competition. All drones in this video are autonomously navigating this complex simulation environment without any human intervention.
The FlightGoggles development environment also enables development in virtual reality using motion-capture facilities. An aerial robotic vehicle with on-board embedded computers is flown in a motion-capture facility. All exteroceptors, including cameras, are synthetically generated, and transmitted to the vehicle while the vehicle is in flight. In this case, all complex dynamics acting on the vehicle, such as the aerodynamics of the vehicle, mechanics of the motors, and electrochemistry of the battery, are real, while all exteroceptive sensors are simulated.
Visit our research page for more technical videos and our research papers.
FlightGoggles is currently under development. More environments and more features will be added. Stay tuned!
