Gaussian Splatting Is Our Secret Ingredient For Safer Cars

For decades, the Volvo brand has been synonymous with safety. But keeping passengers secure is no longer just about a strong cabin or cleverly designed crumple zones. Increasingly, safety is about semi-autonomous driving technology that can mitigate collisions or even avoid them entirely. Volvo intends to be ahead of the game in this era too. Its secret weapon? Something called “Gaussian Splatting”. I asked Volvo’s Head of Software Engineering Alwin Bakkenes and subsidiary Zenseact’s VP Product Erik Coelingh exactly what this is and why it’s so important.

Volvo: Early Application Of Safety Data

“We have a long history of innovations based on data,” says Bakkenes. “The accident research team from the 70s started with measuring tapes. Now in the digital world we’re collecting millions of real-life events. That data has helped us over the years to develop a three-point safety belt and the whiplash protection system. Now, we can see from the data we collect from fleets that a very large portion of serious accidents happen in the dark on country roads where vulnerable road users are involved. That’s why, with the ES90 that we just launched, we are also introducing a function called lighter AES where we have enabled the car to steer away from pedestrians walking on the side of the road or cyclists, which in the dark you can’t see even if you have your high beam on. This technology picks that up earlier than a human driver.” The Volvo EX90 SUV will also benefit from this technology.

“If you want to lead in collision avoidance and self-driving, you need to have the best possible data from the real world,” adds Bakkenes. “But everyone is looking also at augmenting that with simulated data. The next step is fast automation, so we’re using state-of-the-art end-to-end models to achieve speed in iterations. But sometimes these models hallucinate. To avoid that, we use our 98 years of safety experience and these millions of data points as guardrails to make sure that the car behaves well because we believe that when you start to automate it needs to be trusted. For us every kilometer driven with Pilot Assist or Pilot Assist Plus needs to be safer than when you’ve driven it yourself. In the world of AI data is king. We use Gaussian Splatting to enhance our data set.”

What Is Volvo’s Gaussian Splatting?

“Cars are driven all around the world in different weather and traffic conditions by different people,” says Coelingh. “The variation is huge. We collect millions of data points, but it’s still a limited amount compared to reality. Gaussian Splatting is a new technology that some of our PhD students have been developing the last few years into a system where you can take a single data point from the real world where you have all the sensor, camera, radar and LIDAR sequences and then blow it up into thousands or tens of thousands of different scenarios. In that way, you can get a much better representation of the real world because we can test our software against this huge variation. If you do it in software, you can test much faster, so then you can iterate your software much more quickly and improve our product.”

“Gaussian Splatting is used in different areas of AI,” continues Coelingh. “It comes from the neural radiance fields (NeRFs).” The original version worked with static images. “The first academic paper was about a drum kit where somebody took still pictures from different angles and then the neural net was trained on those pictures to create a 3D model. It looked perfect from any angle even though there was only a limited set of pictures available. Later that technology was expanded from 3D to 4D space-time, so you could also do it on the video set. We now do this not just with video data, but also with LiDAR and radar data.” A real-world event can be recreated from every angle. “We can start to manipulate other road users in this scenario. We can manipulate real world scenarios and do different simulations around this to make sure that our system is robust to variations.”

Volvo uses this system particularly to explore how small adjustments could prevent accidents. “Most of the work that we do is not about the crash itself,” says Coelingh. “It’s much more about what’s happening 4-5 seconds before the crash or potential crash. The data we probe is from crashes, but it’s also from events where our systems already did an intervention and in many cases those interventions come in time to prevent an accident and in some cases they come late and we only mitigated it. But all these scenarios are relevant because they happen in the real world, and they are types of edge case. These are rare, but through this technology of Gaussian Splatting, we can go from a few edge cases to suddenly many different edge cases and thereby test our system against those in a way that we previously could not.”

Volvo’s Global Safety Focus

This is increasingly important for addressing the huge variation in global driving habits and conditions a safety system will be expected to encounter. “Neural Nets are good at learning these types of patterns,” says Coelingh. “Humans can see that because of the behavior of a car the driver is talking into their phone, either slowing down or wiggling in the lane. If you have an end-to-end neural network using representations from camera images, LiDAR and radar, it will anticipate those kinds of things. We are probing data from cars all around the world where Volvo Cars are being driven.”

The system acts preemptively, so it can perform a safety maneuver for example when a pedestrian appears suddenly in the path of the vehicle. “You have no time to react,” says Coelingh. Volvo’s safety system will be ready, however. “Even before that, the car already detects free space. It can do an auto steer and it’s a very small correction. It doesn’t steer you out of lane. It doesn’t jerk you around. It slows down a little bit and it does the correction. It’s undramatic, but the impact is massive. Oncoming collisions are incredibly severe. Small adjustments can have big benefits.”

Volvo has developed one software platform to cover both safety and autonomy. “The software stack that we develop is being used in different ways,” says Coelingh. “We want the driver to drive manually undisturbed unless there’s a critical situation. Then we try to assist in the best possible way to avoid collision, either by warning, steering, auto braking or a combination of those. Then we also do cruising or L2 automation.”

Volvo demonstrated how it has been using Gaussian Splatting at NVIDIA’s GTC in April. “We went deeply into the safe automation concept,” says Bakkenes. “Neural nets are good at picking up things that you can’t do in a rule-based system. We’re developing one stack based on good fleet data which has end-to-end algorithms to achieve massive performance, and it has guard rails to make sure we manage hallucinations. It’s not like we have a collision avoidance stack and then we have self-driving stack.”

“There was a conscious decision that if we improve performance, then we want the benefits of that to be both for collision avoidance in manual driving and for self-driving,” says Coelingh. “We build everything from the same stack, but the stack itself is scalable. It’s one big neural network that we can train. But then there are parts that we can deploy separately to go from our core premium ADAS system all the way to a system that can do unsupervised automation. Volvo’s purpose is to get to zero collisions, saving lives. We use AI and all our energy to get there.”