Thermal imaging technology is fast becoming a hot topic in the world of automated vehicles and ADAS. After a fascinating Q&A panel with Bendix De Meulemeester of Umicore, Sebastien Tinnes of Lynred, and Chuck Gershman of OWL at AutoSens Brussels 2023, we asked Bendix to share some thoughts on where thermal is headed as we look forward to AutoSens Detroit 2024. Plus viewpoints from experts at OWL AI, TRIOPTICS, ADASKY, and Teledyne FLIR.
Almost everyone agrees that cars have become much safer compared to 30-40 years ago. It’s amazing that people can leave a high-speed crash unharmed, wiping off the dust from their clothes. Passive safety improvements, the ones that lessen the chance of death or serious injury when an accident is unavoidable, are the most visible in our daily lives. The passengers in the car are getting most of the benefits here. Vulnerable road users (VRU), such as pedestrians and cyclist have remained very … vulnerable.
The only way to really improve this fate of VRU’s is to avoid the accidents to begin with. This is where active safety systems and improved road infrastructure come in. This is the territory of ABS, ESP and the recent Advanced Driver Assistance Systems (ADAS) such as lane keep assist and automatic emergency breaking. Surprisingly enough and contrary to what you might expect with the introduction of these technological marvels, pedestrian fatalities in the US have actually increased significantly the last decade. (fig1)
Fig 1. According to the Governors Highway Safety Association (GHSA) 2022 pedestrian fatality data, the number of pedestrian fatalities in 2022 could be greater than 8,000 for the first time since 1980.
Don’t miss Bendix presentation at AutoSens USA this May. Click here to see the Preliminary Agenda.
There is not a consensus as to why, but it is clear that the use of smartphones inside the cabin and the reliance on driver assistance system play a role. A recent article in the New York Times “Why Are So Many American Pedestrians Dying at Night”, claims that a large part of this increase happens at night.
Fig 2. This picture shows the annual arc of the setting sun. Researchers have found related patterns looking at fatal collisions that occur in the weeks before and after clocks change for daylight saving time. When the 6 p.m. hour abruptly changes from light to dark, for example, even as traffic patterns generally remain the same, that hour becomes abruptly more deadly, too. (Source New York Times)
Night driving is more challenging and to add insult to injury, ADAS systems such as Automatic emergency braking do not work well at night. A particularly intriguing development in this field is the potential addition of thermal imaging technology to the suite of ADAS sensors.
Expanding on the challenges of braking systems, Dave Risdall, Senior Application Engineer at OWL AI says:
ADAS technology in the form of anti-lock braking systems (ABS) has been standard equipment on almost all new vehicles produced since 2004. Traction control, lane departure, adaptive cruise control, and automatic emergency braking are some of the newer technologies being introduced and are increasingly becoming available as standard or optional equipment in today’s production vehicles on the market. The ADAS sensor suite is a vital component in these systems being deployed. Although current sensor technologies generally work, they are far from perfect.
A 2023 study by the NHTSA provided an overview of motor vehicle traffic crashes in 2021. According to the study, “The proportion of people killed outside the vehicle (motorcyclists, pedestrians, pedal-cyclists, and other nonoccupants) has increased from a low of 20 percent in 1996 to a high of 34 percent in 2021”. “The number of pedestrian fatalities has been outpacing the number of motorcyclist fatalities since 2013.” Three-quarters of these fatalities occur at night when ambient light levels are low.
Luckily, a technology that already exists can easily fill the gaps in current ADAS system capabilities. Longwave Infrared, often referred to as thermal night vision is a technology that operates by detecting the inherent energy emitted by objects. Visible light systems like RGB, NIR and SWIR cameras detect light we can see, either emitted, as from a traffic light, or more likely, reflected from various sources by the objects being viewed. To see a pedestrian using a visible light camera, that pedestrian must be illuminated by a separate light source. Without an external source of light, a visible light camera, even one equipped with a High Dynamic Range feature, is blind. That same pedestrian will be visible to a thermal night vision camera regardless of the presence (or lack thereof) of a separate source of illumination. An ADAS system equipped with thermal night vision technology is significantly less affected by external environmental factors such as sun glare, rain, snow, or other weather-related conditions.
The benefits of thermal night vision technology extend beyond seeing in total darkness. In direct contrast to current radar and LiDAR systems, it can provide information on detection (is something there?), classification (what is it?), and in Owl AI’s case ranging (how far away is it?) of vulnerable road users and other objects of interest to the driver and/or ADAS system so that more timely decisions can be made and acted upon.
If a vehicle’s safety components can only operate in ideal lighting and environmental conditions, they cannot be considered reliable. They must operate 24 hours a day, 7 days a week, 365 days a year, and in all environments.
You can catch more from OWL AI at AutoSens USA this May, Click here to see the Preliminary Agenda.
Watch now: See Owl’s CEO Chuck Gershman give a presentation related to this topic here.
For another perspective on thermal imaging and PED, check out this article here from Wilfried Philips, Image Processing and Interpretation, imec – UGent.
The Advantages of Thermal Imaging
The primary advantage of thermal imaging in ADAS lies in its operational independence from external light sources. This means it can function effectively both during the day and at night, making it particularly useful in poorly lit and adverse weather conditions. Furthermore, thermal imaging provides stable and consistent imagery. The temperature of objects is quite static compared to visible light that fluctuates constantly with the change of shadows casted by trees or other objects. Thermal cameras offer a more consistent and reliable data stream, which is crucial to avoid false positives and reliable ADAS performance in general.
We asked Raz Peleg, VP of Business Development at ADASKY why thermal technology is so important:
While daylight cameras, radars, and lidars provide valuable perception capabilities for AEB and other ADAS and autonomous applications, they are not without limitations, leaving dangerous perception gaps in the popular sensor suites. These gaps, or “blind spots”, mainly occur when visibility is at its worst and the road is most dangerous for vehicles, pedestrians, and other vulnerable road users. LWIR cameras can easily fill these gaps due to their precise thermal detection properties and the rich images they produce – agnostic to all environmental conditions.
On top of enhancing the ADAS and ODD performance to challenging visibility conditions, the Thermal cameras present redundancy options to the current sensor suite at more than half the cost of the still expensive lidars and at the lowest power consumption, as they are totally passive.
Last but not least, the public, represented by the regulator, is demanding higher road safety, mainly with VRUs. Thermal cameras are the strongest among all other modalities, with the highest confidence level to declare/classify a VRU even when mostly occluded! This ability will not only allow the AEB to become PAEB (Pedestrian AEB) but also allow the regulator to license ODDs. This goal is not only for innovative service providers but for the public as a whole, still suffering from 1.3M people killed on the roads by other human drivers.
Hear Yonatan and more from ADASKY at AutoSens USA this May. Check out the Preliminary Agenda here.
Understanding Thermal Imaging in ADAS
Thermal imaging has the ability to detect blackbody radiation emitted by objects at room and body temperature. This feature enables thermal cameras to ‘see’ heat, providing a different layer of perception compared to traditional cameras that rely on visible light. The significance of this capability becomes immediately apparent in light of the above night time pedestrian fatality statistics. In scenarios where traditional cameras might struggle, thermal imaging cameras offer a more reliable means of detecting pedestrians.
We caught up with Dirk Seebaum, Product Manager Automation at TRIOPTICS, for his viewpoint:
Visible spectrum (VIS) and Near-infrared (NIR) cameras have become standard in today’s ADAS sensors. To extend the ability to reliably detect and classify objects in difficult visibility conditions such as darkness, fog or sun glare, the use of thermal imaging cameras in the LWIR spectrum can enhance the performance of ADAS and AD sensor systems. To overcome the technical challenges of mass production of thermal cameras, specialized solutions for optical alignment and performance testing are required.
The processes, measurement systems and equipment for mass production must have sufficient precision and repeatability to achieve the best possible production speed. To achieve an optimal solution, TRIOPTICS uses its expertise in testing optical systems at LWIR wavelengths and combines it with the knowledge of developing camera alignment and testing equipment for sensor production in the automotive industry. Special attention is paid to the development of test targets, measurement methods and the definition of hardware requirements, taking into account the physical properties of LWIR camera systems.
Catch Dirk at AutoSens USA. Find out more in the Preliminary Agenda.
Recent Technological Advances
The field of thermal imaging has witnessed significant advancements, primarily in sensor miniaturization and lens manufacturing. The transition from 17µm to 12µm pixel pitches, and now, the introduction of even smaller 8.5µm pixels, allows for the production of more sensors from the same silicon wafer. This miniaturization is key to reducing manufacturing costs.
Another major advancement is in the realm of optics. Traditional optics for thermal imaging systems have been expensive due to material constraints. However, there’s a shift in the industry from using germanium to chalcogenide glasses for lenses. These materials are moldable and significantly cheaper. Additionally, the development of wafer-level optics, where hundreds of lenses are molded simultaneously, further decreases costs and makes these systems more viable for commercial use.
Industry Interest and Challenges
Despite its potential, the integration of thermal imaging into ADAS is not without challenges. One major hurdle is altering the perception of high costs traditionally associated with thermal imaging. This perception has been influenced by its historical use in defense and security applications and the expenses of early thermal automotive night vision systems. However, the recent technological advancements that make thermal imaging more affordable are drawing increased interest from automotive industry leaders. These leaders are considering the integration of thermal imaging into next-generation ADAS systems, recognizing its potential to enhance vehicle safety, especially in terms of pedestrian detection in low-light conditions.
As the industry continues to invest in the development of thermal technology, we asked Jen Hartigan, Senior Marketing Manager at Teledyne FLIR for an update on their roadmap:
Agencies such as NHTSA in the US and others around the world are tracking the significant increase in pedestrian deaths especially at night. A new collaboration brings together Valeo, the world’s leader in automotive camera technologies, with Teledyne FLIR, the world leader in thermal imaging technology, to create the next generation of multispectral sensor fusion systems for automotive safety. The strategic collaboration will bring thermal imaging technology to the automotive industry to enhance the safety of road users. Together, the team already secured a major contract in late 2023 from a leading global automotive OEM to deliver their new thermal imaging cameras as part of a new generation of advanced driver-assist systems (ADAS) driver-aide technology to improve vehicle and road safety. Valeo and Teledyne FLIR will deliver the first Automotive Safety Integrity Level (ASIL) B thermal imaging technology for night vision ADAS. We are looking forward to AutoSens and plan to demonstrate fused visible and thermal data running our Prism AI automotive perception software.
You can read more about Teledyne FLIR and Valeo’s collaboration in our Top Takeaways from CES.
Plus, see what Teledyne FLIR brought to AutoSens Brussels with this demo from John Eggert, Director of Automotive Business Development.
The Role of Lighting in Pedestrian Safety
Coming back to the introduction, It’s important to put the potential of thermal imaging in the context of the broader conversation about pedestrian safety and road lighting. Studies have shown that appropriate lighting can significantly increase the visibility of pedestrian crosswalks and reduce glare for motorists, thereby enhancing overall safety.(2) For instance, overhead lighting at intersections is estimated to reduce all types of injury crashes by 27%, and outside of intersections, the reduction is estimated at 23%.(3) These statistics highlight the importance of environmental factors in pedestrian safety and suggest that thermal imaging could be an additional tool in the arsenal for enhancing safety, particularly in poorly lit areas.
Conclusion
Thermal imaging presents an intriguing avenue for enhancing the effectiveness and safety of ADAS. Its unique ability to operate in various lighting conditions, provide stable imaging, and potentially improve the detection of pedestrians and other objects offers an additional dimension to automotive safety systems. With the ongoing technological developments making it more cost-effective, thermal imaging could play a significant role in the future of vehicle safety. However, it’s essential to view this technology as part of a larger ecosystem of safety measures, including improved road lighting and infrastructure, to create a safer environment for all road users.
Enjoyed this article? You can see Bendix’s 2023 presentation from Brussels here, available to watch with an AutoSens Europe on-demand pass.
AutoSens Partners Joining The Exhibition with Thermal Solutions in Detroit this May
Ready to join the adventure and dive into the technical world of ADAS and AV technology? Our USA exhibition will feature all these companies and more at the cutting edge of Thermal technology.
Join us at AutoSens USA 21st-23rd May 2024. Get your pass here.
Sources:
- “Pedestrians, 2018 Data.” National Highway Traffic Safety Administration (NHTSA) Fatality Analysis Reporting System (FARS). NHTSA Report
- Gibbons, R., et al. “Informational Report on Lighting Design for Midblock Crosswalks.” Publication FHWA-HRT-08-053, FHWA, U.S. Department of Transportation, 2008. FHWA Report
- Harkey, D.L., et al. “NCHRP Report 617: Accident
