With the development of autonomous driving and driving assistance, cameras are more and more significant in the Automotive industry. Whereas the evaluation of image quality for consumer cameras such as smartphone cameras has been well defined for years, the definition of a good image quality for an ADAS camera, fully dedicated to computer vision, is still under debate. It seems obvious that the KPIs for camera evaluation need to be redefined in the light of this new scope. In 2022, DXOMARK has launched an ambitious research program (in partnership with a French institute specialized in detection algorithm) to evaluate various image quality evaluation metrics. The aim of the research program is to correlate the result of three different metrics – Contrast Transfer Accuracy (CTA), Contrast Signal to Noise Ratio (CSNR) and Frequency of Correct Resolution (FRC) – with the success of a license plate reading algorithm based on a neural network. In this talk, we are going to introduce the protocol that we have designed for this purpose and the preliminary conclusions that we draw on the three metrics.

This talk presents our investigation on using Bayer data instead of 3-channel processed RGB images with deep neural network (DNN). The amount of data captured by HDR, high-resolution automotive camera is tremendous. However, is the data format currently used optimal for the intended use case? Camera data can be processed with traditional computer vision methods or DNN algorithms. Often, these algorithms consume 3-colour channel data (Red-Green-Blue), generated from raw/Bayer data captured by the image sensor. By working with Bayer data, it is possible to reduce transmitted amount of data, needed power, and processing time. However, there is currently very little work on optimising DNN for single channel inputs, and there are no complete or fully annotated public Bayer datasets for assisted and automated driving functions. Interestingly, preliminary investigation demonstrates that current DNNs can be used with raw data without a significant performance degradation, and possibly can be further optimised and tuned to achieve even better results. Additionally, different methods to convert (with minimal modifications) existing datasets into different Bayer formats are discussed. In this way, big curated datasets can be re-used for creating Bayer based deep learning models.

In an ever-evolving autonomous driving landscape, the need for efficient, reliable and cost-efficient sensor fusion strategies is paramount. At Perciv AI, we’ve been innovating a novel approach to this challenge, leveraging the power of next-generation 4D radar and monocular RGB camera to generate ‘pseudo-LiDAR 3D point clouds. Cameras and radars have been used for driver assistance for decades. Our solution pushes the fusion of these sensors to the next level: the way we combine these two sensors will challenge the performance of LiDAR sensors for an 80% cheaper price. This ground-breaking technique presents a potential path to a future where high-end LiDARs may not be a necessity in consumer vehicles, only for research and evaluation purposes. Our introduced fusion paradigm is modular by design and can be trained without the reliance on expensive manual annotations. The outcome is a point cloud that mirrors the density of a LiDAR-generated one, but it also seamlessly integrates semantic information from the camera and velocity data from the radar. The variety and depth of this data offer a wide scope for potential applications in the field of autonomous driving, e.g. it can be effectively utilized for multiple downstream tasks such as object detection, free road estimation, or SLAM.

The presentation focuses on the evolving landscape of image sensor technology for automotive applications, driven by the need for High Dynamic Range (HDR) and LED Flicker Mitigation (LFM) while maintaining smaller pixel sizes for enhanced spatial resolution. A notable solution that has emerged is the adoption of Lateral Overflow Integrating Capacitor (LOFIC) Pixels by major automotive image sensor suppliers. The implementation of LOFIC-based sensors requires careful consideration of design trade-offs to optimize Image Quality (IQ). Key challenges include maintaining Signal-to-Noise Ratio (SNR) at elevated temperatures and addressing the increasing demand for higher IQ with shrinking pixel dimensions. The presentation discusses the trade-offs between different pixel approaches, such as the Split-Diode Pixel and the single PD LOFIC, examining their impact on low-light performance, SNR, and color reproduction. By discussing these trade-offs and the presentation aims to show that the optimal solution shifts towards single PD LOFIC as the pixel sizes shrink and provides more insights into the advancements and considerations in automotive image sensor technology.

Developing and testing autonomous driving (AD) systems requires the analysis and storage of more data than ever before. Next-generation Automotive enterprises are fully data-driven. They are focused on unlocking and activating the intrinsic value of the data in their development cycle using the combined practices of continuous integration (CI) and continuous delivery (CD). In this process the data can be created and accessed at the core or on the edge. Modern data loggers capture huge amounts of scenes and scenario data with smart recording technology under the control of a cloud-based AV fleet management solution.

Sensor fusion is key to environment perception in challenging conditions like snow, hail, nighttime, lens flares… Nowadays individual sensor processing relies heavily on machine learning, requiring algorithm developers to create or simulate large amounts of challenging training samples. Unfortunately, this leads to a combinatorically increasing need for training data to create all combinations of challenging conditions (e.g. snow+hail+night+lens flare). We propose “context-adaptive” fusion as a solution: a probabilistic approach wherein an “interpretation layer” translates the output statistic of an existing sensor algorithm to one that is tuned to a particular challenging context like fog, snow, hail, nighttime, lens flares, or even distance. The advantage is that this approach allows to adapt to the challenging context without requiring modification of existing sensor algorithms, using only a very small number of training samples. It is a natural fit for sensor fusion architectures where edge AI is provided with a low-data-rate input of said contexts through what is called “cooperative” sensor fusion. This talk will thus demonstrate how a “cooperative” fusion architecture outperforms a standard sensor fusion pipeline in terms of detection accuracy and tracking performance by adapting to different contexts through the proposed “interpretation layers”.

Surround view camera systems provide drivers with a bird’s eye view of their surroundings, aiding in low-speed maneuvers and enhancing safety. However, these systems have limitations, such as blind spots and geometric distortions, which can hinder the driver’s ability to recognize vulnerable road users (VRUs). In this presentation, we discuss our research on evaluating the image quality of surround view cameras and its impact on VRU safety. Our methodology involved exploring different approaches to characterize performance, including field of view diagrams, object size measurements, and qualitative assessments of image quality. Our findings indicate that while distortion at ground level is limited, significant deformations are observed at different elevations, posing challenges for VRU recognition. We emphasize the importance of including image quality measurements in the safety assessment of surround view cameras, in addition to blind zone definition and object size requirements. Improvements in digital processing algorithms and camera placement are necessary to reduce distortion and enhance VRU safety.

The accuracy of simulation technologies is currently a limiting factor in the development of perception systems for autonomous vehicles. The industry is still heavily dependent on collecting and processing real-world physical data and this process is expensive, time-consuming and doesn’t cover the variety and quantity of edge cases required to provide confidence in the system’s performance.

rFpro’s new ray-traced simulation rendering technology delivers high-fidelity engineering-grade synthetic training data. It has been developed in partnership with leading sensor and perception manufacturers, such as Sony, to integrate and improve the accuracy of virtual camera models as well as validate and correlate the results.

rFpro will showcase how this technology is helping to accelerate sensor development and advance vehicle perception systems, delivering high-quality training data.

The advancement in automotive technology has evolved rapidly from engines using fossil fuel to electric vehicle; from ABS to autonomous driving. An overview of automotive history will be presented in this presentation where many breakthroughs achieved recently will be highlighted. 
By observing facts and current trends, a forecast on the future growth on different segments of automotive market will be discussed. Finally, some new advanced technologies will be shown as examples and evidences supporting the prediction. 

Along with the ADAS development in automotive industry, multiple cameras installation on vehicle is essential. However, the continuous vibration during driving, bumps leading to vertical displacement of the camera modules and unfavorable lighting conditions result in blurry video images captured by the cameras.

The innovative application of MEMS Technology in automotive camera provides a 5-axis Optical Image Stabilization (OIS) solution to improve clarity and achieve stable and precise images. MEMS actuator has higher resonant frequency compared to the regular OIS solution, so it can perform well under the vehicle working conditions.

The MEMS actuator provides a new robust platform for the automotive industry, and this sensor-based technology is more than just OIS. It moves 3 times faster and 10 times more precisely by detecting vibrations of 0.4 micrometers. It consumes up to 50 times less power and function well under 115°C environment. It facilitates high image resolution in low light and video without blur. MEMS actuator also has long lifetime whereas cycling test passed more than 1,600 million and no image degradation. 

In this presentation, attendees will gain valuable insights into the critical role of imaging radars in achieving safe autonomy and next-generation Advanced Driver Assistance Systems (ADAS). Provizio’s innovative approach focuses on leveraging super-resolution imaging radars and Artificial Intelligence (AI) throughout the radar processing chain to unlock scalable, all-weather perception capabilities. By incorporating AI algorithms, attendees will discover how imaging data can be transformed into actionable insights, revolutionizing safety in autonomous systems. The session will showcase point cloud neural network (NN) architectures for super-resolution and denoising, as well as NN architectures for radar-only detection, classification, and freespace estimation. By compounding the advantages of super resolution radars and AI integration at various stages of the chain, attendees will understand how scene understanding is exponentially enhanced, leading to ubiquitous perception on-the-edge.

Until recently, the majority of sensor-based AI processing used vision and speech inputs. Recently, we have begun to see radar, LiDAR, event-based image sensors, and other types of sensors used in new AI applications. And, increasingly, system developers are incorporating multiple, heterogeneous sensors in their designs and utilizing sensor fusion techniques to enable more robust machine perception. In this presentation, we’ll explore some of the heterogeneous sensor combinations and sensor fusion approaches that are gaining adoption in applications such as driver assistance and mobile robots. These sensor fusion techniques are using a mix of AI techniques and traditional DSP processing. We’ll also show how the Cadence Tensilica ConnX DSP, Vision DSP, and AI Accelerator IP families and their associated software tools and libraries support sensor fusion applications with high performance, efficiency, and ease of development.

Considering sensor perception interferences due to adverse weather conditions is decisive for improving the quality of ADAS functions and indispensable for the road release of AD functions / vehicles.
Even more than in the past the “right mix” of test and validation approaches is essential for a reliable and safe sensor and perception approval.
An inverse test strategy from real-world data to indoor testing and virtual simulation is required to ensure a realistic cross validation of the different test and validation approaches (e.g. indoor vs. outdoor, static vs. dynamic SUT).

In a dynamic era of automotive evolution, LGE stands as a pioneer in driving innovation. This session unveils two pivotal strategies that define our commitment. Discover how we’re engineering architectures that seamlessly scale, catering to the evolving demands of autonomous driving, from GSR/NCAP to L3/L4 levels. Moreover, we’ll delve into the crucial realm of identifying and mitigating safety risks, ensuring a secure and reliable foundation for autonomous driving platforms. 

Image sensors act as the eyes of the vehicle and are used in several ADAS functions, such as lane departure warnings, pedestrian detection, and to trigger emergency braking. They also provide input to the fusion system for decision-making to assess the car’s environment and can even be used to monitor driver behavior. In the future, they will assist with identifying and authenticating the car’s users and monitor their vital signs to enable the onboard computer to assume control if the driver becomes incapacitated. It is therefore imperative that image sensors remain functional, especially in the most extreme situations an automobile can encounter.