As cars become reliant on on-board cameras for automated operation, the consequences of any flaws in the quality and interpretation of images sent to decision-making systems starts to become a matter of life and death. So an agreed way to benchmark image quality is imperative, yet no specific product standard for automotive applications currently exists.
This realisation prompted sensor system engineers to initiate the process of writing such a standard, and having analysed existing standards, an industry working group is now looking to open discussion to include representation from across the supply chain.
According to Dr Sven Fleck, “There are image quality standards for other use-cases, such as for surveillance and cell phone cameras, but these have no direct comparison in autonomous vehicle systems where image quality becomes safety relevant.”
Dr Fleck is Vice-Chair of the aptly named Institute of Electrical and Electronics Engineers (IEEE) P2020 Automotive Image Quality Working Group, that currently benefits from the input of around 300 experts, of which around 50 regularly attend face-to-face meetings.
The working group’s first milestone, ahead of its next meeting and coinciding with the AutoSens Conference, in Brussels on 17-20 September 2018, is publication of the IEEE P2020 Automotive Imaging White Paper. View white paper here. This document explains the technical issues being addressed, a process that would benefit, it considers, from collaboration with a broader mix of stakeholders.
As stated by Dr. Marc Geese of Robert Bosch, who leads a sub-group dealing with image quality for computer vision, “In our working group we currently have many active tier 1 and image sensor suppliers involved. We’re approaching the first draft of the standard so, in my opinion, it’s a good time for further OEM and companies involved in the imaging supply chain to review our progress and consider getting involved.”
Listening to industry
Robert Stead, Managing Director of Sense Media, provided much of the initial impetus. “Even prior to starting Sense Media in 2015 I’d had conversations about the need for standards development work”. “We agreed it would be a good initiative, although I wasn’t necessarily planning on being the chair of the group at that point”, he recalled with a hint of stoic laughter.
“I may have got it started in an administrative sense, but the idea and concepts came from experts such as Patrick Denny of Valeo Vision Systems and Ulrich Seger from Robert Bosch who identified the potential benefits of collaboration.”
The IEEE route to standards making was chosen as many of the image quality themes stem from work done under its camera phone image quality standard. The choice of standard number was more fortuitous. “That wasn’t actually the next number available, but our IEEE liaison, Michael Kipness, took a creative leap, and we landed with a very appropriate project name as we work to give future vehicles perfect 20:20 vision.”
As chair of the IEEE P2020 steering committee, Robert is keen to acknowledge the work of other bodies too. “EMVA 1288 by the European Machine Vision Association and IEEE-SA P1858 (CPIQ) for camera phones provided input. There’s quite a lot of relevant standards, KPIs, and test methods for machine vision but no other standard has so far focused specifically on automotive.” “P2020 is trying to avoid duplication. We have relationships with other organisations so can adopt sections of standards from other sources, but many automotive use-cases are unique.”
There are other imperatives too.
According to Dr Sven Fleck, “We felt there was a big gap when we applied to start the working group in December 2015. There was no way to compare automotive imaging systems in numerical terms. You can’t, for instance, quantify how a system will behave in low light, or in low contrast conditions.”
Raising awareness of the consequences
The White Paper features two scientific papers produced by P2020 work so far, both presented at the Society for Imaging Sciences and Technology Electronic Imaging Symposium in January this year.
A major issue highlighted by Brian Deegan, Vision Research Engineer at Valeo, arises from LED flicker. Not a problem for human perception, this characteristic causes a sampling problem for imagining systems: as measured light intensity, such as of traffic lights, varies at a high frequency, leading to a measuring problem with obvious potential for misinterpretation.
The second paper features Marc Geese’s work on a new KPI called Contrast Detection Probability. Sven Fleck pointed to sequential video frames taken from the perspective of a vehicle entering a tunnel, where a combination of light conditions and camera readings produces a veiling glare in the image leading to contrast reduction. This example, he said, “makes the car ahead seemingly invisible for a whole second due to the reduction in contrast”.
Underlining the point, Dr Fleck added, “it’s getting to the point where image quality is what can decide whether you will be dead or alive”. Or putting it more mildly, image quality could be the deciding factor between safe navigation of a tricky ‘edge case’ situation, or a serious accident.
The working group divided its efforts into subgroups to address the priority issues.
Brian Deegan’s group is concentrating on the LED flicker problem, while, as Robert described, “…another subgroup is looking at image quality for visual output, where the output would be displayed on a screen for the driver, such as a reversing camera or a mirror-replacement system. This group is thinking about how to optimise image quality for the human user.”
“A third group is addressing computer-vision: looking at image quality, where data is processed to detect and classify objects by computer rather than humans.”
According to Robert, the scope of the work is being kept deliberately narrow. “Cameras are the most prominent form of sensing involved in ADAS and autonomous vehicles at the moment so they’re the first thing to address in terms of developing a standard specifically for automotive. Longer term, there may be a need for a broader set of standards covering other sensor modalities.”
Taking responsibility while retaining design flexibility
Automotive OEMs, not unexpectedly, tend to differentiate their products as a complete package, and may be inclined to leave component specification definitions to suppliers. Ultimately, however, where these metrics have safety consequences, such an approach may, it could be argued, result in additional legal and reputational exposure.
As Sven Fleck put it, “as image sensors become common the related safety issues becomes more relevant for the OEMs.” “Perhaps it’s unsurprising that OEMs prefer to let suppliers take the lead but we’d like to encourage European OEMs and especially the large German OEMs to become more closely involved, and then others may then want to take part in the standards process too.”
Volvo, Volkswagen and General Motors are listed as contributors to the White Paper, perhaps conscious that lawyers might be waiting in the wings. Indeed, playing a positive role in developing standards might be regarded as showing commitment to reducing the risk of accidents happening in the first place.
“OEMs may be concerned that standards could make it harder to differentiate their products”, added Sven. “But the standard as it’s being written is about defining how to measure image quality and helping define which key performance indicators are important.” “So, they can have flexibility to define their own requirements but will benefit from having KPIs for image quality clearly defined.”
Better communication and better products
Looking ahead Robert sees success as being industry-wide adoption of the standard. “Setting common test methods and a way to measure camera systems against each other will facilitate much, much better communication between people developing camera systems, for suppliers, customers or testing and certification organisations, but particularly OEMs.”
“The long-term benefit will be rather than each company spending time developing their own test methods to evaluate the quality of its camera systems, they will use the IEEE 2020 set of standards. Their time and effort can instead be spent developing better cameras rather than developing testing methods in silos.”
“A commonly understood way of measuring image quality should also have the effect of driving up image quality as a whole, with a knock-on effect of improved safety.”
“There’s also a lot of questions around regulation as testing of autonomous vehicles is happening and we’ve seen some accidents unfortunately. Part of the motivation for the standard is to be proactive and not wait for governments to impose regulations. Being proactive and working together to improve the quality of these systems is the right thing to do.”
Broader representation welcomed
Standards making, according to IEEE, provides opportunities to network, broaden understanding, and gain early insight — so facilitating early compliance and anticipation of market requirements. The objective of P2020 is for engineers developing automotive camera systems to be able to compare systems in a like-for-like manner, while the challenge, according to Sven Fleck, is, “ensuring image quality KPIs in the standard are relevant for all use-cases and making sure these KPIs are really doing what they should be doing”.
Camera sensor systems are being installed as part of the safety features in production models, so the need for consensus is real and urgent. The P2020 working group is therefore keen to connect with more experts that could benefit from engaging with the process of raising image quality standards across the whole automotive sector.
