As vehicle perception systems become more dependent on AI, the image pipeline can no longer be treated as a fixed step before the neural network begins.
In this interview, Arm Lead ISP Architect and Fellow Alexis Lluis Gomez, explains how differentiable ISPs allow image processing parameters to be optimised directly against perception performance. From reducing manual tuning and adapting to new sensors, to helping preserve model accuracy across changing camera platforms, he explores how task-specific imaging could reshape the route from raw sensor data to production-ready automotive AI.
1. What limitations of traditional ISPs motivated a differentiable approach?
Traditional ISPs are powerful, but they were not designed to be tuned as part of an ML training loop.
The first limitation is that humans do not always know what an AI model needs from an image. Traditional ISP tuning is based on human visual preference: good colour, pleasing contrast, controlled noise, natural sharpness, and minimal visible artefacts. Those are valid goals for display, but they are not necessarily the right goals for detection, classification, segmentation, or tracking. Human-vision and computer-vision tunings can be very different when the goal is to maximise task performance rather than produce the most visually pleasing image. A computer vision model may prefer a different tone curve, less denoising, different sharpening, or different HDR handling than a human viewer would.
A fully differentiable ISP cuts out much of the human guesswork. Instead of asking image quality engineers to predict what the network needs, it lets the network drive the tuning process itself. ISP parameters can be optimised against the task objective and cost function, so the system learns which image representation best supports the model’s performance. In other words, humans should not have to decide what is good for the network; the network should be able to learn that for itself.
The second limitation is tuning effort and network robustness. Today, ISP tuning is still largely a manual process carried out by image quality engineers. The resulting image database is then often passed to ML teams for training, without the network having any visibility of how the ISP was tuned or whether those tuning choices are optimal for the task. Tuning an ISP involves many interacting blocks, many registers, and subjective trade-offs between noise, sharpness, colour, tone mapping, detail, and artefacts. At that stage, it is very difficult for a human to know whether a particular trade-off will help or harm network performance.
This process is also time consuming. For a new sensor or product, ISP tuning can take months, and the work may need to be repeated or adjusted when the sensor, optics, ISP configuration, or use case changes. Those changes can shift the image statistics seen by the network and reduce AI performance if the model was trained on a different camera pipeline.
A differentiable ISP turns much of that problem into an optimisation problem. It can tune ISP parameters against a defined ML objective and help adapt the image pipeline to the model, sensor, and task. During training, the ISP parameters can be optimised through backpropagation together with, or alongside, the neural network. The result may not necessarily be the most visually pleasing image, but it can be the image representation that gives the best ML performance for the task.
The third limitation is portability across cameras and platforms. Moving between different sensors, optics, ISPs, platforms, or tuning configurations can create a domain gap for pre-trained networks. A model trained on data from one camera pipeline may not perform as well when deployed on another, and datasets captured on one system may be harder to reuse on another. A differentiable ISP can help map one camera domain to another by learning how the image pipeline should transform sensor data for the target model and task. In that sense, the ISP becomes part of the data generalisation strategy, rather than just a fixed image rendering block.
In effect, a differentiable ISP improves all three sides of the cost-time-accuracy triangle for image quality tuning in AI applications: it reduces manual effort and cost, shortens tuning time, and helps deliver the best image representation for the task.
2. How does differentiable processing accelerate adaptation across sensors and environments?
Sensor and platform changes often shift the input distribution seen by a trained network. A model trained on one camera pipeline may lose performance when the sensor, ISP, tuning, or operating environment changes. Traditionally, recovering that performance can require new data capture, ISP retuning, and retraining.
A typical example is a production vehicle whose network was trained on a large labelled dataset of post ISP camera frames. If the camera sensor is later upgraded, the new sensor will have different noise, colour, dynamic range, HDR, and response characteristics. The ISP will need a new tuning, but standard image quality metrics such as MTF, SNR, sharpness, colour or noise are not always reliable indicators of AI performance, nor are they useful metrics for training. The key question is: how should the ISP be tuned so that the existing network still performs well or even better without substantial retraining? The answer is that the ISP should not just be tuned; it should be trained.
A Differentiable ISP provides three practical flows that use the neural network loss function to guide ISP training.
- Full training: ISP and neural network training. The ISP parameters and CNN weights are trained together when building a new vision chain.
- Domain adaptation: ISP trained, neural network frozen. In this scenario, the CNN is kept fixed and only the ISP parameters are trained. The frozen network still provides gradients through the Differentiable ISP, allowing the ISP to learn a tuning that maps the new camera output into a representation that works well for the existing model and task. This requires a labelled set of raw data from the new camera, but because the ISP has far fewer parameters than a CNN, the amount of data required can be much smaller than the dataset used to train the original network.
- Joint fine tuning: ISP trained, selected neural network weights fine-tuned. In this scenario, the ISP parameters are trained while some of the network weights are also allowed to adapt. For example, most of the network could remain fixed while the first layer adapts to the new sensor characteristics. Alternatively, the network can start from the existing trained weights instead of being trained from scratch. This gives the system more flexibility when the new sensor provides useful information that the original network was not trained to exploit, while still avoiding a full retraining exercise.
In all three flows, the Differentiable ISP acts as a bridge between camera domains. It helps adapt a pretrained network to a new sensor, ISP configuration, or operating environment by learning how the image pipeline should transform raw data for the target task. This improves dataset reuse and reduces the cost of moving between camera platforms.
This is particularly useful when moving to a newer ISP or sensor. The original training dataset does not need to have been captured on the final production ISP. Instead, the differentiable model can be used offline to find a task-optimised tuning for the target pipeline, and the resulting parameters can then be deployed in hardware.
The result is a faster adaptation path: less manual ISP retuning, less dependence on large dataset recapture, and less need to retrain the full network whenever the camera system changes.
3. Which perception tasks benefit most from this capability?
The tasks that benefit most are those where the model’s performance is directly affected by the image formation pipeline. In practice, that means tasks where changes in tone mapping, contrast, denoising, sharpening, colour processing, HDR handling, or local detail can change the task metric in a measurable way.
So far, the main experiments have focused on image classification, object detection, and semantic segmentation. These are useful examples because they respond differently to ISP changes. Classification can benefit when ISP tuning changes the visual features used by the network. Object detection can benefit when the tuning improves object visibility, local contrast, or detail, especially in difficult lighting. Semantic segmentation can be particularly well suited because the loss is usually dense: many pixels contribute to the training signal, so the optimiser gets more information about how the ISP settings affect the task.
The benefit also depends on how much of the performance gap can be recovered through image transformations that the ISP can perform. If the main challenge is exposure, tone mapping, noise, sharpness, colour, or HDR rendering, ISP optimisation can help. If the gap comes from missing labels, insufficient model capacity, poor architecture, or a task mismatch, ISP tuning alone will not solve it. In those cases, neural network fine tuning may be needed to reach the best result.
Another important factor is the starting point. If the initial ISP tuning is already close to optimal for the task, the gain may be modest. If the starting point was tuned mainly for human viewing, or for a different sensor or environment, the Differentiable ISP has more room to improve task performance.
The general expectation is that the best candidates are tasks with a clear performance metric and a direct link between that metric and the input pixel data. If the task can provide a useful training signal back to the image pipeline, then Differentiable ISP can potentially learn a better task specific representation.
4. How close are we to seeing differentiable imaging pipelines in production vehicles?
The Differentiable ISP will not itself be installed in production vehicles. The Mali-C720AE Differentiable ISP is a highly accurate software model of the ISP hardware. In addition to faithfully reproducing the output of the ISP modules, it allows gradients to flow backwards during training, adjusting ISP parameters as needed to achieve optimal performance.
Once the Differentiable ISP model has been trained, the resulting ISP parameters are ported to the ISP firmware and used to drive the hardware ISP in the production vehicle. This means no changes are needed to the vehicle or SoC. The main change is the ISP tuning process: manual tuning is replaced, or complemented, by integrating the Differentiable ISP into the training pipeline.
Mali-C720AE’s computer vision pipeline includes newly designed modules that provide greater flexibility for Differentiable ISP based training. Classical ISP modules are also modelled with a high degree of accuracy and can be optimised with the Differentiable ISP.
Differentiable ISP model enables full benefits of low latency and efficiency provided by fixed-function ISP hardware. The Mali-C720AE differentiable ISP model is available to Arm and Zena CSS partners and OEMs for evaluation.
We already have multiple AE partners implementing Mali-C720AE, and we expect first silicon products to become available 2027.
Don’t miss Alexis’ presentation ‘Differentiable ISP for Rapid Camera Domain Adaptation in Autonomous Driving‘ at AutoSens Europe this year!
Interested in In-Cabin monitoring technology?
With a pass to AutoSens Europe, you’ll also get full access to our co-located sister event, InCabin. Take a look at the full agenda for InCabin here >>
