Modern ADAS and autonomous driving systems generate terabytes of sensor data, creating a need for intelligent systems that can retrieve images containing objects of interest described by natural language queries. Applications of such systems are varied, including rare-object mining from large-scale unlabeled datasets, targeted data annotation, and streamlining system evaluation processes. The previous leading approach relies on aggregating OpenAI CLIP features without any adaptation to the target domain, ultimately limiting its performance. Our work “FOR: Finetuning for Object Level Open Vocabulary Image Retrieval”, WACV 2025, addresses this limitation through fine-tuning on a target dataset using closed-set labels, while preserving the visual-language association that is crucial for open-set retrieval. FOR is based on a combination of dedicated architecture elements based on CLIP, coupled with a multi-objective training framework. Together, these design choices result in a significant accuracy improvement over previous SoTA across multiple datasets.