Going Beyond Nouns With Vision & Language Models Using Synthetic Data

Going Beyond Nouns With Vision & Language
Models Using Synthetic Data

Paola Cascante-Bonilla^1,2, Khaled Shehada^2,3, James Seale Smith^2,4, Sivan Doveh^6,7,
Donghyun Kim^2,7, Rameswar Panda^2,7, Gül Varol⁵, Aude Oliva^2,3,
Vicente Ordonez¹, Rogerio Feris^2,7, Leonid Karlinsky^2,7

¹Rice University, ²MIT-IBM Watson AI Lab, ³MIT, ⁴Georgia Institute of Technology,
⁵LIGM, École des Ponts, ⁶Weizmann Institute of Science, ⁷IBM Research

Paper Data & Code

Abstract

Large-scale pre-trained Vision & Language (VL) models have shown remarkable performance in many applications, enabling replacing a fixed set of supported classes with zero-shot open vocabulary reasoning over (almost arbitrary) natural language prompts. However, recent works have uncovered a fundamental weakness of these models. For example, their difficulty to understand Visual Language Concepts (VLC) that go 'beyond nouns' such as the meaning of non-object words (e.g., attributes, actions, relations, states, etc.), or difficulty in performing compositional reasoning such as understanding the significance of the order of the words in a sentence. In this work, we investigate to which extent purely synthetic data could be leveraged to teach these models to overcome such shortcomings without compromising their zero-shot capabilities. We contribute Synthetic Visual Concepts (SyViC) - a million-scale synthetic dataset and data generation codebase allowing to generate additional suitable data to improve VLC understanding and compositional reasoning of VL models. Additionally, we propose a general VL finetuning strategy for effectively leveraging SyViC towards achieving these improvements. Our extensive experiments and ablations on VL-Checklist, Winoground, and ARO benchmarks demonstrate that it is possible to adapt strong pre-trained VL models with synthetic data significantly enhancing their VLC understanding (e.g. by 9.9% on ARO and 4.3% on VL-Checklist) with under 1% drop in their zero-shot accuracy.

Overview

We finetune a large-scale pre-trained VL model using our SyViC dataset. Our proposed methodology includes domain adaptation and avoiding forgetting through parameterefficient finetuning (LoRA) and model averaging.

Summary of the entire flow, including components and choices of SyViC data synthesis pipeline (left) and the proposed effective finetuning technique (right).

Results

We evaluate our finetuned model on three benchmarks: VL-Checklist, ARO, and Winoground. The compositional reasoning evaluation includes understanding the meaning of the sentence after changing the word order, attributes, and relations of humans/objects.

Poster

BibTeX
                    
                        @misc{cascantebonilla2023going, 

                                title={Going Beyond Nouns With Vision & Language Models Using Synthetic Data}, 

                                author={Paola Cascante-Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gül Varol, Aude Oliva, Vicente Ordonez, Rogerio Feris, Leonid Karlinsky}, 

                                year={2023}, 

                                eprint={2303.17590}, 

                                archivePrefix={arXiv}, 

                                primaryClass={cs.CV}
                        }