\documentclass[english]{jnlp_3.8}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{array}
\usepackage{booktabs}
\usepackage{hangcaption_jnlp}
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%\usepackage{soul}
%%%%\usepackage{amssymb}
%%%%\usepackage{comment} 
%%%%\usepackage[multiple]{footmisc}
\usepackage{multirow}
%%%%\usepackage{color}
%%%%\usepackage{fourier}
%%%%\usepackage{array}
\usepackage{makecell} 
%%%%\newcommand{\blue}[1]{\textcolor{blue}{#1}}
%%%%\newcommand{\red}[1]{\textcolor{red}{#1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{32}
\Number{2}
\Month{June}
\Year{2025}
%%%%\Issuetitle{Special Issue: XXXXXXXX}

\received{2024}{10}{15}
\revised{2025}{2}{6}
\accepted{2025}{3}{18}

\setcounter{page}{598}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Using Linguistic Formalism to Improve Real World Understanding for V\&L Models: Case Study on Image Discrimination for Structurally Ambiguous Language Input}
\eauthor{Lee Sangmyeong\affiref{Author_1}\affiref{Author_2} \and Seitaro Shinagawa\affiref{Author_4}\affiref{Author_1} \and Koichiro Yoshino\affiref{Author_3}\affiref{Author_2}\affiref{Author_1} \and Satoshi Nakamura\affiref{Author_5}\affiref{Author_1} } 
\eabstract{%
In the context of Real World Understanding (RWU) for vision and language (V\&L) models, accurately aligning language with the corresponding visual scene is critical. Since current models typically assume language inputs to be plain text, RWU faces potential issues with structural ambiguity, where a single sentence can have multiple meanings due to various phrase structures. This paper proposes to use linguistic formalism as input, which enriches language information and addresses the issue of structural ambiguity. Our focus is on the Contrastive Language-Image Pre-training (CLIP) model, a prominent V\&L model, focusing on image discrimination tasks of RWU. Our experiments test various approaches to incorporating formalism into the CLIP model, depending on the type of formalism and its processing method. We aim to determine the effectiveness of formalism in discriminating ambiguous images and identify which formalism works best. Additionally, we employ a gradient-based method to gain insights into how formalism is interpreted within the model's architecture.}
\ekeywords{Vision and Language, Structural Ambiguity, Linguistic Formalism}

\headauthor{Lee et al.}
\headtitle{Using Linguistic Formalism to Improve Real World Understanding for V\&L Models}

\affilabel{Author_1}{}{Nara Institute of Science and Technology}
\affilabel{Author_2}{}{Guardian Robot Project, RIKEN}
\affilabel{Author_4}{}{SB Intuitions}
\affilabel{Author_3}{}{Institute of Science Tokyo}
\affilabel{Author_5}{}{The Chinese University of Hong Kong, Shenzhen}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

\renewcommand{\thefootnote}{}
\footnote[0]{A part of this manuscript is based on the authors' previous work at MIRU2023 \cite{Lee2023miru}.}
\renewcommand{\thefootnote}{\arabic{footnote}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

For vision and language (V\&L) models, achieving a clear understanding of the real world they engage with---termed Real World Understanding (RWU)---is of paramount importance. Enhancing RWU bridges the gap between user and system, improving usability and enabling various applications. A key component of RWU is the ability to accurately discriminate visual information with its correct linguistic counterparts. In recent years, significant advancements in this area have been made through large model pre-training, demonstrating high few-shot and zero-shot V\&L discrimination  capabilities \cite{Radford2021LearningTV,Jia2021ScalingUV,Zhai2021LiTZT}. These advancements have been particularly successful in applications requiring precise matching between the two modalities, such as text-to-image generation models \cite{Rombach2021HighResolutionIS,Ramesh2022HierarchicalTI}. 

Current advancements in V\&L models often rely on using plain text for the language module. However, natural language sentences tend to inherently contain ambiguity, having multiple interpretations. One example is structural ambiguity arising from various phrasal grammar structures. Take the sentence ``Danny approached the chair with a bag''---this could mean either \textit{Danny has the bag} or \textit{the bag is on the chair} (Figure \ref{fig:one}). This ambiguity makes it hard to accurately discriminate visual and textual information. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[b]
\begin{center}
\includegraphics{32-2ia7f01.eps}
\end{center}
\hangcaption{Multiple meanings of the text ``Danny approached the chair with a bag.'' based on the syntactic structure (image fragments from \protect\cite{Berzak2015DoYS}).}
\label{fig:one}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Structural ambiguity is crucial in practical applications requiring RWU capability. Think of an example of an image retrieval system, which outputs images based on the user's utterance (Figure \ref{fig:two}). Conventional systems would presume a plain text as input to get the desired image. However, when the input is structurally ambiguous as in Figure 1, it is difficult for the user to guarantee that the visual output corresponds with the intention; the bag in the image might be on the chair, not in the man's arms (Figure \ref{fig:two} (A)). Even by retrieving multiple images so that at least one of them has the target one, it still takes much effort for the user to identify countless images one-by-one to reach the intended output (Figure \ref{fig:two} (B)). 
In this case, leveraging linguistic formalism, a data structure expressing the language's structural information, could help the user get the output as intended in an efficient way (Figure \ref{fig:two} (C)). Given an ambiguous input, a formalism parser parses it into all the possible formalisms (e.g. K-best algorithm). Together with multiple images retrieved from the image retriever, formalisms go into the image formalism discriminator, aligned into groups depending on the precise structural meaning (in this case, a meaning where the bag is in the arms, and on the chair). The users choose the intended group by looking at the formalism and example image, from which they can freely go into further details.      
This method contrasts with traditional text-to-image retrieval system, which often yield numerous potential matches, requiring significant effort from the user to sift through. By integrating linguistic formalisms, the system can pre-filter images according to meaning, reducing the number of options presented to the user and enhancing overall satisfaction. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{32-2ia7f02.eps}
\end{center}
\caption{Example use case of linguistic-formalism-aware V\&L model.}
\label{fig:two}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

From our example case in Figure \ref{fig:two}, we hypothesise that integrating linguistic formalism enhances the V\&L model's RWU ability on image discrimination. We expect that linguistic formalism enriches the language information to match various visual interpretations from ambiguity one by one, strengthening the V\&L alignment.
To examine our hypothesis, we systematically explore various methods for integrating linguistic formalism, 
to identify the most effective approach. We conduct experiments within the context of text-to-image retrieval---a straightforward and precise benchmark to evaluate the model's discrimination capabilities. 
We adopt CLIP model \cite{Radford2021LearningTV} as the implementation core, which has shown great zero-shot V\&L aligning ability. 


Furthermore, we perform a qualitative analysis to validate the effectiveness of linguistic formalism. Using the integrated gradients method, we analyse the model's attention when processing linguistic formalism inputs. This approach provides insights into how the model interprets the underlying linguistic structure \cite{Sundararajan2017AxiomaticAF}, providing valuable guidance for further refinement and application in diverse contexts.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Studies}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Linguistic Formalism}

The syntax tree, a tool for expressing the grammatical structure of a sentence is broadly categorised into two types based on grammar rules: constituency grammar and dependency grammar. Constituency grammar highlights the hierarchical arrangement of words into constituents (Figure \ref{fig:three}(A)), while dependency grammar focuses on word relationships, establishing directed dependencies that specify the syntactic functions of each word. For our research, we opt for the syntax tree based on constituency grammar. This decision is influenced by the prevalence constituency grammar in previous research efforts, reflecting the dominant emphasis on encoding syntax trees.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[b]
\begin{center}
\includegraphics{32-2ia7f03.eps}
\end{center}
\hangcaption{Examples of linguistic formalisms for the text ``Danny approached the chair with a bag.'', with the interpretation that the man is holding a bag. (A) represents the tree structure based on syntactic grammar, and (B) expresses the graph structure from semantic parsing.}
\label{fig:three}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Semantic parsing is a task that involves converting language expressions into a formal representation of meaning. Common representation types include logical forms or structured query language (SQL) queries. Logical forms convey language meaning using a predicate-argument structure to capture the relationships between sentence elements (Figure \ref{fig:three}(B)). For instance, the sentence ``Danny approached the chair with a bag.'' is expressed as a predicate of action (approached) with associated arguments of entities (Danny, bag, and chair), with the order of argument numbers (0, 1, 2) delineating the semantic roles of each entity.
On the other hand, SQL queries structuralise meaning into multiple SQL formats combined with a logical intersect. This approach is effective for tasks involving a knowledge database and has primarily been used in question-answering systems \cite{Hamilton2018EmbeddingLQ,bai-etal-2022-query2particles,Ren2020Query2boxRO}. Given that our task doesn't involve a database, we represent the logical form structure as a graph, terming it a semantic graph. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Encoding Linguistic Formalism}

In this section, we introduce two primary approaches to encoding constituency grammar. The first involves using the syntax tree as a guideline to recurrently merge word vectors, while the second divides the syntax tree into text and phrase labels for separate encoding. Additionally, we introduce a graph-based encoding methodology for natural language processing tasks. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 4
\begin{figure}[b]
\begin{center}
\includegraphics{32-2ia7f04.eps}
\end{center}
\hangcaption{Expressions of the syntax tree from a sentence ``Danny approached the chair with a bag.'' as a bracketed text. (A) shows the original tree, (B) shows the modified version used for the encoding methodology in Section 2.2.1 and (C) shows the one for separate encoding method in Section 2.2.2.}
\label{fig:four}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Syntax tree as a guideline} 

Typically represented as text containing words, corresponding phrase labels, and brackets indicating groupings (Figure \ref{fig:four}(A)), the syntax tree is modified in this approach to retain only the grammar information at the phrase level, emphasising how words are grouped (Figure \ref{fig:four}(B)). Here, the tree serves as a direct guide for combining word vectors during the encoding stage. At the core of this methodology lies Recurrent Neural Network Grammar (RNNG) \cite{Dyer2016RecurrentNN}, which aggregates word embeddings based on the tree structure. Building upon RNNG, subsequent research has aimed to enhance its efficiency. For instance, Unsupervised RNNG \cite{Kim2019UnsupervisedRN} addressed RNNG's dependency on annotated parsed tree corpora by introducing an inference network that acts as a neural parser to compensate for the lack of annotated data. More recently, efforts have extended this principle to the transformer model \cite{Qian2021StructuralGF}, leading to the development of Transformer-Grammar \cite{Sartran2022TransformerGA}. Transformer-Grammar constructs an attention mask based on the parsed tree and integrates it into the transformer encoder, resulting in notably improved results. This methodology is applied in Section 3.2 of our proposal. 



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Separate encoding of syntax tree}

The focal point of this section revolves around a distinctive approach: dividing the input sentence into two components---the original text and the parsed tree, represented textually with brackets and phrase labels. These components undergo separate processing to generate embedding vectors, later combined within the transformer decoder for specific tasks. This methodology had its roots in an LSTM-based approach \cite{Iyyer2018AdversarialEG} and has evolved into a transformer-based methodology as well \cite{Li2020TransformerBasedNT}. In the transformer-based approach, the parsed tree is decomposed into several paths from the root to each leaf node, assisting the model in determining which path to focus on. We apply the principle of this transformer-based methodology in Section 3.3 of our proposal. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Graph encoding for natural language processing} 

Graph Convolutional Network (GCN) \cite{Kipf2016SemiSupervisedCW} has gained attention for its effectiveness in capturing intricate relationships and dependencies within graph-structured data. It achieves this through convolutional layers of node embeddings computation based on the adjacency matrix. The application of GCN extends to the encoding of linguistic formalism, wherein formal structures are represented as graphs. Examples include syntax trees as graphs \cite{Marcheggiani2017EncodingSW,Bastianelli2020EncodingSC}, semantic graphs \cite{Cao2023LeveragingDA,Xu2021DynamicSG}, etc. This idea is applied in Section 3.4 of our proposal, by constructing a semantic graph and applying GCN on it.  


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Limitations of the V\&L model's Plain Textual Inputs}

In Section 1, we discussed how plain textual inputs have limitations in handling structural ambiguity for V\&L tasks. A growing concern in this area is Visio-Linguistic Compositionality--the ability of V\&L models to correctly associate images with the right arrangements of words. For instance, in a study, the CLIP model was shown an image of a purple eggplant and a yellow lemon \cite{Yamada2022WhenAL}. When asked about the colour of the lemon, the model incorrectly responded with ``purple''. This highlights a gap in the CLIP model's ability to align visual scenes and their corresponding textual descriptions. To overcome this limitation, some approaches have incorporated scene graphs---a graphical representation detailing the attributes and relationships among objects within a visual scene \cite{Huang2023StructureCLIPEM,Singh2023CoarsetoFineCL}, to improve this alignment. 
Our research on image discrimination for structural ambiguity shares a similar feature: improving alignment by incorporating additional input beyond plain text. However, while compositionality issues may be fixed by improving the model, structural ambiguity is inherently unsolvable with text alone. We aim to not only resolve this ambiguity but also improve the usability of V\&L models, as shown in Figure \ref{fig:two} (C), where users can successfully extract images that match their intended meaning from countless generated options. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 5
\begin{figure}[b]
\begin{center}
\includegraphics{32-2ia7f05.eps}
\end{center}
\hangcaption{Overview of the proposal along with types of parsers and formalism encoders in the CLIP model's language part.}
\label{fig:five}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed Methods}

Our proposed methodology revolves around integrating various linguistic formalisms into the V\&L model's language processing component (Figure \ref{fig:five}). As mentioned in Section 1, we use the CLIP model, which is trained through contrastive learning to align images with their corresponding text. This helps in tasks that require a strong connection between visual content and language. Compared with the conventional CLIP model, our proposal introduces two key distinctions. Firstly, during fine-tuning, we keep the CLIP model's vision component parameters fixed. Given its extensive pre-training, the vision encoder of the CLIP model is expected to yield high performance without additional training. Therefore, we freeze the vision encoder and focus on refining the language component. Secondly, while conventional CLIP models encode plain text as inputs for the language model, our proposals initially convert the plain text into linguistic formalism and then undergo the encoding process based on the type of linguistic formalism and the chosen encoding methodology. We explore four types of methodologies as proposals for encoding linguistic formalism, listed in Table \ref{table:one}. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[b]
\input{07table01.tex}%
\caption{List of proposals based on the type of linguistic formalism and its encoding methodologies.}
\label{table:one}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The ``Tree as Text'' approach, the first row in Table \ref{table:one}, represents the simplest way of encoding the syntax tree into the CLIP model. It involves fine-tuning the conventional CLIP model by treating the linearised syntax tree treated as texts, without additional alteration to the model architecture. This methodology is discussed in detail in Section 3.1. The ``Phrase Level Tree'', the second row in Table \ref{table:one}, aims to encode the syntax tree following the principle suggested by RNNG \cite{Dyer2016RecurrentNN} and passed down to Transformer-Grammar \cite{Sartran2022TransformerGA}. The syntax tree is modified to the phrase level to reflect how the words are grouped, and a custom attention mask is built. This attention mask is used for additional neural nets beyond the conventional CLIP model's text encoder, as discussed in Section 3.2. ``Path Attention'', the third row in Table \ref{table:one}, applies the method of \citeA{Li2020TransformerBasedNT}. It involves dividing the syntax tree into phrase labels and texts, which are encoded separately and combined at the end. The syntax tree is divided into several paths from the root to each leaf, from which path attention scores are acquired. This process will be explained in detail in Section 3.3. Lastly, ``Semantic GCN'', the last row in Table \ref{table:one}, constructs graphs based on semantic parsing given a sentence and undergoes the GCN encoding algorithm. The graph construction and the architecture of GCN models are explained in Section 3.4. 

While we treat various methodologies regarding the use of syntax tree in our research, we try out only one for the graph structure. For previous research focusing on leveraging graph structures, GCN served as the primary representative method. The reason we speculate is that the message passing paradigm of GCN, to transfer information between neighbouring nodes, accords with the basic nature of the graph structure.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Tree as Text}

As mentioned in Section 2.2, the syntax tree is typically represented as a text containing words, phrase labels, and brackets to denote grouping relationships between constituents (Figure \ref{fig:four}(A)). The methodology presented in this section involves encoding this syntax tree in its simplest form by fine-tuning the language model of the conventional CLIP model with these linearised syntax trees as inputs, without further modifications to the model architecture. This approach is anticipated to have both advantages and drawbacks. The advantage lies in its potential to leverage the pre-trained model's zero-shot ability more effectively compared with other methods. On the flip side, there are potential risks associated with unintentional tokenisation in the linearised syntax tree and the possibility of exceeding input length due to limitations in the CLIP transformer.   


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Phrase Level Tree Encoder}

The methodology presented in this section draws inspiration from Transformer-Grammar \cite{Sartran2022TransformerGA}. The approach involves simplifying the syntax tree into phrase levels and generating an attention mask to organise the words and phrase labels at the same level. Before delving into the overall architecture of the tree encoder, we outline the tree simplification and mask generation process, as depicted in Figure \ref{fig:six}. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 6
\begin{figure}[t]
\begin{center}
\includegraphics{32-2ia7f06.eps}
\end{center}
\hangcaption{Illustration of how the simplified syntax looks like and attention mask is generated. (A) : On the left side is the simplified tree structure used in Transformer-Grammar research, and the right side shows our version of simplification for use in this research. The token marked with a thick underline signifies the duplication of the token immediately preceding it to retain the attention that has already been established. (B) : Tokens and the type of attention operation to form the mask. The tokens are derived from our version of simplification, as depicted on the right side of (A). (C) : Attention mask generated from ``STACK'' and ``COMPOSE'' operations. ``STACK'' is represented in black and ``COMPOSE'' in grey.}
\label{fig:six}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Like in Section 3.1, the syntax tree is essentially a textual representation containing words, phrase labels, and brackets. The accompanying attention mask guides each position in the text on where to focus. The process of generating this mask involves managing a stack of token indices derived from the input syntax tree. Each token in the syntax tree is assigned either ``STACK'', or ``COMPOSE'' as its attention operation. Figure \ref{fig:six}(B) illustrates this procedure, accompanied by the generated mask (Figure \ref{fig:six}(C)). The token index stack accumulates indices until it encounters a token with a closing bracket, where the attention operation is ``COMPOSE''. At this point, the previous token indices at the same level as the current closing bracket token are popped out, and the index of the closing bracket becomes a representative token for a specific phrase level. Since this composed index needs to stay in the stack for further operations, the closing bracket token is duplicated in the syntax tree, each with the operation ``COMPOSE'' and ``STACK''.   
To execute this procedure, the input syntax tree must explicitly express phrase levels, including duplicated closing brackets. The left side of Figure \ref{fig:six}(A) shows the tree structure used in Transformer-Grammar research, which is unsuitable for our methodology. Unlike Transformer-Grammar, we leverage a pre-trained CLIP text encoder to obtain the initial hidden states of the input syntax tree. The text encoder model's vocabulary lacks tokens with phrase labels attached to closing brackets (e.g. ``(VP'' ). Therefore, we eliminate all the opening brackets attached to phrase labels and phrase labels attached to the closing brackets. This allows us to retain grammar information while avoiding noisy tokens. The resulting tree structure is depicted on the right side of Figure \ref{fig:six}(A).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 7
\begin{figure}[t]
\begin{center}
\includegraphics{32-2ia7f07.eps}
\end{center}
\hangcaption{The overall architecture of the phrase-level tree encoder. From a single plain text input the modified syntax tree and the attention mask are acquired, which goes together into the grammar encoder to output a single characteristic vector for a given input sentence. The output characteristic vector serves as the output of the CLIP model's language model.}
\label{fig:seven}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Figure \ref{fig:seven} provides a visual representation of the phrase-level tree encoder, serving as the language model at this new CLIP model. The process begins with parsing input text into a standard syntax tree, which is then transformed into the mentioned phrase-level tree. This modified tree is used to create an attention mask, and the initial hidden states are obtained by integrating the tree into the conventional CLIP model's pre-trained text encoder. The ultimate output is generated by processing the initial hidden states and the attention mask through the grammar encoder. The grammar encoder comprises multiple layers, each containing a multi-head self-attention layer and a feed-forward network. Given the initial hidden states, the output for the kth layer within the encoder layer is expressed as:
\begin{equation}
    h^k = \text{Layer}^k(h^{k-1}, A)  
\end{equation}
Here, 
\begin{itemize}
  \item $\mathrm{h^{k} \in \mathcal{R}^{L*d}}$ represents the embedding of hidden states, where d is the dimension, and L is the number of words in a sentence. 
  \item $A \in {\mathcal{R}^{L*L}}$ is the attention mask. 
\end{itemize}
The multi-head self-attention within the layer follows the original Transformer research \cite{Vaswani2017AttentionIA}, with H as the number of heads:
\begin{align}
    \text{MultiHead}(q, k, v) &= \text{Concat}(\mathit{head}_1, ..., \mathit{head}_H)W^O \\ 
    \mathit{head}_i &= \text{Attention}(q{W_i}^{Q}, k{W_i}^{K}, v{W_i}^{V}) \\ 
    q, k, v &= h 
\end{align} 
Here, $\mathrm{W_i}^{Q}$, $\mathrm{W_i}^{K}$, $\mathrm{W_i}^{V}$, and $\mathrm{W}^{O}$ are learnable parameter matrices. The attention mask $A$ is applied element-wise on the output attention scores for each attention head, ensuring that masked portions are disregarded during processing. Each attention head output is normalised using a softmax activation function, concatenated and linear-transformed to yield the final multi-head self-attention output. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 8
\begin{figure}[b]
\begin{center}
\includegraphics{32-2ia7f08.eps}
\end{center}
\hangcaption{The general framework of the independent tree encoder with path attention. The tree embedding vector is concatenated with the text embedding vector, and the resulting concatenation undergoes a linear transformation. This process transforms it into a characteristic vector, serving as the input for the CLIP model's language model.}
\label{fig:eight}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Syntax Tree Separately Encoded with Path Attention}

In this section, we embrace an approach influenced by GuiG \cite{Li2020TransformerBasedNT}, and the comprehensive structure of the encoder is depicted in Figure \ref{fig:eight}. 
The core strategy of the original research lies in encoding text and syntax trees independently and subsequently merging them using multi-encoder attention within the decoder. The syntax encoder's unique characteristics involves extracting multiple paths from a single tree and obtaining attention outputs for each path. A path is defined as a sequence of phrase labels, starting from the root of the tree and extending to each leaf node. For a single path, the attention output is computed as follows:
\begin{align}
    C &= \text{Concat}(A^1, ... , A^h){W_P}^T  \\
    A^j &= \text{Attention}(PW^T_{Q_j}, PW^T_{K_j}, PW^T_{V_j})
\end{align}

Here, $P$ represents the output embedding from the previous layer, each treated as query, key, and value through learnable parameter matrices. $h$ denotes the number of attention heads, with $j$ as the index. Each attention $A$ from $h$ attention heads go through concatenation and linear transformation by a learnable parameter matrix $W_P$. The overall attention output is determined by averaging the outputs $C$ from each path, effectively replacing the conventional transformer encoder's attention mechanism. In the original GuiG research, a separate decoder was designed to handle text and syntax embeddings independently. However, our research necessitates a unified embedding from the outset. To address this requirement, we merge the two embedding vectors in our CLIP model through concatenation and linear transformation, ensuring a coherent and integrated approach from the beginning.  


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Graph Convolutional Networks for Semantic Graphs}

Our last proposed methodology aims to harness the semantic relationships inherent in a sentence, represented as a graph using the GCN algorithm. Before delving into the specifics of the GCN algorithm we employed for graph encoding, we explain the structure of semantic graphs, which is illustrated in Figure \ref{fig:nine}.
In semantic graphs, the relationships between a predicate and its arguments are articulated through the order in which an argument enters a predicate. This order serves to denote the semantic role played by each argument within the predicate, and rearranging the order can alter its role in shaping the meaning of the sentence. For instance, in Figure \ref{fig:nine}(A), the verb `approach' has two arguments: Danny and chair. In this context, Danny is assigned the number `0' and chair is given the next order, indicating that Danny is the subject, and chair is the object being approached. Switching the order would imply Danny approached by the chair (though contextually implausible). In a previous study that applied GCN to graph-structured linguistic formalism,  \cite{Marcheggiani2017EncodingSW}, relationships were treated as attributes of a graph's edges. For each relationship, a specific weight matrix was defined to encode information transmitted through nodes via edges. However, such an approach could lead to over-parameterisation. To address this concern, we adapt the graph structure by introducing the concept of an edge attribute as another node, which we will refer to as a ``relation node''. Under this modification, we explore two distinct graph structures, illustrated in Figure \ref{fig:nine}. The first structure is more general, where a relation node expressing the edge attribute exists for every single relationship in a graph (Figure \ref{fig:nine}(A)). In the second structure, only one node  exists for each relation node type, managing all relations of the same type around it. We explore this variation to assess whether a single node overseeing all information passed between nodes under the same relation type contributes to a  better understanding of the relationship. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 9
\begin{figure}[t]
\begin{center}
\includegraphics{32-2ia7f09.eps}
\end{center}
\hangcaption{Semantic graphs generated from the sentence ``Danny approached the chair with a yellow bag.'' Abstracts concepts such as adjectives and verbs are expressed as 
predicates where objective concepts like nouns, fill in the slots as arguments. (A) demonstrates the general semantic graph where nodes denoting the order of the arguments exist for every single relationship occurring inside the sentence. On the other hand, (B) shows a graph where a node regarding a certain relationship manages every other relationships with the same category.} 
\label{fig:nine}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The foundation of our graph neural networks draws from Morris et al.'s work, \cite{Morris2018WeisfeilerAL}, where a single node vector undergoes an update according to the formula (from kth layer to (k+1)th):
\begin{equation}
    x^{k+1}_i = {W_1}{x^{k}_i} + {W_2}\bigl(\sum_{j \in \mathit{Neighbours(i)}}e_{j,i}\cdot x_j \bigr)  
\end{equation}
This formulation involves two essential weight matrices: $W_1$ for the self-recurrent computation and $W_2$ for managing the weights of neighbouring edges from the node ($e_{j,i}$). While prior research \cite{Marcheggiani2017EncodingSW} emphasised the multiplication of various weight matrices for different edge attribute types, our adopted methodology assigns the roles of these matrices to edge weights within each edge. This not only makes the model light-weight, but also provides edge weights that can be later analysed to discern the contributions of different parts of the graph to the model's inference.  

In our study, we integrate the GCN methodology into the CLIP pipeline. In contrast to previous research linked with word representation vectors encoded by BiLSTM, we replace it with the pre-trained CLIP model's text encoder. Additionally, we incorporate the text embedding vector obtained from the original text before its conversion into graphs. The text embedding is concatenated with the graph embedding obtained from GCN. This combined representation passes through a linear function to produce the overall characteristic vector, serving as the output of the CLIP model's language model. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}

We conducted a series of experiments to assess how incorporating linguistic formalism into the CLIP model compares to the conventional CLIP model in the context of text-to-image retrieval. Additionally, we compared different proposals, considering the type of linguistic formalism used and its encoding methodology. Our focus centred on two research questions: 
\begin{itemize}
    \item Can the integration of linguistic formalisms enhance the V\&L model's RWU ability, typically in the image retrieval task performance, in case of structural ambiguity? (Section 4.1) 
    \item Does leveraging linguistic formalisms result in sound discrimination quality even in scenarios unrelated to ambiguity? (Section 4.2)
\end{itemize}
Before delving into the experimental settings, we outline and name the models we plan to compare in our experiments. 
\begin{itemize}
    \item $\mathrm{CLIP_{plain}}$ : Conventional CLIP model with plain text inputs, serving as our baseline model. 
    \item $\mathrm{CLIP_{\text{tree as text}}}$ : Conventional CLIP model architecture fine-tuned with textual syntax trees as inputs (described in Section 3.1).
    \item $\mathrm{CLIP_{\text{phrase level tree}}}$ : Phrase-level syntax tree with a customised attention mask (described in Section 3.2). 
    \item $\mathrm{CLIP_{\text{path attention}}}$ : Syntax
    phrase labels encoded separately from the text by computing path attention (described in Section 3.3). 
    \item $\mathrm{CLIP_{\text{GCN unit 1}}}$ : GCN applied to a semantic graph where a relation node exists for every single relationship (described in Section 3.4).
    \item $\mathrm{CLIP_{\text{GCN unit 2}}}$ : Similar to the above model, but the graph's relation nodes are unified so that a single relation node with a specific relation type manages all relationships with the same relation type.
\end{itemize}
These models encompass a diverse range of linguistic formalisms and encoding strategies, enabling a comprehensive exploration of their impact on the V\&L model's RWU ability.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Structural Disambiguation Experimental Setting}

The primary objective of this experimental task is to effectively discern pairs of images and texts characterised by structural ambiguity. This task holds paramount significance among our experiments as it directly addresses the core research question that our study aims to unravel. The focus is on leveraging linguistic formalisms to accurately discriminate between pairs of data, where structural ambiguity poses a challenge. 
\par 
As a dataset, we use Language and Vision Ambiguities (LAVA) corpus \cite{Berzak2015DoYS}. The core components of this corpus include: 
\begin{itemize}
    \item Structurally ambiguous texts featuring two possible interpretations 
    \item Two images corresponding to the possible interpretations for each ambiguous text
    \item Two syntax trees presented as bracketed texts corresponding to the possible interpretations for each ambiguous text 
    \item Two logical forms by semantic parsing corresponding to the possible interpretations for each ambiguous text  
\end{itemize}

This corpus stands out as the sole V\&L dataset specifically addressing structural ambiguity, making it the cornerstone of our study. One notable advantage of this dataset is its provision of accurate language structural information both syntactically and semantically. Consequently, concerns regarding erroneous parsed results are alleviated.
We split the data as 310 pairs for training, 44 pairs for validation, and 89 pairs for testing. One thing to note is that sentences inside the original LAVA corpus don't have strictly all the colour adjectives attached to every object appearing in images. For instance, instead of ``Danny approached the green chair with a yellow bag'', it is recorded as ``Danny approached the chair with a yellow bag''. This lack of full information has potentials to hinder our experiments, so we attached all the objects with its colour adjectives. 

All the models listed in Section 4 leverage the pre-trained CLIP vision and text encoder and undergo fine-tuning with the LAVA corpus. Also in this experiment, we add conventional text-based CLIP models without fine-tuning to compare their performance. This is to demonstrate that the conventional V\&L model cannot discriminate structurally ambiguous V\&L pairs, as mentioned in Section 1. We compare all the pre-trained models released by OpenAI, which will be elaborated in Section 4.3. We employ two key evaluation metrics as depicted in Figure 10.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Discrimination Accuracy}
(Figure \ref{fig:ten}(A)) assesses the model's ability to successfully discern structurally ambiguous V\&L pairs. At every time step, the model is presented with two V\&L pairs, sharing the same plain text but featuring different meaning interpretations due to structural ambiguity. The model is tasked with matching the text (linguistic formalisms for proposals) to the correct image. For each time step, the count of the correctly matched pairs (either 0, 1, or 2) is determined. The accuracy is then calculated as the ratio of the number of correctly matched pairs over the total count. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 10
\begin{figure}[t]
\begin{center}
\includegraphics{32-2ia7f10.eps}
\end{center}
\hangcaption{Illustration of evaluation metrics. (A) demonstrates Discrimination Accuracy. In this scenario, two V\&L pairs from the same structurally ambiguous sentence enter the model. The model matches the pair according to its own inference, and the accuracy is computed as the success ratio. (A) showcases a failure example, and (B) demonstrates the recall@K from text to image. In this case, the model selects the top K search results. The recall@K is then computed as the ratio of having the correct answer among the K searches. (B) showcases the successful example.} 
\label{fig:ten}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Recall@K}
(Figure \ref{fig:ten}(B)) evaluates the models' performance on the LAVA corpus in the context of retrieval. While the Discrimination Accuracy session involves providing the model per time step with ambiguous V\&L pairs from the same plain text, the Recall@K task evaluates more general discrimination ability, requiring the model to identify the correct pair corresponding to the input among all the data in the test set. We set the value of K as 1, 5, and 10. For each K value, we count the number of inputs for which the correct match was found within the top K search results. This session is conducted bidirectionally---from text-to-image, and from image-to-text. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{General Image-text Alignment Ability Evaluation Experimental Settings}

While experimenting with the LAVA corpus is suitable for addressing our specific research question, it has a significant limitation due to its small size. This raises concerns about the general image discrimination ability of leveraging linguistic formalisms into the model across diverse contexts. To explore this concern and ensure broader applicability, we prepared an additional dataset with a more general context. We would like to confirm that our model tuning methodology does not have bad influence on well-known datasets in general image domain irrelevant of semantic ambiguity. COCO \cite{Lin2014MicrosoftCC} is a dataset equipped with a substantial number of V\&L pairs depicting various situations and contexts. In COCO, each visual scene is accompanied by five captions, each describing the same scenery in slightly different ways. For our purpose, we selected the first caption from each set of five to create a single V\&L pair. Unlike the LAVA corpus, COCO lacks inherent linguistic formalisms. To address this, we employed an independent parser, details of which will be discussed in Section 4.3. The data split was performed based on Karpathy split \cite{Karpathy2014DeepVA}. 
\par 
Since COCO is not mainly about structural ambiguity, the Discrimination Accuracy metric, as employed in Section 4.1 lacks meaningfulness in this context. Consequently, for the evaluation of models fine-tuned with COCO, we exclusively utilised the Recall@K metric, calculated in the same manner as in Section 4.1. Under this evaluation framework, we conducted two distinct experiments: 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{COCO session} 
With all models leveraging pre-trained CLIP vision and text encoder, we fine-tuned these models with COCO and subsequently tested their performance on the same COCO dataset based on the Karpathy split \cite{Karpathy2014DeepVA}. This experiment aims to explore the applicability of our proposals in a more general context.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{COCO-to-LAVA session}
With all models fine-tuned with COCO according to the above experiment, we underwent a second round of fine-tuning with the LAVA corpus. The Recall@K metric is measured in the same fashion as above. The objective of this experiment is to gauge the extent to which our models might become overfitted to the specific characteristics of the LAVA corpus. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Implementation Details}

All our implementations were carried out using PyTorch \cite{Paszke2019PyTorchAI}. 
\pagebreak
We utilised the pre-trained CLIP model available from Huggingface,\footnote{\url{https://huggingface.co/openai/clip-vit-base-patch32}} with 512 dimensional hidden states for every model. Also, for the conventional CLIP models without fine-tuning we mentioned in Section 4.1 to demonstrate the CLIP model's limitation in discriminating structurally ambiguous pairs, we added 3 models,\footnote{\url{https://huggingface.co/openai/clip-vit-base-patch16}}$^{,}$\footnote{\url{https://huggingface.co/openai/clip-vit-large-patch14}}$^{,}$\footnote{\url{https://huggingface.co/openai/clip-vit-large-patch14-336}} which count up to 4 in total.
These 4 models are distinguished based on the model's architecture size (base and large), and the patch size by pixels into which image inputs are divided (32, 16, and 14). 336 at the last model indicates the expected size of the image input, so that the inputs are resized before entering the vision model. 
For $\mathrm{CLIP_{\text{phrase level tree}}}$ in Section 4.2 and $\mathrm{CLIP_{\text{path attention}}}$ in Section 4.3, transformers were configured with 8 heads, 6 layers, and a 0.1 dropout rate. The Graph Convolutional Networks for $\mathrm{CLIP_{\text{GCN unit 1}}}$ and $\mathrm{CLIP_{\text{GCN unit 2}}}$ in Section 4.4 each consisted of five GraphConv layers \cite{Morris2018WeisfeilerAL} from PyTorch Geometric \cite{Fey/Lenssen/2019} with a 0.5 dropout rate. All our optimisers adopted the Adam algorithm \cite{Kingma2014AdamAM}, with a learning rate of $\mathrm{1 \times 10^{-5}}$ and a warmup step of zero.  

For the LAVA corpus, we utilised the pre-equipped linguistic formalisms, running the training session for 1000 epochs. In order to equip the COCO dataset with the necessary linguistic formalisms for our proposed methodologies, we employed HanLP \cite{he-choi-2021-stem}, a comprehensive framework that encompasses various parsing techniques including syntax and semantic parsing relevant to our study. The training session with the COCO was run for 10 epochs. The tested models were selected based on the appearance of the lowest validation loss per epoch. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Structural Disambiguation Experiment}

Table \ref{table:two} presents the comprehensive outcomes of our disambiguation experiment, as outlined in Section 4.1. Our major observation can be summarised as follows:
\begin{itemize}
    \item Conventional plain text-based CLIP models were significantly outperformed by all the other models in every respect.
    \item Regarding discrimination accuracy, the proposed models generally outperformed the baseline, with $\mathrm{CLIP_{\text{GCN unit 1}}}$ particularly exhibiting the best performance. 
    \item In terms of Recall@K results, $\mathrm{CLIP_{\text{tree as text}}}$ which leverages most of the conventional CLIP model's pre-trained structure achieved the best results while other proposals experienced a decline. 
\end{itemize}
We delve into these observations in the following sections. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\input{07table02.tex}%
\hangcaption{Disambiguation experimental results on LAVA. The upper five models are baselines, leveraging plain texts as input. Among the five baselines, $\mathrm{CLIP_{\text{plain}}}$ on the bottom is the model fine-tuned with the LAVA corpus, while the rests haven't gone through fine-tuning.}
\label{table:two}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Discrimination accuracy}

Looking at discrimination accuracy, every proposed model surpasses the baselines. It is worth noting that $\mathrm{CLIP_{\text{GCN unit 1}}}$ demonstrates the highest result at 77.53, followed closely by $\mathrm{CLIP_{\text{path attention}}}$ and $\mathrm{CLIP_{\text{GCN unit 2}}}$ at 75.84 and 75.28, respectively. These models share a common trait: linguistic formalism is encoded separately from the original text, and the final characteristic language vector is obtained by combining the text vector with the linguistic formalism vector. The results indicate that integrating linguistic formalisms enhance the V\&L model's RWU ability to discriminate V\&L pairs with structural ambiguity.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Recall@K }

Examining the Recall@K results, $\mathrm{CLIP_{\text{tree as text}}}$ outperforms other models by a significant margin. Other proposals successful in discrimination accuracy experienced a great decline in performance. While surpassing the baseline in Recall@1, the superiority reverses as the value of K increases. Meanwhile, $\mathrm{CLIP_{\text{tree as text}}}$ consistently shows high performance through out every value of K and outperforms the baseline in almost every respect. Additionally, even in discrimination accuracy in Section 5.1.1, $\mathrm{CLIP_{\text{tree as text}}}$ achieves a commendable performance of 74.723. The distinctive aspect of $\mathrm{CLIP_{\text{tree as text}}}$ compared to the other proposals is its full utilisation of the conventional CLIP model's pre-trained structure. Together with the results from Section 5.1.1, it is speculated that combining linguistic formalism with the conventional CLIP model's pre-trained structure is the most effective for disambiguation. On the other hand, it is questionable why $\mathrm{CLIP_{\text{GCN unit 1}}}$, which demonstrated the best performance in discrimination accuracy, showed a drastic decline in other aspects, requiring a thorough analysis. Meanwhile, all the baselines that haven't gone through fine-tuning shows significantly low performances compared to every other model, even $\mathrm{CLIP_{\text{plain}}}$, the fine-tuned model. This shows the conventional plain textual inputs have limitations in discriminating structurally ambiguous pairs, hence showing plausibility in leveraging linguistic formalism. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[b]
\input{07table03.tex}%
\hangcaption{Generality evaluation experimental results on COCO. The Downstream Dataset signifies the type of experiment underwent as described in Section 4.2: COCO denotes the COCO session, and COCO-to-LAVA the COCO-to-LAVA session. In the COCO-to-LAVA sessions, the performance decline from the COCO-session is shown inside a bracket. The best results in COCO-to-LAVA session (the smallest decline) are represented with underlines. $\mathrm{CLIP_{\text{plain}}}$ is the baseline.}
\label{table:three}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{General Image-text Alignment Ability Evaluation Experiment}

Table \ref{table:three} outlines the outcomes of experiments detailed in Section 4.2. Major aspects are as follows:
\begin{itemize}
    \item In the COCO session, only the $\mathrm{CLIP_{\text{tree as text}}}$ exhibited compatible performance with the baseline, while the other proposed models experienced a significant decline. 
    \item In the COCO-to-LAVA session, every model experienced a decrease in its performance, indicating potential overfitting to the LAVA corpus. Among the models, the baseline and $\mathrm{CLIP_{\text{tree as text}}}$ showed the least decline.  
    \item With an observation from the COCO-to-LAVA session that the LAVA corpus has biases, we conducted an additional experiment to train the models to have the same size of samples from the two datasets in every batch. As a result, $\mathrm{CLIP_{\text{tree as text}}}$'s performance increased and outperformed the baseline in several respects.
\end{itemize}
We elaborate on these observation in the following sections.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{COCO session}

With the exception of $\mathrm{CLIP_{\text{tree as text}}}$, our proposed models exhibited performance levels below both the baseline and $\mathrm{CLIP_{\text{tree as text}}}$. Significantly, the baseline, which previously lagged behind $\mathrm{CLIP_{\text{tree as text}}}$ in the disambiguation experiment (Table \ref{table:two}), now demonstrates comparable results. It even outperforms $\mathrm{CLIP_{\text{tree as text}}}$ in the text-to-image direction and maintains similar performance in the opposite direction. This raises concerns about potential overfitting to the LAVA corpus in the disambiguation experiment due to our proposed modifications. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{COCO-to-LAVA session}

The COCO-to-LAVA session represents another round of generality evaluation experiments, where all models underwent a second fine-tuning using the LAVA corpus after initial fine-tuning with COCO. None of our proposals outperformed the baseline in this session. The decline in performance is observed across all models compared to the results in the COCO session, with varying degrees of decline. Models that preserved most of the pre-trained CLIP model's architecture experienced a relatively small decline, while others suffered almost a 50\% drop in their original performance. Particularly noteworthy is the performance of $\mathrm{CLIP_{\text{path attention}}}$, which initially showed strong results in discrimination accuracy in Table \ref{table:two}, but exhibited the worst performance in the COCO session in Table \ref{table:three}, and the most significant decline in the COCO-to-LAVA session. This suggests a potential extreme overfitting issue with the LAVA corpus, for which we would need an analysis of the corpus to address this issue. The results lead us to speculate that additional modifications to the conventional CLIP model may compromise the general image discrimination ability acquired during pre-training. However, $\mathrm{CLIP_{\text{tree as text}}}$ showed the best, and almost the best performance in the COCO session, and showed the minimum decline in performance next to the baseline in the COCO-to-LAVA session. We speculate this achievement attributing to the preservation of the original CLIP model's architecture. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{COCO-and-LAVA session}

Our ideal is to balance the CLIP model's RWU ability while retaining its original retrieval ability across the general domain images.\footnote{For this, it is preferable that the performance evaluation be conducted both on COCO and LAVA, but our paper focuses on COCO to stress the improvement in general image-text alignment ability.} We conducted another experiment to verify whether these two abilities could be successfully maintained in fine-tuning the CLIP model using both the COCO and LAVA datasets. Our method is to construct a single batch of 64 samples, half of which is from the COCO and the other half from the LAVA corpus. Since there is a gap in sample size between the two datasets, we upsample the LAVA corpus to match the COCO's size. We train two models 
under this COCO-and-LAVA session, $\mathrm{CLIP_{\text{tree as text}}}$ and $\mathrm{CLIP_{\text{GCN unit 1}}}$, which have shown better results than the other proposals in the previous experiments. We train each model for 15 epochs and select the best model among 15 checkpoints per epoch. We compare the performance of these models with $\mathrm{CLIP_{\text{plain}}}$, $\mathrm{CLIP_{\text{tree as text}}}$, and $\mathrm{CLIP_{\text{GCN unit 1 }}}$ that were trained only on COCO in Section 5.2.1. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[b]
\input{07table04.tex}%
\hangcaption{Generality evaluation experimental results on COCO. The Downstream Dataset signifies the same as in Table \ref{table:three}. COCO denotes the performance results of COCO session in Table \ref{table:three}, and COCO-and-LAVA means the COCO-and-LAVA sessions in Section 5.2.3, where a single batch contains half the COCO and the other half the LAVA corpus for a single batch. $\mathrm{CLIP_{\text{plain}}}$ is the baseline.}
\label{table:four}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Table \ref{table:four} shows the experimental results, with the bottom two models being the proposals trained on COCO-and-LAVA session. In text-to-image direction, $\mathrm{CLIP_{\text{tree as text}}}$ on COCO-and-LAVA outperforms every other model. In image-to-text direction, it is noteworthy that despite failing to outperform the baseline, it still shows the similar performance, while the same model trained on the COCO-to-LAVA session in Section 5.2.2 showed a drastic decrease in performance. One possible reason is thought to be the order how datasets are fed into the model's training. In the COCO-to-LAVA session in Section 5.2.2, the LAVA corpus is given to the system after all the samples in the COCO are trained. In this way, there's a possibility that the model could have gone through catastrophic forgetting, losing the general image discrimination ability learnt through the COCO. In COCO-and-LAVA session, two datasets are fed to the models simultaneously with the same size, leading to significantly better results than the COCO-to-LAVA counterparts. We speculate from these results that the parallel feeding of both data prevented the model's catastrophic forgetting, and that the models could have both the disambiguation ability and the general image discrimination ability by controlling the training order of the datasets.

During sessions utilising the COCO, models incorporating semantic graphs consistently exhibited inferior performance when compared to sessions employing handcrafted formalisms in the LAVA corpus. This discrepancy is speculated to stem from variations in the effectiveness of the semantic and syntax parsers provided by HanLP \cite{he-choi-2021-stem}. Previous research has extensively evaluated HanLP's performance across multiple natural language processing tasks through multi-task learning. Notably, it has been observed that when Semantic Role Labelling (SRL) and Constituency Parsing (CON) were jointly trained, the performance of SRL decreased, whereas the reverse was true for CON. In light of these findings, it suggests a potential avenue for further exploration into the interplay between semantic and syntax parsers.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Result Summary}

In conclusion, the outcomes indicate that incorporating linguistic formalism into the V\&L model improves its RWU ability to discriminate structurally ambiguous V\&L pairs. The overall performance of $\mathrm{CLIP_{\text{tree as text}}}$ suggests that preserving general image discrimination ability is possible through linguistic formalism. Also, by managing the order of data fed into the model during training, the model can acquire both the disambiguation ability and the general image discrimination ability. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Analysis on $\mathbf{CLIP_{\text{GCN unit 1}}}$}

In this section, we delve into how the model interprets linguistic formalism, using the LAVA corpus. The model to be analysed was trained and tested on the LAVA corpus (as shown in Table \ref{table:two}) to investigate whether the model can distinguish between V\&L pairs with structural ambiguity, focusing on the relevant aspects of linguistic formalism. After considering various models we tried, we have chosen $\mathrm{CLIP_{\text{GCN unit 1}}}$ for the following reasons: 
\begin{itemize}
    \item This model has demonstrated superior performance in the discrimination accuracy metric, which is the primary focus of our research question. 
    \item Despite its high accuracy, this model exhibited low Recall@K for both LAVA and COCO datasets, from which we suspect that the model's way of understanding might be different from the ideal. Observing this model's comprehension might provide us the clues about the reasons to these results. 
    \item The architecture of the GCN model involves combining an additional graph embedding vector with the conventional CLIP model's text embedding through concatenation and linear transformation. Analysing this model provides us intuitive insights into how linguistic formalism influences the model's prediction.    
\end{itemize}
For the analytical approach, we employed the Integrated Gradients method \cite{Sundararajan2017AxiomaticAF}. This method involves interpolating between our input and an empty zero baseline input of the same size, accumulating gradient values for each edge. The formula to obtain attributions for each edge in a graph is expressed as follows: 
\begin{equation}
    \mathit{Attribution}_{e_i} = \int_{\alpha=0}^{1} \frac{dF(x_\alpha)}{dw_{e_i}} \, d\alpha  
\end{equation}
We categorise the types of structural ambiguities involved in the LAVA corpus into two :  
\begin{itemize}
    \item Phrase Attachment Ambiguity : Ambiguity arising from multiple possible attachments of phrases such as prepositional phrases (PP), and verb phrases (VP) (e.g. In the sentence ``He approached the chair with a bag.'', PP is ``with a bag'', and the meaning of the sentence differs whether this PP is attached to the chair or the verb ``approach''.).
    \item Coordination Scope Ambiguity : Ambiguity resulting from the scope of a coordinate such as ``and'', ``or'', etc. (e.g. In the sentence ``He picked up the bag or the chair and the telescope.'', the objects picked up by the man vary depending on the scope made by words ``and'' and ``or''.) 
\end{itemize}
We analysed these two types with examples of success and failure. From our analysis we discovered that while the model showed the possibility of correctly understanding the formalism with successful inference examples, it showed failures as well, which are likely to be caused by noisy data.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Phrase Attachment Ambiguity}

The analysis of $\mathrm{CLIP_{\text{GCN unit 1}}}$ performance on phrase attachment ambiguity from the LAVA corpus revealed two main insights, depending on the number of persons featured in the scenario:

\begin{itemize}
    \item In scenarios featuring one person with multiple objects, the model correctly discriminated the ambiguous V\&L pairs with high inference probability, focusing on the actual difference between the two graphs.   
    \item However, in scenarios with more than two people, each with specific names like Danny or Andrei, the model failed to discriminate even after focusing on the graphical difference. In this case, the model exhibited overfitting tendencies, limiting its ability to generalise beyond exclusively discriminating these specific pairs.  
\end{itemize}

We discuss these insights in more detail in the following sections. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 11
\begin{figure}[b]
\begin{center}
\includegraphics{32-2ia7f11.eps}
\end{center}
\hangcaption{Discriminating examples of $\mathrm{CLIP_{\text{GCN unit 1}}}$, where the scenario features a single person. Both graphs are generated from the same sentence, presented in the bottom. The numbers on the thick arrows indicate the model's inference probability, and nodes enclosed in circles highlight the actual differences between the top and bottom graphs. Two different arrows represent the model's matching and the actual right pair. In this case, both inferences are successful, with the top graph of the high probability and the bottom graph of the low probability.}
\label{fig:eleven}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Linguistic formalism actually contributes to disambiguation (Scenarios with one person)}

In Figure \ref{fig:eleven}, the distinction between the two meanings derived from the same sentence, ``Danny approached the chair with a yellow bag,'' lies in the position of the bag. Represented as a graph, the ``bag'' serves as the second argument inside the predicate ``has'', connected to the relation node ``1''. The difference between the two graphs is the first argument entering the predicate ``has'', connected through the relation node ``0'' (enclosed in circles). 

In the top graph, it is evident that the model accurately matches the correct V\&L pairs with the high probability of 0.988. The overall thickness of edges across the entire graph indicates a seamless flow of information, illustrating the model's thorough understanding. Particularly noteworthy is the relative thickness of edges around nodes enclosed in circles, suggesting that the model effectively comprehends the semantic graph during inference.
The bottom graph demonstrates successful matching with low probability. Compared with the top graph, most edges lack the model's attention, and especially the thickness of edges connecting the nodes within circles is weaker than that of the top graph. While the model correctly matched the bottom graph to the right image, its inference probability is 0.546, showing that the model faced challenges in making this determination. 

The model predominantly exhibited correct discrimination with the right focus and high inference probability, as depicted in the top graph, in most of the prepositional phrase (PP) attachment cases involving one person. From the differences in inference probabilities and the model's focus between the two graphs, we speculate that leveraging linguistic formalism is a plausible strategy for disambiguation. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{The model's overfitting in scenarios with two people}

Figure \ref{fig:twelve} illustrates the instance where the model's matching is unsuccessful. In both graphs, the model focuses on the different parts, as seen in the flow across the nodes within the circles. However, while the matching is successful in the top graph, the bottom graph is matched to the wrong image. This observed tendency is recurrent in scenarios involving two people. One plausible explanation lies in the specific names given to each person. Instead of being described with general explanations like appearances, each person is simply given a specific name, treated as a proper noun. Additionally, the naming strategy in the dataset lacks consistency; for example, the man named ``Danny'' in Figure 11 is named ``Andrei'' in Figure 12. In such scenarios, the model might face difficulties in accurately matching the right pairs even though the focus on graphs is accurate.  

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 12
\begin{figure}[p]
\begin{center}
\includegraphics{32-2ia7f12.eps}
\end{center}
\hangcaption{Discriminating examples of $\mathrm{CLIP_{\text{GCN unit 1}}}$, where the scenario features two people. The Figure structure is as same as that of Figure \ref{fig:eleven}. The top graph is matched to the right image while the bottom graph is matched to the wrong one.}
\label{fig:twelve}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 13
\begin{figure}[p]
\begin{center}
\includegraphics{32-2ia7f13.eps}
\end{center}
\hangcaption{Overfitted discriminating examples of $\mathrm{CLIP_{\text{GCN unit 1}}}$, where the scenario features two people. The Figure structure is as same as that of Figure \ref{fig:eleven}. Both graphs are matched to the right images.}
\label{fig:thirteen}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Figure \ref{fig:thirteen} serves as an illustrative example supporting this explanation. While discrimination in this case is successful, the model's focus on the graphs differs from the previously successful instance in Figure \ref{fig:twelve}. Here, the model concentrates on the right part of the bottom graph, and the information flow is robust. Conversely, the edges connecting the nodes within the circles in the top graph lack focus, and the overall information flow is weaker compared to the bottom graph. This pattern is recurrent in most successful instances of scenarios with two people. These observations lead us to speculate that the model may be overfitted, adopting an extreme strategy exclusively tailored to discriminate this specific type of pairs. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Coordination Scope Ambiguity}

The coordination scope ambiguity arises from the presence of multiple possible scopes within coordinated elements. For instance, in the sentence ``Andrei moved the bag or the chair and the telescope.'', three potential interpretations exist: Andrei moving the bag, Andrei moving the bag and the telescope, and Andrei moving the chair and the telescope. Even so, in the LAVA corpus, each coordination scope ambiguity example involves two possible interpretations. While it is crucial for the model to focus on different links between the two graphs which share the nodes in phrase attachment ambiguity, graphs generated from coordination scope ambiguity examples don't share nodes. Consequently, in this scenario, the model's ability to maintain robust information flow across the entire graph becomes paramount. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 14
\begin{figure}[b]
\begin{center}
\includegraphics{32-2ia7f14.eps}
\end{center}
\hangcaption{Discrimination examples of coordination scope ambiguity. The image structure follows that of Figure 11. The top graph is matched to the wrong image and the bottom to the right image.}
\label{fig:fourteen}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

However, none of the coordination scope ambiguity examples across the entire test set showed complete robust information flow. Figure \ref{fig:fourteen} illustrates a pattern in the model's focus on graphs which was consistent in the entire test set. While there is a strong emphasis on the link between the predicate verb and its subject, the transmission of information between the verb and its object is notably lacking. Regardless of success and failure, the model's focus pattern remains consistent, making it challenging to pinpoint the exact factor contributing to the model's performance. 

It is evident that the model struggles to comprehend the graph in this scenario due to information disconnection. Combining this graph embedding vector with the CLIP model's text embedding might downgrade the pre-trained model's language comprehension ability, potentially explaining the GCN model's inferior performance in generality evaluation experiments in Section 5.2. Given that the information flow is disrupted in the middle by the relation node, one potential solution could involve removing the relation node and directly connecting the predicate with its arguments to address this issue. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Limitation}

Despite our findings, our research has the following limitations that require acknowledgement:
\begin{itemize}
    \item First, we didn't put into our consideration the potential impact of noisy parsed linguistic formalism on the model's performance in actual use. This aspect deserves further investigation to better understand its influence in practical applications. 
    \item Additionally, our attention was exclusively directed towards the CLIP model, without extension to other V\&L models behind. Consequently, the general image discrimination ability of our findings to the broader field of V\&L remains uncertain. 
    \item Despite our depiction of the use case from Figure \ref{fig:two}, our experiments don't take into account the evaluation of formalism parser yielding multiple formalisms from a single sentence from K-best algorithm.
\end{itemize}
These limitations provide avenues for future research endeavours.  


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

In this paper, we introduced an approach to improve the V\&L model's RWU ability to address challenges of text inputs with structural ambiguity. Our method involves integrating linguistic formalism into the CLIP model, employing various strategies tailored to the type of formalism and its encoding methodology. Through extensive experiments, we observed that incorporating a linearised syntax tree into the traditional CLIP model architecture yielded the most promising results in image discrimination, particularly in situations characterised by structural ambiguity, as well as in more general scenarios. On another front, our proposal involving the GCN architecture demonstrated superior discrimination accuracy. Our detailed analysis of the GCN-based model revealed an effective RWU ability in image discrimination using linguistic formalism in specific scenarios, shedding light on its strengths and identifying areas for potential improvement in other contexts.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

This work was supported by JSPS KAKENHI Grant Numbers JP21K17806, 23K24910, and 24H00733.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.7}
\bibliography{07refs}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}
\addtolength{\baselineskip}{-1pt}

\bioauthor[:]{Lee Sangmyeong}{%
Lee Sangmyeong received his bachelor's degree in linguistics from Korea University in 2021, and master's degree in engineering from Nara Institute of Science and Technology (NAIST) in 2024. He's currently pursuing his Ph.D. at NAIST, focusing on visual disambiguation. He is a student member of ANLP.
}

\bioauthor[:]{Seitaro Shinagawa}{%
Seitaro Shinagawa graduated from the Faculty of Engineering at Tohoku University in 2013. He completed his Master's degree in Information Science at the same university in 2015, and his doctoral studies at the Nara Institute of Science and Technology (NAIST) in 2020, earning a Doctor of Engineering. After serving as an Assistant Professor at the same university, he joined SB Intuitions Corp. in 2024, where he is engaged in the research and development of multimodal foundation models.
}

\bioauthor[:]{Koichiro Yoshino}{%
Koichiro Yoshino is an Associate Professor of Institute of Science Tokyo,  Team Leader of RIKEN, and an Affiliate Professor of Nara Institute of Science and Technology (NAIST).
He received his bachelor's degree of arts from Keio University in 2009, master's degree in informatics from Kyoto University in 2011, and Ph.D. in informatics from Kyoto University in 2014. He worked at Kyoto University as a postdoc and NAIST as an assistant professor.
He is working on areas of spoken and natural language processing, especially robot dialogue systems.
Dr. Koichiro Yoshino received several honors, including the best paper award of IWSDS2020, IWSDS2024, and the best paper award of the 1st NLP4ConvAI workshop.
He is a board member of ANLP, a member of IEEE-SLTC, the chair of ELRA-SIGSDS, an AE of ARR, a board member of SIGdial and a member at large of AFNLP.
}

\bioauthor[:]{Satoshi Nakamura}{%
Dr Nakamura is Professor Emeritus and Specially Appointed Professor at the Nara Institute of Science and Technology (NAIST), Japan, the Presidential Chair Professor at the School of Data Science, the Chinese University of Hong Kong, Shenzhen (CUHKSZ), and Honorary Professor at the Karlsruhe Institute of Technology (KIT), Germany. He received his B.S. from Kyoto Institute of Technology in 1981 and his Ph.D. from Kyoto University in 1992. He was an associate professor at NAIST from 1994 to 2000, department head and director of the ATR Spoken Language Communication Research Laboratories from 2000 to 2008. He was Director General of the Keihanna Research Laboratories, NICT, Japan, from 2009 to 2010. He was a professor at NAIST from 2011 to 2024. His interests include modelling and systems of speech-to-speech translation, speech recognition, and spoken language processing. He is one of the leaders in speech-to-speech translation research and has served in various worldwide speech-to-speech translation research projects, including C-STAR, IWSLT, A-STAR, and ISCA SIG on Spoken Language Translation. He was awarded the LREC Antonio Zampolli Prize in 2012. He has also received the Kiyasu Award from the Information Processing Society of Japan, the Docomo Mobile Science Award in 2007, and the ASJ Award for Distinguished Achievements in Acoustics. He was elected to the ISCA Board, the International Speech Communication Association, from 2011 to 2018, the Editorial Board of the IEEE Signal Processing Magazine from 2012 to 2014, and the IEEE SPS Speech and Language Technical Committee from 2013 to 2015. He is an IPSJ Fellow, an ATR Fellow, an IEEE Fellow, and an ISCA Fellow.
}
\end{biography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate




\end{document}



