\documentclass[english,other]{jnlp_3.5}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{29}
\Number{3}
\Month{September}
\Year{2022}
%%%%\Issuetitle{Special Issue: XXXXXXXX}

\received{2005}{1}{2}
\revised{2006}{3}{4}
\rerevised{2006}{5}{6}
\accepted{2006}{7}{8}

\setcounter{page}{1002}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{Society Column}
\etitle{Syntactic-Semantic Dependency Correlation in Semantic Role Labeling: A Shift in Semantic Label Distributions}
\eauthor{Junjie Chen\affiref{UTOKYO}} 

\headauthor{Junjie Chen}
\headtitle{Syntactic-Semantic Dependency Correlation in Semantic Role Labeling}

\affilabel{UTOKYO}{}{The University of Tokyo}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Semantic Role Labeling (SRL) is an important component for natural language understanding \cite{palmer-etal-2013-semantic}.
It identifies the semantic role of words, abstracting away superficial variations caused by ways of expression.
\textit{Semantic dependencies} (upper part of Figure \ref{fig:syn-sem-intro}) embody such information, specifying the semantic role label \textit{actor} of the argument \textit{it} to the predicate \textit{eliminate}.
Semantic dependency provides useful information for tasks such as question answering \cite{yih-etal-2014-semantic}, automatic summarization \cite{DBLP:conf/aaai/JinW020,DBLP:conf/acity/KumarR12}, text mining \cite{DBLP:conf/cicling/PoriaAGHH14}, and beyond. 
On the syntax side, a similar notion of \textit{syntactic dependencies} (lower part of Figure \ref{fig:syn-sem-intro}) arise to encode grammatical relations.
Syntactic dependency specifies the grammatical relation \textit{modifier} of the dependent word \textit{hardly} to the head word \textit{eliminate}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[b]
\begin{center}
\includegraphics{29-3ia24f1.pdf}
\end{center}
\caption{Illustration of the correlation between syntactic and semantic dependencies. }
\label{fig:syn-sem-intro}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The two types of dependencies correlate heavily with each other, creating many parallelisms between them.
Figure \ref{fig:syn-sem-intro} shows an example of such parallelism, where \textit{pigeon} is both an experiencer argument and an object dependent to the verb predicate \textit{eliminate}.
In general, words that are object dependents of a verb predicate often be an experiencer argument of the predicate.
The parallelism motivates a large body of literature enhancing machine learning-based semantic parsers with syntactic dependencies.
The literature includes works such as \citeA{he-etal-2018-syntax} and \citeA{roth-lapata-2016-neural}.
While those works lead to performance improvements, most are satisfied with the score uplift and avoid a more fundamental problem: \textit{what underlines the correlation between syntactic and semantic dependencies}. 
In our paper \cite{chen-etal-2022-modeling}, we study the statistical property underlying the dependency correlation.

We interpreted the dependency correlation as a shift in semantic label distributions.
The label distribution \cite{dozat-manning-2018-simpler} models the distribution over semantic role labels for a dependency spanning the predicate and the argument.
A non-relation label indicates no semantic dependency.
We found that the label distribution changes significantly with the hop patterns of the \textit{shortest syntactic dependency path} (SSDP) connecting the predicate and the argument.
The hop pattern reflects the length of syntactic dependencies.
The distribution shift corresponds to our changing expectations for semantic dependencies.
For example, we would have a high expectation for a semantic dependency given a short syntactic dependency but a low expectation given a long syntactic dependency.
We modeled the distribution shift using a mixture model-based semantic parser, which we will explain in the next section.
Compared to previous syntax-aware semantic parsing methods, modeling the distribution shift improves performance in predicting short-distance semantic dependencies while retaining the performance advantage in long-distance dependencies.
Modeling the distribution shift also provides a small but significant performance uplift compared to syntax-aware semantic parsing baselines, as well as an SRL system competitive with state-of-the-art methods.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Modeling Distribution Shift with Mixture Models}
\label{section:distribution-shift}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{29-3ia24f2.pdf}
\end{center}
\hangcaption{Illustration of the label distribution shift and the mixture model-based semantic parser. The left figure visualize the label distributions for hop patterns. The right diagram illustrates the mixture model-based semantic parser. The dashed arrow indicates the cluster assignment.}
\label{fig:label-distribution-shift}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In this section, we present analyses supporting our interpretation and explain the motivation behind the usage of the mixture model.
We interpreted the dependency correlation as a label distribution shift with SSDP hop patterns. 
SSDP, a salient feature for exploiting the dependency correlation, is the shortest path connecting the predicate and the argument in the syntactic dependency structure (Figure \ref{fig:syn-sem-intro}).
Hop patterns $(\alpha, \beta)$ count the number of transitions going from the predicate to the argument.
$\alpha$ counts the dependent-to-head transition that goes in the opposite direction as syntactic dependencies, while $\beta$ counts the head-to-dependent transition going in the same direction as syntactic dependencies.
The count reflects the syntactic distance between the predicate and the argument.
We refer readers to the paper for more technical details about SSDP and hop patterns.

The left figure in Figure \ref{fig:label-distribution-shift} provides a glimpse of the label distribution shift.
Each vertical bar shows a label distribution for the corresponding hop pattern. 
We see that long hop patterns share a similar label distribution dominated by the non-relation label, whereas short hop patterns have disparate label distributions.
The (0, 1) pattern has the most outstanding label distribution among all patterns.
The non-relation dominating distribution for long patterns doubles down on an old belief that semantic dependencies co-occur mainly with short syntactic dependencies.
Additionally, the figure suggests a variation in label distributions for short patterns.

The two statistical properties motivate the introduction of the mixture model (right diagram in Figure \ref{fig:label-distribution-shift}).
The large number of patterns sharing a similar distribution requires clustering those patterns.
The variation in label distributions requires the separate modeling of distributions with unique properties.
The mixture model method meets the two requirements.
It automatically learns a cluster assignment for hop patterns, as indicated by the dashed arrow in Figure \ref{fig:label-distribution-shift}.
Meanwhile, it learns the label distribution for each cluster, modeling the variation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[t]
\begin{center}
\includegraphics{29-3ia24f3.pdf}
\end{center}
\hangcaption{Illustration of the mutual information analysis and learned cluster assignments. The left heatmap shows the mutual information gain value for hop patterns. The right table shows the cluster assignment extracted from a model. Hop patterns assigned to the same cluster are color-painted in the same color.}
\label{fig:ssdpmi-assignment}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We confirm the label distribution shift using a mutual information analysis.
In the analysis, we compute the mutual information for models aware of hop pattern information and models unaware of such information, and define the gap in mutual information value as \textit{mutual information gain}
The left heatmap in Figure \ref{fig:ssdpmi-assignment} shows the mutual information gain for each hop pattern.
We see that long patterns have a near-zero information gain, whereas short patterns have various information gains.
The (0, 1) pattern has the highest information gain.

The mixture model-based semantic parser learns a cluster assignment agreeing with the mutual information analysis.
The right table in Figure \ref{fig:ssdpmi-assignment} illustrates the learned cluster assignment.
We see that the model assigns one cluster to long patterns sharing the non-relation dominating distribution, and assigns different clusters to short patterns.
The model also assigns a unique cluster to the (0, 1) pattern, the pattern with the most outstanding label distribution and with the highest information gain.
The parallelism between the label distribution visualization, the mutual information analysis, and the learned cluster assignment supports our interpretation that semantic label distributions shift with hop patterns.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}

The abundance of parallelism between the two types of dependencies indicates a strong correlation between syntactic and semantic dependencies.
The parallelism suggests that we can extract a large chunk of semantic dependencies using solely syntactic dependencies.
It builds up a high expectation that syntactic dependencies provide plenty of information for extracting semantic dependencies.

Despite the strong correlation, the impact of syntactic information had decreased over time.
Syntactic information changes from being indispensable \cite{punyakanok-etal-2008-importance,gildea-palmer-2002-necessity} to being optional \cite{he-etal-2017-deep} for high-performance semantic parsers.
The phenomenon is especially vocal in the neural era when the extra syntactic information provides a moderate boost or even does harm to model performance \cite{sachan-etal-2021-syntax}.
\textit{The contrast between the expectation and the reality necessitates the study of the dependency correlation.}
More specifically, the statistical property underlines the dependency correlation.
A deeper understanding of the statistical property would help us better utilize the correlation.

Our interpretation is a generalization over a widely-adopted co-occurrence bias.
The bias suggests that semantic dependencies co-occur mainly with short syntactic dependencies and are unlikely to co-occur with long dependencies.
We showed that the long hop patterns have zero mutual information gain and share a label distribution dominated by the non-relation label.
Previous methods adopt this bias, focusing on the short dependencies rather than the long ones.
\citeA{he-etal-2018-syntax,he-etal-2019-syntax} discriminates short and long syntactic dependencies by heuristics, pruning semantic dependencies that co-occur with long syntactic dependencies.
\citeA{cai-lapata-2019-syntax} employs an auxiliary task predicting whether the word is a direct descendent or ascendant of predicates, implicitly discriminating between one-hop and multi-hop dependencies.
\citeA{shi-etal-2020-semantic} encodes the dependency correlation via a joint syntactic-semantic label scheme designed specifically to handle semantic dependencies co-occurring with short syntactic dependencies.

Our mixture model method provides better interpretability than neural methods for utilizing the correlation.
The mixture model clusters hop patterns with similar label distributions and models disparate distributions for the clusters.
The cluster assignment provides insight for studying the statistical property of the correlation.
In contrast, the neural methods encode the dependency correlation implicitly, offering little insight into its statistical properties.
\citeA{roth-lapata-2016-neural} encode SSDPs as a continuous embedding using a Long-Short Term Memory model.
They found the method learned to cluster semantic dependencies co-occurring with specific linguistic phenomena.
However, the captured phenomena are fragmental and limited to a few syntactic constructions.
This is because Neural encoders, such as the LSTM model, project SSDPs in a high-dimensional space. 
The complex structure of the space renders Euclidean-based clustering analysis less effective.

A better understanding of the correlation would provide insights for better utilization.
Neural methods are notorious for their trial and error practice that tries all possible ways to model the correlation and reports only the successful one.
On the one hand, the score uplift confirms the benefit of utilizing syntactic information in semantic dependency parsing.
On the other hand, the practice reveals little nature about the correlation.
In contrast, a better understanding of the correlation can guide the design of neural models.
Our approach is an example of such a practice where the distribution shift motivates the introduction of the mixture model.
Usage of the mixture model leads to an improvement in scores and, more importantly, confirms the distribution shift as a statistical property underlying the correlation.



However, it is to be noted that our interpretation is only a small step towards a deeper study of the correlation.
The interpretation used only the hop pattern, a feature reflecting the length of syntactic dependencies.
Many other factors, such as the syntactic dependency label, also play an important role in the correlation.
A more comprehensive analysis is needed for a better understanding of the correlation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

The author would like to thank his supervisor, Yusuke Miyao, for the support and insightful suggestions in studying the dependency correlation. The author also wants to express gratitude to his labmates for valuable discussions.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{24refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Junjie Chen}{%
Junjie Chen is currently pursuing his PhD degree at the University of Tokyo under the supervision of Yusuke Miyao. He has a board interest across most areas of Natural Language Processing. Currently, he is actively researching about unsupervised parsing.
}
\end{biography}



\end{document}


