\documentclass[english,other]{jnlp_3.7}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{hangcaption_jnlp}
\usepackage{array}
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
\usepackage{booktabs}
\usepackage{multicol}
\usepackage{multirow}

%%%%%\usepackage{amsfonts}
%%%%%\usepackage{amsthm}
%%%%%\usepackage{amssymb}

\newcommand{\sota}[1]{\textbf{#1}}
\newcommand{\abs}[1]{\vert #1\vert}

%%%%%\newcommand{\ve}[1]{\boldsymbol{#1}}
%%%%%\newcommand{\R}{\mathbb{R}}
%%%%%\newcommand{\B}{\{-1,+1\}}

%%%%%\newcommand{\sign}[1]{\operatorname{sign}}

%%%%%\usepackage{tikz}
%%%%%\usepackage{tikz-qtree}
%%%%%\usepackage{adjustbox}

%%%%%\newcommand\bit[1]{\textbf{\textcolor{red}{#1}}}
%%%%%\newcommand\gpt[1]{\textbf{\textcolor{blue}{#1}}}
%%%%%\newcommand\same[1]{\textbf{\textcolor{blue}{#1}}}
%%%%%\newcommand\tok[1]{{\textcolor{blue}{#1}}}

%%%%%\newcommand{\code}[1]{\\\small\texttt{#1}}
%%%%%\tikzset{level distance=24pt,sibling distance=0pt,every tree node/.style={align=center,anchor=north},edge from parent/.style=
%%%%%  {draw,
%%%%%   edge from parent path={(\tikzparentnode.south)
%%%%%-- +(0,0)
%%%%%                          -| (\tikzchildnode)}}}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{31}
\Number{1}
\Month{March}
\Year{2024}
%%%%\Issuetitle{「　　　　」特集号}




\setcounter{page}{280}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{Society Column}
\etitle{Interpreting Languages with Bits}
\eauthor{Yiran Wang\affiref{nict} \and Taro Watanabe\affiref{naist} \and Masao Utiyama\affiref{nict} \and Yuji Matsumoto\affiref{riken}} 

\headauthor{Wang et al.}
\headtitle{Interpreting Languages with Bits}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\affilabel{nict}{}{National Institute of Information and Communications Technology (NICT)}
\affilabel{naist}{}{Nara Institute of Science and Technology (NAIST)}
\affilabel{riken}{}{RIKEN Center for Advanced Intelligence Project (AIP)}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

The landscape of modern natural language processing has been reshaped by pre-trained language models \cite{devlin-etal-2019-bert,https://doi.org/10.48550/arxiv.1907.11692,lewis-etal-2020-bart,radford2019language,he2021deberta}, which now stand as the default infrastructure. These models have not only markedly boosted performance in various tasks but have also irreversibly transformed the research landscape. However, their inherent lack of interpretability renders them somewhat enigmatic to human understanding, impeding the ability to fully dissect and improve their decision-making processes. In this context, a glimmer of insight has emerged from two recent publications that concentrate on converting continuous representations from pre-trained language models into discrete, interpretable tags, offering a potential breakthrough in this area.\par

On the one hand, \citeA{li-eisner-2019-specializing} propose to compress the contextual representation from pre-trained language models into discrete tags. They utilize the variational information bottleneck \cite{https://doi.org/10.48550/arxiv.1503.02406,alemi2017deep} to nonlinearly interpret high-dimensional continuous vectors into discrete tags, retaining only the information that aids the downstream parsing task. On the other hand, \citeA{kitaev-etal-2022-learned} similarly collapse vectors into discrete tags by employing vector quantization method to limit the size of the discrete token vocabulary.\par

Different from the two methods above, we provide a novel contrastive hashing method to obtain binary codes from high-dimensional hidden states of pre-trained language models. We push the compression limit by further narrowing the information bottleneck to 24 bits to provide a much more compact representation. Besides, the most important difference is that our hashing results not only indicate whether the syntactic properties of two given tokens are different but also distinguish exactly which bits they differ in. Thus, our approach significantly improves the interpretability of models with more compact binary representation.\par


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed Method}


For many tasks, the standard approach in modern language processing involves first feeding the input sentence into a pre-trained language model to assign a continuous vector to each token and then using these vectors in downstream tasks. In this work, our goal is to interpret these continuous vectors as discrete binary codes that only contain necessary task-relevant information. We aim for the compressed vectors to include information from various aspects, ensuring the hash code is as concise as possible while maintaining adequate interpretability.\par

Our idea to incorporate contrastive learning into hashing stemmed from its compelling ability to transform tasks from \emph{wh-questions} to \emph{yes-no questions}. Conventional classification requires specifying target labels for all instances, but contrastive learning only demands knowing whether two instances are identical or not. Answering a single \emph{yes-no question} results in a single bit, while increasing the number of questions to $K$, the model subsequently acquires hashing capabilities by outputting $K$ bits. As our model is based on contrastive learning, training is considerably easier compared to models that rely on variational information bottleneck or vector quantization methods. However, a key challenge that remains in implementing this framework is obtaining scores from $K$ different semantic aspects for hashing.\par

Apart from that, in modern natural language processing, the foundational module is the transformer, with its most crucial component being the multi-head attention mechanism. This mechanism selectively aggregates information from tokens, and the attention scores are supposed to reflect the extent of desired information a token can obtain from others. Particularly, the diagonal entries in the attention matrix indicate the amount of desired information retained within tokens themselves. These characteristics are in perfect alignment with our requirements for hashing. Therefore, we increase the number of heads to $K$, using one head to produce one bit by taking the sign of it. Thus, with $K$ heads each generating one bit from one different semantic aspect, our model becomes capable of compressing vectors into $K$-dimensional binary codes.\par

Furthermore, the full attention matrix is utilized in a dual manner, it not only fulfills its conventional role in the Transformer architecture for computing output vectors but also uses its diagonal entries for learning to hash. Since a portion of the attention matrix is shared between these two objectives, the learned binary codes are likely to retain task-relevant information.\par

In summary, our framework is comprised of two stages. In the first stage, as shown in Figure~\ref{fig:arch}, we focus on converting continuous vectors into discrete tokens. This is achieved by attaching a transformer-based hash layer to the end of a pre-trained language model and fine-tuning the entire model on the downstream task with a contrastive learning objective. A careful selection strategy is employed to exclude potentially false positive and negative instances. After training, the hash layer is utilized to re-annotate the entire dataset, assigning a binary code to each token.\par

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[t]
\begin{center}
%%%%%\includegraphics[width=0.618\textwidth]{31-1ia22f1.pdf}
    \includegraphics{31-1ia22f1.pdf}
\end{center}
    \hangcaption{The architecture of the first-stage model for named entity recognition is designed such that the transformer hash layer generates both contextual representations and ego-attention scores. These are used for task-specific fine-tuning and contrastive hashing, respectively. Solid lines represent positive instances, while dotted lines depict negatives.}
    \label{fig:arch}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In the second stage, we assess whether these binary codes retain task-relevant information or merely represent meaningless bits. To do this, we train a significantly more lightweight model from scratch using only these binary codes as inputs. If our experiments show that this streamlined model still performs comparably to the original full-size model, it would be a demonstration that our method effectively preserves task-relevant information within these binary codes.\par

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}


We conduct experiments on structured prediction tasks, i.e., part-of-speech tagging, named entity recognition, and constituency parsing, to demonstrate the effectiveness of our methods.\par

As presented in Table~\ref{tab:main}, experiments on the part-of-speech tagging show that $32$ bits achieve slightly better results than $16$ bits and $24$ bits on both stages. Besides, we notice that results in the validation stage are constantly superior to hashing stage results, no matter how many bits are used. For named entity recognition, we achieve $90.39$ in F${}_1$ score with $24$ bits, which is even slightly higher than its hashing stage performance, i.e., $90.27$. For $16$ bits and $32$ bits, the validation stage performance also consistently surpasses their hashing stage performance. For constituency parsing, our method outperforms \citeA{kitaev-etal-2022-learned} with $32$ bits in the bidirectional parsing task, even they introduce much more tags, i.e., 256 in total. Besides, our $16$ bits and $24$ bits settings also achieve remarkable performance and are only slightly inferior to theirs. For all these tasks, our tiny model still reproduces comparable or even slightly better performance than the original full-size model. We claim that these results demonstrate that our learned binary codes have properly preserved task-relevant information.\par

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{table}[t]
\input{22table01.tex}%
    \hangcaption{The main results on three datasets. The results of our methods are displayed in two rows, which indicate the performance in hashing and validation stages, respectively. $\abs{\theta}$ columns show the number of parameters, and the bold numbers indicats \textbf{the best validation performance} of each setting.}
    \label{tab:main}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

About the interpretability, we present the hashing and constituency parsing results in Figure~\ref{fig:tree} of our learned binary codes. For comparison with \citeA{kitaev-etal-2022-learned}, we use the same examples as in their paper.\par

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[!t]
\begin{center}
\includegraphics{31-1ia22f2.pdf}
\end{center}
\caption{Derivation of the sentence \emph{Lucas brought the groceries for him}.}
\label{fig:tree}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

In this paper, we have proposed a contrastive hashing method to generate interpretable binary codes from pre-trained language models. We designed a transformer-based hash layer, incorporated it into the contrastive hashing framework, and introduced a novel instance selection strategy to exclude false positives and negatives. Experimental results indicate that our lightweight model achieves superior performance and preserve task-relevant information properly with even fewer bits. Further analyses show that the generated binary codes retain syntactic and semantic information in a highly interpretable and fine-grained format. Although we only focus on structured prediction tasks in this paper, as a novel interpretable representation, our method can be easily adapted to other tasks and may inspire future research on designing efficient architectures.\par

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

This paper outlines the motivation and fundamental workflow of our paper, \emph{24-bit Languages} \cite{wang-EtAl:2023:ijcnlp2}, as presented at \emph{the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics}. For a more comprehensive understanding of the algorithm details and mathematical definitions, please refer to the original paper.\par

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \bibliographystyle{jnlpbbl_1.7}
\bibliography{22refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Yiran Wang}{%
He is a technical researcher of the Advanced Translation Technology Laboratory, at the National Institute of Information and Communications Technology (NICT).
}

\bioauthor[:]{Taro Watanabe}{%
He is the professor of Natural Language Processing Laboratory, at Nara Institute of Science and Technology (NAIST).
}

\bioauthor[:]{Masao Utiyama}{%
He is the director of the Advanced Translation Technology Laboratory, at the National Institute of Information and Communications Technology (NICT).
}

\bioauthor[:]{Yuji Matsumoto}{%
He is the principal investigator of Knowledge Acquisition Team, at RIKEN Center for Advanced Intelligence Project.
}

\end{biography}

%%%% \biodate   %%%% Following settings will be edited by editors.

\end{document}