\documentclass[english]{jnlp_3.5}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%\usepackage{times}
%%%%\usepackage{latexsym}
\usepackage{multirow}
%%%%\usepackage{microtype}
%%%%\usepackage{subfigure}
%%%%\usepackage{here}
%%%%\usepackage{amsmath}
%%%%% \usepackage{natbib}
%%%%\usepackage[dvipdfmx]{color}
\newcommand{\tabref}{Table~\ref}
\newcommand{\figref}{Figure~\ref}
%%%%\newcommand{\citealp}[1]{\citeauthor{#1} \citeyear{#1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{29}
\Number{1}
\Month{March}
\Year{2022}
%%%%\Issuetitle{Special Issue: XXXXXXXX}

\received{2021}{6}{17}
\revised{2021}{9}{7}
%%%%\rerevised{2006}{5}{6}
\accepted{2021}{10}{4}

\setcounter{page}{3}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Automatic Machine Translation Evaluation using a Source and Reference Sentence with a Cross-lingual Language Model}
\eauthor{Kosuke Takahashi\affiref{NAIST} \and Katsuhito Sudoh\affiref{NAIST}\affiref{PRESTO} \and Satoshi Nakamura\affiref{NAIST}} 
\eabstract{%
As the performance of machine translation has improved, the need for a human-like automatic evaluation metric has been increasing. The use of multiple reference translations against a system translation (a hypothesis) has been adopted as a strategy to improve the performance of such evaluation metrics. However, preparing multiple references is highly expensive and impractical. In this study, we propose an automatic evaluation method for machine translation that uses source sentences as additional pseudo-references. The proposed method evaluates a translation hypothesis via regression to assign a real-valued score. The model takes the paired source, reference, and hypothesis sentences together as input. A pre-trained large-scale cross-lingual language model encodes the input to sentence vectors, with which the model predicts a human evaluation score. The results of experiments show that our proposed method exhibited stably higher correlation with human judgements than baseline methods that solely depend on hypothesis and reference sentences, especially when the hypotheses were very high- or low-quality translations.
}
\ekeywords{Automatic Machine Translation Evaluation, Language Model, Sentence Meanings, Source Language, Regression}

\headauthor{Takahashi et al.}
\headtitle{Automatic MT Evaluation using a Source and Reference Sentence}

\affilabel{NAIST}{}{Nara Institute of Science and Technology}
\affilabel{PRESTO}{}{PRESTO, Japan Science and Technology Agency}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Automatic evaluation metrics for machine translation have been studied to replace human evaluation owing to their cost-efficiency and the stability of their results.
In the study of automatic evaluation metrics, a correlation score such as Pearson's is typically used to evaluate the performance of automatic metrics, as reported in the metric shared task of the Fifth Conference on Machine Translation (WMT20) by \fullciteA{mathur-etal-2020-results}.
However, BLEU \cite{bleu}, one of the most popular metrics, correlates more poorly with human evaluation than the recent metrics submitted to the WMT metrics task.
BLEU measures the similarity of references and hypothesis sentences at the surface-level by checking whether word symbols in each sentences are the same, but the meaning of each sentence is ignored in the calculation. 
To address this problem, representation-based metrics have been proposed.
BERTscore \cite{BERTscore}, BERT regressor \cite{BERTregressor}, and BLEURT \cite{BLEURT} use a large pre-trained language model called Bidirectional Encoder Representations from Transformers (BERT) \cite{devlin-etal-2019-bert} to encode the references and hypothesis sentences in embedded vectors as features, and have all achieved much higher correlation with human judgements than surface-level metrics. 

These metrics compare a hypothesis translation to a reference. 
However, a reference translation represents only one possible translation, and these metrics are unlikely to correctly evaluate all candidates that share the same meanings as the reference.
This problem can be mitigated by the use of multiple reference translations, as argued by \citeA{HYTER}, and \citeA{Qin},
but preparing such multiple references is costly and time-consuming.

Since the source and reference sentences should be semantically equivalent, source sentences can be regarded as pseudo references. 
Our prior work \cite{takahashi-etal-2020-automatic} incorporated source sentences into automatic evaluation metrics and showed that source sentences can contribute to the increase of Pearson's correlation coefficient with the results of human evaluation. 
As in our prior work, the proposed method handles source and target languages in a shared sentence embedding space of a cross-lingual language model and trained models with two input styles of using source sentences. In this study, we compared our method to baselines with multiple pre-trained language models and confirmed the improvement in the evaluation performance resulting from using source sentences. Additionally we analyzed benefits of using source sentences and the drawbacks of baseline and the proposed method for future work.
The contributions of this work are summarized below.
\begin{itemize}
    \item Our proposed method surpassed the baseline of BLEURT \cite{BLEURT} in terms of Pearson's correlation coefficient with human judgments, especially when the hypothesis sentence was very low- or high-quality.
    \item The proposed method is robust to finding the similarity of transposed phrases as though they have the same or close meaning.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Surface-level Metrics}

Among automatic evaluation metrics, BLEU \cite{bleu} is the most popular. BLEU calculates its score by the rates of N-gram word matching between reference and hypothesis translations.  Metrics based on this type of feature are called surface-level because they consider only the lexical information at most and deeper linguistic or semantic information such as meanings of words are dismissed. Owing to errors arisen from disregarded meanings of words, BLEU has been shown to make unreliable evaluations such as when a hypothesis translation is equivalent to the reference in meaning but the word lineups are different, or when a system translation has severe translation errors that lose or distort the original meaning \cite{liu-etal-2016-evaluate,novikova-etal-2017-need,chaganty-etal-2018-price}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Embedding-based Metrics}

Embedding-based metrics are proposed to address the problem of surface-level metrics.
In this approach, distributed representation is used as the feature instead of word symbols.
With the recent advance of large-scale language models such as BERT \cite{devlin-etal-2019-bert}, metrics that leverage BERT have expanded the scope of automatic evaluation.
BERT is based on a transformer \cite{TRANSFORMER} and composed of encoder layers of a self-attention network and output layer with a feed-forward network.
BERT is a bidirectional language model and trained with a large number of documents to recover masked tokens. This training of BERT is called masked-language modeling (MLM).
In the downstream tasks of GLEU \cite{GLEU}, BERT has achieved significantly improved performance.
BERTscore \cite{BERTscore}, BERT regressor \cite{BERTregressor}, and BLEURT \cite{BLEURT} are applications of BERT in machine translation evaluation.
BERTscore measures the similarity of reference and hypothesis translation by the cosine similarity of the token embeddings for each token in the reference and hypothesis. 
It implements a pre-trained BERT model and is not fine-tuned further.
Instead, the BERT regressor and BLEURT are fully parameterized and require a human annotated evaluation corpus to fine-tune the models. 
The models of both metrics have the same architecture; a linear layer is attached to top of the BERT encoder. They encode a paired reference and hypothesis sentence using BERT and predict the human evaluation score. 
Additionally, BLEURT conducts warm-up training of BERT before fine-tuning on an evaluation corpus.

Developed independently from this work, a metric called COMET \cite{COMET} was proposed after our prior work. COMET is based on a cross-lingual BERT-like model and uses source language inputs. While COMET is focused on improving the robustness to the high-performing systems and was developed to make rankings of machine translation systems, our prior and this work are focused on the linguistic analysis of the use of source sentences in evaluation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed Method}

In the proposed approach, we extend the BERT regressor \cite{BERTregressor} and use cross-lingual language models to utilize a source sentence as a pseudo-reference.
To obtain a sentence-pair vector from source and target language sentences together, our model encodes input sentences with a cross-lingual language model rather than with monolingual BERT. 
The model procedure is illustrated in \figref{fig:models}.
Our proposed method composes two variants to input each source, reference, and hypothesis sentence: 
\emph{hyp+src/hyp+ref} uses paired inputs denoted as hypothesis-source and hypothesis-reference, and \emph{hyp+src+ref} uses the combination of hypothesis, source, and reference.
The implementation of the two variants is as follows.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[b]
\begin{center}
\includegraphics{29-1ia1f1.pdf}
\end{center}
%%%%\label{fig:model1}
%%%%\label{fig:model2}
\caption{Two input styles of the proposed method}
\label{fig:models}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{description}
%%%%\item[hyp+src/hyp+ref (\figref{fig:model1})]
    \item[hyp+src/hyp+ref (\figref{fig:models}(a))]
introduces another vector as hypothesis-source in addition to the standard vector for the hypothesis-reference pair to perform an ensemble evaluation.
Both sentence vectors obtained from \emph{hyp+src} and \emph{hyp+ref} are concatenated and used to predict the evaluation score in multilayer perceptron (MLP). 

%%%%\item[hyp+src+ref (\figref{fig:model2})]
    \item[hyp+src+ref (\figref{fig:models}(b)]
takes a concatenation of hypotheses, source, and reference sentences as an input to a cross-lingual language model to obtain a sentence-pair vector.
This sentence-pair vector is expected to be directly learned to represent the quality of the translation hypothesis given two correct sentences aligned aside. 
\end{description}

Although the model is distinct in terms of input style, the overall evaluation process is almost the same.
First, the cross-lingual language model encodes an input sentence into a sentence-pair vector. Then, using the sentence-pair vector, an MLP outputs the final evaluation score in a regression manner.
During training, we used the standardized z-score of direct assessment (DA; \citeauthor{DA1} 2013) as the ground truth and updated the model parameters by backpropagation \cite{back} with mean squared error loss (MSE loss).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}

To compare the proposed method with baselines that do not incorporate the sources and to compare the difference between the proposed input styles, we conducted meta-evaluation experiments on the WMT17 metrics task \cite{wmt17results}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Setup}

The experiments were conducted with a corpus of all language pairs to English translations from the segment-level metrics shared task of WMT17 \cite{wmt17results}. 
We divided the corpus of WMT15 and WMT16 into training and development data with a ratio of 9:1, and the whole corpus of WMT17 is used for evaluation of our method and baselines.
The corpus size for each language pair is shown in \tabref{tab:corpus}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[b]
\input{01table01.tex}
\hangcaption{Available corpus size annotated with human judgements in WMT-2017 Metrics Shared Task (to-English)}
\label{tab:corpus}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

As a language model to encode input sentences, we compared BERT and RoBERTa \cite{Roberta} on monolingual \emph{hyp+ref} input, and also compared multilingual BERT \cite{devlin-etal-2019-bert}, XLM \cite{XLM}, and XLMRoBERTa \cite{xlmroberta} for all input variants (\emph{hyp+ref}, \emph{hyp+src}, \emph{hyp+src/hyp+ref}, and \emph{hyp+src+ref}). All of the language models used originated from HuggingFace Transformers.\footnote{\url{https://github.com/huggingface/pytorch-transformers}}

Multilingual BERT, XLM, and XLMRoBERTa are multilingual language models. 
Notably, some of the XLM models are trained with a new objective in addition to BERT, called  Translation Language Modeling (TLM). TLM is used to predict the same masked tokens, but with the help of parallel sentences in other languages.
XLM-RoBERTa is a larger scaled model of XLM and trained with larger corpora.

We used \texttt{BERT-base-uncased} for BERT and \texttt{BERT-base-cased} for multilingual BERT.
For RoBERTa, we used the \texttt{RoBERTa-large} model. 
For more robust multilingual language models,
among the XLM-family models,\footnote{\url{https://github.com/facebookresearch/XLM}} we used \texttt{xlm-mlm-tlm-xnli15-1024} (XLM15) pre-trained by MLM and TLM and \texttt{xlm-mlm-100-1280} (XLM100) pre-trained by MLM.
XLM15 is expected to perform better by the paired bilingual training of TLM,
but the number of available languages is limited.
XLM15 is compatible with only German, Russian, Turkish, and Chinese in the corpus, which confines the model to partial access to the corpus.
Whereas, XLM100 is compatible with all language pairs in the corpus,
while it lacks TLM, which is a supervised bilingual pre-training.
As XLMRoBERTa, we used its largely pre-trained model.

The experiments had two different corpus settings because of the language restriction of XLM15;
one of which was a small-corpus including \{German (de), Russian (ru), Turkish (tr), and Chinese (zh)\} to English (en) language pairs,
and the other of which was an entire-corpus including \{Czech (cz), German (de), Finnish (fi), Latvian (lv), Romanian (ro), Russian (ru), Turkish (tr), and Chinese (zh)\} to English language pairs.  
The evaluation measured  Pearson's correlation coefficient with regard to human judgments in the test set.

As baseline models, we re-implemented BERT regressor \cite{BERTregressor} and BLEURT \cite{BLEURT}. Because the available BLEURT models\footnote{\url{https://github.com/google-research/bleurt}} are fine-tuned with corpus settings differing from those used in this work, we fine-tuned the warmed-up RoBERTa model.
We aimed to reproduce BELEURT's score, but because the best hyperparameters are unknown, the re-implemented BLEURT did not perform as well as the original. However, the difference was not excessively large, and the model remained valid as a baseline, as indicated in the tables showing the results.
Moreover, we compared the results with sentence BLEU \cite{sacrebleu} and BERTscore \cite{BERTscore}.
The base language models of baselines are both monolingual models, using BERT and RoBERTa.
In contrast, the proposed method used the multilingual language models listed above, BERT, XLM, and XLM-RoBERTa, and compared them.
All the monolingual baselines were experimented with \emph{hyp+ref}. 
Whereas, in order to compare the input variants of proposed method, all models of the proposed method were assessed with \emph{hyp+ref}, \emph{hyp+src}, \emph{hyp+src/hyp+ref}, and \emph{hyp+src+ref}.  

The fine-tuning of the proposed methods and re-implemented baselines was based on
MSE loss in the training set.
Note that the parameters of the pre-trained language models were also updated through fine-tuning.
The hyper-parameters were selected through grid search among the following parameters. 
Because models are affected by randomness in training, we ran ten experiments for each of the settings and report the results of the trial in which the Pearson's correlation score of the validation set was the highest.

\begin{itemize}
\setlength{\itemsep}{0pt}
\item Optimizer : \{Adam\}
\item Learning rate : \{$9\mathrm{e}-6$, $6\mathrm{e}-6$, $3\mathrm{e}-6$\}
\item Number of epochs : \{1, ...,10\}
\item Dropout rate of MLP: \{0.1, 0.2, 0.3, 0.4, 0.5\}
\item Batch size : \{4, 8, 16\}
\end{itemize}%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Results}

The results of each experiments on the small-corpus and entire-corpus are shown in Tables \ref{tab:result_all_en_15_17} and \ref{tab:result_all_en_17}, respectively.
Note that XLM15 was not included in the entire-corpus experiment due to its limited language coverage.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\input{01table02.tex}
\hangcaption{Pearson's correlation scores in the small-corpus (\{de,ru,tr,zh\}-en).``Av'' in the table indicates the macro average of the correlation scores for each language pair. ``Al'' shows the correlation score for the aggregated corpus regardless the language pairs.}
\label{tab:result_all_en_15_17}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[t]
\input{01table03.tex}
\hangcaption{Pearson's correlation scores in the entire-corpus (all-en) of WMT17. ``Avg'' in the table indicates the macro average of the correlation scores for each language pair. ``All'' shows the correlation score for the aggregated corpus regardless the language pairs.}
\label{tab:result_all_en_17}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In the small-corpus setting (\tabref{tab:result_all_en_15_17}), XLMRoBERTa with \emph{hyp+src/hyp+ref} performed best among all models. However, among the baselines, the BERT regressor with RoBERT obtained better results than multi-BERT and XLM100. The scores of XLM15 with \emph{hyp+src/hyp+ref} were close to those of RoBERTa. Because our proposed method using XLM-RoBERTa achieved the best score in the small-corpus condition, XLM-RoBERTa with \emph{hyp+src/hyp+ref} may be considered the best option when the fine-tuning resource is scarce.

In the entire-corpus setting (\tabref{tab:result_all_en_17}), RoBERTa was the best among all models with \emph{hyp+ref} conditions. However, although the Pearson correlation of XLM-RoBERTa with hyp+ref was lower than that of RoBERTa, XLM-RoBERTa with \emph{hyp+src/hyp+ref} was almost on par and surpassed the score of BLEURT which was reported in its original paper. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Input variants}
Every model with \emph{hyp+src/hyp+ref} achieved a better score than both \emph{hyp+src} and \emph{hyp+ref}, which indicates that source sentences contributed to the improvement of evaluation.

The results in Tables ~\ref{tab:result_all_en_15_17} and \ref{tab:result_all_en_17} show that \emph{hyp+src/hyp+ref} structure worked better than \emph{hyp+src+ref} in most of the conditions, although we expected \emph{hyp+src+ref} to perform better because it can access two translation answers as references simultaneously. This probably occurred because the cross-lingual language models were not pre-trained to handle three sentences combined in a sequence. Indeed, the score gaps were larger in the small-corpus condition.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Performance of each language model}
As we can see from \tabref{tab:result_all_en_15_17},
the proposed method using XLM15 with the \emph{hyp+src/hyp+ref} structure surpassed the BERT regressor with BERT in the small corpus.
However, XLM100 did not work well in the experiments.
The results were much worse than those of the others in the small-corpus condition,
and it did not compete with the BERT regressor in the entire-corpus condition as shown in \tabref{tab:result_all_en_17}.
One possible reason is that XLM100 lacks of pre-training with the TLM objective.
Because the TLM task allows the model to learn semantically equivalent cross-lingual sentences directly,
tasks related to TLM can be considered important for using source sentences in automatic evaluation metrics.
The results of multilingual BERT were worse than those of BERT regressor with BERT and XLM15, but close to XLM100 or slightly better in general.
Among the cross-lingual language models, XLM-RoBERTa showed the best results. 
Its Pearson correlation scores were better than those of other models in all input styles. 

From this comparison of pre-training objectives and language models, we report that our proposed method is influenced by the multilingualism of a language model. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Contribution of adding source sentences}
We have discussed the benefits of adding source sentences with our proposed method from a quantitative perspective by comparing scores in terms of Pearson's correlation. 
To further investigate the contribution of adding source sentences by a quantitative measurement, we visualized the gap between the model outputs and human judgment scores in scatter plots.
In Figures \ref{fig:scatter_wmt17_all_to_en_RoBERTa} and \ref{fig:scatter_wmt17_all_to_en_XLMRoBERTa},
the shape of the outline becomes sharper and plots are coordinated more lineally in the \emph{hyp+src/hyp+ref} compared to \emph{hyp+ref}. 
Also the sparsity of the plots becomes larger as the DA scores (human scores) of the hypotheses becomes lower, which means that low-quality hypotheses are difficult to evaluate.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{29-1ia1f2.pdf}
\end{center}
%%%%\label{fig:scatter_wmt17_all_to_en_RoBERTa_hyp+ref}
%%%%\label{fig:scatter_wmt17_all_to_en_RoBERTa_hyp+src}
%%%%\label{fig:scatter_wmt17_all_to_en_RoBERTa_hyp+src_hyp+ref}
%%%%\label{fig:scatter_wmt17_all_to_en_XLMRoBERTa_hyp+src+ref}
\hangcaption{RoBERTa scatter plots. Human scores (DA scores) are set in the horizontal axis and the predicted scores are set in the vertical axis.}
\label{fig:scatter_wmt17_all_to_en_RoBERTa}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[t]
\begin{center}
\includegraphics{29-1ia1f3.pdf}
\end{center}
%%%%\label{fig:scatter_wmt17_all_to_en_XLMRoBERTa_hyp+ref}
%%%%\label{fig:scatter_wmt17_all_to_en_XLMRoBERTa_hyp+src}
%%%%\label{fig:scatter_wmt17_all_to_en_XLMRoBERTa_hyp+src_hyp+ref}
%%%%\label{fig:scatter_wmt17_all_to_en_XLMRoBERTa_hyp+src+ref}
\hangcaption{XLMRoBERTa scatter plots. Human scores (DA scores) are set in the horizontal axis and the predicted scores are set in the vertical axis.}
\label{fig:scatter_wmt17_all_to_en_XLMRoBERTa}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Furthermore, we visualized Pearson's correlation and the mean absolute error for each segment of the DA range (\figref{fig:DA_range}).
All models with \emph{hyp+src/hyp+ref} achieved better Pearson's correlation in most areas of the DA than the same models with \emph{hyp+ref}.
In addition, the proposed model using XLM-RoBERTa with \emph{hyp+src/hyp+ref} was competitive overall with the baseline of RoBERTa with \emph{hyp+ref}. Furthermore, it achieved better correlation and error scores in the low- and high-DA areas, while RoBERTa with \emph{hyp+ref} deteriorated its performance in the same DA areas.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 4
\begin{figure}[t]
\begin{center}
\includegraphics{29-1ia1f4.pdf}
\end{center}
%%%%\label{fig:pearson_DArange}
%%%%\label{fig:Mean_abs_error_DArange}
\hangcaption{Pearson's correlation and mean absolute error for each small DA ranges. Each DA segment has 0.5 DA score range. The amount of each segment is written below the range description. }
\label{fig:DA_range}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Case Analysis}

We analyzed some cases by comparing the best-performing models, RoBERTa with hyp+ref and XLMRoBERTa with hyp+src/hyp+ref. 
During the analysis, all the cases were sorted by the absolute error of the predicted score. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Cases in which XLMRoBERTa with hyp+src/hyp+ref performed better}
The cases in which XLMRoBERTa with hyp+src/hyp+ref performed better are described in \tabref{tab:cases_XLMRoBERTa_better}. 
In both cases, the hypotheses exhibited minor errors, but they were both acceptable.
The XLM-RoBERTa model was able to apprehend paraphrase-like phrase-switching in such cases.
We assume that the cause of this phenomenon is that an additional source sentence works as another reference, and finding the similarity of transposed phrases becomes easier.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[t]
\input{01table04.tex}
\hangcaption{Cases in which XLM-RoBERTa with hyp+src/hyp+ref performed betetr than RoBERTa with hyp+ref}
\label{tab:cases_XLMRoBERTa_better}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Cases in which RoBERTa with hyp+ref performed better}
Two example cases in which RoBERTa with hyp+ref is better are described in \tabref{tab:cases_RoBERTa_better}.
In both cases, hypotheses have opposite meanings of sentence to the references, which is affected by the mistranslation or insertion of negation. 
In these scenarios, it is implied that the RoBERTa, which is more robust in the monolingual setting, can better capture the relationship between a reference and hypothesis. 
However, as confirmed in \figref{fig:DA_range}, the ability of RoBERTa to apprehend negation is limited in the middle DA area.
In fact, only a few out of 230 cases from the two worst DA ranges were evaluated suitably using the RoBERTa model.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[tb]
\input{01table05.tex}
\hangcaption{Cases in which RoBERTa with hyp+ref performed better than XLMRoBERTa with hyp+src/hyp+ref}
\label{tab:cases_RoBERTa_better}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Cases in which both models significantly mispredicted}

In \tabref{tab:cases_both_poor}, the cases both models of RoBERTa and XLMRoBERTa failed to evaluate are described. 
The upper case represents the translation error of noun, typically named entity. 
The lower case shows the error of dependency. 
Among the twenty worst cases of the evaluation failure, the most frequent one is due to named entities and all worst cases are assigned to the two worst DA ranges in \figref{fig:DA_range}, which is shown in \figref{fig:scatter_wmt17_all_to_en_RoBERTa} and \ref{fig:scatter_wmt17_all_to_en_XLMRoBERTa} as well. 
The problem of named entity is very tricky since nouns of similar categories are embedded closely in the encoder and difficult to be distinguished. 
Neither the proposed model nor the baseline model have an architecture able to solve this problem.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[t]
\input{01table06.tex}
\hangcaption{Cases in which both of XLMRoBERTa with hyp+src/hyp+ref and RoBERTa with hyp+ref performed poorly}
\label{tab:cases_both_poor}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Analysis with Additional Experiments: Training with All-All Language Pairs}

Our model was evaluated on corpora in which English is always the target language. However, it would be ideal to train the model with mixed languages and to make it robust to multiple target languages.
Thus, to investigate the compatibility of training with a mixed language corpus, we trained the models with English-to-Russian language pairs, which is an available language pair included in the WMT15-16 training corpus, and ran test evaluation on the WMT17 corpus with additional English-to-Russian and English-to-Chinese language pairs.
The results for the entire- and small-corpus are shown in \tabref{tab:result_all_all_17} and \tabref{tab:result_all_all_15_17} respectively. Except for XLMRoBERTa, the correlation was worse than the models trained with to-English corpus. Even for XLMRoBERTa, the result was negligible at best. 
We suppose that the reason the evaluation performance of models trained with multiple target corpora deteriorated is that the source language and reference language were switched during the training and the embedding space was corrupted.
We conclude that a more robust cross-lingual language model and a larger corpus will most likely be required to leverage multiple target language corpora for fine-tuning. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 7
\begin{table}[t]
\input{01table07.tex}
\hangcaption{Pearson's correlation scores on the all-all language pair corpus of WMT17. ``Avg'' in the table indicates the macro average of the correlation scores for each language pair. ``All'' shows the correlation score for the aggregated corpus regardless of the language pairs.}
\label{tab:result_all_all_17}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 8
\begin{table}[t]
\input{01table08.tex}
\hangcaption{Pearson's correlation scores on the small and all-all language pair corpus of WMT17. ``Avg'' in the table indicates the macro average of the correlation scores for each language pair. ``All'' shows the correlation score for the aggregated corpus regardless of the language pairs}
\label{tab:result_all_all_15_17}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

We have proposed the use of source language sentences in machine translation evaluation based on largely pre-trained language models.
From the results of the experiments, we confirmed that the cross-lingual pre-trained language models were adjusted to encode both hypothesis-source and hypothesis-reference input pairs and improve the evaluation performance.   
To exploit our proposed method, it would be better to use a cross-lingual language model, which achieved the best performance in cross-lingual down-streaming tasks such as XNLI \cite{XNLI}.
We have shown that the Pearson correlation and mean absolute error of XLM-RoBERTa with \emph{hyp+src/hyp+ref} were consistently higher than those of the baseline model using RoBERTa with \emph{hyp+ref}. 

With the analysis of the metrics' performance in split DA ranges and some test cases, using source sentences have been indicated to contribute to finding the similarity of transposed phrases as though they have the same or close meaning. However, both proposed and baseline models have a problem of evaluating or penalizing different named entities, which often appear in the low DA area.

The current BERT-like language models are not trained to handle named entities explicitly and other critical errors found in the case analysis. However, we believe that the technique used in other studies is applicable to evaluation metrics. 
Our future work will focus on improving evaluation reliability of critical translation errors such as named entities, negation, and dependency errors.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

This paper is an extended version of our work presented in ACL2020 \cite{takahashi-etal-2020-automatic}. 
This work was supported by JST PRESTO (JPMJPR1856). With the support of JST, we used RIKEN's miniRAIDEN in some of the experiments in this work.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{01refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Kosuke Takahashi}{
He is a Ph.D. student at Nara Institute of Science and Technology. He received his bachelor's degree in engineering from Osaka City University in 2018 and his masterfs degree in engineering from Nara Institute of Science and Technology in 2020.
}

\bioauthor[:]{Katsuhito Sudoh}{%
He is an associate professor at Nara Institute of Science and Technology. He received his bachelor's degree in engineering in 2000 and his master's and Ph.D. degrees in informatics in 2002 and 2015, respectively, from Kyoto University. He worked with the NTT Communication Science Laboratories from 2002 to 2017. He currently works on machine translation and natural language processing. He is a member of ACL, ANLP, IPSJ, ASJ, and JSAI.
}

\bioauthor[:]{Satoshi Nakamura}{
He is a professor at Nara Institute of Science and Technology, team leader of the Tourism Information Analytics Team, AIP Center, RIKEN, and honorer professor at the Karlsruhe Institute of Technology, Germany. He received his B.S. from the Kyoto Institute of Technology in 1981 and a Ph.D. from the Kyoto University in 1992. He was the Director of ATR Spoken Language Communication Research Laboratories in the period 2000--2008 and Vice President of ATR in the period 2007--2008. He was the Director General of Keihanna Research Laboratories and Executive Director of Knowledge Creating Communication Research Center, National Institute of Information and Communications Technology in 2009--2010. He is currently the director of the Augmented Human Communication laboratory and a full professor at the Data Science Center and Information Science Division, Graduate School of Science and Technology, Nara Institute of Science and Technology. He was an Elected Board Member of the International Speech Communication Association, ISCA, in the period June 2011--2019, and IEEE Signal Processing Magazine Editorial Board member in 2012--2015, IEEE SPS Speech and Language Technical Committee Member in 2013--2015. He is an ATR Fellow, IPSJ Fellow, IEEE Fellow, and ISCA Fellow.
}
\end{biography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate




\end{document}

