\documentclass[english]{jnlp_3.5}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{hangcaption_jnlp}
\usepackage{array}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
\usepackage{multirow}
%%%%\usepackage{color}
%%%%\newcommand{\blue}[1]{\textcolor{blue}{#1}}
%%%%\newcommand{\red}[1]{\textcolor{red}{#1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{29}
\Number{2}
\Month{June}
\Year{2022}
%%%%\Issuetitle{Special Issue: XXXXXXXX}

\received{2021}{9}{13}
\revised{2021}{11}{30}
%%%%\rerevised{2006}{5}{6}
\accepted{2022}{1}{12}

\setcounter{page}{344}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Knowledge Distillation for Translating Erroneous Speech Transcriptions}
\eauthor{Ryo Fukuda\affiref{Author_1} \and Katsuhito Sudoh\affiref{Author_1}\affiref{Author_2} \and Satoshi Nakamura\affiref{Author_1}\affiref{Author_2}} 
\eabstract{%
Recent studies consider knowledge distillation as a promising method for speech translation (ST) using end-to-end models. However, its usefulness in cascade ST with automatic speech recognition (ASR) and machine translation (MT) models has not yet been clarified. An ASR output typically contains speech recognition errors. An MT model trained only on human transcripts performs poorly on error-containing ASR results. Thus, it should be trained considering the presence of ASR errors during inference. In this paper, we propose using knowledge distillation for training of the MT model for cascade ST to achieve robustness against ASR errors. We distilled knowledge from a teacher model based on human transcripts to a student model based on erroneous transcriptions. Our experimental results showed that the proposed method improves the translation performance on erroneous transcriptions. Further investigation by combining knowledge distillation and fine-tuning consistently improved the performance on two different datasets: MuST-C English--Italian and Fisher Spanish--English.
}
\ekeywords{Speech Translation, Knowledge Distillation}

\headauthor{Fukuda et al.}
\headtitle{Knowledge Distillation for Translating Erroneous Speech Transcriptions}

\affilabel{Author_1}{}{Nara Institute of Science and Technology}
\affilabel{Author_2}{}{AIP, RIKEN}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Speech translation (ST) converts utterances in a source language into another language, and it has two types of outputs: text and speech; herein, we focus on ST to text.
Conventional ST, called \textit{cascade} or \textit{pipeline} ST, involves two types of models: automatic speech recognition (ASR) and machine translation (MT).
In cascade ST, the error propagation from ASR to MT severely degrades the ST performance.
In contrast, a new ST system called \textit{end-to-end} or \textit{direct} ST uses a single model to directly translate a source language speech into a target language text \cite{berard2016listen}.
Such an end-to-end approach is a new paradigm in ST and is generating significant research interest.
However, simple end-to-end ST without additional training such as ASR tasks is inferior to cascade ST \cite{liu-etal-2018-ustc,salesky-black-2020-phone}.
In addition, it requires parallel data of the source language speech and the target language text, which cannot be easily obtained in practice.

Recent ST studies have incorporated cascade ST techniques into end-to-end ST.
Multitask training with an ASR subtask has been used in end-to-end ST \cite{weiss2017sequence,anastasopoulos-chiang-2018-tied,sperber-etal-2019-attention,9072502,9383496}.
Initializing end-to-end ST with a pretrained ASR or MT model has also become a common approach \cite{berard2018end,bansal2019pre,inaguma-etal-2020-espnet,wang-etal-2020-fairseq,bahar2021tight}.

In this study, we focused on the cascade approach owing to its performance advantage over end-to-end ST.
Moreover, cascade ST models can be incorporated into end-to-end ST, as demonstrated in previous studies.

During the training of an MT model of cascade ST, clean human transcripts can be used as the input for the source language speech.
However, because the MT model of cascade ST typically receives the ASR output during inference, ASR errors are expected to be propagated into the MT model, causing translation errors.
If erroneous speech transcriptions are used in ASR for training, the MT model will be trained to translate \emph{erroneous} transcriptions into \emph{correct} text, which is generally inappropriate.
One possible solution is to use both types of input (clean and erroneous transcriptions) for training. However, questions regarding their use and the appropriate training strategy for cascade STs need to be addressed.

Domain adaptation techniques such as fine-tuning (FT) and mixture training can be used to coordinate clean and ASR-based inputs.
However, overfitting on ASR-based inputs can degrade the performance using clean inputs, which is an ideal case without error.
In addition, these techniques do not consider low-fidelity translations in a speech domain, which increases the difficulty of training the MT model.
Parallel corpora for spontaneous speech frequently include translations of colloquial expressions that reflect the diversity of human languages. Producing translations of colloquial expressions is important for MT to simulate human translations; however,  it is also difficult.

In this study, we developed a knowledge distillation (KD) technique for training of the MT model of cascade ST.
A student model for a realistic scenario that translates ASR-based inputs is supervised by a teacher model that translates clean inputs.
We also investigated combining KD and FT, a domain adaptation technique.

The experimental results revealed that KD improves the translation performance on erroneous transcriptions from ASR and that KD following FT leads to further improvement.
Furthermore, we analyzed the results and examined how our method contributes to cascade ST. 
KD makes it easier to train MT by promoting literal translations, instead of reproducing translations of colloquial spoken utterances.
Additionally, KD combined with FT further improves the robustness against ASR errors without significantly degrading the translation performance using clean inputs.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Studies}

Some ST studies have addressed the problem of ASR error propagation.
N-best hypotheses \cite{zhang2004unified,quan2005integrated}, confusion networks \cite{bertoldi2005new,bertoldi2007speech}, and lattices \cite{matusov2010lattice,sperber2017neural} have been used to include ASR uncertainty in the ST process.
\citeA{osamura2018using} used the weighted sum of embedding vectors for ASR word hypotheses based on their posterior probabilities.
\citeA{sperber2017toward} and \citeA{xue2020robust} showed that the translation accuracy against erroneous speech transcriptions can be improved by introducing pseudo ASR errors in the data for training the MT model.

KD \cite{bucilua2006model,hinton2015distilling} is a method of transferring knowledge from a teacher model to a student model.
Typically, the student model is trained by minimizing the Kullback--Leibler (KL) divergence \cite{kullback1951information} loss between the output probability distributions of the teacher and student models (word-level KD; WKD).
Sequence-level KS (SKD) \cite{kim-rush-2016-sequence} targets the token-sequence generated by the teacher model using a beam search.

The KD technique is extensively used in many machine learning applications, including MT (non-autoregressive MT \cite{gu2017non} and simultaneous translation \cite{ren-etal-2020-simulspeech}).
Typically, it is used to distill knowledge from a larger teacher model to a smaller or faster student model.
Recent studies \cite{furlanello2018born,yang2018knowledge} have shown that the translation accuracy of the student model exceeds that of the teacher model in KD, even when they have the same size.
KD has also been employed in ST.
\citeA{gaido2020knowledge} used KD to an end-to-end ST using an MT model based on clean transcriptions as the teacher of the end-to-end ST model.
We focused on employing KD in cascade ST using a teacher MT model based on clean transcripts for a student MT model that uses erroneous inputs.
This differs from the approach of \citeA{gaido2020knowledge} in that the teacher and student models have the same architecture and allow the use of FT, a standard machine learning technique.
KD and FT can be adapted to the student model simultaneously or sequentially.
In this study, we also investigated the possibility of combining KD and FT for cascade ST.

\citeA{dakwale2019improving} proposed distillation as a solution for effectively using noisy parallel data for MT.
First, they trained the teacher model using only high-quality, clean data.
Subsequently, they fed the source-side of the noisy parallel data into the teacher model and trained the student model to translate using pairs of original source language sentences and their translations using the teacher model.
The main difference between their study and the present one is that we employ loosely equivalent source sentences (clean or erroneous transcriptions), which can be paired with the same target sentence.
Therefore, the student model can be trained with highly reliable objectives obtained by feeding clean transcriptions to the teacher model.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{MT of Cascade ST}

First, we formalize the objective functions of the MT model of cascade ST.
Suppose $W=(w_1, ..., w_J)$, $X=(x_1, ..., x_K)$, and $Y=(y_1, ..., y_L)$ denote sequences of the speech features in the source language, the corresponding transcribed source language tokens, and translated target language tokens.

In the cascade ST, first an ASR model is trained first using $W$ and $X$.
Subsequently, an MT model is trained to translate $X$ to $Y$.
We define the loss function of the MT model, $\mathcal{L}_{MT}$, as the cross entropy between $Y$ and the MT model output, $Y'=({y'}_{1},...,{y'}_{M})$, as follows:
\begin{equation}
  \mathcal{L}_{MT} = -\sum_{l=1}^{L}\sum_{v \in V}^{|V|} \delta(y_l,v) \log p(v|X, {y'}_{<l};\theta),
\end{equation}
where $p(v|X, {y'}_{<l};\theta)$ is the posterior probability of candidate $v$ in target language vocabulary $V$ at time $l$ using the MT model with the parameter $\theta$.
$\delta(y_l,v)$ is an indicator function that returns 1 if $y_l=v$ and 0 otherwise.

When training the MT model, we can also use an ASR output $\hat{X}$.
In this paper, we propose using MT models trained on inputs $X$ and $\hat{X}$ as the teacher and student MT models, respectively.
We introduce a KD loss $\mathcal{L}_{KD}$ between the teacher and student MT model outputs,  $Y'$ and $\hat{Y}'=(\hat{y}'_{1},...,\hat{y}'_{N})$, respectively, as folows:
\begin{equation}
  \label{lkd}
  \mathcal{L}_{KD} = -\sum_{m=1}^{M}\sum_{v \in V}^{|V|} \Bigl( (1-\lambda)\mathcal{L}_{SKD} + \lambda\mathcal{L}_{WKD} \Bigr),
\end{equation}
where $\mathcal{L}_{SKD}$ and $\mathcal{L}_{WKD}$ are SKD and WKD losses defined by the cross entropy and the KL-divergence, respectively, as follows:
\begin{gather}
  \label{skd}
  \mathcal{L}_{SKD} = \delta(y'_m,v) \log p(v|\hat{X}, \hat{y}'_{<m};\theta), \\
  \label{wkd}
  \mathcal{L}_{WKD} = p(v|X, {y}'_{<m};\theta_\text{teacher}) \log \frac{p(v|X, {y}'_{<m};\theta_\text{teacher})}{p(v|\hat{X}, \hat{y}'_{<m};\theta)}.
\end{gather}
Minimizing $\mathcal{L}_{SKD}$ promotes the student model to make the same prediction as the teacher model, given the same context ($\hat{y}'_{<m}$).
However, minimizing $\mathcal{L}_{WKD}$ causes the student model to derive the same output distribution as the teacher model.
$\theta_\text{teacher}$ is a parameter of the teacher MT model. 
$\theta$ is a parameter of the student MT model, and is iteratively updated by using the loss, $\mathcal{L}_{KD}$.
$\lambda$ is a hyperparameter that ensures balance between $\mathcal{L}_{SKD}$ and $\mathcal{L}_{WKD}$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed Method}
\label{sec:proposal}

We refer the model trained with clean input as $X$ $\mathrm{MT_{clean}}$ (Fig. \ref{fig:overview}(a)) and that with ASR-based input as $\hat{X}$ $\mathrm{MT_{asr}}$ (Fig. \ref{fig:overview}(b)).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[b]
\begin{center}
\includegraphics{29-2ia3f1.pdf}
\end{center}
\caption{Overview of key concepts of methods}
\label{fig:overview}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

To most effectively exploit both clean input $X$ and ASR-based input $\hat{X}$, we introduce two training techniques: KD and FT.
In KD, first, the teacher model is trained using $X$, and subsequently the student model is trained using $\hat{X}$ by minimizing the loss, $\mathcal{L}_{KD}$.

FT has been commonly used for domain adaptation in MT \cite{sennrich-etal-2016-improving}.
\citeA{di2019robust} showed that a model fine-tuned with an ASR-based input becomes robust to erroneous ASR inputs while ensuring high performance using clean inputs.
Considering this finding, we employ FT for MT training.
In FT, the student model with $\hat{X}$, which inherits the parameters of the teacher model with $X$, is trained by minimizing $\mathcal{L}_{MT}$.

In addition, we propose the joint use of FT and KD in the training of the ST model. 
The simplest approach to use FT and KD is to apply them simultaneously (\textbf{FT+KD}).
We also consider strategies to use them independently and apply them sequentially.
FT and KD are different tasks, and it is unclear which should be performed first.
The method of performing FT after KD (\textbf{KD→FT}) can be regarded as a type of curriculum learning, and it may be possible to efficiently learn the translations of colloquial utterances.
However, performing FT first (\textbf{FT→KD}) may have the advantage that the model can determine the training direction early in realistic cases.
The details of the three variants of the proposed method are as follows:
\begin{itemize}
 \item \textbf{FT+KD}. The two techniques are applied simultaneously. Unlike regular FT, we use loss $\mathcal{L}_{KD}$ instead of $\mathcal{L}_{MT}$. This is implemented by the following two steps:
 \begin{enumerate}
    \item The teacher model, $\mathrm{MT_{clean}}$, is trained using clean input $X$ and $\mathcal{L}_{MT}$, which is the cross-entropy loss between the model output, $Y'$, and the target text, $Y$.
    \item The student model, $\mathrm{MT_{asr}}$, is fine-tuned with ASR-based input $\hat{X}$ and $\mathcal{L}_{KD}$, which is the KD loss between model output $\hat{Y}'$ and $Y$, inheriting the parameters of $\mathrm{MT_{clean}}$.
 \end{enumerate}
 \item \textbf{KD→FT}. FT is conducted with $\mathcal{L}_{MT}$ applied to the model trained with $\mathcal{L}_{KD}$.
 This is represented by the following three steps:
 \begin{enumerate}
     \item Model $\mathrm{MT_{clean}}$ is trained using $X$ and $\mathcal{L}_{MT}$.
     \item Model $\mathrm{MT_{asr}}$ is trained using $\hat{X}$ and $\mathcal{L}_{KD}$.
     \item Model $\mathrm{MT_{asr}}$ is trained using $\hat{X}$ and $\mathcal{L}_{MT}$.
 \end{enumerate}
\item \textbf{FT→KD}. FT is performed with $\mathcal{L}_{KD}$ on the model trained with $\mathcal{L}_{MT}$.
This is implemented in the following three steps.
\begin{enumerate}
    \item Teacher model $\mathrm{MT_{clean}}$ is trained using $X$ and $\mathcal{L}_{MT}$.
    \item Student model $\mathrm{MT_{asr}}$ is trained using $\hat{X}$ and $\mathcal{L}_{MT}$, inheriting the parameters of the teacher model.
    \item Student model $\mathrm{MT_{asr}}$ is trained using $\hat{X}$ and $\mathcal{L}_{KD}$.
\end{enumerate}
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}

We conducted the experiments to investigate the effectiveness of the proposed method from three viewpoints: differences in the dataset, preprocessing scheme, and types of KD.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Datasets}

We conducted experiments for English--Italian and Spanish--English NMT and investigated the differences in the effectiveness of the proposed method on different datasets.
Table \ref{table:data-statistics} presents the data statistics.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[t]
\input{03table01.tex}
\hangcaption{Number of sentences in the datasets, including training (Train), development (Dev), and test data (Test).}
\label{table:data-statistics}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{MuST-C English--Italian}

In the first experiment, we used the MuST-C \cite{di-gangi-etal-2019-must} dataset, which is a multilingual ST corpus built from TED Talks.
It contains triplets of approximately 250K segments of English speech recordings and transcripts and Italian translations.
Audio and transcript pairs were used to train the ASR model.
To train the MT model, we used transcripts as the clean input and ASR outputs as the noisy input.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Fisher Spanish--English}

In the second experiment, we used the LDC Fisher Spanish speech with new English translations dataset \cite{251fd6c5479b4ae9859dc8f650cf04e5,Salesky_2018}.
It has the following approximately 140K segments of multidirectional parallel data:
\begin{enumerate}
    \item Spanish disfluent speech
    \item Spanish clean transcriptions
    \item Spanish erroneous transcriptions (ASR output)
    \item English disfluent translations
    \item English fluent translations
\end{enumerate}
It contains Spanish telephone conversation speeches and English translations, which include more colloquial expressions than those in the first experiment.
To train the MT model, (5) was used as the output.
For reproducibility, we used (2) and (3) as the clean and noisy inputs, respectively, in the dataset.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Truecasing and Punctuation Restoration}

\citeA{di2019robust} showed that discrepancies in the input formats during the training and inference stages (e.g., presence of punctuation) severely degrade the performance.
They also showed that such a condition mismatch between pre-training and FT could be a cause of reduced accuracy.
Following these findings, we also investigated the impact of the format inconsistency across training and inference or within the training data for the MT model of cascade ST.
In MuST-C English--Italian experiments, we used the following three variants of the MT input preprocessing scheme:

\begin{itemize}
    \item \textbf{ASR-p/Clean-p}. ASR provides the speech transcription in truecasing with punctuation to ensure consistency with clean input (e.g., an ASR-based input is ``Hi there," and the clean input is ``Hi, there.").
    \item \textbf{ASR-np/Clean-p}. ASR provides the speech transcription in lowercases without punctuations (e.g., an ASR-based input is ``hi there" and the clean input is ``Hi, there.").
    \item \textbf{ASR-np/Clean-np}. ASR provides speech transcription in lowercases without punctuations. To overcome the inconsistency with the ASR-based input, we lowercased and removed punctuation marks from the clean input (e.g., an ASR-based input is ``hi there" and the clean input is ``hi there").
\end{itemize}
The input format in the Spanish--English experiments was the same as in ASR-np/Clean-p, because we used option (3) punctuation-free ASR outputs and option (2) clean transcriptions that included punctuation marks and capital letters in the Fisher dataset.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Other Settings}

We preprocessed the text data using byte-pair encoding (BPE) \cite{sennrich-etal-2016-neural} for subword segmentation.
The vocabulary size was set as 8,000 for all the languages.
For the English audio, we extracted 80-channel log mel filterbank features (window size 25~ms and window shift 10~ms) and applied an utterance-level cepstral mean and variance normalization.

To evaluate the performance, we calculated the case-sensitive bilingual evaluation understudy (BLEU) scores using sacreBLEU.\footnote{\url{https://github.com/mjpost/sacreBLEU}}
We measured the BLEU scores for both the ASR-based and clean inputs to evaluate the ASR error robustness and determine the best performance in an ideal scenario of no ASR errors.
We conducted significance tests on the BLEU scores between the baselines and each method using the bootstrap resampling method \cite{koehn-2004-statistical} implemented in util-scripts.\footnote{\url{https://github.com/neubig/util-scripts/blob/master/paired-bootstrap.py}}
The significance level was 5\%, sample size was 50\% test data, and number of trials was 1000.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Models}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Notation}

The MT models used in the experiments are distinguished by the type of training data as follows:
\begin{itemize}
  \item $\mathrm{MT_{clean}}$ is trained using a pair ($X, Y$).
  \item $\mathrm{MT_{asr}}$ is trained using ($\hat{X}, Y$).
  \item $\mathrm{MT_{asr+clean}}$ is trained using mixed data of ($X, Y$) and ($\hat{X}, Y$).
\end{itemize}
Two training techniques were applied to some models:
\begin{itemize}
  \item FT. Additional training with ($\hat{X}, Y$).
  \item KD. Training with ($\hat{X}, Y'$), where $Y'$ is the target text from the teacher MT model, $\mathrm{MT_{clean}}$.
\end{itemize}
The combinations of FT and KD are described in Section \ref{sec:proposal}.
\subsubsection{Types of KD Loss}
As expressed in equation \ref{lkd}, we can control the weights of the SKD loss ($\mathcal{L}_{SKD}$) and WKD loss ($\mathcal{L}_{WKD}$) during training using a hyperparameter $\lambda$.
In the Spanish--English experiments, $\lambda$ was taken as 0, 0.2, 0.4, 0.6, 0.8, and 1.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Implementation}

We used the Transformer \cite{vaswani2017attention} implementation of Fairseq\footnote{\url{https://github.com/pytorch/fairseq}} to construct both the ASR and the MT models.
The hyper-parameters of the models generally follow the Transformer Base settings \cite{vaswani2017attention}.
Each encoder and decoder has six sublayers. We set the word embedding dimensions, the hidden state dimensions, and the feed-forward dimensions as 512, 512, and 2,048, respectively.
We performed dropout of a sublayer with probability 0.1 and employed eight attention heads for both the encoder and the decoder.
The model was trained by Adam optimization with an initial learning rate of 0.0007, $\beta_1=0.9$, and $\beta_2=0.98$, following \citeA{vaswani2017attention}.
We used 4,096 tokens per mini-batch and eight iterations of forward-passes, accumulated gradients, and back-propagated them.
Validation was performed every 1,000 updates, and the test checkpoint with the best loss was stored.

For the English-Italian experiments, we also built several end-to-end ST variants using Fairseq for comparison with the cascade models. 
All settings were identical to those in MT: use of Transformer, as described above, and training with a label-smoothed cross entropy loss.

The text-to-text (MT) and speech-to-text (ASR and end-to-end ST) models took approximetly 3~h and half a day to train on two NVIDIA RTX 2080 Ti GPUs, respectively.
Some MT models involved additional training using FT or KD, which required 10--20~min.
During testing, beam search decoding with a beam size of 4 for the test data required approximately 5~min for all models, including the three variants of the proposed method.
No language model was used for ASR decoding.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Main Results} \label{subsec-mainres}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{MuST-C English--Italian:}
Table \ref{table:whole-result-mustc} summarizes the overall results for the English--Italian NMT.
ASR-p/Clean-p was used for preprocessing, and $\lambda$ was set as zero.
Among the end-to-end systems, the simple model (E1) without any additional technique, such as multitask training, had a much lower performance than the others and was significantly improved by pre-training the ASR encoder (ASR-PT) (E1$\rightarrow$E2).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[b]
\input{03table02.tex}
\hangcaption{ST systems on MuST-C English--Italian dataset. Test BLEU scores are reported.
$\dag$ and $\ddag$ denote significantly higher than $\mathrm{MT_{asr}}$ and $\mathrm{MT_{asr}}+\mathrm{FT}$, respectively ($p<0.05$).
$^1$End-to-end (above) or cascade ST (below) systems using Transformer Base model of Fairseq, which resembles our conditions.
$^2$End-to-end ST system using ESPnet resembles our conditions chosen from a report (\protect\url{https://github.com/espnet/espnet/blob/master/egs/must_c/st1/RESULTS.md}).}
\label{table:whole-result-mustc}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The cascade methods performed better than the end-to-end methods.
Among the cascade ST systems, the performance of that trained using only ASR input (C2) was worse than that trained using only clean input (C1).
The BLEU scores decreased by 0.3 and 2.5 points for the ASR-based and clean inputs, respectively.
The ASR-based training data contained erroneous transcriptions with a word error rate (WER) of 14.49, leading to degradation.

During the evaluation, a significant decrease in the accuracy was due to the erroneous transcriptions.
For example, in C1, there was a considerably large difference between the ASR-based input and clean input cases: 22.4 and 29.7.
One typical cause is the corruption of the meaning of the original text, e.g., mistaking the proper noun, ``LEGO," for ``Legal" and the noun phrase, ``main groups," for ``mangroves."
In contrast, some systems trained using both the ASR and clean inputs were better than C1 when translating clean input.
This suggests that the training with ASR errors can contribute to regularization of the model, leading to improvement.

FT for the ASR-based input (C3) showed improvements for the ASR-based input (+1.1, C2$\rightarrow$C3).
Compared with FT, KD (C6) achieved a small improvement using the ASR-based input (+0.4, C2$\rightarrow$C6).
In KD, the teacher model obtained a BLEU score of 41.6 on the reference for the training data.

C4 is a simple mixture model trained using the ASR-based and clean inputs. 
C4 learned both types of inputs compared with C1 and C2 and performed better than C1 for the ASR input and C2 for the clean input.
FT led to further improvement, as shown by the results of C3 and C5.

With respect to the joint use of FT and KD,
simultaneously applying these techniques (C7) showed only a slight improvement compared to using only FT (C3$\rightarrow$C7); the BLEU scores increased by 0.2 and 0.1 points with the ASR-based and clean inputs, respectively.
Applying FT after KD (C8) was inferior to the other combinations (C7, C9), particularly on clean data, probably because the MT model was not trained with the clean input.
KD after FT (C9) yielded the best scores on both the ASR-based and the clean test data.
FT enables the student model to learn good parameter values, and KD provides the student model with its upper bounds from the teacher model.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[b]
\input{03table03.tex}
\hangcaption{ST systems on Fisher Spanish--English dataset. Test BLEU scores for two fluent references are reported. 
$\dag$ and $\ddag$ denote significantly higher than $\mathrm{MT_{asr}}$ and $\mathrm{MT_{asr}}+\mathrm{FT}$, respectively ($p<0.05$).
}
\label{table:whole-result-fisher}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Fisher Spanish--English:}

Table \ref{table:whole-result-fisher} summarizes the overall results for the Spanish--English cascade ST.
The results were similar to those of the English--Italian case.
FT and KD improved the BLEU scores, and combining them yielded more significant improvements than when employing them separately.
However, the gap was larger for the clean test data between systems only trained only on the ASR-based input (C2) and only on the clean input (C1).
The ASR-based training data contained many erroneous transcriptions with a WER of 36.5, causing more severe degradation than the clean input.
The Spanish--English experiments also differ from the English--Italian experiments in that KD (C6) is superior to FT (C3) on ASR-based test data when used alone.
In KD, the BLEU score using the teacher model as the training data was 48.0, which is
higher than that of 41.6 for the English--Italian experiment.
One possible cause is that a higher upper bound can be trained by KD.
Another difference was the gap between the clean and ASR-based inputs, which had many erroneous transcriptions with a WER of 36.5.
In this case, parameter initialization by FT may not be very helpful.

In spite of the differences in the speech domain between the two experiments, we achieved consistent improvements by combining FT and KD.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Results with Higher WER Conditions}

In the MuST-C English--Italian experiments, we investigated the effectiveness of our method for higher WERs of the ASR output using top-p sampling, in which probabilities are acumulated in order from the most probable candidates.
An output is sampled from the largest possible set of candidates whose cumulative probability does not exceed $p$.
We performed top-p sampling with $p=0.95$ and $p=1$, instead of choosing the highest probability in beam search decoding of the ASR model.
Each ASR result contains erroneous transcriptions with WERs of 24.18 (high), 31.75 (very high), and higher than 14.49 (base).
Table \ref{table:wer-conditions} summarizes the BLEU results for the English--Italian NMT under different WER conditions.
The contributions of FT in each WER condition (base, high, and very high) were 1.1, 1.4, and 1.7 BLEU improvements on the ASR-based input, respectively (C1$\rightarrow$C2). 
With KD, the corresponding improvements were 0.4, 0.3, and 0.8 (C1$\rightarrow$C3).
For the joint use of FT and KD, applying KD after FT was the best under all conditions, with improvements of 1.4, 1.7, and 1.7, respectively (C1$\rightarrow$C6).
Both FT and KD improved the BLEU under all WER conditions, and their combination provided the same or a better improvement on the ASR-based input.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[t]
\input{03table04.tex}
\hangcaption{Dependence of BLEU scores of cascade ST systems with MuST-C English--Italian test set on WER condition.}
\label{table:wer-conditions}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[t]
\input{03table05.tex}
\hangcaption{Dependence of BLEU scores of cascade ST systems with MuST-C English--Italian test set on preprocessing schemes.}
\label{fig:scheme}
\vspace{-0.75\Cvs}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Truecasing and Punctuation Restoration} \label{preprocessing}

Table \ref{fig:scheme} summarizes the BLEU results for the English--Italian NMT for different preprocessing schemes applied for truecasing and punctuation restoration.
ASR-p/Clean-p achieved the best performance among all models with both the ASR-based and clean inputs.
ASR-np/Clean-p showed that an inconsistency between the training and inference times causes performance degradation.
For example, C1, which was trained only with the clean input (including punctuation marks and capital letters), 
\pagebreak
had an extremely small BLEU in the evaluation with the ASR-based input.
ASR-np/Clean-p was also inferior to ASR-p/Clean-p when the models were trained using both the ASR-based and clean inputs (C3).
These results demonstrate that the consistency of the input format across the training and inference times and within the training data are important for MT, similar to the claims of \citeA{di2019robust}.

ASR-np/Clean-np was also consistent in its input format.
However, its BLEU scores were lower than those of ASR-p/Clean-p.
These results are similar to the trends of \citeA{di2019robust}, suggesting that (1) punctuation marks and capital letters in clean human transcripts are essential for MT, (2) it is better for ASR to predict punctuation even if some prediction errors occur, to ensure consistency, and (3) it is difficult to recover capital letters by ASR and predict punctuations using MT.

We also found that the consistency in the input format is important for KD.
Under the ASR-np/Clean-p condition, the model trained with KD (C4) was inferior to that without it (C2).
This result contradicts the Spanish--English experiments.
We used the punctuation-free ASR output provided by the Fisher dataset in the Spanish--English experiments, and found KD to still be effective.
This can be attributed to differences in the speech domains of the datasets.
Punctuation marks are inessential in the Fisher dataset, which is a colloquial dialogue dataset, whereas they are critical in the MuST-C dataset, a relatively literal dataset.
A detailed comparison (including possible combinations of KD and FT) is provided in Appendix.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Types of KD Loss}

Table \ref{table:wkd-result-fisher} summarizes the dependence of the BLEU scores on the balance between the SKD and WKD losses for the Fisher Spanish--English dataset.
A smaller $\lambda$ implies a good BLEU score using the ASR-based input, indicating that SKD is more effective than WKD, in the regression model.
This follows the inference of \citeA{kim-rush-2016-sequence}: ``the sequence distribution is particularly important for NMT, because wrong predictions can propagate forward at test time.''

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[t]
\input{03table06.tex}
\hangcaption{Dependence of BLEU scores of cascade ST systems on the Fisher Spanish--English test sets on balance between SKD and WKD losses.}
\label{table:wkd-result-fisher}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}

We analyzed the results obtained using a Spanish--English dataset to discuss how erroneous transcriptions affect translation results and how KD and FT work.
\paragraph{Erroneous Transcription:}
The example below elucidates the problem of error propagation.
\begin{itemize}
    \setlength{\parskip}{0pt} % 段落間
    \setlength{\itemsep}{5pt} % 項目間
    \item (Clean input) \textit{uno super, super nuevo que salio}
    \item (ASR output) \textit{en un \underline{sur} super nuevo que salio}
    \item (Reference) \textit{One super new that came out}
    \item ($\mathrm{MT_{asr}}$ with ASR-based input) \textit{In the \underline{South}, it came out}
    \item ($\mathrm{MT_{asr}}+\mathrm{KD}$ with ASR-based input) \textit{In a super new one that came out.}
\end{itemize}
Here, the Spanish word, \emph{super}, was misrecognized as \emph{sur} by the ASR model.
This error propagated to the MT model, and $\mathrm{MT_{asr}}$ translated it as \emph{South}.
Although the lexical translation from \emph{sur} to \emph{South} was correct, it was not what was desired.
The model trained by KD ignored this error and generated a better translation.

We observed such ASR error correction phenomena in the results, although KD and FT do not directly address this problem.

In the following example, none of the MT models recovered the ASR error.
\begin{itemize}
    \setlength{\parskip}{0pt} % 段落間
    \setlength{\itemsep}{5pt} % 項目間
    \item (Clean input) \textit{para la computadora, tiene eh para mover el mouse}
    \item (ASR output) \textit{para la computadora tiene eh para mover el \underline{m\'{a}s}}
    \item (Reference) \textit{For the computer, it has to move the mouse.}
    \item ($\mathrm{MT_{asr}}+\mathrm{FT}\rightarrow\mathrm{KD}$ with ASR-based input) \textit{The computer has to move the \underline{most}}
\end{itemize}
Here, \textit{mouse} was mistaken for the word, \textit{m\'{a}s}, which has a similar pronunciation.
The same error was not included in the training data. \\
In addition, ASR results with pronunciations far from the correct transcriptions were incorrectly translated, as exemplified below.
\begin{itemize}
    \setlength{\parskip}{0pt} % 段落間
    \setlength{\itemsep}{5pt} % 項目間
    \item (Clean input) \textit{de, University of Pennsylvania tambi\'{e}n.}
    \item (ASR output) \textit{eh d\'{\i}game ochenta y suce\~{n}a tambi\'{e}n}
    \item (Reference) \textit{From, University of Pennsylvania too.}
    \item ($\mathrm{MT_{asr}}+\mathrm{FT}\rightarrow\mathrm{KD}$ with ASR-based input) \textit{Tell me, eighty Maria too.}
\end{itemize}
These results suggest that the proposed method can recover the ASR error only if the training data contain similar errors and the pronunciation is close to the correct transcription.
In the following, we discuss the individual functions of KD and FT.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Effect of Knowledge Distillation:}
The spoken language parallel data contained translations of colloquial spoken utterances, which increased the difficulty of training the MT model, e.g,
\begin{itemize}
    \setlength{\parskip}{0pt} % 段落間
    \setlength{\itemsep}{5pt} % 項目間
    \item (Clean input) \textit{le ayuda si si, no es, no es interesante pero entonces, a ba- entonces ya des- pues cuando eso termino, tiene que escribir varios asi, ensayos, hacer un analisis}
    \item (Reference) \textit{You have to write some essays like that, to make an analysis}
    \item (KD teacher) \textit{It helps her yes, it's not interesting but then, when I finish, you have to write several, you have to make an analysis}
\end{itemize}
A human translator ignored many disfluent utterances from the original text, resulting in low fidelity.
Some other examples are as follows: 
\begin{itemize}
    \setlength{\parskip}{0pt} % 段落間
    \setlength{\itemsep}{5pt} % 項目間
    \item Inconsistent translations: ``\textit{De Venezuela}" was translated to ``\textit{From Venezuela}" at one time and to ``\textit{Venezuela?}" on another occasion.
    \item All-caps: ``\textit{donde hay problemas}" was capitalized and translated into ``\textit{WHEN TROUBLE ARISES}."
    \item Omission of a part of speech: ``\textit{Porque, tengo el, el bodysuit, pero}" was translated into ``\textit{I have the bodysuit.}." The conjunction ``\textit{pero} (but)" was removed to ensure fluency.
\end{itemize}
Such translations confused the MT model.
KD forced the student model to mimic literal teacher translations that may include some errors instead of reproducing translations of the colloquial spoken utterances.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Effect of Fine-tuning:}
Occasionally the fine-tuned MT model corrected the ASR errors, e.g.,
\begin{itemize}
    \setlength{\parskip}{0pt} % 段落間
    \setlength{\itemsep}{5pt} % 項目間
    \item (Clean input) \textit{Eh, para mi pues, eh, tengo como diez mil canciones en, en el, en la \underline{Ipod}}
    \item (ASR output) \textit{eh para mi pues eh tengo como diez mil canciones en en la \underline{epod}}
    \item (Reference) \textit{I have ten thousand songs in the \underline{Ipod}.}
    \item ($\mathrm{MT_{clean}}$ with ASR-based input) \textit{To me, I have about ten thousand songs in the \underline{ethics}}
    \item ($\mathrm{MT_{asr}}+\mathrm{FT}$ with ASR-based input) \textit{I have about ten thousand songs in the \underline{Ipod}}
\end{itemize}
The ASR model misrecognized ``\textit{Ipod}" as ``\textit{epod}," and the model before FT, which was only trained with the clean input, incorrectly translated it as ``\textit{ethics}."
On FT with the ASR-based inputs, the model successfully translated it as ``\textit{Ipod}."
FT probably provided robustness against erroneous ASR outputs leading to common errors.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

In this paper, we proposed the combined use of KD and FT in the training of the MT model in cascade ST.
Our experimental results showed the advantages of the proposed method on two different datasets: MuST-C and Fisher.
Furthermore, we revealed that the consistency in the formats of the clean and ASR-based inputs is essential and that SKD is more effective than WKD.
In the future, we will incorporate our findings into an end-to-end ST system.
Furthermore, we will also investigate the impact of the proposed method on distant language pairs and incorporate external language models.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

This paper is an extended version of that published in the Proceedings of the 18th International Conference on Spoken Language Translation \cite{fukuda-etal-2021-knowledge},
following further investigation on word-level KD and preprocessing for truecasing and punctuation restoration.
A part of this work was supported by JSPS KAKENHI (Grant Number JP21H05054).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{03refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\appendix

\section{Preprocessing Scheme}


Table \ref{fig:scheme-all} compares the preprocessing schemes for MT input on the MuST-C English--Italian dataset.
As mentioned in section \ref{preprocessing}, ASR-p/Clean-p achieves the best performance among all models with both ASR-based and clean inputs.


\clearpage

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 7
\begin{table}[t]
\input{03table07.tex}
\caption{Comparisons of preprocessing schemes for MT input on the MuST-C English--Italian dataset.}
\label{fig:scheme-all}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Ryo Fukuda}{%
He is a Ph.D. student at the Nara Institute of Science and Technology. He received his bachelor's degree in engineering from Toyohashi University of Technology in 2019 and his master's degree in engineering from the Nara Institute of Science and Technology in 2021.
}

\bioauthor[:]{Katsuhito Sudoh}{%
He is an associate professor at the Nara Institute of Science and Technology. He received his bachelor's degree in engineering in 2000 and his master's and Ph.D. degrees in informatics in 2002 and 2015, respectively, from Kyoto University. He previously worked at the NTT Communication Science Laboratories from 2002 to 2017. He is currently working on machine translation and natural language processing. He is a member of the ACL, ANLP, IPSJ, ASJ, and JSAI.
}

\bioauthor[:]{Satoshi Nakamura}{%
He is a professor at the Nara Institute of Science and Technology, the team leader of the Tourism Information Analytics Team, AIP Center, RIKEN, and an honorary professor at the Karlsruhe Institute of Technology, Germany. He received his B.S. degree from the Kyoto Institute of Technology in 1981 and a Ph.D. degree from the Kyoto University in 1992. He was the Director of the ATR Spoken Language Communication Research Laboratories in the period 2000--2008 and Vice President of ATR during 2007--2008. He was the Director General of the Keihanna Research Center, National Institute of Information and Communications Technology in 2009--2010. He is currently the Director of the Augmented Human Communication laboratory and a full Professor at the Data Science Center and Information Science Division, Graduate School of Science and Technology, Nara Institute of Science and Technology. He was an Elected Board Member of the International Speech Communication Association (ISCA) during June 2011--2019, IEEE Signal Processing Magazine Editorial Board member in 2012--2015, and IEEE SPS Speech and Language Technical Committee Member in 2013--2015. He is an ATR, IPSJ, IEEE, and ISCA Fellow.
}

\end{biography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate




\end{document}


