\documentclass[english]{jnlp_3.8}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{array}
\allowdisplaybreaks

\usepackage{hangcaption_jnlp}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition

%%% by sudoh
%%%%%\usepackage{color}
%%%%%\newcommand{\blue}[1]{\textcolor{blue}{#1}}
\usepackage{bm}
%%%%%\newcommand{\argmax}{\mathop{\mathrm{argmax}}}
%%%%%\usepackage{amssymb,amsmath}
%%%%%\usepackage{amsfonts} 
%%%%%\usepackage{bbm}
%%%%%\usepackage{colortbl}
%%%%%\usepackage{multirow}
%%%%%\usepackage{multicol}

%%%%%\usepackage{graphicx}


% by kano
%%%%%\usepackage{subcaption}
%%%%%\usepackage{here}
% \usepackage[utf8]{inputenc}
% \usepackage{natbib}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{31}
\Number{3}
\Month{September}
\Year{2024}
%%%%\Issuetitle{「　　　　」特集号}

\received{2023}{11}{28}
\revised{2024}{3}{12}
\accepted{2024}{4}{26}




\setcounter{page}{1049}


%%%%   Title, Author, etc.            %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Average Token Delay: A Duration-aware Latency Metric for Simultaneous Translation}
\eauthor{Yasumasa Kano\affiref{Author_1} \and Katsuhito Sudoh\affiref{Author_1} \and Satoshi Nakamura\affiref{Author_1}}
\eabstract{In simultaneous translation, translation begins before the end of an input speech segment. Its evaluation should be conducted based on latency and quality. For users, the smallest possible latency is preferable. Most existing metrics measure latency based on the start timings of partial translations and ignore their duration. This implies that such metrics do not penalize the latency caused by a long translation output, which delays user comprehension and subsequent translations. In this paper, we propose a novel latency evaluation metric for simultaneous translation called the \emph{Average Token Delay} (ATD), which focuses on the duration of partial translations. We demonstrate its effectiveness through analyses that simulate user-side latency based on the Ear-Voice Span (EVS). In our experiments, ATD had the highest correlation with EVS among the baseline latency metrics under most conditions. These results suggest that ATD provides a more accurate evaluation of latency.
}

\ekeywords{Simultaneous Translation, Latency Evaluation}

\headauthor{Kano et al.}
\headtitle{Average Token Delay}

%%%%     Affiliation     %%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\affilabel{Author_1}{}{Nara Institute of Science and Technology}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S1
\section{Introduction}\label{sec:introduction}
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
\footnote[0]{This paper is an extended version of our conference paper \cite{kano2023average} with additional experiments and analyses. A preprint of this paper was posted on arXiv (\url{http://arxiv.org/abs/2311.14353}).}
\renewcommand{\thefootnote}{\arabic{footnote}}
Machine translation (MT) has evolved rapidly due to recent neural network techniques and is now widely used for both written and spoken languages.
MT for speech is used to translate conversations, lectures, and talks.
In situations that require real-time communication, MT should function simultaneously, 
without waiting for the conclusion of the speech.
This task is often referred to as \emph{simultaneous machine translation} (SimulMT).
Hereafter,  the term SimulMT will cover both text and speech inputs.

One crucial challenge in SimulMT is the quality-latency trade-off.
Although a longer input with a longer wait time can improve translation quality, it also creates more latency and vice versa.
In SimulMT studies, researchers evaluate models considering this quality-latency trade-off.
Whether translation latency should be minimized per word, phrase, or sentence depends on the situation.
If the speaker makes a gesture, such as pointing at something, the relationship between the gesture and the word is important, so the translation latency per word should be small.
However, there are also cases where the latency per word is not very important; only the translation latency of the entire sentence matters.

In the research of human interpretation, Ear-Voice Span (EVS) has been used to evaluate latency.
The EVS is basically the time difference between the source and the corresponding target word, although variations can be found \cite{elisa2019}.
By averaging the local latency of EVS in a sentence, the mean EVS includes the viewpoints of both the local latency for each word and the overall latency for each sentence.

Figure \ref{fig:EVS_AL} illustrates an example of the calculation of the mean EVS.
For both cases, the first input chunk is translated into the first output chunk, and the second is translated next.
In this example, EVS is represented as the time distance between the black lines connecting the source and the semantically corresponding target word.
When we calculate the mean values of these EVSs, Case 1 has a smaller value.
Therefore, the latency in Case 1 is lower than in Case 2.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F1
\begin{figure}[b]
\begin{center}
\includegraphics{31-3ia8f1.pdf}
\end{center}
\hangcaption{Example calculation of mean Ear-Voice Span (EVS): Black lines represent word alignment, and the time distance of the lines is EVS.
This example excludes word alignment for stop words and function words.
The mean EVS is smaller in case 1; AL is smaller in case 2.}
\label{fig:EVS_AL}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



EVS has not been used in SimulMT research.
One reason for this is that SimulMT research originated from text-to-text simultaneous translation, while EVS is a latency metric tailored for speech-to-speech simultaneous translation performed by human interpreters.

Hence, alternative latency metrics have been introduced to evaluate the latency of simultaneous text-to-text machine translation.
An example is Average Lagging (AL) \cite{ma-etal-2019-stacl}, which stands as the most commonly used latency metric.
The AL, based on the number of input words available when starting a translation, measures the average number of input words over the entire translation process.
It corresponds to wait-$k$ SimulMT \cite{ma-etal-2019-stacl}, which waits for $k$ input tokens before starting the translation and alternates between reading and writing one token.
In Figure \ref{fig:EVS_AL}, the AL is smaller in Case 2, which contradicts its mean EVS.
This comes from the definition of AL, which counterintuitively assigns a smaller latency value to a long output chunk at a given time step.
Such lengthy output chunks inevitably lead to delays in translating subsequent parts in speech-to-speech SimulST because speech synthesis cannot start a new output while still \emph{speaking}.
Even for closed caption output, long outputs within a short time frame impose a high cognitive load on users.
Therefore, considering that a translation output's length affects user experiences, this factor should be considered in latency measurements.
This observation suggests the need for another latency metric to address these situations effectively.

In this study, we propose a novel latency metric called the \emph{Average Token Delay} (ATD), which focuses on the end timings of partial translations in SimulMT.\footnote{ATD is implemented in \url{https://github.com/facebookresearch/SimulEval}.}
The ATD is calculated as the average distance between the source tokens and their corresponding target tokens, considering the duration of the tokens.
The calculation method for ATD is similar to that of EVS, thus including both local and entire latency viewpoints.
ATD generalizes latency measurements for both speech and text outputs, and it intuitively handles chunk-based outputs that are not properly addressed by AL, as discussed above.
ATD is much simpler than EVS, which involves a long process such as transcribing input and output speech, obtaining word timestamps, and aligning words by annotators.
We present simulation results that clarify ATD's characteristics and
demonstrate its effectiveness through SimulMT experiments, comparing it with baseline latency metrics.
Since no experiments have compared the latency metrics, no evaluation dataset exists.
Therefore, we created a small dataset and used it to compare the correlations between each latency metric and mean EVS.
In our experiments, ATD exhibited the highest correlation among the baseline latency metrics under most conditions.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S2
\section{Simultaneous Machine Translation}\label{sec:simulmt}
First, we review the SimulMT formulation to establish the notations used in this study.

In standard sentence-level NMT,
let $\bm{x} = x_1,x_2,...,x_m$ be an input sentence and $\bm{y} = y_1,y_2,...,${\linebreak}$y_n$ be its translation.
The output probability is denoted as
\begin{equation}
p(\bm{y}|\bm{x}) = \prod_{t=1}^{n} P(y_t | \bm{x}, \bm{y}_{<t}).
\end{equation}

SimulMT uses a prefix as the input for incremental decoding:
\begin{equation}
p(\bm{y}|\bm{x}) = \prod_{t=1}^{n} P(y_t | \bm{x}_{\leq g(t)},\bm{y}_{<t}),
\end{equation}
where $g(t)$ is a monotonic non-decreasing function that represents the number of input tokens read until the prediction of $t$-th output token $y_t$. Thus, $\bm{x}_{\leq g(t)}$ is an input prefix, and $\bm{y}_{<t}$ is the prefix translation predicted so far.
This implies that we can obtain a pair of input and the corresponding output prefixes $(\bm{x}_{\leq g(t)}, \bm{y}_{\leq t})$ at that time.

The incremental decoding can be represented by a sequence of actions: \textit{READ} and \textit{WRITE}.
\textit{READ} is an action that takes a new input, typically one token for the text input or a fixed number of frames for the speech input.
\textit{WRITE} is an action that predicts an output, typically one token for a text output or a corresponding speech signal for the speech output.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S3
\section{Existing Latency Metrics for Simultaneous Translation}
In this section, we review several latency metrics that have been proposed in the SimulMT field before introducing our proposed metric, ATD.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S3.1
\subsection{Text-to-Text Latency Metrics}
Most latency metrics were initially proposed for text-to-text conditions because early SimulMT models were developed under these conditions.
\citeA{gu-etal-2017-learning} proposed a latency metric called the Consecutive Wait length (CW), which
counts the number of consecutive waiting input tokens between two output tokens and measures the local delays in a sentence.
\citeA{cho-2016} proposed an Average Proportion (AP) metric that measures the average latency for an entire sentence.
However, the AP suffers from the following problem: the latency value differs depending on the input and output sequence lengths, even for the same READ-WRITE strategy.

\citeA{ma-etal-2019-stacl} proposed AL, which has recently become the most commonly used method,
denoted as follows:
\begin{equation}
AL_g(\bm{x}, \bm{y}) = \frac{1}{\tau_g(|\bm{x}|)} \sum_{t =1}^{\tau_g(|\bm{x}|)} \left( g(t) - \frac{t -1}{r} \right),
\label{eqn:AL_g}
\end{equation}
where $g(t)$ is a non-decreasing function, as described in Section \ref{sec:simulmt} and $r$ is the length ratio defined as $|\bm{y}|/|\bm{x}|$. $\tau_g(|\bm{x}|)$ is the cutoff step defined as
\begin{equation}
\tau_g(|\bm{x}|) = \min\{t \mid g(t) = |\bm{x}|\},
\label{eqn:tau_g}
\end{equation}
which denotes the index of the output token predicted immediately after the observation of the entire source sentence.

We present an example of the calculation of AL in Figures \ref{fig:AL_case1_timeview} and \ref{fig:AL_case1_actionview} for the wait-1 \cite{ma-etal-2019-stacl} strategy in SimulMT.
Wait-k \cite{ma-etal-2019-stacl} is a SimulMT strategy that reads the first $k$ source tokens before starting the translation and then outputs one target token and reads one source token alternately.
Figure \ref{fig:AL_case1_timeview} shows the time-synchronous view of wait-1, and Figure \ref{fig:AL_case1_actionview} shows the READ/WRITE actions.
The term $\left( g(t) - \frac{t -1}{r} \right)$ in Equation (\ref{eqn:AL_g}) is represented as the dotted line in Figure \ref{fig:AL_case1_actionview}.
Therefore, AL is calculated as the average distance between the ideal and actual policies, and $AL = (1+1+1+1)/4 = 1$.
AL with wait-k was proposed by \citeA{ma-etal-2019-stacl}, and AL equals $k$ for wait-k when the source and target lengths are the same.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F2
\begin{figure}[b]
\begin{center}
\includegraphics{31-3ia8f2.pdf}
\end{center}
\caption{Time synchronous view of wait-1}
\label{fig:AL_case1_timeview}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F3
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f3.pdf}
\end{center}
\hangcaption{Actions of wait-1: Thick solid red arrows represent the actual strategy of wait-1. Thin solid black arrows depict the ``ideal policy,'' which starts to write translation tokens before reading any source token. The dotted lines represent the distance between the actual strategy and the ideal policy.}
\label{fig:AL_case1_actionview}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



However, AL still suffers from an unintuitive latency measurement because it can be negative when the model finishes the translation before reading the entire input.
This occurs when $|\bm{y}| << |\bm{x}|$, resulting in $r << 1$ and the second term $(t-1)/r$ in the subtraction becoming too large.
To mitigate this problem, \citeA{ma-etal-2020-simuleval} modified AL by changing the calculation of the length ratio $r$ to $|\bm{y^*}|/|\bm{x}|$ based on the length of the reference translation $\bm{y^*}$.

In addition, long translation outputs also result in negative AL values.
This is illustrated in Figures \ref{fig:AL_case2_timeview} and \ref{fig:AL_case2_actionview}.
In Figure \ref{fig:AL_case2_timeview}, the first chunk of translation was long,
causing some thick solid red arrows to be above the ideal policy. Consequently, the distance between the actual strategy and the ideal policy becomes negative in certain areas according to Equation (\ref{eqn:AL_g}).
As a consequence, $AL = (1+0.5+0-0.5-1-1.5+1)/7 \approxeq -0.07$.
\citeA{papi-etal-2022-generation} proposed Length-Adaptive Average Lagging (LAAL), which modified $r$ to $max(|\bm{y}|,|\bm{y^*}|)/|\bm{x}|$ to appropriately evaluate the latency of a translation that is longer than the reference.
However, these modifications are not sufficient to address the AL problem, in which the latency decreases for a longer partial output.
When the partial translation output becomes longer, $t$ increases and $(t-1)/r$ becomes larger, although $g(t)$ does not change.
Consequently, the result of subtraction in Equation (\ref{eqn:AL_g}) is reduced.

\citeA{arivazhagan-etal-2019-monotonic} proposed another AL variant called Differentiable Average Lagging (DAL), which can optimize a simultaneous translation model:


\begin{equation}
DAL_g(\bm{x}, \bm{y}) = \frac{1}{|\bm{y}|} \sum_{t =1}^{|\bm{y}|} \left( g'(t) - \frac{t -1}{r} \right),
\label{eqn:DAL_g}
\end{equation}

\begin{equation}
\label{eqn:DAL_g_dash}
g'(t) = \begin{cases} g(t) &  t = 1 \\
\max(g(t), g'(t - 1) + r) & t > 1 \end{cases}.
\end{equation}

DAL replaces the $g(t)$ of AL with $g'(t)$ and does not use the cutoff step $\tau_g(|\bm{x}|)$.
DAL mitigates the AL problem in that a long output chunk reduces the delay, although it remains insufficient.
Suppose $|\bm{x}| = |\bm{y}|$ ($r = 1$).
When the partial translation output becomes longer than the partial input, $g'(t-1)+r$ exceeds $g(t)$ in Equation (\ref{eqn:DAL_g_dash}).
In this situation, every time a new target token is output, $g'(t)$ increases by one, as does $(t-1)/r$.
Therefore, the difference between the two terms in Equation (\ref{eqn:DAL_g}) is not changed by a long output, and the delay does not increase.
Although a long output should delay the start of the next translation chunk, such a delay is not considered in  DAL.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F4
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f4.pdf}
\end{center}
\caption{Time synchronous view of chunk-based SimulMT}
\label{fig:AL_case2_timeview}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F5
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f5.pdf}
\end{center}
\caption{Actions of chunk-based SimulMT}
\label{fig:AL_case2_actionview}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%




The above latency metrics were proposed to evaluate sentence-level SimulMT. Therefore, \citeA{iranzo-sanchez-etal-2021-stream-level} proposed a method to calculate them for streaming input by converting the global delay in the streaming input to the local delay of each sentence.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S3.2
\subsection{Speech-to-Text Latency Metrics}
Although AL was initially designed for text-to-text conditions, it was extended to speech-to-text conditions \cite{ma-etal-2020-simuleval}.
In the speech-to-text condition, each element of the input sequence becomes an audio segment, and the equations for AL are as follows:

\begin{equation}
AL_g(\bm{x}, \bm{y}) = \frac{1}{\tau_g(|\bm{x}|)} \sum_{t =1}^{\tau_g(|\bm{x}|)} \left( g_{speech2text}(t) - \frac{t -1}{r} \right),
\end{equation}
\begin{equation}
g_{speech2text}(t) = \sum_{j =1}^{g(t)} T_{j}
\end{equation}
\begin{equation}
r = \frac{|\bm{y}|}{\sum_{j =1}^{|\bm{x}|} T_{j}}
\end{equation}
\begin{equation}
\tau_g(|\bm{x}|) = \min\{t \mid g_{speech2text}(t) =\sum_{j =1}^{|\bm{x}|} T_{j}\},
\end{equation}
where $T_{j}$ denotes the duration of the $j$-th source audio segment.
$g_{speech2text}(t)$ is similar to $g(t)$ but uses the length of input audio instead of the number of input tokens.
AL variants such as LAAL for the speech-to-text condition are also defined similarly in the implementation of SimulEval.

In actual speech-to-text conditions, we need to consider computation time as a delay.
The computation-aware version of AL is defined by replacing $g_{speech2text}(t)$ with $g_{speech2text}(t) + elapsed(t)$ in the implementation of SimulEval. $elapsed(t)$ represents the difference between the actual start time for reading the input sequence and the actual end time for outputting the $t$-th target token.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S3.3
\subsection{Speech-to-Speech Latency Metrics}
The latency metrics described so far were proposed without assuming speech-to-speech conditions.
Therefore, simpler metrics called Start Offset and End Offset \cite{agrawal-etal-2023-findings} have been used for this condition.
The Start Offset is the time difference between the start of the source speech input and the target speech output.
The End Offset uses the ends instead of the starts.
Unlike AL and its variants, these two metrics only measure the delay at a single point in the sequence; therefore, the delay in the middle of the translation cannot be considered.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S4
\section{Proposed Metric: Average Token Delay}
We propose a novel latency metric ATD which includes the delay caused by a long output chunk in the latency measurement.
We begin with the ATD calculation for the case of speech-to-speech SimulMT and generalize it to speech-to-text and text-to-text cases.
In the following explanation, we first assume that the computation time is included in ATD's latency calculation.
Next, we explain the non-computation-aware version of ATD, which excludes the computation time.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S4.1
\subsection{ATD for simultaneous speech-to-speech translation}
Figure \ref{fig:s2s_simulate} illustrates the step-by-step workflow of the speech-to-speech SimulMT.
In this figure, the white boxes with an order (1st, 2nd) represent the duration of the speech segments, the orange ones labeled as `$j$' represent the processing time to encode the input prefixes and decide whether we should \emph{READ} or \emph{WRITE}, and the blue ones labeled as `$d$' represent the decoding time.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F6
\begin{figure}[b]
\begin{center}
\includegraphics{31-3ia8f6.pdf}
\end{center}
\caption{Step-by-step example of simultaneous speech-to-speech MT: Time passes from left to right.}
\label{fig:s2s_simulate}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



To calculate ATD, we divide each speech segment into subsegments of length $\tau$ from the beginning of the segment, assuming that one word is uttered in duration $\tau$.
Intuitively, ATD is the average of the time differences between the points of the same color (black, white, red, and blue) at the ending time of subsegments (Step 7).

ATD is defined as follows:
\begin{equation}
\textrm{ATD}(\bm{x}, \bm{y})= \frac{1}{|\bm{y}|}\sum_{t=1}^{|\bm{y}|} \left( T(y_{t}) - T(x_{a(t)}) \right)
\label{eqn:atd_definition},
\end{equation}
where
\begin{equation}
\label{eqn:a_definition}
a(t) = \min(t - d(t), g(t))
\end{equation}
\begin{equation}
\label{eqn:d_definition}
d(t) = (t-1) - a(t-1).
\end{equation}


$T(\cdot)$ in Equation (\ref{eqn:atd_definition}) represents the end time of each input or output token, shown as colored points in Figure \ref{fig:s2s_simulate}.
A token is a subsegment of speech; it can be a character or word in the text.
$a(t)$ represents the index of the input token corresponding to $y_{t}$ in the time difference calculation and $a(0)$ = 0.
$d(t)$ in Equation (\ref{eqn:d_definition}) represents the length of the previous translation prefix, which is longer than that of the previous input prefix.
As shown in Equation (\ref{eqn:a_definition}), if $d(t) > 0$, $a(t)$ becomes smaller than the output index $t$.
This means that the previous long output increases the time difference between the input and corresponding output tokens.

ATD is the average delay of the output tokens relative to their corresponding input tokens, considering the latency required for the inputs and outputs.
Although the input-output correspondence does not necessarily denote semantic equivalence, especially for language pairs with large differences in word order and number of tokens, we use this simplified formulation for latency measurements, similar to AL.


Figure \ref{tab:equation1} presents examples of Equation (\ref{eqn:a_definition}) for the first chunk translation.
Here, we simplify the durations of the input and output tokens to be the same length.
In Figure 7a, we measured the token delay on $y_3$.
Here, $d(3)= 0$, and we obtained $a(3) = t-0 = 3$; thus, $y_3$ corresponds to $x_3$.
In Figure 7b, we measured the token delay on $y_4$
 and obtained $d(4) = 0$ and $a(4) = g(4) = 3$, so indicating that $y_4$ corresponds to $x_3$.
In Figure 7a, the input and output lengths are identical, as are their corresponding token indices.
Similarly, in Figure 7b, since the output is longer than the input,
we correspond  $y_3$ and subsequent tokens like $y_4$ to an identical input token $x_3$.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F7
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f7.pdf}
\end{center}
\caption{Examples of 1st chunk translation}
\label{tab:equation1}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



Figure \ref{tab:equation2} shows examples to explain Equation (\ref{eqn:a_definition}) for the second chunk translation.
Consider that we measure the token delay on $y_5$.
In Figure 8a and Figure 8b, $d(5) = 0$, hence $a(5) = t-0 = 5$.
Therefore, $y_5$ corresponds to $x_5$.
In Figure 8c, $d(5) = 4 - a(4) = 1$, according to the result of Figure 7b.
Thus, $a(5) = t - 1= 4$ and $y_5$ corresponds to $x_4$.
Unlike Figure 8a and Figure 8b, the first chunk output, which is longer than the first chunk input in Figure 8c, produces the difference of the corresponding indices of the input and output tokens (Figure 7b).
This difference accumulates and propagates to subsequent tokens (Figure 8c).



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F8
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f8.pdf}
\end{center}
\caption{Examples of 2nd chunk translation}
\label{tab:equation2}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S4.2
\subsection{ATD for simultaneous \{speech, text\}-to-text translation}
Figure \ref{fig:s2t_simulate} illustrates the latency measurement for the speech-to-text SimulMT, assuming that the text output duration is negligible.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F9
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f9.pdf}
\end{center}
\caption{Summary view for latency measurement for simultaneous speech-to-text translation}
\label{fig:s2t_simulate}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


For speech-to-text SimulMT cascading incremental automatic speech recognition (ASR) and text-to-text SimulMT, we sometimes need to investigate only the latency of the text-to-text SimulMT.
Figure \ref{fig:t2t_simulate} shows the latency measurement for the text-to-text SimulMT.
We reserve the input duration of each text token here (shown by yellow rectangles labeled as `$a$') as the computation time of the incremental ASR because the input for text-to-text SimulMT should come from the output of incremental ASR.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F10
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f10.pdf}
\end{center}
\caption{Summary view for latency measurement for simultaneous text-to-text translation}
\label{fig:t2t_simulate}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F11
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f11.pdf}
\end{center}
\hangcaption{Summary view for non-computation-aware latency measurement for simultaneous text-to-text translation}
\label{fig:t2tNCA_simulate}
{\vskip -10pt}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S4.3
\subsection{Non-computation-aware ATD}
We sometimes use a latency measurement that is independent of the computation time for theoretical analyses.
In Figures \ref{fig:s2s_simulate}, \ref{fig:s2t_simulate}, and \ref{fig:t2t_simulate}, we remove the parts labeled as `$j$', `$d$', and `$a$', and only include the duration of the speech segments to calculate the delay.
However, this means that all terms in text-to-text SimulMT become zero.
We followed the conventional stepwise latency measurement for AP and AL by letting each input and output word spend one step (Figure \ref{fig:t2tNCA_simulate}).
Here, we assume that the model can read the next input and output partial translations in parallel.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S5
\section{Simulation}
Prior to conducting the experiments using real data,
we present simulations that compare AL, DAL, and ATD under different conditions in text-to-text SimulMT to investigate how the output length affects latency.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S5.1
\subsection{Cases 1 and 2}\label{subsec:casex}
In Figure \ref{fig:EVS_AL}, we show that a long output chunk yields a smaller AL for Case 2.
Additionally, we calculated other latency metrics for the two cases in Figure \ref{fig:EVS_AL} and show the calculated values in Table \ref{tb:case1and2}.
AL and DAL were larger in Case 1, whereas ATD was larger in Case 2, as was EVS.
This example shows that ATD sufficiently considers the delay caused by long outputs.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%T1
\begin{table}[b]
\input{08table01.tex}
\caption{Latency values for cases in Figure \ref{fig:EVS_AL}}
\label{tb:case1and2}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S5.2
\subsection{Case 3: (10+10)-($\bm{L_1}$+10)}\label{subsec:case3}
To analyze the effect of the output length in more detail, we simulated another case.
In Case 3, we assumed that the input is divided into two chunks, each of which with ten tokens, and each corresponding output consists of $L_1$ and ten tokens with varying $L_1$.
Figure \ref{fig:case3} shows the results.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F12
\begin{figure}[b]
\begin{center}
\includegraphics{31-3ia8f12.pdf}
\end{center}
\caption{Case 3 (10+10 input and $L_1$+10 output tokens with varying $L_1$)}
\label{fig:case3}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


If $L_1 < 10$, some target tokens corresponding to the first input chunk appear after reading the second chunk begins.
Figure 8b illustrates this scenario, where $y_2$ is in the second chunk output, and the corresponding $x_2$ is in the first chunk input.
If $y_2$ is in the first output chunk, then the time difference between $x_2$ and $y_2$ decreases. This also reduces the time difference between the latter token pairs, such as  $x_3$ and $y_3$.
Therefore, as $L_1$ decreases, the delay within this range increases.

If $L_1 > 10$, then the translation of the second chunk is delayed because of the long translation outputs of the first chunk (Figure 8c).
Therefore, as $L_1$ increases, the delay within this range increases. ATD reflects this phenomenon; however, AL decreases monotonically with $L_1$ while DAL does not increase.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S5.3
\subsection{Case 4: 20-20}\label{subsec:case4}
AL was proposed with wait-k, which reads one token and outputs one token alternately; therefore, outputting multiple tokens simultaneously was not sufficiently considered in AL.
To analyze the latency metrics for outputting multiple tokens simultaneously, we conducted another type of simulation by assuming an input of 20 tokens and an identical amount of output.
We simulated two simultaneous translation strategies: wait-k and fixed-size segmentations.
In fixed-size segmentation, for simplicity, we assume that the length of the input and output chunks is $k$ until the end-of-sentence token is predicted.
We call this simple strategy chunk-$k$ in this simulation.
Hyperparameter $k$ for wait-$k$ and chunk-$k$ varied from 1 to 20.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F13
\begin{figure}[b]
\begin{center}
\includegraphics{31-3ia8f13.pdf}
\end{center}
\caption{Case 4 (20 input and 20 output tokens)}
\label{fig:case4}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



Figure \ref{fig:case4} shows the simulation results.
Except for the AL evaluating chunk-$k$, because all the metrics have identical values, they were plotted using the same color.
There is a gap between wait-$k$ and chunk-$k$ when using AL, although ATD and DAL result in the same values.
One serious problem is the large jump in AL at $k=20$.
For chunk-$19$, AL uses $r=1$, $\tau_{g_{\textrm{chunk-}19}}(|\bm{x}|) = 20$, $g_{\textrm{chunk-}19}(\tau)=19$ ($1 \leq \tau \leq 19$), and $g_{\textrm{chunk-}19}(\tau)=20$ ($\tau = 20$).
The AL value becomes $\frac{1}{20} \left\{ \left( \sum_{\tau=1}^{19} \tau \right) + 1 \right\} = \frac{191}{20} = 9.55$.
However, for chunk-$20$, $g_{\textrm{chunk-}20}(\tau) = 20$ for all $\tau$ and  $\tau_{g_{\textrm{chunk-}20}}(|\bm{x}|) = 1$, according to Equation (\ref{eqn:tau_g}).
As a result, the AL value becomes 20.
This phenomenon demonstrates the problem of AL, in which a long partial translation output reduces latency.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S6
\section{SimulMT Analyses }
We analyzed the actual SimulMT results to investigate the effectiveness of ATD.
As most existing latency metrics were originally proposed for text-to-text SimulMT, we compared text-to-text models for the non-computation-aware condition in this experiment.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S6.1
\subsection{Settings}
We used data from the shared task of simultaneous English-to-German translation in the IWSLT 2022 evaluation campaign \cite{anastasopoulos-etal-2022-findings}.
We used the WMT 2014 training set (4.5~M sentence pairs) for pre-training and the IWSLT 2017 training set (206~K sentence pairs) for fine-tuning.
The development set consisted of dev2010, tst2010, tst2011, and tst2012 (5,589 sentence pairs in total), and the evaluation set consisted of tst2015 (1,080 sentence pairs).

Following the experimental settings in \cite{kano-etal-2022-simultaneous}, several SimulMT methods were compared: wait-$k$ \cite{ma-etal-2019-stacl}, Meaningful Unit (MU) \cite{zhang-etal-2020-learning-adaptive}, Incremental Constituent Label Prediction (ICLP) \cite{kano-etal-2021-simultaneous}, and Prefix Alignment (PA) \cite{kano-etal-2022-simultaneous}.
For MU and PA, we used boundary predictors with one-look-ahead (\texttt{MU-1}, \texttt{PA-1}) and no-look-ahead (\texttt{MU-0}, \texttt{PA-0}) approaches.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S6.2
\subsection{Results}
Figure \ref{fig:en-de} shows the comparison results.
Compared to the AL shown in Figure 14a, the ATD in Figure 14b demonstrated clear differences in delays among the models.
MU and ICLP were affected by changes in the latency metric.
We scrutinized their results and found that this degradation was caused by over-translation, as suggested by observations of the length ratio results shown in Figure \ref{fig:ratio_de_al}.
MU and ICLP generated long translations exceeding a length ratio of 1.0 when they worked with small latency.
One interesting finding was the correlation between BLEU scores and ATD by MU;
a larger latency did not always increase BLEU. Over-translation increased ATD but simultaneously decreased BLEU.
In contrast, the wait-$k$ strategy generated only one output token at a time and did not suffer from this issue.
The PA also worked well with the latency measurement by ATD because it fine-tuned the translation model to prevent over-translation.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F14
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f14.pdf}
\end{center}
\caption{Latency and BLEU}\label{fig:en-de}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%F15
\begin{figure}[t]
\begin{center}
\includegraphics{31-3ia8f15.pdf}
\end{center}
\caption{Length ratio and AL}\label{fig:ratio_de_al}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7
\section{Using EVS as Reference}
Because the above analyses were not sufficient to verify the effectiveness of ATD, we conducted the following experiments to compare the different latency metrics.
To verify the superiority, we compared the correlation between each latency metric and the mean EVS, as calculated from the outputs of real SimulMT models.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.1
\subsection{Methodology to Calculate EVS}
The EVS can be measured in several ways \cite{elisa2019}.
We used an automatic word-alignment tool to assist humans in choosing the correct alignment, because of the high cost of word alignment from scratch.
EVS was calculated using the following process:

\paragraph{1.} Run speech-to-speech SimulMT models for the test dataset and output target speech.
\paragraph{2.} Use awesome-align \cite{dou2021word} to obtain automatic word alignment in each sentence transcription of the source and target speech. Transcription is obtained through automatic speech recognition (ASR) using Whisper \cite{radford2022robust}.
\paragraph{3.} Human annotators select the correct alignment from the awesome-align output.
\paragraph{4.} Obtain timestamps for the source and target words from speech and its transcription using WhisperX \cite{bain2022whisperx}.
\paragraph{5.} Calculate EVS by determining the time difference between the start time of aligned source and target words. The mean EVS for a sentence is then computed by averaging these EVS values. 

For the analysis, we also calculated the mean value of all EVSs, including incorrect word alignments, and called this the mean automatic EVS.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.2
\subsection{Settings}
We used the following models and evaluation dataset in our experiments:


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.2.1
\subsubsection{Models}
We used a cascaded speech-to-speech SimulMT model by connecting speech-to-text SimulMT and incremental text-to-speech (TTS) models, following \citeA{fukuda-etal-2023-naist}.
We used their TTS model \cite{fukuda-etal-2023-naist} for the following three speech-to-text SimulMT models and their variations:

\begin{itemize}
  \item \textbf{Test-time wait-k \cite{ma-etal-2019-stacl}:} This method first reads \texttt{k} input speech segments and alternately repeats outputting one word and reading one input speech segment.
Hyperparameter \texttt{k} of test-time wait-k was set to 3 and 5. 
The source speech is input to the translation model in 250-ms segments.
  \item \textbf{Fixed-size segmentation:} This method immediately starts a translation every time a new input speech segment comes.
We compared segment sizes of 750 and 2500~ms.
  \item \textbf{System by \citeA{fukuda-etal-2023-naist}:} This scheme uses PA \cite{kano-etal-2022-simultaneous} and local agreement \cite{liu2020lowlatency} to mitigate over-translation with fixed-size segmentation.
We compared segment sizes of 250 and 1250~ms.
\end{itemize}


The settings of the training NMT models followed those in published research \cite{fukuda-etal-2023-naist}.
For fixed-size segmentation and test-time wait-k, we shared a single NMT model that was not fine-tuned with the PA pairs.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.2.2
\subsubsection{Evaluation Data}
We used the first 30 sentences of the English-Japanese tst-COMMON in MuST-C.
We used three types of SimulMT methods, each with two variations, as described above.
A total of 180 sentences were evaluated.

In the streaming SimulMT, the input sequence sometimes becomes longer than a sentence because there are no obvious sentence boundaries.
This long translation can delay the translation of the subsequent inputs.
Therefore, we also evaluated the latency involved in concatenating the two sentences.

We used Spearman's $\rho$ to measure the correlation between EVS and each latency metric.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.3
\subsection{Compared latency metrics}
We compared ATD	with the baseline metrics for speech-to-speech, speech-to-text, and text-to-text latency.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.3.1
\subsubsection{Speech-to-Speech}
We used two baseline latency metrics in the speech-to-speech condition, Start Offset and End Offset, both of which were officially used in the Simultaneous Translation Task of IWSLT 2023 Evaluation Campaign \cite{agrawal-etal-2023-findings}.
For ATD, we set $\tau$ to 300~ms for both input and output speech.

We evaluated the latency metrics using speech-to-speech SimulMT outputs and
conducted experiments under both computation-aware (CA) and non-computation-aware (NCA) conditions.
In the former, the output speech reflected the actual computation time as silence.
Therefore, the target token timestamps under CA should differ from those under the NCA condition.
We used one NVIDIA RTX TITAN throughout all experiments.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.3.2
\subsubsection{Speech-to-Text}
We used four baseline latency metrics for the speech-to-text condition: AL \cite{ma-etal-2019-stacl}, AL with reference \cite{ma-etal-2020-simuleval}, LAAL \cite{papi-etal-2022-generation}, and DAL \cite{arivazhagan-etal-2019-monotonic}.
As in the speech-to-speech condition, we conducted an experiment under both the CA and NCA conditions.
In ATD calculation, we set $\tau$ to 300~ms for the speech input, and the duration of the target text token to 0~ms.

We evaluated the latency metrics using speech-to-text SimulMT without cascading incremental TTS.
EVS was calculated from the timestamps of the speech-to-text SimulMT outputs.
In the speech-to-text condition, the target text tokens had no duration and multiple target tokens had the same timestamps.
However, EVS was proposed for speech-to-speech simultaneous interpretation, and it was not assumed that the interpreter would speak multiple words simultaneously.
Therefore, in this study, we denote the EVS for speech-to-text as $\text{EVS}_\text{speech2text}$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.3.3
\subsubsection{Text-to-Text}
The same baseline latency metrics were used in the text-to-text and speech-to-text conditions.
In ATD calculation, we provided the duration of one step for the input and output text tokens.

We used the results of the speech-to-speech experiment to evaluate the latency metrics in the text-to-text condition.
We evaluated text-to-text latency metrics only under the NCA condition because the CA version was not implemented in SimulEval.
The reference EVS was identical to that used in the speech-to-speech experiment.
Using the word timestamps obtained by WhisperX, we corresponded speech segment boundaries to text boundaries.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.4
\subsection{Results}
We conducted the experiments under the conditions described thus far and obtained the following results.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.4.1
\subsubsection{Speech-to-Speech}
Table \ref{tb:s2s_human_res} lists the correlation between the mean EVS and latency metrics for the speech-to-speech SimulMT results.
In this evaluation, we excluded the sentences with no correct word alignments.
Regardless of the input sequence length and computational awareness, ATD has the highest correlation among all the latency metrics that do not use word alignment.
The mean automatic EVS is the best because the mean EVS is calculated based on word alignments, which are part of the alignments used to calculate the mean automatic EVS.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%T2
\begin{table*}[b]
\input{08table02.tex}
\hangcaption{Speech-to-speech: Spearman's correlation between mean EVS and latency metrics: Value in parentheses is p-value. ``\#sample'' represents the number of sentences used in this evaluation}
\label{tb:s2s_human_res}
\end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%




For the two sentences with both CA and NCA, the correlation of the Start Offset largely decreased compared to the result for one sentence.
When the input increased, the impact of the delay at the initial start-of-translation decreased for the entire delay.
Therefore, the Start Offset had a lower correlation, and in the CA condition, there was no significant correlation.
ATD maintained a relatively high correlation, similar to that of the mean automatic EVS.
When the input sequence is lengthened, the accumulated delay caused by the previous long-chunk translation output increases.
Because ATD adequately considers the output length, it still has a high correlation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.4.2
\subsubsection{Speech-to-Text}
Table \ref{tb:s2t_human_res} shows the  correlation between $\text{EVS}_\text{speech2text}$ and speech-to-text latency metrics.
Regardless of the input sequence length, ATD was the best under the CA condition.
Under the CA condition, we can regard the computation time as a type of duration for the target text token.
Therefore, the CA speech-to-text and speech-to-speech conditions are similar in that the source and target tokens have similar durations.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%T3
\begin{table*}[b]
\input{08table03.tex}
\caption{Speech-to-text: Spearman's correlation between mean $\text{EVS}_\text{speech2text}$ and latency metrics}
\label{tb:s2t_human_res}
\end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



In the NCA condition, DAL had the highest correlation with $\text{EVS}_\text{speech2text}$.
However, the target text tokens have no duration, and multiple target tokens are simultaneously output immediately after reading the speech chunk under this condition, which is not realistic.
EVS was not designed for this type of situation; therefore, we might need another reference latency metric for this condition.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.4.3
\subsubsection{Text-to-Text}
Table \ref{tb:t2t_human_res} lists the correlations between the mean EVS and the text-to-text latency metrics. 
ATD has the highest correlation among all the text-to-text latency metrics.
AL is the most common latency metric, but its correlation was worse than ATD and its variants (AL-ref, LAAL, and DAL).
DAL showed the strongest correlation among the AL variants.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%T4
\begin{table}[t]
\input{08table04.tex}
\caption{Text-to-text: Spearman's correlation between mean EVS and latency metrics}
\label{tb:t2t_human_res}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


For two sentences, the correlations of all latency metrics significantly decreased, but the differences between ATD and the other latency metrics were larger than those for one sentence.
This is because ATD includes a sufficient delay caused by the end time of the previous sentence translation, as explained in the speech-to-speech results.

Compared with the speech-to-speech results in Table \ref{tb:s2s_human_res}, ATD has a lower correlation with EVS in the text-to-text condition.
This suggests that we should directly evaluate the latency of SimulMT from speech inputs and outputs without ASR.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S7.4.4
\subsubsection{Text-to-Text (Character-Level)}
Text-to-text latency metrics assume that each text token is a word. 
However, languages such as Japanese and Chinese do not have spaces between words. 
Therefore, SimulEval \cite{ma-etal-2020-simuleval}, which is commonly used to evaluate simultaneous translation systems, regards characters as tokens in Japanese for simplicity, without tokenizing them into words.
Additionally, we conducted an evaluation using character-based text tokens.

Table \ref{tb:char_word_num} shows the statistics for the IWSLT 2017 English-Japanese training dataset.
The number of English and Japanese words is similar, although the number of Japanese characters is twice as large.
Therefore, to calculate the latency metrics, we compared one character as one token and two characters as one token.\footnote{For two characters as one token, we regard the last one character in the output chunk as one token if the number of characters in the chunk is odd.}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%T5
\begin{table}[t]
\input{08table05.tex}
\caption{Statistics of IWSLT2017 English-Japanese training dataset}
\label{tb:char_word_num}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Table \ref{tb:t2t_human_res_char} presents the comparison results.
The word-level latency calculation in Table \ref{tb:t2t_human_res} has a higher correlation than the character-level calculation, and two characters as one token perform better than one character as one token, especially for two sentences.
This is because the amount of information conveyed by one Japanese word is closer to that conveyed by one English word than that conveyed by one or two Japanese characters.
For the same reason, two Japanese characters as one token outperformed one Japanese character as a token, as shown in Table \ref{tb:char_word_num}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%T6
\begin{table*}[t]
\input{08table06.tex}
\hangcaption{Text-to-text (character-level output): Spearman's correlation between mean EVS and latency metrics}
\label{tb:t2t_human_res_char}
\end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


ATD was the best except for the condition of one character as one token for one sentence.
ATD is the latency metric that is most affected by the output length.
The output of one Japanese character has less correspondence with the input of one English word than with the other output levels, and the output length per character is much longer than the input length per word.
Consequently, the ATD added an excessive delay caused by this long output, and its correlation decreased.

According to the character-level results, we must carefully address tokenization, not only for ATD but also for other latency metrics.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%S8
\section{Conclusion}
We propose a novel latency metric, ATD for SimulMT, that addresses the problem of latency evaluation for long chunk outputs by considering the output length.
Unlike AL, ATD gives a large latency value to a long output based on the assumption that the output also causes a delay.
We identified the effectiveness of ATD through simulations and found that it had the highest correlation with EVS among the token-based latency metrics.
Despite being much simpler than the mean automatic EVS and not requiring a long process such as ASR, word alignment, or obtaining timestamps, ATD correlates well with mean EVS.
Therefore, it can be easily used to evaluate the latency of simultaneous translation.
The implications of this study are significant for the development of more accurate and user-friendly latency metrics in SimulMT. 
In future work, we will investigate the correlation between latency metrics and the delays experienced by listeners.





% Start Acknowledgment
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment
Part of this work was supported by JSPS KAKENHI Grant Numbers JP21H05054 and JP21H03500.

% Start bibliography
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.7_for08}
%%%%%\bibliography{08refs}
\input{08kano.bbl.tex}



% Start biography
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Yasumasa Kano}{%
He is a Ph.D. student at Nara Institute of Science and Technology. He received his Bachelor's degree in Economics from Yokohama National University in 2019 and his Master's degree in Engineering from Nara Institute of Science and Technology in 2021.
}

\bioauthor[:]{Katsuhito Sudoh}{%
He is an Associate Professor at the Nara Institute of Science and Technology. He received his bachelor's degree in engineering in 2000, and his master's and Ph.D. degrees in informatics in 2002 and 2015, respectively, from Kyoto University. He worked with NTT Communication Science Laboratories from 2002 to 2017. He is currently working in machine translation and natural language processing. He is a member of ACL, ISCA, ANLP, IPSJ, ASJ, and JSAI.
}

\bioauthor[:]{Satoshi Nakamura}{%
He is a professor at the Nara Institute of Science and Technology and an honorary professor at the Karlsruhe Institute of Technology, Germany. He received his B.S. degree from the Kyoto Institute of Technology in 1981 and a Ph.D. degree from Kyoto University in 1992. He was the Director of ATR Spoken Language Communication Research Laboratories from 2000--2008 and Vice President of ATR from 2007--2008. He was the Director General of Keihanna Research Laboratories and the Executive Director of the Knowledge Creating Communication Research Center, National Institute of Information and Communications Technology from 2009--2010. He is currently the director of the Augmented Human Communication Laboratory and a professor at the Data Science Center and Information Science Division, Graduate School of Science and Technology, Nara Institute of Science and Technology. He was an Elected Board Member of the International Speech Communication Association, ISCA, from June 2011 to 2019, an IEEE Signal Processing Magazine Editorial Board member from  2012 to 2015, and IEEE SPS Speech and Language Technical Committee Member from 2013 to 2015. He is an ATR, IPSJ, IEEE, and ISCA fellow.
}
\end{biography}

\biodate   %%%% Following settings will be edited by editors.


\end{document}


