\documentclass[english,tombow]{jnlp_3.2}
\usepackage{jnlpbbl_1.4}
\usepackage[dvipdfm]{graphicx}
\usepackage{gentombow}
\usepackage{amsmath}
\settombowbleed{5mm}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
%%%%\makeatletter
%%%%\def\@{\spacefactor3000\space}
%%%%\makeatother

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%\usepackage{times}
%%%%\usepackage{latexsym}
%%%%\usepackage{graphicx}
%%%%\usepackage{multirow}
\usepackage{url}
%%%%\usepackage{comment}
%%%%\usepackage{float}
\usepackage{booktabs}
%%%%\usepackage{color}
%%%%\usepackage{appendix}
\newcommand*{\sptoken}[1]{\texttt{#1}}
%%%%\setlength\abovecaptionskip{3pt}
%%%%\renewcommand{\figurename}{Figure}
\newcommand*{\TODO}[1]{{#1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\renewcommand\UrlFont{\rmfamily}


\Volume{27}
\Number{4}
\Month{December}
\Year{2020}
%%%%\Issuetitle{「　　　　」特集号}

\received{2020}{5}{1}
\revised{2020}{8}{1}
%%%%\rerevised{2006}{5}{6}
\accepted{2020}{8}{27}

\setcounter{page}{781}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Multi-dialect Neural Machine Translation for 48 Low-resource Japanese Dialects}
\eauthor{Kaori Abe\affiref{Tohoku}\affiref{Riken} \and Yuichiroh Matsubayashi\affiref{Tohoku}\affiref{Riken} \and Naoaki Okazaki\affiref{TIT} \and Kentaro Inui\affiref{Tohoku}\affiref{Riken}} 

\eabstract{%
We present a multi-dialect neural machine translation (NMT) model tailored to Japanese. Although the surface forms of Japanese dialects differ from those of standard Japanese, most of the dialects have common fundamental properties, such as word order, and some also use numerous  same phonetic correspondence rules. To take advantage of these properties, we integrate multilingual, syllable-level, and fixed-order translation techniques into a general NMT model. Our experimental results demonstrate that this model can outperform a baseline dialect translation model. In addition, we show that visualizing the dialect embeddings learned by the model can facilitate  the geographical and typological analyses of the dialects.
}
\ekeywords{Neural Machine Translation, Multilingual, Dialect, Typological Analysis, \linebreak Dialectometry}

\headauthor{Abe et al.}
\headtitle{Multi-dialect Neural Machine Translation for 48 Low-resource Japanese Dialects}

\affilabel{Tohoku}{}{Tohoku University}
\affilabel{Riken}{}{RIKEN}
\affilabel{TIT}{}{Tokyo Institute of Technology}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\label{intro}

With the use of automated personal assistants (e.g., Apple's Siri, Google Assistant, or \linebreak
Microsoft Cortana) and smart speakers (e.g., Amazon Alexa or Google Home) becoming increasingly widespread, the demand to bridge the gap between the standard form of a given language and its dialects has also enlarged. The importance of dealing with dialects is particularly evident in a rapidly aging society, like that in Japan, where older people use them extensively.

To address this issue, we consider a system for machine translation (MT) between Japanese dialects and standard Japanese. If such a system can yield correct dialect-to-standard translations, then other natural language processing systems (e.g., information retrieval or semantic analysis) that adopt standard Japanese as the input could also be applied to dialects.
In addition, if a standard-to-dialect translation system becomes available, then smart speakers could respond to the native speakers of a dialect using that dialect. We believe that sympathetic interactions of this type might lead to such systems gaining more widespread acceptance of in the Japanese society.

In this paper, we present a multi-dialect neural MT (NMT) system tailored to Japanese. Specifically, we employ {\it kana}, a Japanese phonetic lettering system, to provide the basic units in the encoder--decoder framework to avoid the \TODO{followings}: ambiguity in converting {\it kana} to {\it kanji} (characters in the Japanese writing system), difficulties in identifying word boundaries especially for dialects, and data sparseness problems due to dealing with numerous words originating from different dialects. Because Japanese dialects almost always use the same word order as standard Japanese, we employ \textit{bunsetsu} (a Japanese phrase unit) as a unit of sequences, instead of a sentence, which is more commonly used in NMT.

One issue for Japanese dialects is the lack of training data. To deal with this, we build a unified NMT model covering multiple dialects, inspired by the studies on multilingual NMT \cite{Johnson2016a}. This approach utilizes \textit{dialect embeddings}, i.e., vector representations of Japanese dialects, to inform the model of the input dialect. An interesting by-product of this approach is that the dialect embeddings that the system learns illustrate the difference between different dialect types from different geographical areas. In addition, we present an example of using these dialect embeddings for 
%%%%dialectometry \cite{Nerbonne:2011,Kumagai:2016,Guggilla:2016,Rama:2016}.
    dialectometry (Nerbonne and Kretzschmar 2011; Kumagai 2016; Guggilla 2016; Rama and \c{C}\"{o}ltekin 2016)\nocite{Nerbonne:2011,Kumagai:2016,Guggilla:2016,Rama:2016}.

Another advantage of adopting a multilingual architecture for multiple related languages is that it can enable gaining knowledge of their lexical and syntactic similarities.
For example, \TODO{\citeA{Lakew2018}} reported that including several related languages in supervised training data can improve multilingual NMT.
Our results confirm the effectiveness of using closely related languages (i.e., Japanese dialects) in multilingual NMT.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}
\label{sec:related_work}

Dialectal text is scarcely available because dialects are generally spoken, instead of being written.
For this reason, many dialect MT researchers study in low-resource settings \cite{Zbib2012,Scherrer,Hassan2017}. 

The use of similar dialects has been found to be helpful in learning translation models for particular dialects.
Several previous studies have investigated the characteristics of translation models of closely related dialects \cite{Meftouh,Honnet}.
For example, \TODO{\citeA{Honnet}} reported that a character-level NMT model trained on one Swiss-German dialect performed moderately well for translating sentences in closely related dialects.

Therefore, in view of the above, we use multilingual NMT \cite{Johnson2016a} to learn the parameters that encode the knowledge of the shared lexical and syntactic structures of dialects.
Some researchers \cite{Gu2018,Arivazhagan2019MassivelyMN} demonstrated that multilingual NMT could be useful for low-resource language pairs, additionally \TODO{\citeA{Lakew2018}} found that a multilingual NMT system trained on multiple related languages showed an improved zero-shot translation performance.
We believe that multilingual NMT can be effective for closely related dialects, and can compensate for the lack of translation data for the different associated dialects.

Multilingual NMT can also assist in analyzing the characteristics of each considered language.
\TODO{\citeA{Ostling2016}} found that clustering the language embeddings learned by a character-level multilingual system provided an illustration of the language families involved.
In the light of this, we also examine our dialect embeddings to investigate whether our multi-dialect model can capture the similarities between the dialects (Section 5).

Previous studies reported that character-level statistical machine translation (SMT) using words as translation units is effective for translating between closely related languages \cite{Nakov2012,Scherrer}. There are two reasons for this: character-level information enables the system to exploit lexical overlaps, whereas using words as translation units takes the advantage of the syntactic overlaps of the related languages. 
%%%%\TODO{To utilize these overlaps, Pointer Networks \cite{NIPS2015_5866,DBLP:journals/corr/GulcehreANZB16}, 
  \TODO{To utilize these overlaps, Pointer Networks (Vinyals, Fortunato, and Jaitly 2015; G\"{u}l\c{c}ehre, Ahn, Nallapati, Zhou, and Bengio 2016)\nocite{NIPS2015_5866,DBLP:journals/corr/GulcehreANZB16}, 
which can copy some parts of input sequences to output sequences, also seem to be effective for dialect translation.
In this study, we conduct an experiment with a simple long short-term memory (LSTM) architecture to train a multilingual NMT model for multiple dialects. 
Adopting a copy architecture similar to Pointer Networks will be conducted in a future study.}

In this study, we present a method of translating between Japanese dialects by combining three ideas: multilingual NMT, character-level NMT, and using base phrases (i.e., \textit{bunsetsu}) as translation units. We believe this enables our approach to fully exploit the similarities among dialects and standard Japanese, even under low-resource settings.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Data: Japanese Dialect Corpus}
\label{sec:data}

Japanese is a dialect-rich language, with dozens of dialects used in everyday conversations in most Japanese regions.
They can be characterized in terms of the differences in their content words (vocabulary) and regular phonetic shifts, mostly in their postpositions and suffixes. Specifically, they share most words with standard Japanese, and mostly use common grammatical rules, such as for the word order, syntactic marker categories, and connecting syntactic markers.\footnote{For details, \textit{Linguistic Atlas of Japan Database} (\url{https://www.lajdb.org/TOP.html}) published by NINJAL provides an overview of the Japanese dialect distribution.} 
Some dialects also share the dialect-specific vocabulary.
For example, the word, しゃっこい (\textit{shakkoi}, meaning ``cold''), is shared among some dialects in the Tohoku region, such as Aomori and Akita.

In this study, we used a collection of parallel textual data for the dialects and standard Japanese, called as the \textit{National Dialect Discourse Database Collection} \cite{furusato}. 
This corpus includes $48$ dialects, one from each of the $47$ prefectures and an additional dialect from the Okinawa Prefecture.
For each dialect, the texts consist of transcribed 30-minute conversations between two native speakers of that dialect. The total number of dialect sentences (each paired with a translation into standard Japanese) is 34,117. Figure~\ref{fig:data_size_graph} shows the number of sentences in each dialect.
The amount of each dialect data is less than 1,500 sentences and vary.

Japanese texts are generally written in a mix of \textit{kanji} and \textit{kana}; therefore, we converted the \textit{kanji} in the sentences into \textit{kana}, and subsequently, segmented them into \textit{bunsetsu}s.\footnote{This is the smallest Japanese phrase unit, containing a single content word and attached postpositions.}
\TODO{In this study, we used the \textit{bunsetsu} segmentation annotated in the original corpus.}
After preprocessing, the average sentence lengths were of 14.62 and 15.57 characters for the dialects and the standard Japanese, respectively.
The average number of \textit{bunsetsu}s per sentence was 3.42.\footnote{\TODO{The total number of \textit{bunsetsu}s is 116,928.}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[t]
\begin{center}
\includegraphics{27-4ia3f1.eps}
\end{center}
\caption{Number of sentences in each dialect in {\it National Dialect Discourse Database Collection}}
\label{fig:data_size_graph}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{NMT Model}
\label{sec:nmt}

Figure~\ref{fig:dialectNMT} presents an overview of our network structure of the multi-dialect NMT system. Because our focus is on examining the effectiveness of the multi-dialect NMT and its detailed behavior, rather than on creating a novel translation model, we used OpenNMT \cite{Klein}.
OpenNMT is a stacking LSTM encoder--decoder model with a multilingual extension similar to that of Johnson's method \cite{Johnson2016a}.
However, to improve its direct translation accuracy, we introduce the following three modifications.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Dialect labels: }
Following a previous multilingual NMT study \cite{Johnson2016a}, 
we train the unified model that deals with all the 48 dialects simultaneously using the dialect embeddings including auxiliary dialect labels.
\TODO{\citeA{Johnson2016a}} added a label to the beginning of each sequence to specify the output language. 
We modify this approach to specify both the input and output dialects of the model, and examine the four different placements for these labels, as listed in Table~\ref{tab:label}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[b]
\begin{center}
\includegraphics{27-4ia3f2.eps}
\end{center}
\caption{Proposed multi-dialect NMT model}
\label{fig:dialectNMT}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[t]
\caption{Dialect label input order variants}
\label{tab:label}
\input{03table01.tex}
\vspace{4pt}\small
For example, when we translate Aomori dialect into standard language, we test using input sentence that replaced \sptoken{<SRC>} with 青森 (\textit{Aomori}) and \sptoken{<TGT>} with 標準語 (\textit{Hyoujungo}).
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Syllable-to-syllable translation: }

As mentioned in Section~\ref{sec:data}, a key to translate between two closely related languages, especially in our case, dialects, is modeling the phonetic correspondences between them. 
Thus, to consider syllable-level translation rules that may be shared by similar dialects, we defined our translation task as a syllable-to-syllable translation.

We realize syllable-to-syllable translation by representing the inputs and the outputs as \textit{kana} sequences and performing character-based MT.
A similar approach was used to normalize Japanese text from Twitter, where the main issue was phonological transliteration \cite{Saito2017a}.
In our dataset, all the dialect expressions are transcribed using \textit{kana}; however, the standard Japanese translations use a mix of \textit{kanji} and \textit{kana} characters. 
Therefore, in order to conform to the syllable-to-syllable task, we also convert them into \textit{kana} sequences by automatically analyzing the pronunciation of each \textit{kanji} character and replacing it with the corresponding \textit{kana} sequence.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Translation without distortion: }
Finally, we attempt to remove the word-order distortion modeling from NMT.
In a standard MT, systems adopt a single sentence as the input and yield a translated sentence in an appropriate word order for the target language.
However, in the dialect translation, the input and output word orders are mostly the same.
To test this, we manually checked 100 randomly-selected sentence pairs from the training set, and found no differences in the ordering (distortion).
This fact suggests that we do not require sentence-by-sentence supervision data, because it does not need to learn a distortion model. 
Based on this intuition, we split each input sentence into base-phrase parts, i.e., \textit{bunsetsu} sequences, translate each chunk from the source to the target language and, subsequently, output the translated chunks in the same order.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}

Using parallel text data (standard Japanese and 48 regional dialects), we trained both a single dialect-to-standard translation model and a reverse (standard-to-dialect) model, measuring the translation quality using BLEU scores \cite{papineni_bleu_2002}.
In addition, we analyzed the trained dialect embeddings in detail and conducted data ablation tests.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Experimental Setup}

For these experiments, we split the corpus into training, development, and the test sets in an $8$:$1$:$1$ ratio. 
We oversampled the translation pairs to ensure that every dialect had the same amount of training data, because there were different numbers of training and test instances for each dialect (in Figure \ref{fig:data_size_graph}).
\TODO{For the oversampling, we randomly sampled the existing sentence in each dialect dataset. 
Finally, all the training sets for each dialect consisted of 1,042 sentences, the same size as the largest original training set of the Iwate dialect. }

Because Japanese dialects mostly share the same vocabulary and there are few distortions (word order changes), we expect that the translation between a Japanese dialect and standard Japanese is relatively easy compared to that between other languages. 
Thus, the main focus of the following experiments was to evaluate how well the model captured the phonological shifts between the dialects and the standard Japanese.
Therefore, we employed syllable-level (i.e., character-level) BLEU scores as the evaluation measures. 
We calculated the syllable-level BLEU for each sentence by concatenating the chunk-wised translations.
Note that this evaluation measure generally yields higher scores than those calculated at the word level.
Finally, we macro-averaged the scores over all the dialects.
\TODO{For Multi NMT or SMT models, we generated the translation output for the entire test set, which contained all the dialects, and we divided it into the 48 dialect test sets. 
Subsequently, we calculated the macro-averaged BLEU scores using the 48 local BLEU scores obtained on the test set for all the dialects. 
For Mono NMT or SMT models, we trained 48 local NMT/SMT models using a local training/valid set (also a subset of the entire training/valid set) and subsequently evaluated it with a local test set. 
For all the settings, for the evaluation, we used the test set written as one sentence per line.
The difference between the dialect-to-standard and standard-to-dialect translations is simply the exchange of the source and target languages. }
In fact, the macro-averaged BLEU score reached 35.10 even when we simply output the dialect sentences without translation. 

We used OpenNMT-py\footnote{https://github.com/OpenNMT/OpenNMT-py} with its default hyper-parameter settings, except for the number of training epochs (which we set to $20$), and 
selected the model that performed best on the development set.
\TODO{For the details, we list the hyperparameter settings in Table \ref{tab:parameters}.}
In addition, we employed Moses\footnote{http://www.statmt.org/moses} \cite{Koehn2007} as the baseline SMT model and set the distortion limit to $0$. The standard Japanese language model used in Moses was trained with KenLM \cite{Heafield2011}.
For the syllable-to-syllable translation, we used MeCab 0.996\footnote{http://taku910.github.io/mecab/} to analyze the pronunciations of the {\it kanji} characters.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\caption{Experimental settings of OpenNMT such as hyper parameters}
\label{tab:parameters}
\input{03table02.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Regarding the dialect label order used for the input, our preliminary experiments \TODO{on the validation set} indicated that the best models were obtained using input sequence (d) (Table 1) for the dialect-to-standard translation and input sequence \TODO{(b)} for the standard-to-dialect translation.\footnote{\TODO{See Appendix B for more details.}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[t]
\caption{Descriptions of each model}
\label{tab:model_features}
\input{03table03.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\TODO{A brief description of each model we used in our experiments is provided in Table \ref{tab:model_features}. 
Except for the multi-sentence NMT models, we used each \textit{bunsetsu} as a translation unit. 
Note that, as we mentioned in Sec.~\ref{sec:nmt}, it is practically unnecessary to model the word order distortion in Japanese dialect translation. 
The individual dialects are distinguished by two methods: addition of dialect labels to the multilingual NMT, and training the local models for each dialect separately. 
In comparison, the multilingual NMT and SMT systems without the labels (Multi NMT w/o labels, Multi SMT w/o labels) do not distinguish dialects, and therefore, are disadvantageous. 
}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Multi-Dialect NMT Model Performance}
\label{sec:impact}

Table \ref{tab:main_result} summarizes the results of the dialect translation performance of all the considered models, with the first row group comprising their scores for dialect-to-standard translation under different input settings.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[t]
\caption{Syllable-level BLEU scores of all models}
\label{tab:main_result}
\input{03table04.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Monolingual vs. multilingual: }
For comparison, we first considered a model that was trained using only a single set of dialect-standard parallel data (Mono NMT). 
It performed quite poorly compared to the other models that used data for all the dialects (Multi NMT) and was even worse than simply outputting the dialect sentences unchanged ($35.10$).
This indicates that training independent NMT models for each language pair with a limited amount of training data is extremely inefficient.
In contrast, the multi-dialect model presented a drastically improved the translation performance. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Dialect labels: }
Including dialect labels improved the Multi-NMT BLEU score by $4.37$ points (\TODO{fifth} row of Table~\ref{tab:main_result}) compared to that of the same model without the dialect labels (\TODO{third} row). 
Figure~\ref{fig:BLEU_graph} shows for these two models the BLEU scores for all the dialects in ascending order of translation difficulty. 
Here, the translation difficulty is defined as the average normalized Levenshtein distance over all the sentence pairs (dialect and standard Japanese) for a given dialect. 
As expected, the BLEU scores for all the dialects present a strong negative correlation ($\rho = - 0.82$) with the translation difficulty. 
In addition, we can observe that the model with language labels consistently outperforms that without the labels, except for the Tottori dialect, for which there is an extremely small amount of text data (Figure \ref{fig:data_size_graph}). 
This result indicates that explicit information of the source and target dialects with dialect labels can improve the encoding and decoding accuracy.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[t]
\begin{center}
\includegraphics{27-4ia3f3.eps}
\end{center}
\caption{BLEU scores of Multi NMT models and translation difficulty for all dialects}
\label{fig:BLEU_graph}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Fixed-order translation: }
Comparing the proposed model (Multi NMT) with the same model trained via the standard approach of using entire sentences as input/output sequences (Multi NMT-sentence) shows that Multi NMT outperforms Multi NMT-sentence by $5.92$ points.
One disadvantage of the chunk-wise translation is that it cannot capture the context beyond the boundary of each chunk; however, despite this disadvantage, our Multi NMT model can still outperform the model with an access to a broader context (Multi NMT-sentence).
This indicates that our fixed-order translation approach is suitable for translating Japanese dialects, despite its limited context sensitivity.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{NMT vs. SMT: }
\TODO{\citeA{Zoph2016a}} found that SMT models largely outperformed state-of-the-art NMT models for low-resource languages.
The second-row group in Table~\ref{tab:main_result} summarizes the results for a fixed-order character-based SMT baseline. In these experiments, the NMT model trained using a single dialect (Mono NMT) yielded the poorest performance; however, the one with dialect labels outperformed the baseline Multi SMT model, achieving the best performance overall.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Example of Translation Results}

To demonstrate how each of the proposed component contributes to generating accurate translations, we now present some concrete examples of the translation results of our models for the Hyogo, Kagoshima, and Nigata dialects (Table \ref{tab:exp_translation}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[p]
\caption{Example dialect-to-standard translations for Hyogo, Kagoshima, and Nigata dialects}
\label{tab:exp_translation}
\input{03table05.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Comparing the Multi NMT models with and without dialect labels, we noted that adding labels enables the models to better translate the chunks that required dialect-specific knowledge.
In Example 1, the source sentence includes a local name, おー(\textit{O -}), for a certain area, あいおい(\textit{Aioi}), in Hyogo, which only the model with dialect labels can successfully translate.
In addition, in Example 2, the dialect labels enable the model to capture a dialect-specific transliteration rule for the functional suffix (``ta ra''), a conditional-mood marker in the last chunk of the reference sentence (i.e., ``ta ya'' to ``a ra'').

Similarly, because the Multi SMT model could not take advantage of the dialect labels, it failed to capture dialect-specific translation rules.

In the previous section, we noted that our fixed-order translation approach is suitable for translating Japanese dialects, despite its limited context sensitivity.
However, this becomes a problem in Example 3, where the chunk-wise translation models cannot correctly translate a \TODO{\textit{bunsetsu}} owing to the lack of context.
Here, none of the models, except the Multi NMT-sentence, can translate the \TODO{\textit{bunsetsu}}, ``mi zu n'', in the Nigata dialect to the correct standard Japanese \TODO{\textit{bunsetsu}} ``mi zu no.''
Because the translation of the functional word, ``n,'' in the Nigata dialect is ambiguous, it can be translated as either ``ga'' (nominative marker) or ``no'' (of) depending on the following context. This example exposes the limitations of our chunk-wise translation models and suggests potential future directions: extending the fixed-order translation to incorporate contextual information.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Visualizing Attention Weights}

Here, to investigate how the proposed model translated the {\it kana} sequences in various dialects, we visualized the attention weights of the best-performing model for some correctly-translated examples.

Figure \ref{fig:attention}(a) shows the attention history of the model for an example where a part of the target language (standard Japanese) \TODO{\textit{bunsetsu}} changes from the source language (Aomori dialect) according to a simple regular rule.
In such cases, the model tends to weight the dialect label heavily when applying the rule (``da'' → ``do ha'').
Conversely, Figure \ref{fig:attention}(b) shows the attention history for an example where almost all the syllables are transcribed. 
In these cases, the model needs to disambiguate the morpheme-level definitions to create a correct translation, and thus, tended to focus on the entire sequence of semantically- or grammatically-related morphemes.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig.4
\begin{figure}[t]
\begin{center}
\includegraphics{27-4ia3f4.eps}
\end{center}
\hangcaption{Attention weight examples. (a) {\it Aomori-to-standard} translation of ``next time''. Aomori word {\it konda} is formed by linking syllables from two standard Japanese words, {\it kondo} (next time) and {\it ha} (topic marker). (b) {\it Okinawa-to-standard} translation of ``we''. Okinawa word {\it watta-} combines two standard Japanese words, {\it watashi} (I) and {\it tachi} (plural marker), with {\it watt} and {\it ta-} roughly corresponding to {\it watashi} and {\it tachi}, respectively. }
\label{fig:attention}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Visualizing Dialect Embeddings}

\TODO{\citeA{Ostling2016}} reported that clustering the language embeddings used to train a multilingual language model produced a language cluster structure similar to those of established relationships among the language families.
Motivated by their work, we decided to examine the relationships between the dialect embeddings and the typology of the dialects. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 5
\begin{figure}[t]
\begin{center}
\includegraphics{27-4ia3f5.eps}
\end{center}
\hangcaption{T-SNE projection of dialect label vectors. Dialects belonging to same region are shaded using same background color.}
\label{fig:heat_map}
\vspace{-0.5\Cvs}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Figure \ref{fig:heat_map} shows the t-Distributed Stochastic Neighbor Embedding (t-SNE) projection of the dialect embeddings. 
It indicates that dialects from neighboring regions tend to form a single cluster.
Furthermore, we can observe an interesting agreement between the cluster distances and the predictions of the dialectological typology theory, known as {\it center versus periphery} \cite{Yanagida1980}, wherein new language use trends gradually propagate from the cultural center (the old capital, Kyoto) to less culturally influential areas. This potentially explains why the dialects in the Tohoku region (E) are similar to those in the Kyushu region (D), despite their large geographical separation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Effect of the Nearby Dialects}

To investigate in more detail how jointly learning multiple dialects contributed to the dialect-to-standard translations for each dialect, we performed an ablation study on all the dialect regions.
As presented in the previous section, the dialects in geographically close regions are generally more similar to each other than those in other regions.
Therefore, we assumed that the impact of sharing data from other dialects would differ depending on their geographical distances from the target dialect.

To investigate this assumption, we prepared two Multi NMT models per dialect, trained on the data that excluded the five geographically nearest or farthest dialects\footnote{The distances between the dialect pairs were calculated using the Euclidean distances between the points where the dialogs were recorded. } for the given dialect region, and calculated the differences in the BLEU scores of these models and the original model, for all the dialect regions.
For example, the $-\mathit{nearest 5}$ model for the Tokyo dialect is the Multi NMT model learned with the training data excluding the Chiba, Kanagawa, Saitama, Gumma, and Ibaraki dialects.
Subsequently, we compared this BLEU score of the model for the Tokyo dialect in the test set to that of the original model trained with full data.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[t]
\hangcaption{Impact of excluding nearest or farthest five dialect regions from training data when calculating BLEU score for each dialect region}
\label{tab:ablation}
\input{03table06.tex}
\vspace{4pt}\small
``Avg. $\Delta$'' denotes the average BLUE score difference compared to that using all the data.
\par\vspace{-0.5\Cvs}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Table~\ref{tab:ablation} lists the average results over all the 48 models for both the cases. 
Both the models trained without the nearest five dialects, and those without the farthest five dialects yielded lower average BLEU scores for their target dialects compared to the full models.
This suggests that even very distant dialects still assist in training other dialects.
In addition, we note that removing the nearest five dialects had a more significant impact than removing the farthest five dialects, indicating that similar dialects contribute more to assisting a multi-dialect NMT to learn effectively.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

We have examined the effectiveness of a multilingual, syllable-based, fixed-phrase-order NMT model for translating Japanese dialects into standard Japanese. The results showed that each component of our multi-dialect NMT model successfully improved the translation accuracy when using a limited amount of supervised training data.
In addition, we demonstrated the potential benefit of analyzing dialect embeddings for dialectological analysis applications, and have also analyzed how the multi-dialect NMT leverages the training data involving similar dialects to translate a given dialect.

One limitation of the proposed model is that it cannot consider longer-range dependencies beyond the chunk level.
Therefore, our future research plans include incorporating contextual information, \TODO{e.g., n-to-1 translation \cite{tiedemann-scherrer-2017-neural}}, into fixed-order translation models and investigating the characteristics of the dialect embeddings further.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

We would like to thank the Center for the Study of Dialectology, Tohoku University for sharing the dataset with us.
This work was supported by JSPS KAKENHI Grant Numbers JP15K16045, JP15H05318, and JP15H01702.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.5}
\bibliography{03refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%\newpage
\appendix
\section{Data size for each dialect }

\TODO{
We show the details of the dataset size in Figure \ref{fig:each_datasize}. 
In our experiments, we used the largest available Japanese dialect dataset; however, the number of sentences in some dialects (e.g., \linebreak
Tottori) was small in our dataset. 
Owing to the lack of the training or test dataset, the BLEU score of the Tottori dialect is actually inconsistent with those of the other dialects, as shown in Figure \ref{fig:BLEU_graph}. 
However, this result suggests that the proposed method is effective for almost all the dialects, except in a very low-resource scenario. 
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 6
\begin{figure}[b]
\begin{center}
\includegraphics{27-4ia3f6.eps}
\end{center}
\caption{Size of each dataset (training/valid/test) in our experimental setting per dialect}
\label{fig:each_datasize}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments of dialect label order variants }

\TODO{We summarize the results of the preliminary experiments to examine the best setting of the dialect labels in Table~\ref{tab:result_dialect_label_val} (validation set) and Table~\ref{tab:result_dialect_label_test} (test set). 
As can be seen from both the tables, the best models are obtained using input sequence (d) for the dialect-to-standard translation and input sequence (b) (in Table \ref{tab:label}) for the standard-to-dialect translation. 
Label set (b) in the dialect-to-standard translation and set (a) in the standard-to-dialect translation do not contain the dialect information, in contrast with the other settings. They present a quite lower performances.
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 7
\begin{table}[h]
\hangcaption{Syllable-level BLEU scores for each dialect label in Multi NMT in both translation directions on the validation sets. We showed the details of dialect labels (a)--(d) in Table \ref{tab:label}.}
\label{tab:result_dialect_label_val}
\input{03table07.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 8
\begin{table}[h]
\hangcaption{Syllable-level BLEU scores for each dialect label in Multi NMT in both translation directions on the test sets.}
\label{tab:result_dialect_label_test}
\input{03table08.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Kaori Abe}{%
is a first-year doctoral student at the Graduate School of Information Sciences, Tohoku University. She received the M.Sc degree in information science from Tohoku University, Japan, in 2020. She is supported by Research Fellowships for Young Scientists (DC1), Japan Society for the Promotion of Science. Her research interests include natural language processing, especially dialogue systems and machine translation.
}

\bioauthor[:]{Yuichiro Matsubayashi}{%
is an associate professor at the Graduate School of Education, Tohoku University. He received a Ph.D. degree in Information Science and Technology from the University of Tokyo in 2010. His work focuses specifically on modeling a communication process of humans, building a semantics of the natural language, and semantic processing of natural language text. 
}

\bioauthor[:]{Naoaki Okazaki}{%
 is a professor of the School of Computing, Tokyo Institute of Technology. He received a Ph.D. degree in the Graduate School of Information Science and Technology, the University of Tokyo in 2007. He is also a visiting research scholar of the Artificial Intelligence Research Center (AIRC), AIST. His research interests include natural language processing, text mining, and machine learning. 
}

\bioauthor[:]{Kentaro Inui}{%
is a professor of the Graduate School of Information Sciences at Tohoku University, where he is head of the Natural Language Processing Lab. He also leads the Natural Language Understanding Team at the RIKEN Center for the Advanced Intelligence Project. His research areas include natural language processing and artificial intelligence. He currently serves as Vice-chairperson of the Association for Natural Language Processing, Member of the Science Council of Japan, and Director of NPO FactCheck Initiative Japan.
}

\end{biography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate



\end{document}

