\documentclass[english]{jnlp_3.3}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{array}
%%%%\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
\usepackage{enumerate}
\usepackage{autobreak}
\usepackage{multirow}
%%%%\usepackage{colortbl}
\usepackage{booktabs}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{28}
\Number{4}
\Month{December}
\Year{2021}
%%%%\Issuetitle{Special Issue: XXXXXXXX}


\received{2021}{2}{26}
\revised{2021}{6}{1}
%%%%\rerevised{2006}{5}{6}
\accepted{2021}{7}{3}

\setcounter{page}{1270}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{System Paper}
\etitle{A Selection Support System for Enterprise Resource Planning Package Components using Ensembles of Multiple Models with Round-trip Translation}
\eauthor{Masao Ideuchi\affiref{AffiRef1} \affiref{AffiRef2} \affiref{AffiRef3} \and Yohei Sakamoto\affiref{AffiRef4} \and Yoshiaki Oida\affiref{AffiRef1} \affiref{AffiRef5} \and Isaac Okada\affiref{AffiRef1} \affiref{AffiRef5} \affiref{AffiRef6} \and Shohei Higashiyama\affiref{AffiRef2} \affiref{AffiRef3} \and Masao Utiyama\affiref{AffiRef2} \and Eiichiro Sumita\affiref{AffiRef2} \and Taro Watanabe\affiref{AffiRef3}}
\eabstract{%
An enterprise resource planning (ERP) package consists of software to support day-to-day business activities and contains multiple components. System engineers combine the most appropriate software components for system integration using ERP packages. Because component selection is a very difficult task, even for experienced system engineers, there is a demand for machine-learning-based systems that support appropriate component selection by reading the text of requirement specifications and predicting suitable components. However, sufficient prediction accuracy has not been achieved thus far as a result of the sparsity and diversity of training data, which consist of specification texts paired with their corresponding components. We implemented round-trip translation at both training and testing times to alleviate the sparsity and diversity problems, adopted pre-trained models to exploit the similarity of text data, and utilized an ensemble of diverse models to take advantage of models for both the original and round-trip translated data. Through experiments with actual project data from ERP system integration, we confirmed that round-trip translation alleviates the problems mentioned above and improves prediction accuracy. As a result, our method achieved sufficient accuracy for practical use.
}
\ekeywords{Round-trip Translation, Data Augmentation, Text Classification, Business Data}

\headauthor{Ideuchi et al.}
\headtitle{A Selection Support System for ERP Package Components using EMMRT}

\affilabel{AffiRef1}{}{FUJITSU LIMITED}
\affilabel{AffiRef2}{}{National Institute of Information and Communications Technology}
\affilabel{AffiRef3}{}{Nara Institute of Science and Technology}
\affilabel{AffiRef4}{}{Ridgelinez Limited}
\affilabel{AffiRef5}{}{The University of Tokyo}
\affilabel{AffiRef6}{}{Senshu University}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

An enterprise resource planning (ERP) package consists of software components for constructing an integrated system for managing day-to-day business activities such as finance, human resources, sales, and logistics. To integrate an ERP system for each customer, system designers must select and combine appropriate components from tens of thousands of choices. In addition to the problem of selecting components from such a large number of candidates, this selection process requires a deep knowledge of business areas and customer requirements. Therefore, only experienced experts can perform this task. For example, to select components for a ``Clearing Accounts Payable'' operation, a system designer must know its sub-operations, such as ``Cash Payment'' and ``Change Accounting Document.'' Additionally, the system designer must be familiar with each business function that is performed by an ERP package's standard functions, such as ``Vendor Line Items'' and ``Post with Clearing.''

Previous studies have developed machine learning systems that automatically suggest components for ERP system developers \cite{RN87,RN88,RN90}. These systems take functional requirement sentences (FRSs), which are Japanese natural language descriptions referring to customer requirements, as inputs and select ERP package components that correspond to the functionality described in the FRSs. However, these systems face two major difficulties caused by the specification of components in FRSs. The first problem is data sparsity, meaning that insufficient data exacerbate the zero-frequency problem. The second problem is the expression diversity of FRSs, which means that each customer may use different expressions to refer to identical components.

We cast the task of suggesting ERP components as a classification task in which a set of suitable components is predicted based on given input descriptions. To solve the problems of data sparsity and expression diversity, we propose a method that uses ensembles of multiple models with neural machine translation (NMT) round-trip translation. The proposed method consists of three steps. 1) Create augmented training and testing data through the round-trip translation of training and testing data, respectively, using domain-adapted N-best translation. 2) Create multiple models that are trained using either the original data or augmented data. 3) Use ensembles of prediction results from each model: one from the original testing data predicted by a model trained on the original training data and the other from the augmented testing data predicted by a model trained on the augmented training data.

We performed experiments using practical data constructed from real documents on past projects. The experimental results demonstrate that the proposed techniques improve accuracy and the final system achieved practical performance. Our main findings can be summarized as follows:

\begin{itemize}
\item An ensemble method that combines models trained separately on original data and augmented data performs better than a single model trained on a mixture of both types of data.
\item The domain adaptation of translation models and N-best translation improve the quality of augmented data.
\item Our ensemble method with round-trip translation is applicable to different neural architectures, including models based on both long short-term memory (LSTM) and bidirectional encoder representations from transformers (BERT).
\end{itemize}

Our error analysis reveals that the domain-adapted NMT generates diverse expressions for a single component, which may increase generalization ability for unseen inputs. Subjective analysis revealed that our method achieves accuracy levels that can satisfy the requirements for practical use.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{ERP Package Component Selection}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Background}

ERP packages are very flexible in their ability to combine various functions and components, and it is simple for a system designer to construct a system at a low cost using such packages. Based on these features, ERP packages have been widely used since the 1990s for the purposes of standardization and system cooperation \cite{RN115,RN116}. The number of components has also increased with the introduction of new technologies such as automation and prediction \cite{RN117}. Therefore, the number of components has become increasingly large, reaching as high as tens of thousands of enterprise system functions provided by a single ERP package. Furthermore, the detailed functionality of each component can be changed by changing parameter settings. Therefore, no individual engineers can understand all components in detail.

When constructing a system, system designers must select components that satisfy customer requirements for every project because requirements differ from customer to customer. Expert system designers can select and set parameters effectively based on their knowledge of the business area and past projects. However, non-experts, even if they are expert software engineers, cannot narrow the scope of investigation to select appropriate components. The incorrect selection of components leads to additional implementation work and inferior system quality, as well as increased cost and delivery time. System designers require many years of training to work as experts in this field based on the need to acquire knowledge and experience from many projects. Therefore, the number of experts cannot increase rapidly.

Selecting components is an important part of a project when designing an ERP system and it tends to be a bottleneck. ERP package component selection support systems are designed to support designers by automatically suggesting appropriate components for a project. Once a candidate component is selected, it is trivial for standard software engineers, even non-experts, to verify whether the selected component satisfies customer requirements by actually operating the component.

To support software development, various recommendation systems for software libraries or APIs have been developed \cite{RN200,RN118,RN198,RN199}. \citeA{RN200} found that conversation co-mentions and text matching from Drupal.org online forums are effective for software module recommendation based on collaborative filtering. In particular, co-mentions work well with popular modules and text matching works well with less popular modules. \citeA{RN118} and \citeA{RN198} used software descriptions as a data source to calculate the similarity between third party libraries. \citeA{RN199} used tags for online articles and constructed a knowledge base to find analogical libraries in other programming languages. The main targets of previous studies on software development support are open-source software, so the usage histories and descriptions of software can be easily obtained. In contrast, the usage histories and descriptions of ERP components are difficult to obtain because many ERP components are used to construct proprietary systems and the information of such systems is kept secret. Therefore, to support software development using ERP components, it is necessary to use limited resources effectively.

In a previous study on ERP software development support, \citeA{RN87} used a business function ontology to suggest related components and graphically display the relationships between components as an explanation for suggestions. They combined business domain knowledge and the text similarity of ERP component descriptions and customer requirements, and achieved a high recall because their business function ontology covered related functions, but the precision was low because all components in the same category were not necessary for every project. \citeA{RN90} simplified ERP software development support as a classification task that uses ERP component text descriptions and introduced round-trip translation to increase the amount and variation of FRSs. Round-trip translation is a method for augmenting data by translating text from one language to another language, and then translating backward. This is a well-known data augmentation technique in the field of machine translation. Their work demonstrated that data augmentation and ensemble prediction yield improved accuracy, although the final accuracy still did not satisfy the requirements for practical use. In this study, we developed a system that performs a classification task, similar to \citeA{RN90}, to suggest ERP components.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Approaches to Data Sparsity and Diversity Problems}

There are three well-known approaches to improving the accuracy of text classification: data augmentation, using a pre-trained model with fine-tuning, and using ensembles of prediction results. To solve the problem of data sparsity and diversity, data augmentation increases the amount and diversity of data, and the use of a pre-trained model exploits the semantic similarity of text from a large amount of pre-trained data. Finally, using ensembles of prediction results enhances the generalization ability of models.

Data augmentation methods for text data include the use of round-trip translation and word replacement \cite{RN119}. Most existing data augmentation methods \cite{RN83,RN86,RN89} increase the accuracy of learning models on general public data such as conversation data. However, we focus on how data augmentation contributes to increasing the accuracy of business data.

Round-trip translation increases the amount of data by obtaining paraphrases using machine translation \cite{RN121,RN123,RN125,RN126,RN124}. Recently, it has become easier to perform NMT locally because NMT libraries \cite{RN94} and parallel corpora \cite{RN85} can be obtained easily. Therefore, applying NMT round-trip translation to business data has also become easier because local NMT maintains data confidentiality. Furthermore, we can expect to obtain high-quality translations of target domain text through the domain adaptation of NMT \cite{RN196,RN197}, which leverages an in-domain corpus in addition to a general domain corpus.

Word replacement generates similar text data by either substituting a word in text \cite{RN122,RN102}, or by swapping or deleting words \cite{RN102}. \citeA{RN102} introduced easy data augmentation (EDA), which improves text classification accuracy and consists of four simple operations. Two of the operations use synonyms to replace or insert words in the text data. The other operations are the deletion and swapping of words.

Pre-trained language representation models, which are trained on large amounts of text, have been shown to improve various tasks in natural language processing (NLP). The transformer model \cite{RN98} was recently proposed and has become the state-of-the-art method in NMT. Additionally, pre-trained models such as BERT \cite{RN105}, XLNet \cite{RN109}, RoBERTa \cite{RN110}, ALBERT \cite{RN111}, and ELECTRA \cite{RN112} are used in various NLP tasks. For handling Japanese, there are some BERT models available that are pre-trained on Japanese text \cite{RN104,RN107,RN108}.

Constructing ensembles of multiple models yields greater generalization ability than using each model individually \cite{RN129}. Different ensemble methods vary in complexity. Simpler methods calculate the linear sum of model results \cite{RN128,RN127}. The linear sum is analogous to the ``or'' operation and works well for integrating different predictions. The product of experts (PoE) \cite{RN113} multiplies similar probability distributions from multiple models. PoE is analogous to the ``and'' operation and works well for integrating similar predictions. More complex methods use deep learning in ensemble models \cite{RN130}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{EMMRT: Ensemble of Multiple Models with Round-trip Translation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Task Definition of ERP Package Component Selection Support}

For an ERP package component selection support system, we considered component selection as a classification task. The data used for training were labeled FRSs from past projects and the data for prediction were FRSs without labels from new projects. Each labeled FRS is a sample $(x, y)$, where $x$ is an FRS that specifies a component with the necessary function, $y \in L$ is a component ID, which is treated as a class label, and $L$ is the set of all labels. The task requires the return of $(x, y^*)$ from an input $x$, where $x$ is an FRS from a new project and $y^*$ is the predicted label.

To develop an ERP package component selection support system, we divided the labeled data $D$ into training data $D_{\mathrm{train}}$ and testing data $D_{\mathrm{test}}$. The prediction for $x$ is defined as
\[
  y^* = \mathrm{Predict}(\mathrm{Model}(M, D), x),
\]
where $\mathrm{Model}(M, D)$ is the model trained for $D$ using method $M$. Similarly, the predictions $Y^*$ for a set of FRSs $X = \{ x_1 , ... , x_n \}$ are denoted as 
\begin{align}
  Y^* &= \mathrm{Predict}(\mathrm{Model}(M, D), X) \nonumber \\
      &= \{\mathrm{Predict}(\mathrm{Model}(M, D), x_1), ..., \mathrm{Predict}(\mathrm{Model}(M, D), x_n) \}. \nonumber
\end{align}
The ensemble result of predictions from two models for $x$ is defined as
\[
  y^* = \mathrm{Ensemble}(E, \{ \mathrm{Predict}(\mathrm{Model}(M_1, D_1), x), \mathrm{Predict}(\mathrm{Model}(M_2, D_2), x) \}),
\]
where $E$ denotes the ensemble method. For data augmentation, we denote the augmented data as $\mathrm{Aug}(D)$, where Aug represents augmentation and is replaced by the name of the augmentation method. For example, if the augmentation method is round-trip translation ($\mathrm{RT}$ in the equation) and the LSTM model is trained using augmented data combined with original data, we define the prediction for $x$ as
\[
  y^* = \mathrm{Predict}(\mathrm{Model}(\mathrm{LSTM}, \{ D_{\mathrm{train}} \cup \mathrm{RT}( D_{\mathrm{train}} ) \}), x).
\]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[b]
\begin{center}
\includegraphics{28-4ia13f1.pdf}
\end{center}
\caption{Overview of EMMRT}
\label{fig:Overview_of_EMMRT}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Overview of EMMRT: Ensemble of Multiple Models with Round-trip Translation}

We propose a method called EMMRT to improve classification accuracy based on text data. EMMRT uses ensembles of multiple models, where each model is trained either on the original data or on data augmented by round-trip translation. An overview of EMMRT is presented in Fig. \ref{fig:Overview_of_EMMRT}.

EMMRT has the following features.
\begin{enumerate}[1.]
  \item It uses pre-trained BERT for computing contextualized embedding and a fully connected linear network that minimizes cross-entropy loss for multi-class classification.
  \item Each of the two classifiers is fine-tuned for FRSs using the augmented training data from round-trip translation based on domain-adapted translation models.
  \item It uses ensembles of separately trained models: one trained on the original training data and the other trained on the augmented training data.
\end{enumerate}

First, we prepared an NMT system trained on a large general bilingual corpus and a small ERP-related bilingual corpus, and then obtained the round-trip translation data for the training and testing data. Next, we separately trained models on the original training data and augmented training data. Finally, we applied the ensemble principle to the prediction scores of the models. One score was obtained from the original testing data and predicted by the model that was trained on the original training data. The other score was the average score from the augmented testing datasets that was predicted by the model that was trained on the augmented training data. We defined a hyperparameter to determine the best mixing ratio based on a linear sum of prediction scores.

EMMRT can be defined as follows according to our notation:
\begin{align}
  \begin{autobreak}
    \mathrm{Ensemble}(\mathrm{Linear}\text{($a$:$b$)}, \{
    \mathrm{Predict}(\mathrm{Model}(\mathrm{jBERT}, D_{\mathrm{train}}), D_{\mathrm{test}}),
    \mathrm{Predict}(\mathrm{Model}(\mathrm{jBERT}, \mathrm{adaptRT}n(D_{\mathrm{train}})), \mathrm{adaptRT}n(D_{\mathrm{test}})\}), \nonumber
  \end{autobreak},
\end{align}
where $\mathrm{Linear}\text{($a$:$b$)}$ denotes a linear sum of the prediction scores with a ratio of $a$:$b$, $\mathrm{jBERT}$ denotes BERT trained on Japanese text, and $\mathrm{adaptRT}n(D)$ denotes the round-trip translation result of $D$ from the domain-adapted NMT using N-best translation. The mixture of the original data and round-trip translated data is denoted as $\{D \cup \mathrm{adaptRT}n(D)\}$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Data Augmentation for Training Data}

Improving the accuracy of prediction requires a large amount and diversity of training data. In other words, the prediction of component IDs requires more labeled FRSs as training data. However, it is difficult to prepare a large amount of business data. To increase the number of labeled FRSs, we used ja-en and en-ja round-trip translation to obtain FRSs with similar meanings, but different expressions. By using N-best translation, we controlled the number of FRSs in the augmented training data. The number of FRSs became N$\times$M times larger when N-best translation for ja-en and M-best translation for en-ja were performed.

As mentioned previously, $D_{\mathrm{train}}$ denotes the original training data and $\mathrm{adaptRT}n(D_{\mathrm{train}})$ denotes the training data augmented by the domain-adapted NMT using N-best translation. For comparisons to a baseline method, we also used $\mathrm{RT}(D_{\mathrm{train}})$ and $\mathrm{RT}n(D_{\mathrm{train}})$, which are outputs from round-trip translation without domain adaptation. The character $n$ following $\mathrm{RT}$ indicates that N$\times$M-best round-trip translation is used for augmentation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Data Augmentation for Testing Data}

In EMMRT, data augmentation is applied to both the training data and testing data, which are round-trip translated and fed into the model trained on round-trip translated data. This means that round-trip translation is also executed in the prediction phase. The final prediction is the ensemble result of two predictions: one from the FRSs in a new project and the other from the round-trip translation results of the FRSs. Because the NMT model is trained on both an ERP-related corpus and general corpus, the round-trip translated FRSs are expected to contain more diverse expressions than the original FRSs. Because we apply round-trip translation to both the training data and testing data, we can narrow the gap between the expressions in the training data and those in the testing data. N$\times$M-best round-trip translation is also useful for testing data augmentation because translation candidates further reduce this gap.

Similar to the training data, we denote the original testing data as $D_{\mathrm{test}}$ and augmented testing data (augmented by domain-adapted NMT using N-best translation) as $\mathrm{adaptRT}n(D_{\mathrm{test}})$. $\mathrm{RT}(D_{\mathrm{test}})$ and $\mathrm{RT}n(D_{\mathrm{test}})$ are the outputs of round-trip translation without domain adaptation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}

In this section, we demonstrate that each element of EMMRT contributes to improving the accuracy of the LSTM and BERT methods by comparing it to a baseline method and the results of an ablation study. The experimental settings are explained first, followed by the experimental results.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Experimental Settings}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Data Preparation}

We collected data from actual ERP package system integration data and explanatory descriptions of standardized business activities published on a website by SAP\footnote{\url{https://www.sap-tcodes.org/}} and METI.\footnote{\url{https://www.meti.go.jp/policy/servicepolicy/contents/management_support/files/download.html}} These data consist of FRSs and their corresponding components. Each component has a component ID, which is an identifier of that component and is attached to each FRS as a class label. Examples of the training data are presented in Table \ref{table:examples_of_training_data}. We gathered FRSs from three types of documents: 765 FRSs from requirement documents for real projects, 3,318 FRSs from explanations of components written in operational workflow and incident logs of specific components, and 1,067 FRSs from public information provided by SAP and METI.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[b]
\input{13table01.tex}
\caption{Examples of training data}
\label{table:examples_of_training_data}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The total number of classes in the training data was 521 and multiple labels were assigned to many instances. However, more than 85\% of the instances had only a single label, so we simplified the task to single-label classification by expanding a multi-label instance into multiple single-label instances, where all labels were correct. For example, a training instance x with two labels $y_1$ and $y_2$ was treated as two instances ($x$, $y_1$) and ($x$, $y_2$), and both expanded instances were assessed as correct for loss calculation if either $y_1$ or $y_2$ was predicted. There were 314 classes with one to five instances, 107 classes with 6 to 10 instances, and 100 classes with 11 or more instances. Therefore, more than half of the classes contained five or fewer instances and there was significant imbalance between classes. To alleviate this imbalanced distribution, we increased the number of instances of minority classes by applying N$\times$M-best round-trip translation to the training data according to the number of instances in the original training data to obtain more than 10 instances of each class after round-trip translation. For example, a 4$\times$3-best round-trip translation was applied to a class that had only one instance in the training data and a 3$\times$2-best round-trip translation was applied to a class that had two instances in the training data. We used GiNZA\footnote{\url{https://megagonlabs.github.io/ginza/}\\Our preliminary experiments revealed that GiNZA performed better than Janome. Therefore, we employed Ginza for the entire study using the LSTM-based model reported in this paper.} \cite{RN106} as a tokenizer for LSTM-based models.

The testing data consisted of FRSs extracted from the same source documents as the training data under the condition that the components had two or more instances. A total of 506 instances were included. We applied a 2$\times$2-best round-trip translation to the testing data and calculated an average score from four testing sets.

To prepare NMT systems, we used the JParaCrawl Parallel corpus (v2.0)\footnote{\url{http://www.kecl.ntt.co.jp/icl/lirg/jparacrawl/}} as a general corpus for training and an approximately 40,000 word in-house ERP parallel corpus (collected from different sources of FRSs) for domain adaptation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{NMT Systems for Round-trip Translation}

We prepared two NMT systems for our experiments. One was a general NMT system that was trained on a general corpus and the other was a domain-adapted NMT system that was trained on a general corpus and an ERP corpus. We selected fairseq\footnote{\url{https://github.com/pytorch/fairseq}} as a modeling toolkit and SentencePiece\footnote{\url{https://github.com/google/sentencepiece}} \cite{RN93} as a sentence tokenizer. We split sentences with a vocabulary size of 32,000 and set the learning parameters based on the WMT19 NMT robustness task \cite{RN100,RN99,RN98}. We then removed sentences with lengths exceeding 250 sub-words. The validation and testing sets, both of which contained 10,000 sentences, were extracted from the complete data of JParaCrawl. After training the translation model for 50 epochs on approximately 10 million sentences, the BLEU scores \cite{papineni-etal-2002-bleu} on the testing set were 29.4 for ja-en and 30.9 for en-ja.

We also investigated the effects of NMT domain adaptation. First, we split the ERP corpus into a training set (99.0\%), validation set (0.5\%), and testing set (0.5\%). We oversampled (by 250 times) the ERP training set and merged it with the JParaCrawl training set before training the domain-adapted translation model. This is because some previous studies \cite{RN85,RN101} reported that oversampling is better than fine-tuning for domain adaptation. The BLEU scores for the testing set of the ERP corpus increased to 46.8 for ja-en and 34.6 for en-ja. These scores are 15.2 and 8.2 points higher, respectively, than those of the model trained using only JParaCrawl.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Classification using BERT Models}

We used the simple transformers\footnote{%
\url{https://simpletransformers.ai/}} implementation of a BERT-based classifier. In this model, the BERT output layer is followed by a linear layer that transforms an input vector into another vector whose dimension is equal to the number of classes, as shown in Fig. \ref{fig:Overview_of_EMMRT}. Japanese text preprocessing for each BERT model was performed based on the experimental procedure used in the NICT BERT.\footnote{\url{https://alaginrc.nict.go.jp/nict-bert/Experiments_on_RCQA.html}}

First, we fine-tuned the BERT models using the original data. All BERT models produce outputs with poor accuracy after one epoch, so we changed the number of fine-tuning epochs to 50. To compare accuracies more precisely, because the fine-tuned BERT base model sometimes produced better results than the fine-tuned BERT large model using all training data, we divided the training data into multiple training and validation sets using a stratified 10-fold split \cite{RN114} and calculated the average accuracy of experiments using the BERT models. We then calculated the prediction scores of multiple models that were fine-tuned either on the original data or augmented data for the ensemble based on the testing data. In this paper, we report the results of the Kyoto large BERT model.\footnote{%
\url{https://nlp.ist.i.kyoto-u.ac.jp/index.php?ku_bert_japanese}\\We examined BERT models that were pretrained on Japanese or multiple languages from Google, Kyoto University, Tohoku University, and NICT in our preliminary experiments. The Kyoto large BERT model performed the best in most experiments.}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Evaluation Metrics}

We adopted an accuracy indicator that can evaluate ranking accuracy because it outputs the ranking of the predicted component IDs ordered by certainty. Specifically, the results are evaluated by the P@k indicator, which is defined as the proportion of results in which the correct component ID is within the top-k-ranked component IDs predicted by the proposed system. We used P@1, P@3, and P@20 for comparisons to the baseline method and from the perspectives of whether the predictions of EMMRT are correct, whether the correct predictions lie near the top, and whether system designers can find correct components by examining prediction results. The target prediction accuracy for EMMRT is P@3=80\% (80\% within the top three) because poor prediction accuracy would waste subsequent work. We also calculated the mean reciprocal rank (MRR) as a total ranking score.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Main Results}

We selected the method from \citeA{RN90} as a baseline method and refer to their method as the mixed training method with round-trip translation (MTRT). MTRT uses a unidirectional LSTM followed by a linear layer with softmax activation as a classification model and Janome\footnote{\url{https://mocobeta.github.io/janome/en/}} as a tokenizer. Notably, MTRT extracts only nouns from original FRSs and uses noun sequences as inputs for its classification model. It merges round-trip-translated data using general NMT with original data for training and separates the round-trip-translated data from the original data for testing. To calculate results, it combines the scores of each component ID from the original testing data and round-trip-translated testing data with a ratio of 2:1. The formulation for their method is defined as follows:
\begin{align}
  \begin{autobreak}
    \mathrm{Ensemble}(\mathrm{Linear}\text{(2:1)}, \{
    \mathrm{Predict}(\mathrm{Model}(\mathrm{LSTM}, \{D_{\mathrm{train}} \cup \mathrm{RT}(D_{\mathrm{train}})\}), D_{\mathrm{test}}),
    \mathrm{Predict}(\mathrm{Model}(\mathrm{LSTM}, \{D_{\mathrm{train}} \cup \mathrm{RT}(D_{\mathrm{train}})\}), \mathrm{RT}(D_{\mathrm{test}})\}), \nonumber
  \end{autobreak}
\end{align}
where $\{D_{\mathrm{train}} \cup \mathrm{RT}(D_{\mathrm{train}})\}$ represents the mixture of the original training data and round-trip-translated training data. The baseline method constructs one model from the mixed training data and combines the prediction scores from the original testing data and the round-trip-translated testing data.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[b]
\input{13table02.tex}
\caption{Comparison of text classification accuracies using all training data}
\label{table:comparison_all_training_data}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We also compared our results to those of three other methods. The first was fine-tuned BERT in the form of two fine-tuned pre-trained BERT models using the original training data and an ensemble of two models with a mixing ratio of 0.5 for fair comparison because EMMRT uses two BERT models. The second was EDA as an alternative data augmentation method. And the third was PoE as an alternative ensemble method. For comparisons to the baseline method, we trained each model for 50 epochs on all training data. The results of comparisons in terms of classification accuracy are presented in Table \ref{table:comparison_all_training_data}. For most experiments, the table contains two rows that represent the linear sum ensemble source scores from the different sets of training data and testing data described in the first and second rows. The third column presents the ratio of Linear(1\textit{st}:2\textit{nd}). The results of the EDA experiment are presented in a single row because it is not an ensemble method and ``-'' reflects this difference in the mixing ratio and ID cells. We used PoE to integrate the three ensemble results (A to C) of the linear sum and each linear sum setting is presented in two rows. The detailed experimental settings for EDA and PoE are discussed in Sections \ref{comparison_to_other_augmentation} and \ref{comparison_with_other_ensemble}, respectively. The final ensemble source of PoE was trained pm translated data using RoBERTa, as described in Section \ref{combination_with_other_data}.

EMMRT improves the accuracy by 13.6 points for P@1, 13.8 points for P@3, and 4.4 points for P@20, compared to the baseline method. EMMRT also outperforms the fine-tuned BERT without data augmentation by a significant margin ($p < 0.001$) based on the Wilcoxon signed-rank test. EDA cannot improve the accuracy of fine-tuned BERTs without data augmentation.\footnote{We also confirmed the experiment settings for fine-tuned BERT without an ensemble and fine-tuned BERT + EDA with an ensemble, but their accuracy did not exceed the accuracy of EMMRT.} PoE works well when it is used to integrate the results of a linear sum. PoE improves the P@20 accuracy, but the calculation cost is three times greater than that of EMMRT.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Ablation Studies}

In this subsection, we report the results of ablation studies using LSTM-based and BERT-based models. We prepared a general NMT model trained using only JParaCrawl and used the same neural network architecture as the baseline method for classification. For comparison to the baseline method, we trained an LSTM-based model for 50 epochs using all training data. To confirm the effects of EMMRT, we changed the experimental settings one by one from EMMRT to the baseline method. Oversampling is denoted by $\mathrm{oversample}(D)$ in the table, which increases the number of instances by the same amount as N$\times$M-best round-trip translation performed on the training data. The results of the ablation study for the LSTM-based model are presented in Table \ref{table:ablation_study_all_training_data}, whose format is the same as that of Table \ref{table:comparison_all_training_data} (``baseline+DS+DA'' corresponds to the same model as EMMRT, except for the use of LSTM instead of BERT). In Table \ref{table:ablation_study_all_training_data}, DS indicates data separation, meaning an ensemble of two models separately trained using either the first or second set of training data (designated in the ID column in Table \ref{table:ablation_study_all_training_data}), whereas the vanilla baseline is an ensemble of two models, both of which were trained on a mixture of two types of training data. DA indicates domain adaptation and OS indicates oversampling.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[b]
\input{13table03.tex}
\caption{Ablation study on LSTM-based model using all training data}
\label{table:ablation_study_all_training_data}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[b]
\input{13table04.tex}
\caption{Ablation study on BERT using stratified 10-fold data}
\label{table:ablation_study_stratified_training_data}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In our experiment on the LSTM-based model, the oversampled training data improved classification accuracy. The accuracy was also improved by the domain adaptation of NMT and by handling augmented data separately. The results from models that combined the original data with augmented data were worse than those of models using separate data. Compared to the results in Table \ref{table:comparison_all_training_data}, the most effective improvement was obtained by using BERT, which improved P@1 by more than 8 points.

Similar to the LSTM-based model, the results of an ablation study on the BERT-based model are presented in Table \ref{table:ablation_study_stratified_training_data}, whose format is the same as that of Table \ref{table:ablation_study_all_training_data}. It is worth noting that these scores cannot be directly compared to those in Table \ref{table:ablation_study_all_training_data} because stratified 10-fold training data were used in these experiments.

Similar to the LSTM-based model, accuracy was improved by domain adaptation and by handling augmented data separately. In contrast to the experiment on LSTM, adjusting the number of instances by oversampling improved the P@1 score, but failed to improve the P@3, P@20, and MRR scores.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Comparison to Other Augmentation Methods}\label{comparison_to_other_augmentation}

We compared EMMRT to two other data augmentation methods: fine-tuned BERT and EDA \cite{RN102}. For fine-tuned BERT, we fine-tuned two pre-trained BERT models using the original training data and used an ensemble of the two models with a mixing ratio of 0.5. For EDA, we used the original EDA source code.\footnote{%
\url{https://github.com/jasonwei20/eda_nlp}} EDA provides four simple operations for data augmentation in English and all of them have positive effects on prediction. We adopted a random swap from EDA because this operation does not depend on WordNet and is applicable to Japanese text, and half of the EDA operations are based on WordNet \cite{RN103} to obtain synonyms. We then ensembled the augmented data using EDA instead of round-trip translated data. In Table \ref{table:comparison_to_EDA_stratified_training_data}, $\mathrm{EDA}(D_{\mathrm{train}})$ denotes the randomly swapped stratified 10-fold training data.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[b]
\input{13table05.tex}
\caption{Comparison to EDA using stratified 10-fold data}
\label{table:comparison_to_EDA_stratified_training_data}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Applying the EDA and EDA + ensemble methods yielded similar accuracy to the fine-tuned BERT without data augmentation, but the scores were inferior to those of EMMRT according to the Wilcoxon signed rank test ($p < 0.001$). This experiment revealed that the ensemble component of EMMRT can also be effective with other data augmentation methods and that EDA can be used as an alternative ensemble method.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Combination with Other Data}\label{combination_with_other_data}

We examined the extent to which accuracy can be improved by EMMRT if fewer data are available. Table \ref{table:comparison_with_less_training_data} presents the classification accuracy of the fine-tuned BERTs when the amount of training data is reduced. $D_{\mathrm{train1K}}$ indicates that the amount of training data is reduced to approximately 1,000 instances and $D_{\mathrm{train2K}}$ indicates that it is reduced to approximately 2,000 instances. In both cases, EMMRT improves all P@k scores and improves the MRR score by more than 0.004 points.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[b]
\input{13table06.tex}
\caption{Combination with less data}
\label{table:comparison_with_less_training_data}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 7
\begin{table}[b]
\input{13table07.tex}
\caption{Comparisons using 10-fold translated data and English language representation models}
\label{table:comparison_using_translated_stratified_training_data}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Additionally, we evaluated methods using English language representation models and English augmented data translated by the ja-en translator instead of Japanese language representation models and Japanese data augmented by round-trip translation. Because various pre-trained language representation models are available for English, we used BERT, XLNet, RoBERTa, ALBERT, and ELECTRA. We used the Simple Transformers implementation in the same manner as the other experiments. The ensemble results are listed in Table \ref{table:comparison_using_translated_stratified_training_data}. $\mathrm{adaptT}n(D)$ denotes the stratified 10-fold translated training data, which are increased by N-best translation with domain-adapted NMT. We omitted the results from ALBERT and ELECTRA because they could not improve accuracy.

Similar to round-trip translation, the classification accuracy of the ensemble results for BERT, XLNet, and RoBERTa, when fine-tuned with translated data, is higher than that of the fine-tuned BERT without data augmentation or ensembling. Although it is not reflected in the table, domain adaptation improves the classification accuracy of the translated data and has a good effect on using ensembles of prediction results. For example, BERT and XLNet are improved by approximately three points in terms of P@1. In particular, RoBERTa significantly improves the accuracy (approximately 9 points at P@1) with domain adaptation. This indicates that the improvement in NMT translation quality facilitates the use of RoBERTa and increases the availability of language representation models.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Combination with Other Ensemble Methods}
\label{comparison_with_other_ensemble}

We also investigated whether multiple ensembles could improve accuracy. Based on the large number of potential combinations, we used PoE \cite{RN113} to aggregate multiple results. In our experiments, the linear sum worked better than PoE for integrating predictions from the models trained on the original data and augmented data, but PoE worked well with linear sum ensemble results because PoE is good at integrating similar predictions. This implies that the characteristics of predictions using round-trip translated data differ from those of predictions using the original data. The linearly summed predictions of both types of data with different ratios are similar. Therefore, we used PoE to integrate the ensemble results produced by the linear sum. The sources used for PoE can be summarized as follows.
\begin{enumerate}[A.]
  \item Use a linear sum ensemble to integrate the results of predictions from the original data and N-best round-trip translated data with domain adaptation using the fine-tuned BERT. The formulation is
  \begin{align}
    \begin{autobreak}
      \mathrm{Ensemble}(\mathrm{Linear}\text{(8:2)},
      \mathrm{Predict}(\mathrm{Model}(\mathrm{jBERT}, D_{\mathrm{train}}), D_{\mathrm{test}}),
      \mathrm{Predict}(\mathrm{Model}(\mathrm{jBERT}, \mathrm{adaptRT}n(D_{\mathrm{train}})), \mathrm{adaptRT}n(D_{\mathrm{test}})). \nonumber
    \end{autobreak}
  \end{align}
  \item Use a linear sum ensemble to integrate the results of predictions from the original data and N-best round-trip translated data without domain adaptation using the fine-tuned BERT. The formulation is
  \begin{align}
    \begin{autobreak}
      \mathrm{Ensemble}(\mathrm{Linear}\text{(8:2)},
      \mathrm{Predict}(\mathrm{Model}(\mathrm{jBERT}, D_{\mathrm{train}}), D_{\mathrm{test}}),
      \mathrm{Predict}(\mathrm{Model}(\mathrm{jBERT}, \mathrm{RT}n(D_{\mathrm{train}})), \mathrm{RT}n(D_{\mathrm{test}})). \nonumber
    \end{autobreak}
  \end{align}
  \item Use a linear sum ensemble to integrate the results of predictions from the original data using the fine-tuned BERT and predictions from the N-best translated data with domain adaptation using the fine-tuned RoBERTa. The formulation is
  \begin{align}
    \begin{autobreak}
      \mathrm{Ensemble}(\mathrm{Linear}\text{(8:2)},
      \mathrm{Predict}(\mathrm{Model}(\mathrm{jBERT}, D_{\mathrm{train}}), D_{\mathrm{test}}),
      \mathrm{Predict}(\mathrm{Model}(\mathrm{RoBERTa}, \mathrm{adaptT}n(D_{\mathrm{train}})), \mathrm{adaptT}n(D_{\mathrm{test}})). \nonumber
    \end{autobreak}
  \end{align}
\end{enumerate}

We selected these settings to confirm whether a synergistic effect could be obtained from the combination of various models that performed well in the previous experiment. The results of PoE with stratified 10-fold training data are presented in Table \ref{table:comparison_and_combination_of_PoE_stratified_training_data}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 8
\begin{table}[b]
\input{13table08.tex}
\caption{Comparison and combination of PoE and 10-fold data}
\label{table:comparison_and_combination_of_PoE_stratified_training_data}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

According to these results, PoE performs poorly when it is used instead of a linear sum, but works well when it is used to integrate the results from a linear sum. P@3 and P@20 are improved by combining the ensemble results from multiple data and models. Although it is not reflected in the table, we also attempted to use an ensemble for integrating EDA results (from Table \ref{table:comparison_to_EDA_stratified_training_data}) and found that this also improved accuracy slightly. However, the accuracy at low epochs was lower than that before the calculation of PoE, so PoE should be used only after learning has converged. Furthermore, the combination of more predictions makes interpretation more difficult and increases the amount of work required to update a model, so the best balance for practical use should be considered. We judged that round-trip translation and ensemble are the best combinations for ERP packages because interpretation can be performed in Japanese and it is possible to increase the amount of training data while continuing to use a model. In future work, additional ensemble methods could be used to combine various models. For example, Scikit-learn StackingClassifier\footnote{\url{https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html}} would be useful for this purpose.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Discussion}

In this subsection, we evaluate the classification accuracy of the proposed method from the perspective of its practical use for ERP package system integration and describe its prospects for additional benefits. We received a response from experienced system designers that ``83\% of P@3 is useful for work.'' This indicates that EMMRT exceeds the target accuracy of P@3=80\% and achieves sufficient accuracy for practical use. As mentioned in Section 1, after referring to the suggestions from EMMRT, system designers conduct subsequent work using the suggested components by selecting and operating them to check whether the components satisfy the requirements of customers. Improving the accuracy of P@1 and P@3 compared to the baseline method leads to a reduction in the time required for subsequent work and improves the quality of work. Additionally, we received positive feedback that the proposed method can speed up the training of system designers by helping them understand and notice related functions.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Analysis}

First, we confirmed the relationship between accuracy improvement and the number of instances in the training data. We divided all classes into three groups: classes containing five or fewer instances, classes containing 6 to 10 instances, and classes containing 11 or more instances. Compared to fine-tuned BERT without an ensemble and round-trip translation, EMMRT improved the MRR of classes containing five or fewer instances in the training data without reducing the MRR of classes containing 11 or more instances in the training data.

Second, we analyzed the predictions based on round-trip translated data to check the number of classes whose classification accuracies changed. Because linear summation produced a better result than PoE in ensembles of predictions based on round-trip translated data, the characteristics of the round-trip translation data appear to be different from those of the original data. According to the differences between the predictions from the original data and those from the round-trip translated data, we analyzed how round-trip translation affects accuracy. We also analyzed the classes and instances that were significantly different before and after the domain adaptation of NMT. Because the data considered in this study contained a large number of classes, over half of the classes contained five or fewer instances, so we counted the number of times the correct labels (P@1) were predicted during 10-fold iterations. The number of correct labels varied between 0 and 10 because in the worst case, none of the 10-fold models would predict the correct label and in the best case, all of the 10-fold models would predict the correct label. We then calculated the differences between the models. The results are listed in Table \ref{table:differences_of_correct_predictions_in_10fold_iterations}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 9
\begin{table}[b]
\input{13table09.tex}
\caption{Differences in number of correct predictions (P@1) between two models using 10-fold iterations}
\label{table:differences_of_correct_predictions_in_10fold_iterations}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Table \ref{table:differences_of_correct_predictions_in_10fold_iterations} presents the model performance before adaptation, model performance after adaptation, and differences between the two results from left to right. The comparison source and comparison destination are described by the training data and testing data that they use. The comparison results are divided into stages from ``$>$+5'' to ``$<$$-5$,'' where ``$>$+5'' indicates that the comparison destination is dramatically improved and ``$<$$-5$'' indicates that the comparison destination is dramatically deteriorated compared to the source. For example, the case in which the correct label is predicted in 2 of 10 iterations in the source and 8 of 10 iterations in the destination is categorized as ``$>$+5.'' The results of this analysis demonstrate that classification based on round-trip translated data is inferior to that based on original data, but new correct answers are acquired in some cases. A comparison between the results before and after domain adaptation reveals both improvement and deterioration, but the improvement is greater than the deterioration. This confirms that the superior prediction results from round-trip translated data also apply to ensemble results.

We also compared the unique word n-grams in the original data and round-trip translated data to examine whether the sparsity and diversity problems were alleviated. For diversity, we examined how many new n-grams that were not in the original data were generated by round-trip translation. For sparsity, we calculated the average number of shared n-grams between the training data and round-trip translated data for each class of ERP components. The results are listed in Table \ref{table:unique_ngrams_in_original_and_RT_data}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 10
\begin{table}[t]
\input{13table10.tex}
\caption{Unique n-grams in original and round-trip translated data}
\label{table:unique_ngrams_in_original_and_RT_data}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

As a result of round-trip translation, the number of unique n-grams in the training data decreased and those of the testing data increased. In the training data, approximately 31\% and 47\% of 1-grams and 2-grams were new expressions, respectively, and approximately 45\% and 70\% of 1-grams and 2-grams in the testing data were new expressions, respectively. The number of shared n-grams between training data and testing data per class of ERP components increased by approximately 3\% for 1-grams and decreased by approximately 22\% for 2-grams. In particular, in classes where the accuracy was improved during the 10-fold iterations, the number of shared n-grams per class increased significantly ($p < 0.001$) according to the Mann-Whitney U test.\footnote{The same trends were observed in shared 3-grams and 4-grams.} In contrast, in classes where the number of shared n-grams per class increased, the accuracy during the 10-fold iterations did not improve in all cases. These results indicate that round-trip translation produces new and more similar expressions for training data and testing data. Therefore, round-trip translation alleviates the problems of diversity and sparsity in some cases. For example, the original expression ``資金繰り表'' (shikin guri hyo) `a cash flow table' is changed to ``資金スケジュール'' (shikin sukejuru) `a schedule of funding' by round-trip translation and another original expression ``入出金予定'' (nyusyukkin yotei) `a plan of receiving and making payment' is changed to ``入出金スケジュール'' (nyusyukkin sukejuru) `a schedule of receiving and making payment.' Both of the original expressions are related to cash flow and funding management, and new expressions are relatively similar. Conversely, in other cases, expressions were changed to new expressions that had different meanings by round-trip translation and the classification accuracy deteriorated. For example, ``照会'' (sho-kai) `refer' was changed to ``問い合わせ'' (toiawase) `inquiry' because ``照会'' is also used to express inquiry. Consequently, in some cases, more accurate predictions that could not be derived from the original data were produced from round-trip translated data and the ensemble model took advantage of both types of results.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

To support system designers in ERP package integration, we improved the accuracy of the automatic suggestion of software components. To improve accuracy, we applied data augmentation through round-trip translation to both training data and testing data using a domain-adapted NMT and proposed an effective ensemble method for round-trip translated data. As a result, we achieved a sufficient level of accuracy for practical use.

The proposed method can also be applied to text classification tasks other than ERP component suggestion and may improve accuracy in various business areas where accuracy is currently inadequate based on a lack of data for machine learning.

This study demonstrated that machine translation, which has already been used to support human-to-human communication, can also be used to support human-machine interaction. We found that NLP can even benefit areas with limited data by using a large-scale trained model with rich text data, such as BERT, and using NMT round-trip translation for data augmentation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

We are very grateful to Keiichi Zukeran, Masaaki Abe, and Ryuzo Horiguchi, who are SAP consultant experts at FUJITSU LIMITED and provided intensive support for the arrangement of experiments, including gathering data, labeling training data, and advising on SAP-related resources. They shared a variety of business knowledge related to ERP and contributed significantly to the development and evaluation of our system.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{13refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Masao Ideuchi}{%
He received a B.S. degree in Environment and Information Studies from Keio University, Japan, in 2004, and an M.S. degree from the Graduate School of Media and Governance of the Cyber Informatics Program of Keio University, Japan, in 2006. From 2006 to 2020, he worked on the development of products and services related to natural language processing at FUJITSU LIMITED and the Fujitsu Research Institute. He has been a member of the National Institute of Information and Communications Technology (NICT) and a Ph.D. student at the Nara Institute of Science and Technology (NAIST) since 2020.
}

\bioauthor[:]{Yohei Sakamoto}{%
He received an M.S. degree in Particle Physics at Kyoto University, Japan, in 2013. Currently, he is a data scientist at Ridgelinez Ltd., proposing and supporting PoC for AI implementation projects and business process improvement in companies.
}

\bioauthor[:]{Yoshiaki Oida}{%
He received a B.S. degree in Engineering and a M.S. degree in Human and Engineered Environmental Studies from the University of Tokyo in 2010 and 2012, respectively. He worked on the development of a discrete event simulator for resource management across multiple information system development projects and earned a Ph.D. from the Graduate School of Frontier Sciences, the University of Tokyo, in 2019. He is currently a managing engineer at FUJITSU LIMITED and since 2020, he has been a part-time lecturer in the Faculty of Engineering of the University of Tokyo. His research interests include model-based systems engineering in the manufacturing and information systems industries.
}

\bioauthor[:]{Isaac Okada}{%
He received his Ph.D. in Science from the Graduate School of Frontier Sciences, the University of Tokyo, in 2019. He is now the Executive Engineer of the Digital Transformation Business Unit at FUJITSU LIMITED, a Visiting Professor in the Faculty of Commerce of Senshu University, Special Lecturer in the Faculty of Engineering of the University of Tokyo, and Part-time Lecturer at the School of Computing of the Tokyo Institute of Technology. His research interests include system dynamics, artificial intelligence (machine learning), natural language processing, knowledge management technologies, semantic technologies, and metadata for business.
}

\bioauthor[:]{Shohei Higashiyama}{%
He received a B.S. degree in Engineering and M.S. degree in System Informatics from Kobe University in 2012 and 2014, respectively. He was a researcher at the NEC Corporation from 2014 to 2019. He is currently a technical researcher at the National Institute of Information and Communications Technology (NICT) and Ph.D. student at the Nara Institute of Science and Technology (NAIST).
}

\bioauthor[:]{Masao Utiyama}{%
He is an executive researcher at the National Institute of Information and Communications Technology, Japan. He completed his doctoral dissertation at the University of Tsukuba in 1997. His main research field is machine translation.
}

\bioauthor[:]{Eiichiro Sumita}{%
He is an NICT fellow, the Associate Director-General of the Advanced Speech Translation Research and Development Promotion Center (ASTREC), and also serves as the president of the International Association for Machine Translation (IAMT) and the president of the Asia-Pacific Association for Machine Translation (AAMT). His research interests include machine translation and e-learning.
}

\bioauthor[:]{Taro Watanabe}{%
He received his B.E. and M.E. degrees in Information Science from Kyoto University, Kyoto, Japan in 1994 and 1997, respectively, and obtained an M.S. degree in language and information technologies from the School of Computer Science, Carnegie Mellon University in 2000. In 2004, he received a Ph.D. in informatics from Kyoto University, Kyoto, Japan. He also worked as a researcher at ATR, NTT, NICT, and Google Inc. He is a professor at the Nara Institute of Science and Technology (NAIST). His research interests include natural language processing, machine learning, and statistical machine translation.
}
\end{biography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate






\clearpage


\clearpage





















\end{document}

