\documentclass[tombow,english]{jnlp_2.3}
\usepackage{jnlpbbl_1.3}
\usepackage[dvipdfm]{graphicx}
\usepackage{gentombow}
\usepackage{amsmath}
\settombowbleed{5mm}
\usepackage{hangcaption_jnlp}
\usepackage{array}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
%%%%\makeatletter
%%%%\def\@{\spacefactor3000\space}
%%%%\makeatother

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
\usepackage{multirow}
%%%%\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{url}
%%%%%\usepackage{pgfplots}
%%%%\usepackage{subcaption}
%%%%\usepackage{arydshln}
%%%%\newdimen\figrasterwd
%%%%\figrasterwd\textwidth
\DeclareMathOperator{\clip}{clip}
\DeclareMathOperator{\softmax}{softmax}
\DeclareMathOperator{\dist}{dist}
\DeclareMathOperator{\mode}{mode}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\renewcommand\UrlFont{\rmfamily}

\Volume{27}
\Number{2}
\Month{June}
\Year{2020}
%%%%\Issuetitle{「　　　　」特集号}

\received{2019}{11}{26}
\revised{2020}{2}{29}
%%%%\rerevised{2006}{5}{6}
\accepted{2020}{4}{6}
%%%%\Issuetitle{「　　　　」特集号}

\setcounter{page}{445}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Syntax-based Transformer for Neural Machine Translation}
\eauthor{Chunpeng Ma\affiref{HIT} \and Akihiro Tamura\affiref{Ehime} \and Masao Utiyama\affiref{NICT} \and Eiichiro Sumita\affiref{NICT} \and Tiejun Zhao\affiref{HIT}} 
\eabstract{%
The Transformer \cite{vaswani2017attention}, which purely depends on attention mechanism, has achieved state-of-the-art performance on machine translation (MT). However, syntactic information, which has improved many previous MT models, has not been utilized explicitly by Transformer. We propose a syntax-based Transformer for MT, which incorporates source-side syntax structures generated by the parser into the self-attention and positional encoding of the encoder. Our method is general in that it is applicable to both constituent trees and packed forests. Evaluations on two language pairs show that our syntax-based Transformer outperforms the conventional (non-syntactic) Transformer. The improvements of BLEUs on English-Japanese, English-Chinese and English-German translation tasks are up to 2.32, 2.91 and 1.03, respectively. Furthermore, our ablation study and qualitative analysis demonstrate that the syntax-based self-attention does well in learning local structural information, while the syntax-based positional encoding does well in learning global structural information.}
\ekeywords{Machine Translation, Deep Learning, Syntactic Information}

\headauthor{Ma et al.}
\headtitle{Syntax-based Transformer for Neural Machine Translation}

\affilabel{HIT}{}{Harbin Institute of Technology}
\affilabel{Ehime}{}{Ehime University}
\affilabel{NICT}{}{National Institute of Information and Communications Technology}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

In recent years, neural machine translation (NMT) has been developing rapidly and has became the de facto standard approach for MT. Dominant NMT models follow the encoder-decoder paradigm. Based on networks used by the encoder/decoder, these models can be divided into several categories: convolutional neural network (CNN) based models \cite{gehring2017convolutional}, recurrent neural network (RNN) based models \cite{sutskever2014sequence}, etc. Attention mechanism \cite{bahdanau2014neural,luong2015effective} plays an important role in these models. \citeA{vaswani2017attention} proposed the Transformer, which is based solely on attention mechanisms. Compared with CNN/RNN-based NMT models, Transformer has two major advantages: (1) it avoids the sequential dependencies in RNN-based NMT models, so that training can be parallelized and can be done efficiently, and (2) it uses self-attention mechanism and positional encoding, so that a wide range of context can be used. Therefore, we focus on Transformer here.

Effectiveness of explicit usage of syntactic information has been proved by plenty of previous research. Syntactic information can be used in various models, including models in statistical machine translation \cite{mi2008forest2}, RNN-based NMT \cite{kuncoro2018lstm} and CNN-based NMT \cite{watanabe2017cky}. Syntactic information can be used on either the source-side \cite{eriguchi2017learning}, or the target-side \cite{aharoni2017towards}, or both \cite{wu2018dependency}. Syntactic information can be represented as constituent trees \cite{eriguchi2016tree}, packed forests \cite{ma2018forest}, or graphs \cite{hashimoto2017neural}. There are also some attempts at using syntactic information explicitly in Transformer \cite{strubell2018linguistically}. Their method utilized syntactic information from one single tree, while our proposed method is applicable to forests as well.

It is not straightforward to adapt RNN-based syntactic NMT methods for the Transformer, especially forest/graph-based methods, because of the inherent difference on network structures. Here, we propose a syntax-based Transformer for MT, which encodes syntactic structures (constituent trees or packed forests \cite{huang2008forest}) on the source-side. In particular, we propose a syntax-based self-attention mechanism (Section \ref{ssec:forestsa}) and syntax-based positional encoding (Section \ref{ssec:forestpe}) for the encoder of the Transformer. The proposed syntax-based self-attention mechanism determines which words and/or constituent labels to pay attention to, on the basis of syntactic structures, rather than paying attention to all the words and constituent labels in the syntactic structure. The proposed syntax-based positional encoding encodes the syntactic distance between nodes in the syntactic structures, rather than encoding the absolute position \cite{vaswani2017attention} or relative word distance \cite{shaw2018self}.

This paper makes the following contributions.
\begin{itemize}
\item We propose a general syntax-based Transformer, which is applicable to both constituent trees and packed forests. It can encode large amount of syntactic information efficiently in parallel.
\item Experiments on two datasets demonstrate the good performance of our syntax-based Transformer compared with the non-syntactic counterpart. This indicates the usefulness of using syntactic information explicitly.
\item Qualitative analysis shows that our syntax-based Transformer learns syntactic characteristics successfully, and the proposed two modules learn syntactic information from different aspects.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Background}

Different architectures in NMT, especially the Transformer, are reviewed in this section.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Architectures in NMT}
\label{ssec:bgtrans}

Generally, models in NMT follow the encoder-decoder paradigm, which first encodes the input (plain sentences, syntactic structures, etc.) of the source language with a neural network (RNN or CNN, etc.) and generates the representation (a fixed-length vector or a sequence of vectors) of the input, and then generates the target sentence by decoding the representation. 

The RNN-based NMT models are composed of two RNNs for the encoder and decoder, respectively. 
This kind of models has been adopted into industrial systems \cite{wu2016google}. Despite these success, the sequential nature of RNNs makes the models difficult to be trained in parallel. Therefore, the training process is very time-consuming.

The CNN-based NMT models stack multiple convolutional layers on both encoder and decoder \cite{gehring2017convolutional}. 
Training can be fully parallelized and can be greatly accelerated by using GPUs. However, CNNs capture only limited context and thus long-term dependencies are difficult to learn, which may deteriorate the translation of long sentences.

The Transformer \cite{vaswani2017attention} is designed to address the disadvantages of the above two models. The encoder is a stack of layers. Each layer consists of two submodules: a self-attention network and a feed-forward network. The decoder is also a stack of layers. Compared with layers in the encoder, the decoder counterpart contains one more submodule: the encoder-decoder attention. At the bottom of the encoder and the decoder, positional encodings are added to the input embeddings. With the help of layer normalization \cite{ba2016layer} and residual connection \cite{he2016deep}, the Transformer achieves state-of-the-art translation performance.

It should be noted that: (1) the Transformer uses self-attention mechanism in both encoder and decoder, and pays attention to all the inputs, so that long-term dependencies can be learned easily as RNN-based models, and (2) the Transformer encodes the positional information by using the positional encoding, so that training can be parallelized as CNN-based models. We incorporate syntactic information in both of them. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Self-attention Mechanism}
\label{ssec:bgsa}

Assume that there are $N$ layers in the encoder. For the $l$-th layer, denote the input and output sequence of the self-attention submodule as $\mathbf{x}^{(l)}=(x_1^{(l)},\ldots,x_n^{(l)})$ and $\mathbf{z}^{(l)}=(z_1^{(l)},\ldots,z_n^{(l)})$, respectively, where $n$ is the length of the input sequence and $l\in[1,N]$. For $i\in[1,n]$, we calculate $z_i^{(l)}\in\mathbb{R}^{d}$ as follows:
\begin{align}
z_i^{(l)} & =\sum_{j=1}^{n}\alpha_{ij}(x_{j}^{(l)}W^{V}), \label{eq:zi}\\
\alpha_{ij}^{(l)} & =\softmax(\beta_{ij}^{(l)}), \label{eq:alpha}\\
\beta_{ij}^{(l)} & =\frac{(x_{i}^{(l)}W^{Q})(x_{j}^{(l)}W^{K})^{\top}}{\sqrt{d}} \label{eq:eij},
\end{align}
where $W^{V}, W^{Q}, W^{K} \in\mathbb{R}^{d\times d}$ are parameter matrices. Note that each $z_{i}^{(l)}, i\in[1,n]$ is calculated from all the tokens in $(x_1^{(l)},...,x_n^{(l)})$.

Denote the source-side word sequence as $(w_1,\ldots,w_n)$, then $x_i^{(1)}$ is calculated as follows:
\begin{equation}
x_i^{(1)}=W^{E}[w_i],\label{eq:emb}
\end{equation}
where $W^{E}\in\mathbb{R}^{d\times\vert V\vert}$ is the embedding matrix and $\vert V\vert$ is the vocabulary size. $[\cdot]$ is an operator to get the specific column of a matrix, therefore $x_i^{(1)}\in\mathbb{R}^{d}$ is the $w_i$-th column of $W^{E}$.

The output sequence $(z_1^{(l)},\ldots,z_n^{(l)})$ of the self-attention submodule is fed to the feed-forward submodule, and the output of the feed-forward submodule is fed to the next layer as $(x_1^{(l+1)},\ldots,x_n^{(l+1)})$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Positional Encoding}
\label{ssec:bgpe}

Different from RNN-based and CNN-based models, the self-attention mechanism itself is unable to encode the order of words in the input sequence. To make use of word order information, positional information (either absolute position or relative position) should be inserted into the model.

\citeA{vaswani2017attention} encodes absolute positional information by modifying Equation (\ref{eq:emb}):
\begin{equation}
\begin{aligned}
x_i^{(1)}& =W^{E}[w_i]+PE(i),\\
PE(i)_{2k} & =\sin(i/10000^{2k/d}), \\
PE(i)_{2k+1} & =\cos(i/10000^{2k/d}),
\end{aligned}
\end{equation}
where $2k$ and $2k+1$ are even and odd dimension indices, respectively. They hypothesize the choice of sinusoidal function makes the model be able to learn relative positions, because of the periodicity of the function. This property is also shared by the relative positional encoding introduced below.

On the other hand, \citeA{shaw2018self} encodes relative positional information by modifying Equations (\ref{eq:zi}) and (\ref{eq:eij}) as follows, respectively:
\begin{align}
z_i^{(l)} & =\sum_{j=1}^{n}\alpha_{ij}(x_{j}^{(l)}W^{V}+a_{ij}^{V}), \label{eq:relpe1} \\
\beta_{ij} & =\frac{(x_{i}^{(l)}W^{Q})(x_{j}^{(l)}W^{K}+a_{ij}^{K})^{\top}}{\sqrt{d}}, \label{eq:relpe3}
\end{align}
where $a_{ij}^{K}, a_{ij}^{V}\in\mathbb{R}^{d}$ are vectors representing the relative positions between words. They are calculated as follows:
\begin{align}
a_{ij}^{K} & =p_{\clip(j-i,k)}^{K}, \label{eq:aijk9}\\
a_{ij}^{V} & =p_{\clip(j-i,k)}^{V}, \label{eq:aijk0}\\
\clip(m,k) & =\max(-k,\min(k,m)), \label{eq:relpe2}
\end{align}
where $k$ is the window size, $(p_{-k}^{K},\ldots,p_{k}^{K})$ and $(p_{-k}^{V},\ldots,p_{k}^{V})$ are relative position representations to be learned.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Syntax-based NMT}
\label{ssec:bgforest}

As for tree-based NMT models, a lot of different methods have been proposed.
Trees can be used in either the source side \cite{li2017modeling} or the target side \cite{aharoni2017towards}, or both \cite{wu2018dependency}, can be encoded either using tree-structured neural networks \cite{eriguchi2016tree} or with the help of linearization \cite{sennrich2016linguistic}, and can be either constituent trees \cite{chen2017improved} or dependency trees \cite{wu2017sequence}.
As for forest-based NMT models, \citeA{ma2018forest} is the first attempt, where linearized packed forests are encoded using RNNs in order to make the model robust to parsing errors. \citeA{zaremoodi2018incorporating} encode the packed forests using tree structured neural networks.

One important issue of syntax-based NMT models is the training efficiency. Since most of the models are based on RNNs, encoding input tokens sequentially, it is difficult to parallelize training. Furthermore, syntactic structures (especially packed forests) contain much more information than plain sentences. This aggravates the efficiency problem. On the contrary, the Transformer can be trained efficiently. However, as far as we know, syntactic information has never been used explicitly in the Transformer for MT. Therefore it is quite appealing to combine the advantages of both, which is the focus of this paper.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Methodology}

We modify two modules of the encoder for the syntax-based Transformer: syntax-based self-attention mechanism (Section \ref{ssec:forestsa}) and syntax-based positional encoding (Section \ref{ssec:forestpe}). Syntactic information is incorporated in both of them.

Note that although our syntax-based Transformer receives a sequence (i.e., a linearized sequence of a syntax structure) as a conventional Transformer does, our syntax-based Transformer is independent of the linearization algorithm used for generating the input sequence. This is a remarkable difference from previous syntax-based NMT models based on linearization \cite{li2017modeling,chen2017improved}. Since there is no evidence for the preference of any linearization methods, it is more reasonable to propose an encoding method which is equivalent to them all.

Only the encoder is involved in the proposed method, while the decoder is identical to the baseline Transformer.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Syntax-based Self-attention}
\label{ssec:forestsa}

Since a constituent tree can be regarded as a specific instance of a packed forest, we first give the description of our method based on packed forests. 

Given the source-side word sequence $(w_1,\ldots,w_n)$, the parser generates the packed forest $\mathcal{F}=\langle\mathcal{V},\mathcal{E}\rangle$. We assume that the nodes in $\mathcal{V}$ are sorted such that all words are located in front of constituent labels, i.e., $\mathcal{V}=(w_1,\ldots,w_n,v_{1},\ldots,v_{m})$. Here, $w_i(i\in[1,n])$ is a word, and $v_i(i\in[1,m])$ is a constituent label accompanied with its span, which means that forest nodes like ``NP$_{[2,3]}$'' and ``NP$_{[2,4]}$'' are different. $(v_{1},\ldots,v_{m})$ is sorted by some kind of sorting algorithm (e.g., top-down style linearization \cite{vinyals2015grammar} or bottom-up style linearization \cite{ma2017deterministic}, etc.).\footnote{%
Different ordering of words and constituent labels correspond to different input sequences, which further influence the final output.} $\mathcal{E}=\{e_1,\ldots,e_{l}\}$ is the set of hyperedges, and each $e_i(i\in[1,l])$ is a three-tuple $e_i=\langle\mathcal{H}_i,\mathcal{T}_i,\mathcal{S}_{i}\rangle$, where $\mathcal{H}_i,\mathcal{T}_i\in\mathcal{V}^{*}$ are heads and tails of $e_i$, and $\mathcal{S}_i\in\mathbb{R}_{\leq 0}$ is the score given by the parser, reflecting the confidence of the parser to the hyperedge.\footnote{%
Although the original definition of a head in the packed forest is $\mathcal{H}_{i}\in\mathcal{V}$, the parser released by \citeA{huang2008forest}, who defines the original packed forests, permits multiple parents for each node. Therefore, we follow the extended definition (i.e., $\mathcal{H}_{i}\in\mathcal{V}^{*}$).} The score is the logarithm of a probability, therefore it is non-positive.

Then, $(x_1^{w},\ldots,x_n^{w},x_1^{v},\ldots,x_m^{v})$ is calculated:\footnote{%
For simplicity of notations, henceforth, we omit the superscript $(l)$, the index of layers in the encoder.}
\begin{equation}
x_{i}^{w}=W^{Ew}[w_i],~~~~x_{i}^{v}=W^{Ev}[v_i],
\end{equation}
where $W^{Ew}\in\mathbb{R}^{d\times\vert V^{w}\vert}, W^{Ev}\in\mathbb{R}^{d\times\vert V^{v}\vert}$ are embedding matrices. $V^{w}$ and $V^{v}$ are source language dictionary and constituent label dictionary, respectively. $\vert V^{w}\vert$ is the vocabulary size of the source language, and $\vert V^{v}\vert$ is the total number of constituent labels. For convenience, we denote $X^{w}=(x_1^{w},\ldots,x_{n}^{w})$ and $X^{v}=(x_{1}^{v},\ldots,x_{m}^{v})$.

Then, we calculate the sequence $(z_1^{w},\ldots,z_{n}^{w},z_{1}^{v},\ldots,z_{m}^{v})$ on the basis of the source-side syntactic information given by the constituent tree or packed forest. In particular, rather than using Equation (\ref{eq:eij}), we compute $\beta_{ij}$ as follows: 
\begin{equation}
\beta_{ij}=
\begin{cases}
\frac{1}{\sqrt{d}}(x_{i}W^{Q})(x_{j}W^{K})^{\top}, & x_j\in\mathcal{N}(x_{i}), \\
-\infty, & x_j\not\in\mathcal{N}(x_{i}),
\end{cases}
\end{equation}
where $x_i,x_j\in X^{w}\cup X^{v}$. The function $\mathcal{N}(x)$ returns the syntax-based neighborhood of $x$. Intuitively, only the words (constituent labels) which are near in the tree or packed forest should be paid attention to, and the other words (constituent labels) are safe to be ignored. The syntax-based neighborhood function is as follows:\footnote{%
For simplicity of notations, we discriminate neither $x^{w}_{i}$ and $w_{i}$, nor $x_i^{v}$ and $v_{i}$, for the case with no ambiguity.}
\begin{equation}
\mathcal{N}(x)=
\begin{cases}
X^{w}\cup\mathcal{N}^{\downarrow}(x,d^{\downarrow}), & x\in X^{w}, \\
\mathcal{N}^{\uparrow}(x,d^{\uparrow})\cup\mathcal{N}^{\downarrow}(x,d^{\downarrow}), & x\in X^{v},
\end{cases}
\end{equation}
where $d^{\downarrow}$ and $d^{\uparrow}$ are the hyperparameters reflecting the syntactic depths in top-down and bottom-up directions, respectively. For words, we assume that all the words are useful, and syntactic neighbors are also useful. For constituent labels, we assume that only syntactic neighbors are useful. The functions $\mathcal{N}^{\uparrow}(x,d)$ and $\mathcal{N}^{\downarrow}(x,d)$ are syntactic neighbors in the bottom-up and top-down directions, respectively, and are defined recursively as follows:
\begin{align}
\mathcal{N}^{\uparrow}(x,d) &= x\cup\bigg(\bigcup_{y\in\mathcal{T}(x)}\mathcal{N}^{\uparrow}(y,d-1)\bigg),\\
\mathcal{N}^{\downarrow}(x,d) &= x\cup\bigg(\bigcup_{y\in\mathcal{H}(x)}\mathcal{N}^{\downarrow}(y,d-1)\bigg),
\end{align}
and the terminal conditions are:
\begin{align}
\mathcal{N}^{\uparrow}(x,0)=\mathcal{N}^{\downarrow}(x,0) &=\{x\},\\
\mathcal{N}^{\uparrow}(\emptyset,d)=\mathcal{N}^{\downarrow}(\emptyset,d) &=\emptyset,~~~~\forall d\in\mathbb{Z}_{+}.
\end{align}
$\mathcal{H}(x)$ and $\mathcal{T}(x)$ are the parents set and children set of $x$, respectively. They are defined as follows:
\begin{align}
\mathcal{H}(x) &= \{v\in\mathcal{V}\mid \exists \langle \mathcal{H}, \mathcal{T}, \mathcal{S}\rangle\in\mathcal{E}, x\in \mathcal{T}\land v\in\mathcal{H}\}, \\
\mathcal{T}(x) &= \{v\in\mathcal{V}\mid \exists \langle \mathcal{H}, \mathcal{T}, \mathcal{S}\rangle\in\mathcal{E}, x\in \mathcal{H}\land v\in\mathcal{T}\}, 
\end{align}
Therefore, the top-down syntactic neighborhood of a node $x$ should consist of $x$ itself and all the ancestors within depth $d^{\downarrow}$, and similarly for the bottom-up syntactic neighborhood. Note that for constituent trees, there is only one hyperedge connected to one node. In this way, syntactic information is incorporated into the calculation of the self-attention.

Furthermore, as demonstrated by \citeA{ma2018forest}, scores are important for the performance of syntax-based NMT systems. To utilize the scores, we calculate the attention by Equation (\ref{eq:alpha23}) rather than Equation (\ref{eq:alpha}), which is similar to the Score-on-Attention model of \citeA{ma2018forest}.
\begin{equation}
\alpha_{ij}=
\begin{cases}
s(x_i,x_j)\cdot\softmax(\beta_{ij}),& x_j\in\mathcal{N}(x_i), \\
0,& x_j\not\in\mathcal{N}(x_i),
\end{cases}
\label{eq:alpha23}
\end{equation}
where $s(x_i,x_j)$ is calculated as follows:
\begin{equation}
s(x_i,x_j)=\sum_{\mathcal{E}_{k}\in\mathcal{H}_{e}(x_i,x_j)\cup\mathcal{T}_{e}(x_i,x_j)}\prod_{\langle\cdot,\cdot,\mathcal{S}_{l}\rangle\in\mathcal{E}_{k}}\exp(\mathcal{S}_{l}),
\end{equation}
where the functions $\mathcal{H}_e$ and $\mathcal{T}_e$ return the sets of hyperedge sequences, and are defined as follows:
\begin{gather}
\begin{aligned}
\mathcal{H}_{e}(x,y)=\{(&\langle\mathcal{H}_{1},\mathcal{T}_{1},\mathcal{S}_{1}\rangle,\ldots,\langle\mathcal{H}_{t},\mathcal{T}_{t},\mathcal{S}_{t}\rangle)\mid x\in\mathcal{T}_{1}\land y\in\mathcal{H}_{t}\land\langle\mathcal{H}_{1},\mathcal{T}_{1},\mathcal{S}_{1}\rangle\in\mathcal{E}\\
&\land(\forall i\in\mathbb{Z}\cap[2, t],\langle\mathcal{H}_{i},\mathcal{T}_{i},\mathcal{S}_{i}\rangle\in\mathcal{E}\land\mathcal{H}_{i-1}\cap\mathcal{T}_{i}\neq\emptyset)\},
\end{aligned}
\\
\begin{aligned}
\mathcal{T}_{e}(x,y)=\{(&\langle\mathcal{H}_{1},\mathcal{T}_{1},\mathcal{S}_{1}\rangle,\ldots,\langle\mathcal{H}_{t},\mathcal{T}_{t},\mathcal{S}_{t}\rangle)\mid x\in\mathcal{H}_{1}\land y\in\mathcal{T}_{t}\land\langle\mathcal{H}_{1},\mathcal{T}_{1},\mathcal{S}_{1}\rangle\in\mathcal{E}\\
&\land(\forall i\in\mathbb{Z}\cap[2, t],\langle\mathcal{H}_{i},\mathcal{T}_{i},\mathcal{S}_{i}\rangle\in\mathcal{E}\land\mathcal{T}_{i-1}\cap\mathcal{H}_{i}\neq\emptyset)\},
\end{aligned}
\end{gather}
where $t$ is the length of the sequence. Intuitively, each element in $\mathcal{H}_{e}$ (or $\mathcal{T}_{e}$) is a path between node $x$ and $y$ in the bottom-up (or top-down) direction. In this way, scores are utilized by the syntax-based self-attention mechanism.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Syntax-based Positional Encoding}
\label{ssec:forestpe}

The syntax-based self-attention mechanism does not capture the position of each node in the constituent tree/packed forest. Therefore, we propose a syntax-based positional encoding for directly encoding syntactic positional information. We can adapt the absolute/relative positional encoding to the syntax-based machine translation model straightforwardly. We first linearize the constituent tree/packed forest, and calculate the positional encoding of each word/label based on its position in the sequence. However, this naive adaptation of the absolute/relative positional encoding depends on the order of nodes in the sequence, while the optimal syntactic sequence for NMT is an open problem. Therefore, we propose a syntax-based positional encoding that relies only on pure syntactic structures (constituent trees/packed forests) and does not depend on the linearization method of the syntactic structures.

The proposed syntax-based positional encoding is defined by modifying Equation (\ref{eq:zi}) and (\ref{eq:eij}) as follows:
\begin{align}
z_i & =\sum_{x_j\in\mathcal{N}(x_i)}\alpha_{ij}(x_{j}W^{V}+b_{ij}^{V}), \\
\beta_{ij}&=
\begin{cases}
\frac{1}{\sqrt{d}}(x_{i}W^{Q})(x_{j}W^{K}+b_{ij}^{K})^{\top},&x_j\in\mathcal{N}(x_{i}), \\
-\infty,&x_j\not\in\mathcal{N}(x_{i}),
\end{cases}
\end{align}
where $b_{ij}^{K},b_{ij}^{V}\in\mathbb{R}^{d}$ are two kinds of distributed representation of syntactic distance between two nodes $x_i$ and $x_j$. They are calculated as follows:
\begin{equation}
b_{ij}^{K}=
\begin{cases}
q^{wK}_{\dist(i,j)}, & x_i\in X^{w},\\
q^{vK}_{\dist(i,j)}, & x_i\in X^{v},
\end{cases}
~~~
b_{ij}^{V}=
\begin{cases}
q^{wV}_{\dist(i,j)}, & x_i\in X^{w},\\
q^{vV}_{\dist(i,j)}, & x_i\in X^{v}.
\end{cases}
\end{equation}
Here, every $q$ is the distributed representation of syntactic distance to be learned. Similar to $p$ in Equation (\ref{eq:aijk9}) and (\ref{eq:aijk0}), they are vectors in $\mathbb{R}^{d}$. $\dist(i,j)$ is defined as follows:
\begin{equation}
\dist(i,j)=\mode\big(\{\vert\mathcal{E}\vert:\mathcal{E}\in\mathcal{H}_{e}(x_i,x_j)\cup\mathcal{T}_{e}(x_i,x_j)\}\big),
\end{equation}
where $\mode(\cdot)$ returns the element that appears most often in the argument set.\footnote{%
When several different elements meet the condition, the minimum one is returned.} Intuitively, $\dist(i,j)$ can be regarded as the most frequent path length between $x_i$ and $x_j$ in two directions according to the syntactic structures. According to their definitions, $d^{\uparrow}$ and $d^{\downarrow}$ are the maximum syntactic distances in two directions, respectively, while $\dist(i,j)$ is the syntactic distance of the most frequent path. Therefore, we can see that
\begin{equation}
\max_{i,j}\dist(i,j)\leq\max(d^{\uparrow},d^{\downarrow})\equiv d^{\updownarrow}, \forall i,j,
\end{equation}
therefore our syntax-based positional encodings to be learned is a $4\times(d^{\updownarrow}+1)$ weight matrix:
\begin{equation}
Q\equiv\left(
\renewcommand{\arraystretch}{1}
\begin{array}{ccc}
q_{0}^{wK} & \ldots & q_{d^{\updownarrow}}^{wK} \\
q_{0}^{wV} & \ldots & q_{d^{\updownarrow}}^{wV} \\
q_{0}^{vK} & \ldots & q_{d^{\updownarrow}}^{vK} \\
q_{0}^{vV} & \ldots & q_{d^{\updownarrow}}^{vV}
\end{array}
\right),
\end{equation}
where empty elements may exist.

Note that the values of $b_{ij}^{K}$ and $b_{ij}^{V}$ are independent of the orders of elements in $X^{v}$ (i.e., different linearization methods of trees/forests).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Configuration}

To evaluate the effectiveness of our syntax-based Transformer models, we did the experiments on three language pairs: English(En)-to-Japanese(Ja), English(En)-to-Chinese(Zh) and English(En)-to-German(De).\footnote{%
We choose English as the source language in all experiments, because high-quality packed forests can be obtained. Note that the proposed methods are applicable to any source language, as long as the packed forests are available.} The sizes of training corpora are roughly $10^5, 10^6, 10^7$ sentence pairs, respectively, so that we can investigate the effects with respect to the scales of training corpora. ASPEC corpus is introduced in \citeA{nakazawa2016aspec}. Only 100k high-quality sentence pairs were used in our experiments. LDC corpus is composed of the following parts: LDC2002E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08, and LDC2005T06. WMT 2019 (En-De) is preprocessed following the steps in \citeA{ng2019facebook} and 10M sentences are randomly chosen.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[t]
\caption{Statistics of the corpora}
\label{tab:corpora}
\input{12table01.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The constituent trees and packed forests of English sentences are obtained by the constituent parser proposed by \citeA{huang2008forest}.\footnote{%
\url{http://web.engr.oregonstate.edu/~huanlian/software/forest-reranker/forest-charniak-v0.8.tar.bz2}}$^{,}$\footnote{%
We also confirmed the effectiveness of our models when using the Egret parser (\url{https://github.com/neubig/egret}), and found little difference in experimental results.} The parser converts a tree/packed forest to a sequence where all words are located in front of constituent nodes and all constituent nodes are located in front of their parents, which is the input of our syntax-based Transformer. Forests whose total numbers of words and constituent nodes are larger than 150 are filtered out. To make the comparison fair, we also filtered out these long sentences for other state-of-the-art models. Chinese word segmentation is done by the Stanford segmentation tool.\footnote{%
\url{https://nlp.stanford.edu/software/stanford-segmenter-2017-06-09.zip}} For Japanese sentences, we followed the preprocessing steps recommended in WAT 2017.\footnote{%
\url{http://lotus.kuee.kyoto-u.ac.jp/WAT/WAT2017/baseline/dataPreparationJE.html}}

We implemented our framework based on \texttt{OpenNMT}\footnote{%
\url{http://opennmt.net/}} \cite{2017opennmt}. $d^{\uparrow}$ and $d^{\downarrow}$ are tuned on the development data and the other hyperparameters are chosen as  recommended by the Transformer model implemented by \texttt{OpenNMT}. Both the encoder and the decoder have 6 layers. The dimensions of hidden vectors and word embeddings are 512. The multi-head attention has 8 heads, and the dropout probability is 0.1.

The vocabulary sizes for source language and target language are both 50,000. Because it is difficult to obtain subword-based packed forests, all the experiments are word-based. As for optimization, we used the Adam optimizer \cite{kingma2014adam}, with $\beta_{1}=0.9,\beta_{2}=0.998$, and $\epsilon=10^{-9}$. Warmup and decay strategy for learning rate are also used, with $8,000$ warmup steps. We also used the label smoothing strategy \cite{szegedy2016rethinking} with $\epsilon_{ls}=0.1$. The number of training epochs is fixed to 50, and we selected the best model on the development set for testing. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Experimental Results}

Table \ref{tab:finalresult} summarizes the experimental results of our models and compares our models with previous state-of-the-art models. As stated in Section 3.1, for all syntax-based configurations, constituent trees/packed forests are linearized as $\mathcal{V}=(w_{1},\ldots,w_{n},v_{1},\ldots,v_{m})$, i.e., all constituent labels are located after all words. See Figure 3 for an illustration. Furthermore, absolute positional encoding is adapted to each models straightforwardly (see Section 3.2 for details). ``Line Input'' means that the model structure is identical to the plain Transformer \cite{vaswani2017attention}, while the source-side input is linearized trees/forests. ``Syntax SA'' is the proposed model that uses the syntax-based self-attention mechanism (Section \ref{ssec:forestsa}), and ``+Syntax PE'' is the proposed model that uses the syntax-based positional encodings (Section \ref{ssec:forestpe}) in addition to the syntax-based self-attention mechanism. Results of Syntax-based RNN systems are extracted from \citeA{ma2018forest}, while Plain Transformer is implemented on \texttt{OpenNMT}. The translation performance is evaluated by character-level BLEU score \cite{papineni2002bleu} for En-Ja and En-Zh, and by SacreBLEU score \cite{post2018call} for En-De. The brevity penalties (BP) are identical for all language pairs.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[b]
\caption{Machine translation performance}
\label{tab:finalresult}
\input{12table02.tex}
\vspace{4pt}\small
Paired bootstrap resampling significance test has been done. $*$: significantly better w.r.t. Vaswani+ 17. $\dagger$: significantly better w.r.t. ``Line Input''. $p<0.05$. Values of $(d^\uparrow, d^\downarrow)$ are $(0,1)$, $(1,1)$, $(0,1)$ for En-Ja, En-Zh, En-De, respectively, which are tuned on the development set.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We can see that the results of ``Line Input'' are all worse than plain Transformers, even though syntactic information is encoded straightforwardly in the input. This means that the Transformer failed to learn syntactic information automatically with the naive linearized input. This is because the input sequence is quite long, which is problematic for the Transformer \cite{dai2019transformer}. In contrast, ``Syntax SA'' outperforms ``Line Input'', and moreover ``+Syntax PE'' is better than ``Syntax SA''. This indicates that both the proposed syntax-based self-attention mechanism and the syntax-based positional encoding improve the translation performance significantly. 

The improvement of performance on En-to-De is smaller than the improvements on En-to-Ja and En-to-Zh. This is because fairly enough syntactic information has already been captured by the large training corpus. However, the proposed methods still help improving the BLEU scores, and the improvement is statistical significant. ``Line Input'' is significantly worse than the baseline, while the difference between ``Syntax SA'' and ``+Syntax PE'' is not significant.

We find that there is a gap of performance between RNN-based models and Transformer models. Although the number of sentences and target languages are different among the corpora we used, we still found some interesting facts. Our syntax-based Transformer achieves the best performance on En-to-Zh and En-to-De tasks, although ours is worse than RNN-based forest NMT models on En-to-Ja task. This indicates that there are inherent differences between RNN-based models and Transformer models w.r.t. different sizes of datasets. RNN-based models are more robust to small-scale datasets, while Transformer performs well for large-scale datasets. Furthermore, difficulty in hyperparameter tuning is different, especially when syntactic information are utilized. Even if \citeA{ma2018forest} succeed to incorporate packed forests into an RNN-based NMT model, it is far from trivial to incorporate packed forests into the Transformer. Previous studies \cite{lakew2018comparison} reported that the configuration of hyperparamters of the Transformer is much more tricky than that of the RNN-based NMT models.  However, we simply use the same hyperparameters on all language pairs in our experiments. 

Despite the relatively worse BLEU score on En-to-Ja task, our system is much faster. On En-to-Ja tasks, it takes about 22 hours with one GPU to train our model, while for the system of \citeA{ma2018forest}, the training time is about 240 hours (11 times slower than ours).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Hyperparameter Tuning and Ablation Studies}

We use ASPEC corpus for hyperparameter tuning and ablation studies, because of its small size. The results are summarized in Table \ref{tab:1bestejself} and \ref{tab:ablation}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[t]
\caption{Results with respect to $d^{\uparrow}$ and $d^{\downarrow}$}
\label{tab:1bestejself}
\input{12table03.tex}
\vspace{4pt}\small
Left: constituent trees. Right: packed forests. In theory, the search space is $[0,\infty)$. In practice, because the depth of trees/forests is limited, $d^\uparrow=d^\downarrow=9$ can be regarded as $\infty$ for most sentences.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[t]
\caption{Results of ablation experiments}
\label{tab:ablation}
\input{12table04.tex}
\vspace{4pt}\small
Abs. PE = Absolute positional encoding. $d^{\uparrow}=0$, $d^{\uparrow}=1$. Left: constituent trees. Right: packed forests.
\par\vspace{-1\Cvs}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Influence of $d^{\uparrow}$ and $d^{\downarrow}$}

Table \ref{tab:1bestejself} shows the result of adjusting $d^\uparrow$ and $d^\downarrow$.\footnote{%
When $d^\uparrow=d^\downarrow=0$, all terminals are neighbors to each other, and non-terminals have no neighbors except themselves. In this case, syntactic attention scores are zeros, so that syntactic information are blocked, although syntactic labels indeed appeared in the input sentence.} We can see that the best $d^{\uparrow}$ is 0, and the best $d^{\downarrow}$ is 1 and 2 for the case of using constituent trees and packed forests, respectively. If $d^{\uparrow}$ and $d^{\downarrow}$ are too large, then too much noise is involved. The difference of optimal values of $d^{\uparrow}$ and $d^{\downarrow}$ indicates different effects of parents and children in a syntax tree (forest). Parents bring more noise and less information into the model, because the global coarse structure of different sentences are similar. For example, almost all English sentences consist of the subject, the verb, and the object.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Influence of scores}

Comparing the first two rows in Table \ref{tab:ablation}, 
\pagebreak
we can see that for both constituent trees and packed forests, using scores improves the translation performance. Note that we used absolute positional encoding instead of syntax-based positional encoding, so we can conclude that the improvement on performance was completely from syntax scores. This indicates that scores are important for the syntax-based Transformer, as was the case with the syntax-based NMT based on RNNs as demonstrated in \citeA{ma2018forest}. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Effect of different positional encodings}

Comparing the 2nd, 3rd, and 4th rows of Table 4, we can see that for the syntax-based Transformer, $b_{ij}^{K}$ is useful while $b_{ij}^{V}$ decreases translation performance significantly. This might follow the conclusion by \citeA{shaw2018self} that positional encodings on keys (i.e., $a_{ij}^{K}$ in Equation (\ref{eq:relpe3})) are useful while positional encodings on values (i.e., $a_{ij}^{V}$ in Equation (\ref{eq:relpe1})) are useless.

By comparing the 2nd, 3rd, 5th, and 6th rows of Table \ref{tab:ablation}, we can see that without any positional encodings, the performance is
quite bad. Using either of positional encodings improves the performance significantly, and the combination of absolute positional encoding and $b_{ij}^{K}$ performs the best. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Tree or forest?}

Although \citeA{ma2018forest} concludes that for RNN-based NMT systems, using packed forests is definitely better than using constituent trees, we found that it is not necessarily true for our syntax-based Transformer. For some configurations, using forests performs worse than using trees. For different models, noises in the forest affect the performance in different manners.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Qualitative Analysis}

Figure \ref{fig:example} demonstrates the translation results of an English sentence, where only ``+Syntax PE'' correctly translates the source sentence. Figure \ref{fig:attention} shows the self-attention learned by the encoders of different models, where attentions with small values are filtered out. For the cases of ``Plain Transformer'' and ``Line Input'', only few lines are left. Even for these lines, the quality\footnote{%
One may wonder the meaning of ``quality''. Here, when we say the quality of the attention mechanism is high, we mean that a high attention score is assigned to semantically/syntactically closely related words/labels, and vice versa. This argument is supported my many previous researches \cite{bahdanau2014neural,luong2015effective,vaswani2017attention}, where they show the attention matrices and illustrate the similarity of attention matrices and word alignments. Therefore, it is reasonable to interpret the attention mechanism as the strength of connections between words/labels, and a high-quality attention mechanism should capture the connection to some extent.} is still quite bad. For example, in Figure \ref{fig:attention}(a), ``\textit{many}'' pays attention to ``\textit{sprinkler}'', and in Figure \ref{fig:attention}(b), ``\textit{There}'' pays attention to ``\textit{which}''. These do not make any sense. Furthermore, in Figure \ref{fig:attention}(b), constituent nodes have hardly ever been used. This means that little syntactic information has assisted MT.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[t]
\begin{center}
\includegraphics{27-2ia12f1.eps}
\end{center}
\caption{Reference and translation results of an English sentence. Mistaken phrases have been marked.}
\label{fig:example}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{27-2ia12f2.eps}
\end{center}
\caption{Constituent tree of the sentence in Figure \ref{fig:example}.}
\label{fig:tree}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[t]
\begin{center}
\includegraphics{27-2ia12f3.eps}
\end{center}
\hangcaption{Self-attention of the last encoder layer. Upper sequences are keys and lower sequences are queries. 
	We set $d^{\uparrow}=d^{\downarrow}=\infty$, to let the models determine by themselves how much syntactic information should be used. 
	Attentions with values less than 0.2 are filtered out. 
	Darker line indicates larger attention. In (b), on the right side of ``$_{[3]}$VBP'', there are no lines, so we omit this part for clarity. 
	The corresponding constituent tree is shown in Figure \ref{fig:tree}. 
	Nodes are indexed according to the linearization order.}
\label{fig:attention}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In contrast, attentions learned by the encoders of syntax-based Transformers are much better. Using syntax-based self-attention mechanism (Figure \ref{fig:attention}(c)), although we set $d^{\uparrow}=d^{\downarrow}=\infty$, the encoder learns to pay more attention to neighborhood nodes. For example, ``$_{[12]}$DT'' in the key sequence are paid attention to by two nodes in the query sequence: ``\textit{the}'' and ``$_{[25]}$NP''. These are the child and parent, respectively. On the other hand, the grandparent ``$_{[32]}$VP'' is not paid attention to.

When syntax-based positional encoding is also used (Figure \ref{fig:attention}(d)), the connections become much stronger and much clearer. Unlike the many-to-many connections in Figure \ref{fig:attention}(c), most of the connections in Figure \ref{fig:attention}(d) are one-to-one. Part-of-speech tags pay attention to the corresponding words, and words in the query pay attention to the corresponding ancestor nodes near the root.

Comparing Figure \ref{fig:attention}(c) and Figure \ref{fig:attention}(d), we can conclude that syntax-based self-attention is good at learning local syntax structures, while syntax-based positional encoding is good at learning global syntax structures. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion and Future Works}

We proposed a syntax-based Transformer. It combines the advantages of the Transformer and syntax-based NMT system, two state-of-the-art systems in machine translation. It can utilize rich syntactic information from constituent trees or packed forests, meanwhile can be trained efficiently in parallel. It consists of two important modules: syntax-based self-attention mechanism and syntax-based positional encodings. Both of them encode syntactic information, and improve the translation performance. The proposed syntax-based Transformer outperforms the RNN-based syntax NMT models on En-to-Zh tasks, and outperforms the non-syntactic Transformer baseline on all language pairs. However, too much syntactic information may import noise, and forests are not necessarily better than trees. Even when the training corpus is quite large, the proposed  method is effective although the improvement becomes small. Qualitative analysis indicates that syntax-based self-attention and syntax-based positional encodings learn different aspects of syntax structure. In future, we would like to incorporate target-side syntax structures into our model. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.5}
\bibliography{12refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Chunpeng Ma}{%
is currently a researcher in Fujitsu Laboratory. He received his B.E., M.E., Ph.D. from Harbin Institute of Technology in 2010, 2014 and 2020, respectively. His advisor was Prof. Tiejun Zhao. From July 2013 to January 2014, he interned at Baidu Inc. with instruction from Xianchao Wu. From October 2015 to September 2016, he interned at National Institute of Information and Communications Technology, with the instruction from Akihiro Tamura, cooperating with Lemao Liu. His research interests include syntactic parsing, machine translation and deep learning.
}

\bioauthor[:]{Akihiro Tamura}{%
received his B.E., M.E., and Dr. Eng. from the Tokyo Institute of Technology in 2005, 2007, and 2013, respectively. After serving as a researcher at NEC and NICT, he worked as an assistant professor at Ehime University from 2017. Since 2020, he has been working as an associate professor at Doshisha University. His research interests include natural language processing and machine learning. He is a member of the Association for Computational Linguistics, the Information Processing Society of Japan, the Japanese Society for Artificial Intelligence, and the Association for Natural
Language Processing.
}

\bioauthor[:]{Masao Utiyama}{%
received the Ph.D. degree in engineering from the University of Tsukuba, Tsukuba, Japan, in 1997. He is currently a Research Manager in the National Institute of Information and Communications Technology, Tokyo, Japan. He is a member of the Association for Natural Language Processing, and Asia-Pacific Association for Machine Translation.
}

\bioauthor[:]{Eiichiro Sumita}{%
received the M.S. degree in computer science from the University of Electro-Communications in 1982 and the Ph.D. degree in engineering from Kyoto University in 1999. He is the Associate Director General, ASTREC (Advanced Speech Translation Research and Development Promotion Center). He is currently the fellow of NICT. He is a member of the Association for Natural Language Processing, director of Japan Translation Federation, and the president of Asia-Pacific Association for Machine Translation.
}

\bioauthor[:]{Tiejun Zhao}{%
Doctor, professor of Research Center of Language Technology, School of Computer Science and Technology, Harbin Institute of Technology. He is associate dean of SCST and director of Ministry of Education-Microsoft Key Laboratory of NLP \& Speech in HIT. He is associate director of Machine Translation Subject Committee of Chinese Information Processing Society, the member of editorial board of Journal of Chinese Information Processing, Journal of Automation, the senior member and associate director of Chinese Computing Committee of China Computer Federation. His research fields include: natural language understanding, content-based web information processing, applied artificial intelligence.
}

\end{biography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate


\end{document}

