\documentclass[tombow,english]{jnlp_2.3}
\usepackage{jnlpbbl_1.3}
\usepackage[dvipdfm]{graphicx}
\usepackage{gentombow}
\usepackage{amsmath}
\settombowbleed{5mm}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{array}
    \usepackage{dcolumn}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
%%%%\makeatletter
%%%%\def\@{\spacefactor3000\space}
%%%%\makeatother

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%\documentclass[english,dvipdfmx]{jnlp_2.2}  % author modified
%%%%\usepackage{jnlpbbl_1.3}
%%%%\usepackage{graphicx}
%%%%\usepackage{amsmath}
%%%%\usepackage{latexsym}
\usepackage{amssymb}
\usepackage{url}
\usepackage{multirow}
%%%%\usepackage{subfig}
%%%%\usepackage{hhline}
\usepackage{colortbl}
%%%%\renewcommand{\thesubfigure}{\roman{subfigure}}
%%%%\renewcommand{\thesubtable}{\roman{subtable}}
\newcommand{\bgc}{\cellcolor[gray]{0.8}}
\newcommand{\bd}[1]{\textbf{#1}}
%%%%\newcommand{\ul}[1]{\underline{#1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\renewcommand\UrlFont{\rmfamily}
\renewcommand{\refname}{References}
\allowdisplaybreaks

\Volume{27}
\Number{3}
\Month{September}
\Year{2020}

\received{2019}{11}{28}
\revised{2020}{2}{28}
%%%%\rerevised{2006}{5}{6}
\accepted{2020}{4}{3}
%%%%\Issuetitle{「　　　　」特集号}

\setcounter{page}{499}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%   Title, Author, etc.            %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Character-to-Word Attention for Word Segmentation}
\eauthor{Shohei Higashiyama\affiref{Author_1}\affiref{Author_2} \and Masao Utiyama\affiref{Author_1} \and Eiichiro Sumita\affiref{Author_1} \and Masao Ideuchi\affiref{Author_3} \and \\
	Yoshiaki Oida\affiref{Author_3} \and Yohei Sakamoto\affiref{Author_3} \and Isaac Okada\affiref{Author_3} \and Yuji Matsumoto\affiref{Author_2}\affiref{Author_4}}
\eabstract{%
 Although limited effort has been devoted to exploring neural models in Japanese word segmentation, much effort has been actively applied to Chinese word segmentation  because of the ability to minimize effort in feature engineering. In this work, we propose a character-based neural model that makes joint use of word information useful for disambiguating word boundaries. For each character in a sentence, our model uses an attention mechanism to estimate the importance of multiple candidate words that contain the character. Experimental results show that learning attention to proper words leads to accurate segmentations and that our model achieves better performance than  existing statistical and neural models on both in-domain and cross-domain Japanese word segmentation datasets.
}
\ekeywords{Word Segmentation, Neural Networks, Attention Mechanism}

\headauthor{Higashiyama et al.}
\headtitle{Character-to-Word Attention for Word Segmentation}

\affilabel{Author_1}{}{National Institute of Information and Communications Technology}
\affilabel{Author_2}{}{Nara Institute of Science and Technology}
\affilabel{Author_3}{}{FUJITSU RESEARCH INSTITUTE}
\affilabel{Author_4}{}{RIKEN Center for Advanced Intelligence Project}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Word segmentation is the first step of natural language processing (NLP) for most 
East Asian languages, including Japanese and Chinese.
In recent years, neural network models---character-based or word-based---have been 
widely applied to Chinese word segmentation because they can minimize effort 
in feature engineering.
Character-based models \cite{xzheng2013,mansur2013,pei2014,chen2015gated}
efficiently predict optimal label sequences by treating the task as sequence 
labeling that assumes first-order dependency between labels.
Word-based models \cite{mzhang2016,cai2016,cai2017,yang2017} directly segment 
a character sequence into words and can easily achieve the benefits of word-level 
information.
These neural models have achieved great success in Chinese word segmentation 
performance.

In contrast, limited effort has been devoted to exploring neural models in 
Japanese word segmentation. 
Instead, popular approaches are based on statistical learning algorithms, 
such as conditional random fields (CRFs) \cite{kudo2004} and logistic 
regression \cite{neubig2011}.
More recently, \citeA{morita2015} achieved improved performance by 
integrating a word-level language model, based on a recurrent neural network 
(RNN), into their statistical baseline model.
However, \citeA{kitagawa2018} demonstrated better performance than with an 
existing statistical segmenter \cite{neubig2011} by applying a pure neural 
model based on a general long short-term memory (LSTM) architecture for sequence 
labeling.
Even so, more specialized neural architectures for word segmentation leave room 
for further research to clarify neural models' potential.
Therefore, we investigate a novel neural word segmentation model that utilizes 
word information expected to be useful in resolving the ambiguity in word 
segmentation.

Within a sentence, a character has multiple candidate words that contain the 
character, but a candidate word's plausibility differs within the target character's 
context.
For example, more than three candidate words exist for characters $x_3$, $x_4$, 
and $x_5$ in the sentence $x_{1:5}$ in Fig. \ref{fig:word_search}, so the proper 
word $w_8$ must be identified from among the candidates.\footnote{$w_6$ and $w_5$ 
can be correct depending on a segmentation criterion.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[t]
\begin{center}
\includegraphics{27-3ia1f1.eps}
\end{center}
\hangcaption{An example of candidate words $w_{1:8}$ retrieved from a vocabulary for the sentence $x_{1:5}$. Strings in angle brackets ``$\langle\rangle$'' and in parentheses ``()'' respectively indicate words' (typical) readings and English translations. The value in each $(i,j)$ represents whether the $i$-th character is contained by the $j$-th word, i.e., $\delta_{ij}$ in Eq.~(\ref{eq:alpha}).}
\label{fig:word_search}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Motivated by that consideration, we propose a character-based word segmentation 
model that incorporates word information into a BiLSTM-CRF architecture 
\cite{huang2015,chen2015long}.
Different from similar work in Chinese word segmentation \cite{wang2017,jyang2019},
we apply an attention mechanism \cite{bahdanau2015,luong2015} to learn and 
distinguish all possible candidate words' importance for a character within a context.
Note that although we demonstrate only experimental results with Japanese datasets,
we use no features specific to Japanese, so our model can be applied to other 
unsegmented languages such as Chinese.

Our contributions are as follows:
\begin{itemize}
%%%% \setlength{\itemsep}{0cm}
 \item To distinguish and leverage possible candidate words' importance in different 
       contexts, we introduce word information and an attention mechanism into a 
       character-based word segmentation framework.
 \item We empirically demonstrate that learning accurate attention to proper candidate 
       words leads to correct segmentations.
 \item Compared with existing Japanese word segmentation models, our model achieves 
       better performance on Japanese datasets in both in-domain and cross-domain 
       scenarios.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Task Definition}

Word segmentation can be regarded as a character-level sequence labeling task.
Given a sentence $x=x_{1:n}=(x_1,\cdots,x_n)$ of length $n$, each character $x_i$ will 
be assigned a segmentation label $y_i$ of tag set $\mathcal{T}$; then a label sequence 
$y=y_{1:n}=(y_1,\cdots,y_n)$ will be predicted.
We employ tag set $\mathcal{T}=\{{\rm B},{\rm I},{\rm E},{\rm S}\}$, where B, I, and E 
represent the beginning, inside, and end of a multi-character word, and S represents a 
single-character word \cite{xue2003}.\footnote{As for predicted label sequences 
that violate the tagging scheme, such as ``$\cdots${\rm IB}$\cdots$'' and 
``$\cdots${\rm EI}$\cdots$'', we adopt the same correction rules as the script used in 
the CoNLL-2003 shared task \cite{sang2003}
(\url{https://www.clips.uantwerpen.be/conll2003/ner/bin/conlleval}).}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Baseline Model}

For our baseline model, we use a BiLSTM-CRF architecture that has been successfully 
applied to sequence labeling tasks, including word segmentation \cite{chen2015long}.
The model consists of a character embedding layer, recurrent layers, and a CRF layer,
as illustrated in Fig. \ref{fig:model_arch}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{27-3ia1f2.eps}
\end{center}
\hangcaption{Architectures of our baseline and proposed models, with common components (light gray) and additional components for the proposed model (dark gray).}
\label{fig:model_arch}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Character Embedding Layer}

Let $\mathcal{V}_c$ be a character vocabulary.
In a given sentence, each character is transformed into a character embedding 
$\mathbf{e}^c$ of a $d_c$-dimensional vector by a lookup operation that retrieves the 
corresponding column of the embedding matrix 
$E_c \in \mathbb{R}^{d_c \times |\mathcal{V}_c|}$~.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Recurrent Layers}

A sequence of character embeddings 
$\mathbf{e}^c_{1:n}=(\mathbf{e}^c_1,\cdots,\mathbf{e}^c_n)$ is fed into 
an RNN to derive contextualized representations 
$\mathbf{h}_{1:n}=(\mathbf{h}_1,\cdots,\mathbf{h}_n)$, which we call 
\textit{character context vectors}.
We adopt a stacked (multi-layer) and bidirectional variant of an LSTM 
\cite{hochreiter1997} network, which addresses the issue of learning long-term 
dependencies and the gradient vanishing problem.

Hidden vectors $\mathbf{h}_{1:n}^{(l)}$ of the $l$-th bidirectional LSTM (BiLSTM) layer 
are calculated by forward LSTM (${\rm LSTM}_f$) and backward LSTM (${\rm LSTM}_b$):
\begin{align*}
 \overrightarrow{\mathbf{h}}_i^{(l)} &= {\rm LSTM}_f(\mathbf{h}_i^{(l-1)}, \overrightarrow{\mathbf{h}}_{i-1}^{(l)})~,\\
 \overleftarrow{\mathbf{h}}_i^{(l)} &= {\rm LSTM}_b(\mathbf{h}_i^{(l-1)}, \overleftarrow{\mathbf{h}}_{i+1}^{(l)})~,\\
 \mathbf{h}_i^{(l)} &= \overrightarrow{\mathbf{h}}_i^{(l)} \oplus \overleftarrow{\mathbf{h}}_i^{(l)}~,
\end{align*}
where $\oplus$ denotes a concatenation operation and $\mathbf{h}_i^{(0)}=\mathbf{e}^c_i$.

More concretely, each forward LSTM calculates forward hidden vectors 
$\overrightarrow{\mathbf{h}}_{1:n}$ from an input sequence $\mathbf{v}_{1:n}=(\mathbf{v}_1,\cdots,\mathbf{v}_n)$ of 
$d_v$-dimensional vectors as follows:
\begin{align*}
 \overrightarrow{\mathbf{h}}_i &= {\rm LSTM}_f(\mathbf{v}_i,\mathbf{h}_{i-1} ) := \mathbf{o}_i \odot {\rm tanh}(\mathbf{c}_i)~,\\
 \mathbf{c}_i &= \mathbf{i}_i \odot \mathbf{t}_i + \mathbf{f}_i \odot \mathbf{c}_{i-1},\\
 \mathbf{o}_i &= \sigma(W_o \mathbf{v}_i + U_o \mathbf{h}_{i-1} + \mathbf{b}_o)~,\\
 \mathbf{i}_i &= \sigma(W_i \mathbf{v}_i + U_i \mathbf{h}_{i-1} + \mathbf{b}_i)~,\\
 \mathbf{f}_i &= \sigma(W_f \mathbf{v}_i + U_f \mathbf{h}_{i-1} + \mathbf{b}_f)~,\\
 \mathbf{t}_i &= {\rm tanh}(W_t \mathbf{v}_i + U_t \mathbf{h}_{i-1} + \mathbf{b}_t)~, 
\end{align*}
where $\odot$ denotes element-wise multiplication, $\sigma$ is the sigmoid function, 
$\mathbf{i}$, $\mathbf{f}$, and $\mathbf{o}$ indicate an input gate, a forget gate, 
and an output gate, $W_i, W_o, W_f, W_t, U_i, U_o, U_f, U_t \in \mathbb{R}^{d_r \times d_v}$ 
and $\mathbf{b}_i, \mathbf{b}_o, \mathbf{b}_f, \mathbf{b}_t \in \mathbb{R}^{d_r}$ are 
trainable parameters, and $d_r$ is a hyperparameter.
Note that the input vector $\mathbf{v}_i$ (and its dimension $d_v$) corresponds to 
$\mathbf{h}^{(0)}_i=\mathbf{e}^c_i$ ($d_c$) for the first layer and to 
$\mathbf{h}^{(l-1)}_i$ ($2d_r$) for the second and subsequent layers.
Backward LSTMs similarly calculate backward hidden vectors $\overleftarrow{\mathbf{h}}_{1:n}$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{CRF Layer}

A character context vector 
$\mathbf{h}_i$ is mapped into a $|\mathcal{T}|$-dimensional vector representing 
scores of segmentation labels:
\[
 \mathbf{s}_i = W_s \mathbf{h}_i + \mathbf{b}_s~, 
\]
where $W_s \in \mathbb{R}^{|\mathcal{T}| \times 2d_r}$ and 
$\mathbf{b}_s \in \mathbb{R}^{|\mathcal{T}|}$ are trainable parameters.
Following previous sequence labeling work \cite{collobert2011}, we introduce a 
CRF \cite{lafferty2001} layer, which has a transition matrix 
$A \in \mathbb{R}^{|\mathcal{T}| \times |\mathcal{T}|}$ to give transition scores 
between adjacent labels.
Thus, the score of a label sequence $y=y_{1:n}$ for a sentence $x=x_{1:n}$ is 
calculated as follows:
\[
 {\rm score}(x,y;\theta) = \sum_{i=1}^n (A_{y_{i-1},y_i} + \mathbf{s}_{i}[y_i])~,
\]
where $\theta$ denotes all parameters and $\mathbf{s}[y]$ indicates the dimension of 
a vector $\mathbf{s}$ corresponding to a label $y$.
We can find the best label sequence $y^\star$ by maximizing the sentence score:
\begin{equation}
 y^\star = {\rm argmax} {}_{y \in \mathcal{T}^n}~{\rm score}(x, y;\theta) ~.
\label{eq:decode}
\end{equation}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Training Objective}

During training, parameters $\theta$ of the network are learned by minimizing the 
negative log likelihood over all sentences in training data $\mathcal{D}$ w.r.t 
$\theta$:
\begin{gather}
 \min_{\theta} \quad -\sum_{(x,y) \in \mathcal{D}} \log p(y|x,\theta)~, \nonumber \\
 p(y|x,\theta) = \frac{{\rm score}(x,y;\theta)}{\sum_{y^\prime} {\rm score}(x,y^\prime;\theta)}~.
\label{eq:prob}
\end{gather}
Note that the Viterbi algorithm can be used for efficient calculation of 
a label sequence's probability in Eq.~(\ref{eq:prob}), similar to decoding in 
Eq.~(\ref{eq:decode}).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed Model}

To disambiguate word boundaries more effectively than earlier models, we integrate 
word information into the character-based framework.
More specifically, we transform embeddings of multiple candidate words for each 
character into a fixed-size word vector, which we call a \textit{word summary vector},
by a \textit{word feature composition function} (see Fig. \ref{fig:model_arch} for our
model's architecture).
In addition to the baseline model's layers, the model comprises a word embedding 
layer and a word feature composition function.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Word Embedding Layer}

Given a character sequence $x=x_{1:n}$, we search for all words within a maximum 
word length corresponding to subsequences of the input sequence from a word vocabulary 
$\mathcal{V}_w$.
Then, we obtain a set $\mathcal{W}_x=\{w_1,\cdots,w_m\}$ of all candidate words.
For example, for the given sentence $x_{1:5}$ in Fig. \ref{fig:word_search}, candidate 
words $\{w_1,\cdots,w_8\}$ are found.
The embedding matrix $E_w \in \mathbb{R}^{d_w \times |V_w|}$ transforms each word 
$w \in \mathcal{W}_x \subseteq \mathcal{V}_w$ into a $d_w$-dimensional vector 
$\mathbf{e}^w$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Composition Functions of Word Features}

For character $x_i$, a composition function aggregates embeddings of all candidate 
words that contain the character into word summary vector $\mathbf{a}_i$.
We introduce two attention-based composition functions, weighted average (WAVG) and 
weighted concatenation (WCON), that enable a model to pay more or less attention 
according to candidate words' importance.

Both functions calculate the importance score $u_{ij}$ from character $x_i$ to 
word $w_j$ in $\mathcal{W}_x$ through bilinear transformation that indicates the 
interaction between the character context vector $\mathbf{h}_i$ and the word embedding 
$\mathbf{e}^w_j$.
Then to normalize scores, a softmax operation obtains the weight $\alpha_{ij} \in [0, 1]$:
\begin{align}
 u_{ij} &= \mathbf{h}_i^T W_a \mathbf{e}^w_j~, \label{eq:imp} \\
 \alpha_{ij} &= \frac{\delta_{ij} \exp(u_{ij})}{\sum_{k=1}^m  \delta_{ik} \exp(u_{ik})}~, \label{eq:alpha}
\end{align}
where $W_a \in \mathbb{R}^{2d_r \times d_w}$ is a trainable parameter.
To simplify equations, we introduce an indicator variable $\delta_{ij} \in \{0, 1\}$
that indicates whether the character $x_i$ is included in the word $w_j$ 
(Fig. \ref{fig:word_search}).

Next, WAVG and WCON calculate a word summary vector $\mathbf{a}_i$ as the weighted 
average and the weighted concatenation of word embeddings, respectively:
\begin{gather}
 \mathbf{a}_i = {\rm WAVG}(x_i, \{w_j\}_{j=1}^m) = \sum_{j=1}^m \alpha_{ij} \mathbf{e}^w_{j}~,
\label{eq:wave}\\
 \mathbf{a}_i = {\rm WCON}(x_i, \{w_j\}_{j=1}^m) = \bigoplus_{l=1}^L \alpha_{i,i_l} \mathbf{e}^w_{i_l}~,
\label{eq:wcon}
\end{gather}
where $\{w_j\}_{j=1}^{m}=\mathcal{W}_x$ and $\bigoplus(\cdot)$ indicates the 
concatenation of given arguments.
Let $K$ be the maximum word length, $L=\sum_{k=1}^{K}k$, and $i_l$ for the character 
$x_i$ denotes the corresponding index in $\mathcal{W}_x$ of $l$-th words $w^\prime_l$ 
in the list 
$\{w^\prime_1,\cdots,w^\prime_L\}=\bigcup_{k=1}^K \bigcup_{p=-k+1}^{0} \{x_{i+p:i+p+k-1}\}$.
If $w^\prime_l \notin \mathcal{V}_w$, we use a zero vector as the $l$-th argument in 
Eq.~(\ref{eq:wcon}).
For example, if $K=3$, WCON concatenates embeddings of words corresponding to $x_i$ 
(length 1), $x_{i-1:i}$, $x_{i:i+1}$ (length 2), $x_{i-2:i}$, $x_{i-1:i+1}$ and 
$x_{i:i+2}$ (length 3) in that order, into a single vector for the character $x_i$.
WAVG and WCON  finally output a summary vector of size $d_w$ and $Ld_w$, respectively.
The obtained summary vector $\mathbf{a}_i$ and the context vector $\mathbf{h}_i$ are 
together fed into the CRF layer.
We use a zero vector as a summary vector if no candidate words for a character are found. 

We also use two more variants of composition functions without the attention mechanism, 
the average function (AVG) and the concatenation function (CON).
AVG is a special case of WAVG, where $\alpha_{ij}=\delta_{ij}/\sum_k \delta_{ik}$ for all 
$(i,j)$ in Eq.~(\ref{eq:wave}).
CON is the equivalent function to the word features used in \citeA{wang2017} and a 
special case of WCON, where $\alpha_{i,i_l}=1$ for all $(i,i_l)$ in Eq.~(\ref{eq:wcon}).

Note that our importance score function in Eq.~(\ref{eq:imp}) has the same form 
as the bilinear variant of the global attention model in \citeA{luong2015},
used to calculate alignments between source and target hidden vectors in machine translation.
They further evaluate the input-feeding approach that uses previous alignment information in 
next-time steps.
To take into account attended words from previous characters, similar word segmentation 
approaches might also be useful.
We leave this for future work.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Settings} \label{sec:settings}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Datasets}%%%%\qquad
Using three Japanese datasets, we evaluated our model in both in-domain and cross-domain 
settings.
First, the Balanced Corpus of Contemporary Written Japanese\footnote{
\url{https://pj.ninjal.ac.jp/corpus_center/bccwj/en/}} (BCCWJ) Version 1.1 
\cite{maekawa2014} follows the short unit word (SUW) criterion defined by the 
National Institute for Japanese Language and Linguistics (NINJAL) and consists of several 
domains, for example, news, magazines and blogs. 
We used this mixed domain data for in-domain experiments.

Second, the Japanese Dependency Corpus\footnote{
\url{http://www.ar.media.kyoto-u.ac.jp/data/word-dep/home-e.html}} (JDC) 
\cite{mori2014} follows the criterion that extends NINJAL's SUW by separating 
inflectional words' endings from their stems. 
JDC consists of six domains: BCCWJ (parts of sentences in BCCWJ), economy newspaper 
articles, dictionary example sentences, information processing journal abstracts, patent 
specifications, and recipe texts. 
We used sentences from the former three domains as source domain data and sentences
from other domains (i.e., journal, patent, or recipe) as target domain data.

The third dataset comprises three corpora that follow the same criterion 
for the Japanese morphological analyzer JUMAN:
Kyoto University Text Corpus\footnote{http://nlp.ist.i.kyoto-u.ac.jp/EN/index.php?Kyoto\%20University\%20Text\%20Corpus} 
(KTC) Version 4.0 \cite{kawahara2002}, 
Kyoto University Web Document Leads Corpus\footnote{
\url{http://nlp.ist.i.kyoto-u.ac.jp/EN/index.php?KWDLC}} (KWDLC) 
\linebreak
\cite{hangyo2012}, 
and Kyoto University and NTT Blog Corpus\footnote{
\url{http://nlp.ist.i.kyoto-u.ac.jp/kuntt/KNBC_v1.0_090925.tar.bz2}} (KNBC) 
\cite{hashimoto2011}.
We refer to this mixed data as JUMAN Mixed Corpus (JMC).
We used news sentences in KTC and Web sentences in KWDLC as source domain data and 
blog texts with a specific topic (sports, mobile phones, dining, or travel in Kyoto) in KNBC 
as target domain data.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[t]
\caption{Dataset statistics}
\label{tab:data_stat}
\input{01table01.tex}
\vspace{4pt}\small
Train/dev/test size indicates the number of sentences in the corresponding set.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We used training/development/test splits as shown in Table \ref{tab:data_stat}.
The BCCWJ test set comprises the same test sentences as used in the Project Next NLP.\footnote{
\url{http://www.lsta.media.kyoto-u.ac.jp/mori/research/topics/PST/NextNLP.html}}
We randomly selected a certain number (500, 1,000, or 2,000, as in Table \ref{tab:data_stat})
of sentences for the development set of BCCWJ, and for the respective development and test 
sets of the JDC and JMC source domain data.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Word vocabulary construction}%%%%\qquad
Apart from given training and development sets for each dataset, we assumed that no annotated 
information, including external dictionaries and third-party segmenters, was available in our 
experiments.
Therefore, we used the training set and large unlabeled texts to obtain a word vocabulary for 
our proposed model.

First, we trained a baseline model from each training set and applied it to unlabeled texts.
Then, we regarded the union of auto-segmented words from the texts and gold words from the 
training set as a word vocabulary.
From the auto-segmented texts, we discarded words occurring less than the minimum word frequency 
threshold $f$ of five, the default value in the Word2Vec used for pre-training word embeddings, 
as described later in this subsection.
We used the non-core section of BCCWJ (BCCWJ-NC),\footnote{
We restored provided auto-segmented texts, which were segmented by another segmenter,
to the original raw sentences and used them as unlabeled texts.}
which contains about 5.9 million sentences, as common unlabeled texts for the three datasets.

In the implementation of our proposed model, a word dictionary manages all words in a given 
word vocabulary.
In the training phase, however, the model holds embedding parameters for only a part of those 
words corresponding to gold words or substrings in training sentences.
In the test phase, the model searches substrings in the test sentences from the dictionary and
dynamically loads their word embeddings from the external word embedding model used for 
initialization.
This strategy reduces model size while handling hundreds of thousands of words in the dictionary,
as shown later in \S \ref{sec:exp_freqlen}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Pre-training of embedding parameters}%%%%\qquad
Following previous work \cite{collobert2011}, we pre-trained word embeddings from large 
texts and used them to initialize the word embedding matrix in our proposed segmenter.
To pre-train word embeddings, we applied the gensim \cite{rehurek2010} implementation of 
Word2Vec \cite{mikolov2013a} to the same texts as those used to construct word vocabularies,
i.e., auto-segmented sentences in BCCWJ-NC processed by baseline segmenters.
We used the toolkit with the skip-gram algorithm, embedding size 300, the number of iterations 1, 
and other default parameters, including minimum frequency 5.
For words occurring only in a training set, we randomly initialized their embeddings.
We fine-tuned all word embeddings during training of the proposed segmenter.

In contrast, we randomly initialized all character embeddings, since pre-trained character 
embeddings did not improve performance in our preliminary experiments.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Hyperparameter setting}%%%%\qquad
Table \ref{tab:hyp} displays the proposed model's hyperparameters.
\pagebreak
We set the maximum word length $K$ to 4 because this value covered 99\% of words in the BCCWJ 
and JDC development set and because larger values did not further improve performance in our 
preliminary experiments.\footnote{However, an optimal value of $K$ can vary depending on domain, 
as we show later in \S \ref{sec:exp_freqlen}.}
The same dropout strategy as in \citeA{zaremba2014} was applied to non-recurrent connections 
of recurrent layers.
Besides that, we used word vector dropout, which randomly replaces a word embedding
to a zero vector when calculating a word summary vector in Eqs.~(\ref{eq:wave}) or (\ref{eq:wcon}).
We used a mini-batch stochastic gradient descent to optimize parameters and decayed the learning 
rate with a fixed decay rate every epoch after the first five; we trained models for up to 20 
epochs and selected the best model on the development set.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\hangcaption{Hyperparameter values common between the baseline and the proposed model (top) and specific to the proposed model (bottom)}
\label{tab:hyp}
\input{01table02.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Main Results}
\label{sec:results}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Comparison of proposed model variants}%%%%\qquad
We evaluated our baseline and proposed model variants on the three datasets, and Table 
\ref{tab:dev} displays means and standard deviations of  F1 scores of three runs for each 
model variant.
Among the proposed model variants, WCON achieved the best performance in almost all cases.
From the results, we observed the following three findings.
First, the four word-integrated model variants outperformed the pure character-based baseline on 
almost all datasets. 
Each variant's improvement over the baseline was significant at the 0.01 level for WAVG, CON, and 
WCON in 12 of 16 cases, according to McNemar's tests \cite{gillick1989} on differences 
between two systems' word-level predictions for gold words (i.e., true positive or false negative).
Second, the attention-based variants achieved performance equivalent to or better than their 
counterparts without attention.
According to McNemar's tests, the improvement of WAVG over AVG on the JDC news set and that of WCON 
over CON on the JMC Web set were statistically significant.
Third, the concat-based variants performed better than their average-based counterparts in all 
cases, probably because CON and WCON retain word length and character position information.
For example, ($d_w\!+\!1$)-th to $2d_w$-th dimensions of a summary vector always represent a 
word with a length of two ending with a target character (namely $x_{i-1:i}$ for $x_i$),
while AVG and WAVG lose such information.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[t]
\caption{In-domain performance on development sets}
\label{tab:dev}
\input{01table03.tex}
\vspace{4pt}\small
The table shows the means and standard deviations (denoted with $\pm$) of F1 scores of three runs for the baseline (BASE) and proposed model variants. The symbols $\dagger$ and $\ddagger$ indicate statistical significance at the 0.01 level over the baseline and over the variant without attention, respectively.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In Table \ref{tab:dev}, we also show the model size (i.e., the number of parameters; see Appendix 
\ref{sec:param} for more details).
The proposed model's variants  have 109--135\% more parameters than the baseline, and WCON has 
11--12\% more parameters than other variants.
Then, for a fair comparison of model size, we examined the performance of a larger size of model 
variants, BASE\_L, WAVG\_L, and CON\_L, with numbers of parameters similar to WCON, by increasing 
the number of BiLSTM hidden units $d_r$.
Large model variants exhibited small performance improvements or degradations, but none achieved 
better performance than WCON, indicating that model variants' performance differences were due not
to the different model size but to the different model structures.
Note that the current size model's performance might not be saturated in the Web domain, because 
all large model variants consistently show performance improvements.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[t]
\caption{In-domain and cross-domain performance (mean F1 scores of three runs) on the test sets}
\label{tab:test}
\input{01table04.tex}
\vspace{4pt}\small
$\Delta$ denotes the difference of the average score among target domains from that among source domains.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[t]
\caption{Dataset statistics and performance of models for out-of-vocabulary (OOV) words}
\label{tab:test_unk}
\input{01table05.tex}
\vspace{4pt}\small
The OOV rate of a test set for vocabulary $\mathcal{V}$ indicates the rate of words that occur in the test set but are not in $\mathcal{V}$. OOV recall indicates recall for words not in the training vocabulary $\mathcal{V}_{\rm train}$ (mean scores of three runs).
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Test set performance and comparison with existing methods}%%%%\qquad
We evaluated WCON, our best model variant, on test sets and compared it with the 
baseline.
In Table \ref{tab:test}, we show F1 scores and out-of-vocabulary (OOV) recalls
together with, in Table \ref{tab:test_unk}, OOV rates of datasets.
WCON improved F1 scores over the baseline by 0.25--0.55 points on source domains 
and approximately 0.5--2.1 points on target domains.
As indicated by its smaller performance drops from source domain results ($\Delta$ 
values), WCON showed robust performance for domain shifts in cross-domain settings.
Among target domains, the model obtained the largest gain in the patent domain.
This can be explained in terms of OOV words shown in Table \ref{tab:test_unk}.
Although in particular, this domain has many OOV words not in the training vocabulary 
$\mathcal{V}_{\rm train}$, a large portion (77\%) were covered by the auto-segmented 
word vocabulary $\mathcal{V}_{\rm auto}$ (with $f=5$ and $K=4$), and the model 
also greatly improved OOV recall.
Thus, our model exploited word information not covered by training data while also
reducing substantive unknown words.

As shown in Table \ref{tab:test}, we also compared our model with other Japanese 
word segmenters without additional annotated data---a popular statistical Japanese 
segmenter KyTea \cite{neubig2011} and a recent LSTM-based model by 
\citeA{kitagawa2018}.\footnote{
The result with * is cited from their paper and the other results of compared models 
are from our single run of their codes.}
Due to effective information on candidate words, our model achieved better performance 
than these in all domains, because neither used direct word information except for 
word indicator features.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Detailed Evaluation}
\label{sec:anldis}

In this section, we provide the detailed proposed model's evaluation and analysis.
We use the same experimental settings as in \S\ref{sec:settings} and 
report the mean score of three runs for each model unless otherwise specified.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Effect of semi-supervised learning}
\label{sec:exp_puresl}

Our proposed model is a semi-supervised learning method that uses unlabeled texts 
for pre-training of word embedding parameters.
To investigate unlabeled texts' contribution, we evaluated both a pure-supervised 
version of the proposed model, which began from randomly initialized word embedding 
parameters, and a semi-supervised version of the baseline model, which additionally 
used auto-segmented texts through self-training.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[t]
\hangcaption{Performance (mean F1 scores of three runs) on development sets in pure-supervised and semi-supervised settings}
\label{tab:pure_sl}
\input{01table06.tex}
\vspace{4pt}\small
``Random'' and ``pre-trained'' indicate models started respectively with randomly initialized word embeddings and with pre-trained word embeddings.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

As Table \ref{tab:pure_sl} shows, both supervised versions of proposed model variants
CON and WCON, with and without attention, could obtain little performance improvement 
over the baseline and seriously underperformed their semi-supervised versions.
This suggests that, from the performed task that predicts segmentation labels from 
character representations incorporated with word vectors, learning meaningful word 
representations is difficult.
Rather than using an external method such as the skip-gram, another effective method 
might be to train a model from an auxiliary task, like word-level language modeling, 
along with the segmentation task.
Either way, considering the sparse distribution of words, large amounts of text are 
probably necessary.
From comparison between baseline and proposed models in the semi-supervised setting,
we observed limited performance improvements through self-training,
thus indicating that our proposed model more effectively utilizes unlabeled texts.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Effect of word frequency and length thresholds}
\label{sec:exp_freqlen}

We analyzed how the WCON model's performance changed in various domains according to 
different word vocabularies in both minimum frequency and maximum length of words.
For the minimum frequency threshold $f$, a model's vocabulary excludes words occurring 
fewer time than the threshold value in auto-segmented texts.
For the maximum length threshold $K$, a model ignores words whose length exceeds the 
threshold value.
Namely, a smaller frequency threshold and a larger length threshold lead to a larger 
vocabulary.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Minimum word frequency threshold}%%%%\qquad
First, we fixed the length threshold to 4 and changed the word frequency threshold $f$ 
among \{1, 5, 10\}.
Then we examined vocabulary size, OOV rates, and the WCON model's performance (Table 
\ref{tab:freq}).
The frequency threshold's increase from 1 to 10 resulted in approximately an 80\% smaller 
vocabulary (about 35\% smaller number of embedding parameters) in exchange for drops of 
0.2--0.3 points for source domains and 0.3--1.8 points for target domains in OOV rates.
The model with $f=1$ achieved (close to) the best performance on all datasets.
The model with the larger frequency threshold achieved similar performance on some 
source domains but yielded lower performance on most target domains with relatively 
higher OOV rates.
Thus, we confirmed that infrequent words in a source domain could be effective information 
especially for out-of-domain texts.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 7
\begin{table}[t]
\hangcaption{Vocabulary size, OOV rate, and performance (mean F1 scores of three runs) of the WCON model with each minimum word frequency threshold $f$ on test sets}
\label{tab:freq}
\vspace{-4pt}
%%%%\subfloat[Vocabulary size]{
\input{01table07.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Maximum word length threshold}%%%%\qquad
Second, we fixed the frequency threshold to 5, changed the word length threshold $K$ 
among \{1, 2, $\cdots$, 6\}, and evaluated performance for each length of gold words 
in evaluation sentences.
We picked up several test sets from JDC and JMC data, which are source domain data 
and target domain data with higher OOV rates (Table \ref{tab:len}).
We also show performance of the baseline as the model with ``$K=0$'' for reference.

OOV rates for the model with the largest length threshold $K=6$ decreased up to 7 
points from the model with $K=1$, and performance also varied greatly.
For each length $k$ of gold words, the model using words of the length (i.e., the 
model with $K\geq k$), tended to outperform the model not using those words (i.e., 
the model with $K<k$), as highlighted by gray background in Table \ref{tab:len}.
Moreover, the model with the larger threshold value often improved performance for 
shorter words, and, therefore, overall performance.
These results suggest that information on words of a particular length
was effectively used to disambiguate character sequences of the same or shorter length.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 8
\begin{table}[p]
\hangcaption{Performance (mean F1 scores of three runs) of the WCON model with maximum word length $K$ and that of the baseline denoted with $K=0$ for each word length $k$}
\label{tab:len}
\vspace{-4pt}
\input{01table08.tex}
\vspace{4pt}\small
Values in ``()'' denote percentages of words of length $k$ in the data. We highlighted with gray background the results of the model with $K\geq k$ that outperformed the model with $K<k$ for each $k$.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

For each data domain, performance was saturated when $K=5$ for the news domain (the 
rate of words whose length $k\geq 6$ is 0.35\%) and the dining domain (0.6\%).
In contrast, better performance was obtained when $K=6$ for the patent domain (0.9\%) 
and the Web domain (1\%).
Especially for domains with many long words, such as loanwords written with 
\textit{katakana}, we can expect to achieve robust segmentations by a model with 
larger maximum word length.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Effect of attention for segmentation performance}
\label{sec:exp_att}

To analyze how the attention mechanism affects segmentation performance, in Fig. 
\ref{fig:att}, we show the WCON model's segmentation and attention accuracies for 
the BCCWJ development set.
Segmentation accuracy indicates character-level accuracy of segmentation label 
prediction.
Attention accuracy is defined as the rate of characters that correctly attend to gold 
words.\footnote{
We regarded that a character attended to the word with the largest weight among all 
candidate words, and judged as correct if the attended word corresponded to the gold 
segmentation.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[t]
\begin{center}
\includegraphics{27-3ia1f3.eps}
\end{center}
%%%%\label{fig:att_a}
%%%%\label{fig:att_b}
%%%%\label{fig:att_c}
\hangcaption{Effect of the WCON model's attention for segmentation performance on the BCCWJ development set.}
\label{fig:att}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%In Fig. \ref{fig:att} \subref{fig:att_a}, 
    In Fig. \ref{fig:att}~(i), 
we show the count of corresponding 
characters and accuracies for each case of attention status: 
(a) there are no candidate words (then word vectors are not available);
(b) there are candidate words except a gold word (then attention is always 
incorrectly paid);
(c) only a gold word is a candidate (then attention is always correctly paid); 
and (d) candidate words consist of both a gold and other words (then attention 
can be correctly paid if weights are properly calculated).
Compared to overall accuracy, the model resulted in much poor segmentation accuracy 
when there were no gold words (case b).
However, it achieved better accuracy when gold words were available (cases c and d),
with the benefit of proper word information.
Case (c), in which almost perfect segmentation accuracy was achieved, might have the 
most easily identified correct labels due to less ambiguity.
In case (d), the model successfully paid attention at a rate of more than 93\% and 
achieved much higher segmentation accuracy than in case (b). 

%%%%In Fig. \ref{fig:att} \subref{fig:att_b}, 
    In Fig. \ref{fig:att}~(ii), 
we investigated detailed performance in 
case (d); we divided all examples of characters into intervals from $[0, 0.1)$ to 
$[0.9, 1.0]$\footnote{We omitted intervals from $[0, 0.1)$ to $[0.3, 0.4)$ with 
infrequent examples from the figure. They had only 21 examples in total (0.1\% of 
all), and the corresponding segmentation accuracies were close to 90\%.}
on the basis of the maximum value of attention weights to (one of the) candidate 
words and evaluated both accuracies for each interval.\footnote{For example, 
if there are two characters that one attends to its gold word with the weight 
of 0.95 and the other attends to an incorrect word with the weight of 0.95, 
then attention accuracy for $[0.9, 1.0]$ is $1/2$.}
As Fig. \ref{fig:att} illustrates, distribution of maximum weight 
$\alpha_{i,j^\star}$ was biased toward a higher value, that is, the case in which
$\alpha_{i,j^\star} \geq 0.9$ corresponded to about 89\% of all cases.
Both attention and segmentation accuracy improved with increased value of 
$\alpha_{i,j^\star}$, and therefore, this confidence score properly reflected 
the model's certainty of prediction.
We obtained high segmentation accuracy (99.7\%) in the most confident case in
which $\alpha_{i,j^\star} \geq 0.9$.

To examine whether a direct relationship exists between attention and segmentation 
accuracies, we controlled correctness of attention by artificially changing values 
of attention weights of the trained model and evaluated segmentation accuracy for 
each ``correct attention probability''.
Specifically, on the basis of the correct attention probability threshold 
$p_t \in [0, 1]$, a random variable $p\sim \mbox{Uniform}(0,1)$, and gold labels,
we changed a weight value $\alpha_{ij}$ of the trained WCON model for a character 
$x_i$ and a word $w_j$ as follows:
\[
 \begin{cases}
  \alpha_{i,j\neq g}=\frac{1}{L}   \quad \mbox{and} \quad \alpha_{i,g}=1-  \frac{m-1}{L} & (p<p_t),\\
  \alpha_{i,j\neq j_c}=\frac{1}{L} \quad \mbox{and} \quad \alpha_{i,j_c}=1-\frac{m-1}{L} & (\mbox{otherwise}),
 \end{cases}
\]
where $m$ denotes the number of candidate words $\{w_j\}_{j=1}^m$ for the character 
$x_i$, $L=\sum_{k=1}^K k=10$ indicates the maximum number of candidate words, $g$ 
indicates the gold word's index, and $j_c$ indicates the index of a randomly chosen 
candidate word except for the gold word.
Namely, given the threshold value $p_t$, the model (forcibly) pays correct attention 
with the probability $p_t$, while assigning small weights to other candidate words.
%%%%As in Fig. \ref{fig:att} \subref{fig:att_c}, 
    As in Fig. \ref{fig:att}~(iii), 
segmentation accuracy monotonically 
improved according to the increase of correct attention probability, indicating that 
our model tends to adopt candidate word information emphasized by attention weights
for segmentation decisions and that learning accurate attention to proper words leads 
to correct segmentations.
Possibly, therefore, overall segmentation performance can be further improved by 
learning more accurate attention or discarding words with low confidence.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Effect of additional word embeddings from target domains}

Aiming to improve cross-domain performance, we tried and evaluated a simple 
method to enhance our model with target domains' unlabeled texts.
Specifically, for each JDC and JMC data, we merged unlabeled texts of source 
and all target domains, obtained auto-segmented texts by applying the same 
baseline segmenter described in \S \ref{sec:settings},
and then trained word embeddings from auto-segmented texts with the Word2Vec toolkit.
Finally, we trained from scratch the WCON model initialized with learned 
word embeddings.
We used resources in Table \ref{tab:rsrc_tgtul} as unlabeled texts for target 
domains\footnote{
For unlabeled texts of journal, patent, recipe, and blog domains, we used 
computer science paper abstracts published on IPSJ Digital Library
(\url{https://www.ipsj.or.jp/e-library/digital_library.html}),
NTCIR-8 PATMT Test Collection 
(\url{http://research.nii.ac.jp/ntcir/permission/ntcir-8/perm-en-PATMT.html}),
the Cookpad dataset (\url{https://www.nii.ac.jp/dsc/idr/cookpad/}), and Yahoo! 
Chiebukuro data (3rd edition) (\url{https://www.nii.ac.jp/dsc/idr/yahoo/chiebkr3/Y_chiebukuro.html}), 
respectively.}
in addition to BCCWJ-NC texts for source domains used in previous experiments.
For comparison, we also evaluated the baseline model enhanced with 
the same source and target unlabeled texts by self-training.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 9
\begin{table}[t]
\caption{Unlabeled texts of source and target domains}
\label{tab:rsrc_tgtul}
\input{01table09.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 10
\begin{table}[t]
\hangcaption{Cross-domain performance (mean F1 scores of three runs) of the model with and without additional unlabeled texts}
\label{tab:test_tgtemb}
\input{01table10.tex}
\vspace{4pt}\small
Methods denoted with ``S'' or ``S\&T'' used unlabeled texts in source domains or those in both source and target domains, respectively.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Table \ref{tab:test_tgtemb} shows results for baseline and proposed models
with and without additional unlabeled texts.
Although the baseline model using self-training from source unlabeled texts 
improved performance over the pure-supervised baseline, its performance was 
inferior to that of WCON with the same additional resource.
Besides that, we did not obtain further improvements on most domains by adding 
target domain texts to the baseline.
The enhanced WCON model achieved performance similar to that of the original WCON 
model on many domains and also achieved improvements by more than 0.1 points on 
journal, dining, and travel domains.
However, its performance greatly (0.5 points) decreased on the patent domain,
perhaps because of poor quality of word embeddings in this domain, whose 
occurring words differ greatly from those of the source domain (Table \ref{tab:test_unk}).
Additionally, the baseline segmenter might generate noisy inputs to train word 
embeddings.
Thus, simple addition of unlabeled texts in a target domain did not necessarily 
contribute to further improvement.
A possible solution is iterating self-training steps; we can use a trained WCON 
model for training a new WCON model by generating higher quality of word embeddings,
despite the high cost of doing so.
Another prospective method for constructing a more reliable vocabulary is to 
combine annotated resources such as lexicon and unlabeled texts in a target domain.
We leave this for the future, however.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Segmentation examples}
\label{exp_examples}

To examine actual sentences' segmentation results by the different methods, we 
pick up sentence segments (a)--(l) from the JDC's target domain test sets.
%%%%In Fig. \ref{fig:ew} \subref{fig:example} we show WCON's results, 
    In Fig. \ref{fig:ew}~(i) we show WCON's results, 
in addition to
BASE and CON's results for reference.
Regarding parts of sentences, 
%%%%in Fig. \ref{fig:ew} \subref{fig:weight}, 
    in Fig. \ref{fig:ew}~(ii), 
we also show weight values $\alpha_{ij}$ learned by WCON.

WCON predicted correct segmentations for (a)--(f) but predicted wrong results 
for (g)--(l).
%%%%Too, as shown in Fig. \ref{fig:ew} \subref{fig:weight},
    Too, as shown in Fig. \ref{fig:ew}~(ii),
the model attended to proper words for most characters in the former correct examples
(attention accuracy for each segment ranged from 66--100\%), but it often failed to 
pay correct attention in the latter incorrect examples (accuracy 0--50\%).
Besides that, we examined segmentation results by the oracle model WCON\_O,
in which attention weights are set to 1 for gold words (if existing in the 
vocabulary) and 0 for other word candidates, while other parameters are fixed to 
the trained WCON model's original values. 
The oracle model predicted correct segmentations for most sentences that the 
original model incorrectly segmented.
These observations, along with performance improvement results by the ``probabilistic 
oracle model'' discussed in \S \ref{sec:exp_att}, suggest that the model generated 
segmentation results depending on calculated attention weights.
Note that the vocabulary does not contain the gold words (``segmentation'' and 
``\textit{yude}'') for (i) and (k)\footnote{
As for (i), there exists the gold word (``segmentation'') in the original vocabulary 
without the length limit but the trained model with the maximum word length of 4 
excluded it.}
among the examples of the oracle model's incorrect segmentations.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 4
\begin{figure}[t]
\begin{center}
\includegraphics{27-3ia1f4.eps}
\end{center}
\label{fig:example}
\label{fig:weight}
\caption{Examples of segmented sentences in the BCCWJ test set}
\label{fig:ew}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}
\label{sec:related_work}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Word segmentation}%%%%\qquad
For both Chinese and Japanese, word segmentation has been traditionally addressed 
by applying statistical learning algorithms, such as maximum entropy 
\cite{uchimoto2001,xue2003}, CRFs \cite{kudo2004,peng2004,zhao2008}, and
logistic regression \cite{neubig2011}.

To reduce the burden of manual feature engineering for Chinese word segmentation,
various neural network architectures have been explored.
Specifically, character-based neural models have been developed to model the task
as a sequence labeling problem, beginning with \citeA{xzheng2013} and 
\citeA{mansur2013}'s earlier work that applied feed-forward neural networks.
Furthermore, \citeA{pei2014} used a neural tensor network to capture interactions 
between tags and characters.
More sophisticated architectures have also been used as standard components of word 
segmentation models to derive effective features automatically.
For instance, \citeA{chen2015gated} proposed gated recursive neural networks 
(GRNN) to model complicated combinations of characters, and \citeA{chen2015long} 
used LSTM to capture long distance dependencies.
\citeA{xu2016} combined LSTM and GRNN to capture long term information better 
by utilizing chain and tree structures.
Additionally, convolutional neural networks (CNNs) have also been used to extract 
complex features such as character $n$-grams \cite{chen2017feature} and 
Chinese characters' graphical features \cite{shao2017}.
\citeA{ma2018} showed that a standard BiLSTM model can achieve state-of-the-art 
results when combined with deep learning best practices.
Finally, \citeA{gan2019} showed that a self-attention network achieved 
competitive results with BiLSTM and demonstrated further improvements by integrating 
a state-of-the-art model of contextualized representations BERT \cite{devlin2019}.

On the other hand, word-based neural models have also been proposed.
Typical word-based models \cite{mzhang2016,cai2016,cai2017,yang2017} sequentially 
determine whether or not to segment each character on the basis of word-level features 
and segmentation history, while retaining multiple segmentation candidates by beam 
search decoding.
\citeA{liu2016} combined neural architectures for segment (i.e., word) 
representations into a semi-CRF framework that searches for an optimal segmentation 
sequence of variable length segments.
Using a deep CNN consisting of more than ten layers, \citeA{sun2017} proposed 
a gap-based model to predict whether or not to segment two consecutive characters.

Recent works have utilized word information on a character-based framework.
Using word boundary information from auto-segmented texts, for instance,
\citeA{zhou2017} pre-trained character embeddings.
\citeA{wang2017} explicitly introduced word information into their CNN-based 
model and concatenated embeddings of a character and multiple words corresponding to 
$n$-grams ($n$ ranging from 1--4) that include the target character.
Moreover, \citeA{jyang2019} proposed a lattice LSTM model with subsequence 
(i.e., word or subword) information.
Their model integrates information on a character and the word ending with the 
character into an LSTM cell vector for the character using a gate-mechanism.

In Japanese word segmentation, popular approaches are based on statistical learning 
algorithms.
For example, \citeA{kudo2004} used CRFs for Japanese morphological analysis
that simultaneously predicts words and parts of speech (POS) by searching an optimal 
word and POS sequence over a word lattice that enumerates possible candidate sequences 
for a sentence.
\citeA{neubig2011} used logistic regression for their pointwise segmentation method 
that makes an independent segmentation decision at each pair of characters in a sentence.
Other recent work employed neural models.
\citeA{morita2015} integrated an RNN language model into a statistical Japanese 
morphological analysis framework, while \citeA{kitagawa2018} applied a pure neural 
model based on LSTM to word segmentation treated as character-level sequence labeling.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Semi-supervised learning for word segmentation}%%%%\qquad
Especially to improve performance on OOV words, semi-supervised learning with unlabeled 
data has been explored for word segmentation.
Typical approaches include self-training \cite{yliu2012}, co-training \cite{lzhang2013},
statistical features such as accessor variety \cite{wsun2011}, and frequent 
substrings \cite{mshen2016}.
As a common practice in recent neural models, large unlabeled texts have been used to 
pre-train character/subword/word embeddings 
\cite{xzheng2013,chen2015long,mzhang2016,jyang2019}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{LSTM-CRF}%%%%\qquad
LSTM-CRF is a popular neural architecture that has been applied to various tagging tasks,
including word segmentation \cite{chen2015long}, POS tagging, and NER 
\cite{huang2015,ma2016,rei2016}.
In contrast to our work introducing candidate word information of characters on the 
character-level labeling task, \citeA{ma2016} and \citeA{rei2016} introduced 
words' internal character information on word-level labeling tasks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Attention mechanism}%%%%\qquad
An attention mechanism \cite{bahdanau2015,luong2015} was first introduced in machine 
translation to focus on appropriate parts of a source sentence during decoding.
This mechanism has been widely applied to various NLP tasks, including question answering 
\cite{sukhbaatar2015}, relation extraction \cite{lin2016}, and natural language 
inference \cite{parikh2016}.
To determine the relative importance of a word itself and the word's internal characters,
\citeA{rei2016} introduced a gate-like attention mechanism on their word-based 
sequence labeling model.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

Aiming to contribute to word boundaries' disambiguation, we proposed a word segmentation 
model that integrates word-level information into a character-based framework.
Experimental results show that our model with an attention-based composition function
achieved better performance than the model variant without attention and also than the 
existing Japanese segmentation models on Japanese datasets.

The main findings from our analysis are, first, word information from auto-segmented 
texts alleviated the unknown word problem and also contributed to robust performance 
for cross-domain segmentation.
Second, the attention mechanism learned appropriate weights for words, leading to 
accurate segmentation.
Third, due to learned attention weights, our model can generate intuitively 
interpretable segmentation results.
In future work, we will explore a more robust method for various domains' texts by 
exploiting available resources in target domains, such as lexicon and unlabeled or 
partially-labeled data.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

We would like to thank our anonymous reviewers for their constructive comments.
We used the Cookpad dataset provided by Cookpad Inc. and Yahoo! Chiebukuro 
data (3rd edition) provided by Yahoo Japan Corporation via IDR Dataset Service 
of the National Institute of Informatics.
Part of this work was published in the 2019 Conference of the North American 
Chapter of the Association for Computational Linguistics: Human Language Technologies
\cite{higashiyama2019}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.5}
\bibliography{01refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\appendix

\section{Model Parameters}
\label{sec:param}

In Table \ref{tab:param}, we show details on the numbers of parameters in the baseline 
and proposed model variants with the hyperparameter values in Table \ref{tab:hyp}.
Those in the character and word embedding matrices are average values for the three 
datasets. BCCWJ, JDC, and JMC.

\clearpage

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 11
\begin{table}[t]
\caption{Numbers of parameters in model variants}
\label{tab:param}
\input{01table11.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%\newpage
\begin{biography}
\bioauthor[:]{Shohei Higashiyama}{
received a B.S degree in engineering and an M.S. degree in system informatics 
from Kobe University in 2012 and 2014, respectively.
He was a researcher at NEC Corporation from 2014 to 2019.
He is currently a Technical Researcher at the National Institute of Information 
and Communications Technology (NICT) and a Ph.D. student at Nara Institute of 
Science and Technology (NAIST).
His main research interests are natural language processing and machine learning.
}
\bioauthor[:]{Masao Utiyama}{
received a B.S. in Computer Science from the University of Tsukuba, Japan, in 1992,
an M.S. in Computer Science from the University of Tsukuba, Japan, in 1994,
a Ph.D. in Engineering from the University of Tsukuba, Japan, in 1997.
From 1997 to 1999, he was a Research Associate at Shinshu University, Japan.
He has been a member of the National Institute of Information and Communications 
Technology, Japan, since 1999, and is currently an Executive Researcher at NICT.
}
\bioauthor[:]{Eiichiro Sumita}{
received his Ph.D. in Engineering from Kyoto University in 1999,
and a master's and bachelor's in Computer Science from the University of 
Electro-Communications in 1982 and 1980, respectively.
He is now in the National Institute of Information and Communication Technology 
(NICT), its fellow and the associate director-general of Advanced Speech Translation 
Research and Development Promotion Center (ASTREC).
His research interests cover machine translation and e-learning.
He is a co-recipient of the Maejima Hisoka Prize in 2013, the Commendation for 
Science and Technology by the Minister of Education, Culture, Sports, Science and 
Technology, Prizes for Science and Technology in 2010, and the AAMT Nagao Award 
in 2007.
}
\bioauthor[:]{Masao Ideuchi}{
received a B.S. in Environment and Information Studies from Keio University, 
Japan, in 2004, and an M.S. in the Graduate School of Media and Governance, 
Cyber Informatics Program from Keio University, Japan, in 2006.
From 2006 to 2020, he worked on the development of Japanese natural language 
processing related products and services at FUJITSU LIMITED and 
the Fujitsu Research Institute.
He has been a member of the National Institute of Information and Communications 
Technology (NICT), Japan, since 2020.
}
\bioauthor[:]{Yoshiaki Oida}{
received a bachelor degree's in Engineering and a master's degree in Human and 
Engineered Environmental Studies from the University of Tokyo in 2010 and 2012 
respectively. 
He worked on the development of a discrete event simulator for resource management 
across multiple information system development projects and earned a Ph.D. from 
the Graduate School of Frontier Sciences, the University of Tokyo in 2019.
He is currently a managing engineer at Fujitsu Ltd., as well as, since 2020, 
a part-time lecturer at the Faculty of Engineering of the University of Tokyo. 
His research interests include model-based systems engineering in manufacturing and 
information systems industry.
}
\bioauthor[:]{Yohei Sakamoto}{
received an M.S. in Particle Physics at Kyoto University, Japan, in 2013.
Currently, he is a data scientist at Ridgelinez Ltd., proposing and supporting PoC 
for AI implementation projects and business process improvement in companies.
}
\bioauthor[:]{Isaac Okada}{
received his Ph.D. in science from the Graduate School of Frontier Sciences, 
the University of Tokyo in 2019.
He is now Executive Engineer, Digital Transformation Business Unit, FUJITSU LIMITED 
and also Visiting Professor, Faculty of Commerce, Senshu University.
His research interests cover System Dynamics, Artificial Intelligence 
(Machine Learning), Natural Language Processing, Knowledge Management Technologies, 
Semantic Technologies and Metadata for Business.
}
\bioauthor[:]{Yuji Matsumoto}{
is currently a team leader at RIKEN Center for Advanced Intelligence Project (AIP). 
He received his M.S. and Ph.D. degrees in information science from Kyoto University 
in 1979 and in 1990, respectively.
He joined Machine Inference Section of Electrotechnical Laboratory in 1979.  
He has been an academic visitor at Imperial College of Science and Technology, 
the deputy chief of First Laboratory at ICOT, an associate professor at Kyoto University, 
and a professor at Nara Institute of Science and Technology.  
His main research interests are natural language understanding and machine learning.  
He is an ACL fellow and a fellow of Information Processing Society of Japan.
}
\end{biography}

\biodate



\end{document}

