\documentclass[english,tombow]{jnlp_3.2}
\usepackage{jnlpbbl_1.4}
\usepackage[dvipdfm]{graphicx}
\usepackage{gentombow}
\usepackage{amsmath}
\settombowbleed{5mm}
\usepackage{hangcaption_jnlp}
\usepackage{array}
\usepackage[multi]{otf}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
%%%%\makeatletter
%%%%\def\@{\spacefactor3000\space}
%%%%\makeatother

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's defitnion
%%%%\usepackage{latexsym}
\usepackage{amssymb}
\usepackage{url}
\usepackage{multirow}
%%%%\usepackage{subfig}
%%%%\usepackage{hhline}
%%%%\usepackage{color}
\usepackage{colortbl}
%%%%\renewcommand{\thesubfigure}{\roman{subfigure}}
\newcommand{\ck}{\checkmark}
\newcommand{\bd}{\textbf}
\newcommand{\bgc}{\cellcolor[gray]{0.8}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\makeatletter
\let\MYcaption\@makecaption
\makeatother

\usepackage{subcaption}
\captionsetup{compatibility=false}

\makeatletter
\let\@makecaption\MYcaption
\makeatother

\setlength{\captionwidth}{\textwidth}
\renewcommand\UrlFont{\rmfamily}
\renewcommand{\refname}{References}
\allowdisplaybreaks


\Volume{27}
\Number{3}
\Month{September}
\Year{2020}
%%%%\Issuetitle{Special Issue: XXXXXXXX}

\received{2020}{5}{1}
%%%%\revised{2006}{3}{4}
%%%%\rerevised{2006}{5}{6}
\accepted{2020}{6}{17}

\setcounter{page}{573}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Auxiliary Lexicon Word Prediction for\\ Cross-Domain Word Segmentation}
\eauthor{Shohei Higashiyama\affiref{Author_1}\affiref{Author_2} \and Masao Utiyama\affiref{Author_1} \and Yuji Matsumoto\affiref{Author_3} \and \\
	Taro Watanabe\affiref{Author_2} \and Eiichiro Sumita\affiref{Author_1}} 

\eabstract{%
Recent work has explored various neural network-based methods for word segmentation and has achieved substantial progress mainly in in-domain scenarios. There remains, however, a problem of performance degradation on target domains for which labeled data is not available. A key issue in overcomming the problem is how to use linguistic resources in target domains, such as unlabeled data and lexicons, which can be collected or constructed more easily than fully-labeled data. In this work, we propose a novel method using unlabeled data and lexicons for cross-domain word segmentation. We introduce an auxiliary prediction task, Lexicon Word Prediction, into a character-based segmenter to identify occurrences of lexical entries in unlabeled sentences. The experiments demonstrate that the proposed method achieves accurate segmentation for various Japanese and Chinese domains.
}
\ekeywords{Word Segmentation, Neural Networks, Domain Adaptation}

\headauthor{Higashiyama et al.}
\headtitle{Auxiliary Lexicon Word Prediction for Cross-Domain Word Segmentation}

\affilabel{Author_1}{}{National Institute of Information and Communications Technology}
\affilabel{Author_2}{}{Nara Institute of Science and Technology}
\affilabel{Author_3}{}{RIKEN Center for Advanced Intelligence Project}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\label{intro}

Word segmentation is a fundamental step for many NLP applications in Asian languages 
such as Japanese and Chinese, in which there are no explicit word delimiters.
In recent years, neural network models have been widely applied to word segmentation, 
especially for Chinese, to reduce the burden of manual feature engineering.
These models are categorized as character-based or word-based.
Character-based models \cite{xzheng2013,mansur2013,pei2014,chen2015gated}
predict segmentation label sequences by treating the task as character-level sequence 
labeling, while typical word-based models \cite{mzhang2016,cai2016,cai2017,yang2017}
directly segment a character sequence into words by sequentially evaluating partial 
segmentation hypotheses.
These neural models demonstrated large performance improvements for in-domain word 
segmentation.
However, these models are based on supervised learning and require a large amount of 
manually labeled data to obtain satisfactory performance.
Therefore, the main challenge in word segmentation is to achieve robust performance for 
out-of-domain texts.

Supervised and semi-supervised methods using various linguistic resources in target 
domains have also been explored for cross-domain word segmentation.
Lexicons and unlabeled data are especially good complementary resources, both of which 
can be collected or constructed more easily than fully-labeled data.
A lexicon feature \cite{neubig2011,mzhang2014,gwu2014} is a well-known 
technique that uses occurrence information of lexical entries in a given sentence.
However, models based on lexical features may not sufficiently adapt to target domains
since they cannot learn the proper relationship between feature values and segmentation 
labels for unlabeled target sentences. 
Another technique, called distant supervision \cite{mintz2009}, 
uses pseudo labeled data generated from unlabeled data and a lexicon.
\citeA{yliu2014} and \citeA{lzhao2018} augmented labeled data with pseudo 
partially-labeled data generated by matching lexical entries with unlabeled sentences.
However, pseudo labeled data can be noisy because a heuristic matching method, e.g., 
the longest matching, may not correctly resolve the ambiguities that different lexical 
entries can match within overlapping spans in a sentence.

In this work, we propose a novel word segmentation method using unlabeled data and lexicons.
We introduce an auxiliary task, which we call Lexicon Word Prediction (LWP), into a 
character-based segmenter to predict whether each character in a sentence corresponds to 
a particular position of a word retrieved from a lexicon.
With the help of the auxiliary task, a model learns word indicators from unlabeled 
(source and) target sentences, together with segmentation label information from source 
labeled sentences.
This method can naturally handle conflicts of lexicon matching by introducing multiple 
LWP tasks to predict different positions; a character in a sentence can be the beginning, 
middle, or end of different words simultaneously.

The contributions of this research are as follows:
\begin{itemize}
 \setlength{\itemsep}{0cm}
 \item Introduction of a novel word segmentation method to learn explicit signals of 
       word occurrence with surrounding contexts from unlabeled sentences.
 \item A demonstration that the method improves performance for various target domains,
       while preventing performance degradation for source and other domains.
 \item Achieving better or competitive performance on Japanese and Chinese 
       datasets, compared with existing methods for cross-domain word segmentation.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Character-Based Word Segmentation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Task Definition}

Word segmentation can be seen as a character-level sequence labeling task.
Labeled data $\mathcal{D}_l=\{(x,t)\}$ is a set of pairs of a sentence 
$x=x_{1:n}=(x_1,\cdots,x_n)$ and its gold label sequence $t=t_{1:n}=(t_1,\cdots,t_n)$,
where a character $x_i$ is in a character vocabulary $\mathcal{V}_c$ and a label $t_i$ 
is in a tag set $\mathcal{T}$.
Given a sentence $x$, a segmentation model is required to predict a label sequence 
$y=y_{1:n}=(y_1,\cdots,y_n) \in \mathcal{T}^n$.
We employ tag set $\mathcal{T}=\{{\rm B},{\rm I},{\rm E},{\rm S}\}$,
where B, I and E represent the beginning, inside and end of a multi-character word,
and S represents a single character word \cite{xue2003}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{BiLSTM Model for Word Segmentation}
\label{sec:baseline}

As a baseline, we use a BiLSTM model \cite{huang2015,chen2015long}
that is a standard architecture for character-based word segmentation.
The model consists of a character embedding layer, BiLSTM layers, and an inference layer.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Character Embedding Layer}
For each character $x_i$ in a given sentence $x$, a character embedding $\mathbf{e}_i$ 
of a $d_c$-dimensional vector is retrieved from a character embedding matrix 
$E_c \in \mathbb{R}^{d_c \times |\mathcal{V}_c|}$.
We randomly initialized character embeddings, since pre-trained character embeddings 
did not improve performance in our preliminary experiments.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{BiLSTM Layers}
Embedding vectors $\mathbf{e}_{1:n}=(\mathbf{e}_1,\cdots,\mathbf{e}_n)$ for a sentence 
are fed into an $N$-layer recurrent neural network (RNN) to derive contextualized 
representations $\mathbf{h}^{(l)}_{1:n}=(\mathbf{h}^{(l)}_1,\cdots,\mathbf{h}^{(l)}_n)$
($l \in \{1,\cdots,N\}$).
We adopt a bidirectional variant of a long short-term memory (LSTM) 
\cite{hochreiter1997} network, which addresses the issue of learning long-term 
dependencies and the gradient vanishing problem.
Bidirectional LSTM (BiLSTM) consists of forward and backward LSTMs.
An $l$-th BiLSTM layer concatenates a forward hidden vector 
$\overrightarrow{\mathbf{h}}_i^{(l)} \in \mathbb{R}^{d_r}$ and a backward hidden vector 
$\overleftarrow{\mathbf{h}}_i^{(l)} \in \mathbb{R}^{d_r}$\,, which are calculated by 
forward LSTM (${\rm LSTM}_f$) and backward LSTM (${\rm LSTM}_b$). 
The outputs is a hidden vector $\mathbf{h}_i^{(l)} \in \mathbb{R}^{2d_r}$ for each time step $i$:
\begin{align*}
 \overrightarrow{\mathbf{h}}_i^{(l)} &= {\rm LSTM}_f(\mathbf{h}_i^{(l-1)}, \overrightarrow{\mathbf{h}}_{i-1}^{(l)})~, \\
 \overleftarrow{\mathbf{h}}_i^{(l)} &= {\rm LSTM}_b(\mathbf{h}_i^{(l-1)}, \overleftarrow{\mathbf{h}}_{i+1}^{(l)})~, \\
 \mathbf{h}_i^{(l)} &= \overrightarrow{\mathbf{h}}_i^{(l)} \oplus \overleftarrow{\mathbf{h}}_i^{(l)}~, 
\end{align*}
where $\oplus$ denotes a concatenation operation, $\mathbf{h}_i^{(0)}=\mathbf{e}_i$, and 
$d_r$ is a hyperparameter that corresponds to the number of rows in the LSTM parameter 
matrices. Note that the number of columns in the parameter matrices is equal to the 
dimension of an input vector $\mathbf{h}_i^{(l-1)}$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Inference Layer}
An output vector $\mathbf{h}_i=\mathbf{h}^{(N)}_i$ of the BiLSTM for character $x_i$ is 
mapped into a $|\mathcal{T}|$-dimensional vector representing scores of segmentation 
labels via an affine layer:
\[
 \mathbf{s}_i = W_s \mathbf{h}_i + \mathbf{b}_s~,
\]
where $W_s \in \mathbb{R}^{|\mathcal{T}| \times 2d_r}$ and 
$\mathbf{b}_s \in \mathbb{R}^{|\mathcal{T}|}$ are trainable parameters.
The model outputs a predicted label sequence $y=y_{1:n}$ for a sentence $x$,
where each label $y_i \in \mathcal{T}$ corresponds to the dimension of the score 
vector $\mathbf{s}_i$ with the largest value.
Note that we did not adopt the CRF-based prediction since it did not saliently outperform 
softmax-based one in our preliminary experiments.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Training Objective}
During training, the parameters of the network are learned 
by minimizing the loss function $L_{\rm seg}$ for the segmentation task, 
which is defined as the cross entropy between gold and predicted label distributions:
\begin{align}
 L_{\rm seg}({D}_l) &= -\sum_{(x,t) \in \mathcal{D}_l} \sum_i \mathbf{t}_i \log~\mathbf{y}_i~, \label{eq:lseg} \\ 
 \mathbf{y}_i &= \frac{{\rm exp}(\mathbf{s}_i)}{\sum_k {\rm exp}(s_{i,k})}~, \nonumber
\end{align}
where $\mathbf{t}_i$ indicates the one-hot vector of the gold label $t_i$,
$\mathbf{y}_i$ indicates predicted label distribution, and $s_{i,k}$ indicates the 
$k$-th element of the score vector $\mathbf{s}_i$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Lexicon Word Prediction for Cross-Domain Word Segmentation}
\label{sec:lwp}

In addition to labeled data $\mathcal{D}_l$, we assume that unlabeled data 
$\mathcal{D}_u=\{x\}$ and a word lexicon $\mathcal{L}=\{w\}$ are available.
A word $w$ in a lexicon is a sequence of characters.
The $j$-th character of a word $w$ is denoted as $w_j$ and the length of a word as $|w|$.
Although the proposed auxiliary task is introduced to a BiLSTM model, it can be 
used with any neural architectures including convolutional neural networks (CNNs) 
\cite{lecun1998,collobert2008} and self-attention networks (SANs) 
\cite{vaswani2017}. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Lexicon Word Prediction}
An explicit lexicon feature is expected to capture a word occurrence in a context.
However, labeled data in a new domain is necessary to learn the proper weights of the 
features for sentences in that domain.
Instead, we introduce a novel auxiliary task to adapt to a new domain using unlabeled 
data and a lexicon.
This method performs joint learning of the segmentation task based on labeled data
and the auxiliary task based on unlabeled data.

The Lexicon Word Prediction (LWP) auxiliary task is defined as follows: 
a model predicts whether each character in a sentence corresponds to a particular 
position of a word in a lexicon.
Formally, an auxiliary label sequence $u^{(B)}=u^{(B)}_{1:n} \in \{0,1\}^n$ is generated
for each (labeled or unlabeled) sentence $x=x_{1:n}$ by matching substrings of $x$ with 
any words in a lexicon $\mathcal{L}$.
Given a character position $j$ and a word length $k$ such that $j=1<k$, an auxiliary 
label $u^{(B)}_i$ for a character $x_i$, which indicates whether $x_i$ corresponds to 
the beginning of a word $w$ (B), is defined as follows:
\begin{equation}
 u^{(B)}_i =
 \begin{cases}
  1 \quad (\exists w \in \mathcal{L}, x_i=w_j~\mbox{and}~|w|=k)~, \\
  0 \quad (\mbox{otherwise})~.
 \end{cases} \label{eq:aux_label}
\end{equation}
Similarly, auxiliary label sequences $u^{(I)}$, $u^{(E)}$, and $u^{(S)} \in \{0,1\}^n$
are generated
to indicate the inside of a word (I), the end of a word (E), and a single character word (S).
Each label $u^{(I)}_i$, $u^{(E)}_i$, and $u^{(S)}_i$ is similarly defined by letting 
$1<j<k$, $j=k>1$, and $j=k=1$ in Eq.~(\ref{eq:aux_label}), respectively.
Fig. \ref{fig:aux_lab} illustrates an example of auxiliary label sequences for 
a sentence and a lexicon.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[t]
\begin{center}
\includegraphics{27-3ia4f1.eps}
\end{center}
\hangcaption{An example of auxiliary label sequences $u^{(B)}$, $u^{(I)}$, $u^{(E)}$, and $u^{(S)}$ of the B, I, E, and S positions for a lexicon $\mathcal{L}$ and a sentence $x$, which means ``create a database''}
\label{fig:aux_lab}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The loss of the auxiliary task is defined for each position by the cross entropy based 
on auxiliary labels, similarly to the segmentation loss $L_{\rm seg}$ in 
Eq.~(\ref{eq:lseg}).
Then, the auxiliary loss $L_{\rm aux}$ is the sum of the losses for four 
LWP tasks w.r.t B, I, E and S positions:
\[
 L_{\rm aux}(\mathcal{D}_l \cup \mathcal{D}_u) = -\sum_{(x,t) \in \mathcal{D}_l \cup \mathcal{D}_u} \sum_{p\in\{B,I,E,S\}} \sum_i \mathbf{u}^{(p)}_i \log~\mathbf{y}^{(p)}_i~,
\]
where $\mathbf{u}^{(p)}_i$ ($p \in \{B, I, E, S\}$) is the one-hot vector of the 
auxiliary label and $\mathbf{y}^{(p)}_i$ is the predicted label distribution.
Note that any lexical information except for auxiliary labels are not given for 
solving the LWP task.

Finally, the weighted sum of the loss functions of both tasks is minimized:
\begin{equation}
 L_{\rm seg}(\mathcal{D}_l) + \lambda L_{\rm aux}(\mathcal{D}_l \cup \mathcal{D}_u)~,
\label{eq:loss_merged}
\end{equation}
where $\lambda$ is a hyperparameter to control the importance of the auxiliary task.
Note that, as for labeled sentences, the model is trained not only on the segmentation 
task but also on the auxiliary task.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Task-Specific MLPs}
As additional components, we introduce multi-layer perceptrons (MLPs) to learn 
task-specific representations.
Let $d_m$ be a hyperparameter. 
A hidden vector $\mathbf{h}_i$ from the BiLSTM layers is transformed into task-specific 
vectors 
$\mathbf{m}^{({\rm seg})}_i,\mathbf{m}^{({\rm aux})}_i \in \mathbb{R}^{d_m}$ via 
different MLPs with one hidden layer for the main task (${\rm MLP}_{\rm seg}$) and 
the auxiliary task (${\rm MLP}_{\rm aux}$):
\begin{align*}
 \mathbf{m}^{({\rm seg})}_i &= {\rm MLP}_{\rm seg}(\mathbf{h}_i) ~=~ g \left (U_s \mathbf{h}_i + \mathbf{v}_s \right )~,\\
 \mathbf{m}^{({\rm aux})}_i &= {\rm MLP}_{\rm aux}(\mathbf{h}_i) ~=~ g \left (U_a \mathbf{h}_i + \mathbf{v}_a \right )~,
\end{align*}
where $g$ indicates the ReLU activation function, and 
$U_s, U_a \in \mathbb{R}^{d_m \times 2d_r}$ and 
$\mathbf{v}_s,\mathbf{v}_a \in \mathbb{R}^{d_m}$ are trainable parameters.
Then, the task-specific vector for each task is transformed into a score vector
$\mathbf{s}^{({\rm seg})}_i \in \mathbb{R}^{|\mathcal{T}|}$ or 
$\mathbf{s}^{{(\rm aux\_}p)}_i \in \mathbb{R}^2$ for $p \in \{B,I,E,S\}$, respectively:
\begin{align*}
 \mathbf{s}^{({\rm seg})}_i &= W_s \mathbf{m}^{({\rm seg})}_i + \mathbf{b}_s~,\\
 \mathbf{s}^{({\rm aux\_}p)}_i &= W_{a,p} \mathbf{m}^{({\rm aux})}_i + \mathbf{b}_{a,p}~,
\end{align*}
where 
$W_s \in \mathbb{R}^{|\mathcal{T}| \times d_m}$, $W_{a,p} \in \mathbb{R}^{2 \times d_m}$,
$\mathbf{b}_s \in \mathbb{R}^{|\mathcal{T}|}$, and $\mathbf{b}_{a,p} \in \mathbb{R}^2$
are trainable parameters.
The model outputs predicted label distributions for the segmentation and 
auxiliary tasks similarly to the baseline method described in \S \ref{sec:baseline},
and the loss is calculated by Eq.~(\ref{eq:loss_merged}).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Language Resources}
\label{sec:data}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Datasets}
For Japanese experiments, we used Japanese Dependency Corpus\footnote{
\url{http://www.ar.media.kyoto-u.ac.jp/data/word-dep/JDC-2015-06.tar.gz}} 
(JDC) \cite{mori2014}. 
The corpus consists of the core data of BCCWJ\footnote{
\url{https://pj.ninjal.ac.jp/corpus_center/bccwj/en/}}${}^{\rm ,}$\footnote{
BCCWJ is a Japanese balanced corpus including books, news articles, magazines, 
and blogs etc.} \cite{maekawa2014}, economic newspaper articles, dictionary example 
sentences, computer science journal abstracts (JNL), patent specifications (JPT), and 
recipes (RCP).
We used sentences in the former three domains as the source domain data, which we called
GEN, and randomly selected 500 and 2000 sentences for the development set and the 
in-domain test set.
We used JNL, JPT,\footnote{We converted HTML entities (e.g., ``\&gt;'')
contained in JPT to corresponding characters (e.g., ``$>$'').} and RCP domain data 
for cross-domain evaluation.

For Chinese experiments, we used two source domain data: Chinese Treebank (CTB) 
5.0\footnote{\url{https://catalog.ldc.upenn.edu/LDC2005T01}} 
and the SIGHAN Bakeoff 2005 \cite{emerson2005} PKU data.
As evaluation data for CTB, we used an internet novel dataset Zhuxian (C-ZX) 
\cite{mzhang2014}.
As evaluation data for PKU, we used three internet novel datasets \cite{lqiu2015w},
Zhuxian (P-ZX), FanrenXiuxianZhuan (FR), and DouluoDalu (DL) along with two science and 
technology datasets \cite{lqiu2015c} dermatology (DM) and patent (CPT).
These combinations of source and target domain data were adopted for comparison with 
previous work.
C-ZX and P-ZX were from the same data source but had the different data splits introduced 
by previous work \cite{mzhang2014,lqiu2015w}.
We followed the training/development/test split of CTB by \citeA{yzhang2011} and
the official training/test split of PKU.
We used randomly sampled 90\% of sentences of the PKU training data as the training set and 
the remaining sentences as the development set.
We normalized texts in the Chinese datasets by converting single-byte characters to 
double-byte ones as preprocessing.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[t]
\caption{Dataset statistics}
\label{tab:data_stat}
\input{04table01.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Table \ref{tab:data_stat} shows the dataset statistics.
Values in the train, dev, test, and unlabeled rows indicate the numbers of 
sentences, and values in the lexicon row indicate the number of entries.
Note that the development sets of C-ZX and P-ZX were not used in our experiments.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Unlabeled Data}
For in-domain experiments, 
\pagebreak
we used unlabeled data in source domains:
the non-core data of BCCWJ Version 1.1\footnote{We restored the provided 
auto-segmented data to the original raw sentences and used them as unlabeled data.} 
for GEN and Chinese Gigaword Fifth Edition\footnote{
\url{https://catalog.ldc.upenn.edu/LDC2011T13}} for CTB and PKU.

As unlabeled data for cross-domain experiments, we used Japanese computer science 
paper abstracts published on IPSJ Digital Library\footnote{
\url{https://www.ipsj.or.jp/e-library/digital_library.html} (in Japanese)} for JNL,
NTCIR-8 PATMT Test Collection\footnote{
\url{http://research.nii.ac.jp/ntcir/permission/ntcir-8/perm-en-PATMT.html}} for JPT,
the ``steps'' portion of the Cookpad Dataset\footnote{
\url{https://www.nii.ac.jp/dsc/idr/cookpad/} (in Japanese)} for RCP.
We used the unlabeled data provided by \citeA{yye2019}\footnote{
\url{https://github.com/vatile/CWS-NAACL2019}} for the target domains of the Chinese 
datasets. We used the same unlabeled data from Zhuxian for C-ZX and P-ZX.
The unlabeled data of the novel domains included raw test sentences.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Lexicon}
We used UniDic\footnote{
\url{https://unidic.ninjal.ac.jp/unidic_archive/cwj/2.1.2/unidic-mecab-2.1.2_src.zip}}
\cite{den2007}
and Jieba dictionary\footnote{\url{https://github.com/fxsjy/jieba/blob/master/jieba/dict.txt}}
as source lexicons for the Japanese and Chinese datasets, respectively.
In addition, we constructed target lexicons from keywords in Japanese computer science 
papers for JNL, from JST thesaurus\footnote{
\url{https://dbarchive.biosciencedbc.jp/en/mecab/data-1.html}} for JPT, from the 
``ingredients'' portion of the Cookpad Dataset for RCP, and from the articles on the 
novels in Baidu Baike and Chinese Wikipedia for the Chinese novel domains.
We also used the Zhuxian name lexicon\footnote{
\url{https://github.com/egrcc/Cross-Domain-CWS/blob/master/dataset/preprocess_data/zx/zx_dict.txt}} 
for C-ZX and P-ZX.

The preprocessing steps of lexicon construction were as follows.
For the JNL lexicon, we split each keyword in computer science papers by predefined 
expressions and used separated strings.\footnote{We investigated frequent functional 
expressions that occurred in between noun phrases in keywords and used 
``および''/``及び'' (and), ``とその'', (and that) ``における'' (on), ``による'' (by), 
``への'' (to), ``からの'' (from), ``のための'' (for), ``に向けた'' (toward), ``する'' 
(do), and single-character particles, such as ``が'' (nominative case) and ``と'' 
(coordinate conjunction), as the predefined expressions.
E.g., an original keyword ``機械学習と自然言語処理'' (machine learning and natural 
language processing) is split into ``機械学習'' and ``自然言語処理.''} 
For the JPT lexicon, we used both the original entries in JST thesaurus and the 
auto-segmented results of the entries.
For the RCP lexicon, we split each ingredient description by punctuation and coordinate 
conjunctions\footnote{For example, an original description ``牛肉または豚肉'' (beef 
or pork) is split into ``牛肉'' and ``豚肉.''}
and adopted strings occurring at least 10 times.
For each novel domain lexicon, we collected entity names by extracting strings 
surrounded by particular XML tags from the corresponding encyclopedia pages.
While these semi-automatically constructed (or extended) lexicons often contain 
multi-words or phrases rather than words, we avoided the manual checking cost 
by using them as they were.

We merged a source lexicon and corresponding target lexicon(s) into a single 
lexicon for each target domain and used it to train each domain-specific model.
Since there were no target lexicons for DM and CPT, a single model for these domains was 
trained using the source lexicon and the merged target unlabeled data in two domains.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Settings}
\label{sec:settings}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Baseline Methods}
We used the following three baselines.
\begin{itemize}
 \item A na\"{i}ve baseline (BL): A BiLSTM model, which is described in \S \ref{sec:baseline}, 
       trained from source labeled data.
 \item A self-training baseline (ST): A BiLSTM model trained from labeled data in source domain 
       and auto-segmented data in each target domain. 
       BL was applied to target unlabeled data to obtain auto-segmented data.
 \item A lexicon feature baseline (LF): A BiLSTM model enhanced with lexicon features and 
       trained from source labeled data.
       Binary lexicon features were defined that indicated whether a character corresponded 
       to a particular position (immediate left, immediate right, beginning, middle, 
       or end) of any lexical word of length $k$ ($2 \leq k \leq 3$, $4 \leq k \leq 5$, 
       or $6 \leq k \leq 10$).
       A fixed-sized\footnote{
       The dimension was $15=5~{\rm positions} \times 3~{\rm length~groups}$.}
       vector $\mathbf{l}_i$ was constructed for a character $x_i$ and used 
       $\mathbf{e}^\prime_i=\mathbf{e}_i \oplus \mathbf{l}_i$ as input to BiLSTM layers,
       instead of the character embedding $\mathbf{e}_i$.
       Differing from the previous work \cite{qzhang2018} that extended a standard 
       LSTM  architecture to incorporate lexicon features, these features were used in the 
       above simple manner.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Training Setting}
Suppose there were $n_l$ labeled sentences and $n_u$ unlabeled sentences.
To keep training time manageable for a large amount of unlabeled data, 
for each training epoch, we used $2n_l$ training sentences 
consisting of all labeled sentences and randomly sampled $n_l$ unlabeled sentences.
In each iteration, we alternately made a mini-batch consisting only of labeled or 
unlabeled sentences.
Only $L_{\rm aux}$ was calculated in Eq.~(\ref{eq:loss_merged}) for mini-batches consisting 
of unlabeled sentences.
In this way, we trained a domain-specific model for each target domain.
We adopted a similar strategy for training the ST baseline using auto-segmented 
sentences instead of raw unlabeled sentences.

Table \ref{tab:hyp} gives the hyperparameters for the baseline and proposed methods.
We used lexical words whose length was less or equal to 6 when generating auxiliary 
labels, because there were many cases in which longer lexical entries in target lexicons 
were not single words.\footnote{
For example, the target lexicon for RCP has multi-word entries such as 
{``トマトジュース''} (tomato juice),\\
{``しょうゆ大さじ１''} (a tablespoon of soy sauce), 
and {``中華スープの素''} (Chinese soup mix).}
We applied dropout \cite{zaremba2014} to non-recurrent connections of recurrent layers.
We used a mini-batch stochastic gradient descent to optimize parameters and decayed the 
learning rate with a fixed decay rate every epoch after the first five epochs.
We trained models for up to 20 epochs and used early stopping based on the F1 score on the 
development set.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\caption{Hyperparameter values for the baseline and proposed methods}
\label{tab:hyp}
\input{04table02.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[t]
\caption{Performance on source domain test sets}
\label{tab:src_test}
\input{04table03.tex}
\vspace{4pt}\small
The resource column lists resources used by each method: source unlabeled data $\mathcal{U}_s$ and source lexicon $\mathcal{L}_s$.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Main Results}
\label{sec:results}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{In-domain results}
We evaluated the baseline methods and the proposed method with source domain resources 
(LWP-S) in the in-domain setting, expecting our auxiliary task to encode word 
occurrence information and to work similarly to a lexicon feature in source domains.
Table \ref{tab:src_test} shows the mean F1 score of three runs for each method and 
each dataset.
We conducted McNemar's tests \cite{gillick1989} on the differences between 
word-level predictions (true positive or false negative) of two systems for gold 
words.
The symbols $\star$, $\dagger$, and $\ddagger$ in Table \ref{tab:src_test}
indicate statistical significance at the 0.001 level over the BL, ST, and LF, 
respectively.
The symbol $\underline{\ddagger}$ indicates that the performance is significantly 
lower than that of the LF.

The improvements of the ST and the LWP-S over the BL were significant on JDC and PKU, 
and those of the LF over the BL were significant on three datasets.
The performance differences between the ST and the LWP-S were significant on three 
domains,\footnote{Our recall-oriented significance tests showed that the improvement 
of the LWP-S (recall of 97.6) over the ST (97.3) on CTB was significant, although 
the ST performed better in terms of F1 score as in Table \ref{tab:src_test}.} 
and that between the LF and the LWP-S was significant only on PKU.
Compared to the LF, the proposed method achieved similar but slightly lower 
segmentation performance; 
the LF has the advantage that it accesses information on all words in a lexicon,
while the proposed method only uses information encoded in the model via pseudo labels 
during training.

Table \ref{tab:aux_acc} shows the mean accuracy of three runs of the LWP-S on auxiliary 
label classification on the test sets.
Our method yielded at least 97\% accuracy for each position while the overall 
performance was biased toward the easiest S position.
These results supported the expectation; our method successfully learns word occurrence 
information and exploits it for segmentation decisions.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[t]
\caption{Accuracy of LWP-S on auxiliary label classification on the test sets}
\label{tab:aux_acc}
\input{04table04.tex}
\vspace{4pt}\small
Values in ``()'' indicate the percentage of positive labels ($u_i^{(p)}=1$) in all labels for each position $p \in \{B,I,E,S\}$.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Cross-domain results}

In the cross-domain setting, we evaluated the baseline methods and the proposed method 
with source domain resources (LWP-S) or target domain resources (LWP-T) on the 
target domain test sets.
Table \ref{tab:test} shows the mean F1 score of three runs for each method and each 
dataset.
The symbols $\star$, $\dagger$, and $\ddagger$ in Table \ref{tab:test} indicate 
statistical significance at the 0.001 level over the BL, ST, and LF, respectively,
according to the McNeamer's tests similar to the in-domain experiments.
The symbol $\underline{\star}$ indicate that the performance is significantly lower 
than that of BL.

The ST showed limited improvements (+0.2 points over the BL on average).
The LF showed a more clearly improved performance (+2.0 points over the BL).
The proposed method, LWP-T, achieved larger improvements (+3.2 points over the BL) 
than the other enhanced baselines on all domains.
These results validate our auxiliary task that enables it to learn word indicators in 
target contexts.
The performance of the ST, LF, and LWP-T was significantly better than that of the BL on 
6, 8, and 9 out of 9 datasets, respectively.\footnote{Our significance tests showed that 
the improvement of the ST (recall of 94.9) over the BL (94.8) on JPT, and the degradation 
of the LF (recall of 94.5) over the BL (95.0) on RCP were significant, although the BL 
performed better in the former case and the LF performed better in the latter case 
in terms of F1 score as in Table \ref{tab:test}.}
Moreover, the performance of the LWP-T was significantly better than that of the ST and 
the LF on all domains. Note that the performance of the LWP-S was also significantly better 
than that of the BL and the ST on all domains and than that of the LF on the three Japanese 
domains.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[t]
\caption{Performance on target domain test sets}
\label{tab:test}
\input{04table05.tex}
\par\vspace{4pt}\small
The resource column lists resources used by each method: target unlabeled data $\mathcal{U}_t$, target lexicon $\mathcal{L}_t$, and oracle lexicon $\mathcal{V}_{w:{\rm test}}$ (i.e., the set of gold words in the test set). Both LF and LWP-T used only the source lexicon on DM and CPT (i.e., $\mathcal{L}_t=\emptyset$ for these domains).
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[t]
\caption{OOV rate of a test set for a vocabulary}
\label{tab:oov_rate}
\input{04table06.tex}
\vspace{4pt}\small
$\mathcal{V}_{w:{\rm train}}$ indicate the set of gold words in the corresponding training set. $\Delta$ indicates the difference of the rates between $\mathcal{V}_t$ and $\mathcal{V}_0$.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Table \ref{tab:oov_rate} shows the out-of-vocabulary (OOV) rate of a test set for a 
vocabulary $\mathcal{V}$.
OOV rate indicates the percentage of OOV word tokens, which are word tokens not 
contained in $\mathcal{V}$, in all word tokens in the test set.\footnote{
For example, the OOV rate of a test set consisting of 6 words, ``A,'' ``A,'' ``B,'' 
``C,'' ``D'' and ``E,'' for a vocabulary $\mathcal{V}=\{ {\rm A}, {\rm B}, {\rm C} \}$ 
is $2/6=33.3\%$.}

The results led to the following findings.
First, there was a tendency for the proposed method to yield larger performance improvements
on domains in which the OOV rate largely decreased by adding a lexicon. We observed 
improvements of more than 3 F1 points on C-ZX, P-ZX, and DM (OOV rate reduction of 
more than 11 points), those of more than 2.5 F1 points on JPT, FR, DL, and CPT (OOV 
rate reduction of more than 7 points), and those of about 1 F1 point on JNL and RCP 
(OOV rate reduction of about 5 or 6 points).
Second, the proposed method is not sensitive to the size of unlabeled data; the smaller 
size of unlabeled data used for Chinese domains as in Table \ref{tab:data_stat} 
did not stop the large improvements on these domains, probably due to the OOV rate 
reduction as above.
This also suggests that a reasonable size of unlabeled data covers frequent words 
in a target domain.
Third, the proposed method using only a source lexicon was effective when combining source 
unlabeled data (LWP-S on all domains) or target unlabeled data (LWP-T on DM and 
CPT).
This concludes that the proposed method is applicable to broad domains including low resource 
domains where off-the-shelf lexicons are not available.

For reference, we evaluated the proposed method using the oracle lexicon (LWP-O), 
i.e., the set of gold words in the test set, instead of the original target lexicons.
The results are shown in the last column of Table \ref{tab:test}.
The higher performance of the LWP-O demonstrates that the proposed method can achieve 
further improvements using a higher-coverage lexicon.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 7
\begin{table}[t]
\caption{Comparison with state-of-the-art methods on test sets}
\label{tab:sota}
\input{04table07.tex}
\vspace{4pt}\small
``UL'' and ``lex'' indicates whether a method uses additional unlabeled data and lexicons, respectively. Non-neural methods are marked with ``${}^\circ$''.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Comparison with state-of-the-art methods}

Table \ref{tab:sota} shows results of state-of-the-art methods.
The results of the three methods in the second block are from our run on their 
implementations, and those of the methods in the third block are cited from their 
papers (the result of \citeA{yzhang2010} is cited from \citeA{yye2019}). 
We cited the results of \citeA{lgan2019}'s method that did not rely on POS 
information for comparison of methods based on unlabeled data and/or word lexicons.

Our method achieved better performance than existing methods on some domains (JNL, 
JPT, DM, and CPT) and competitive performance on the other domains, while 
direct comparison was difficult since each method relied on different unlabeled 
data or lexicons.
Note that our method was on a par with \citeA{lzhao2018}'s method that 
incorporated partially-labeled target sentences (F1 of 91.6 on C-ZX) but their 
method obtained further gains (+1.3 points as in Table \ref{tab:sota}) by 
integrating a character-level language model.
Similarly, \citeA{lgan2019} showed improvements by introducing BERT 
\cite{devlin2019} character embeddings.
Our method may also obtain benefits from combining language modeling-like 
information learned from a huge amount of data.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{27-3ia4f2.eps}
\end{center}
\caption{Performance of LWP-T for each $\lambda$ value on the JPT and C-ZX test sets}
\label{fig:lambda}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Analysis}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Influence of weight for auxiliary loss}

We investigated the influence of the hyperparameter $\lambda$ to control the 
importance of the LWP task in {Eq. (\ref{eq:loss_merged})}.
Fig. \ref{fig:lambda} shows F1 scores of single runs of the LWP-T with different 
$\lambda$ values (0.01, 0.05, 0.1, 0.25, 0.5, 0.75, 1, and 2) on the JPT and 
C-ZX test sets.
According to the increase of the value of $\lambda$, the performance of the proposed 
method improved from the baseline performance with $\lambda=0$.
However, it was saturated when $\lambda$ was moderate (around 0.1) and gradually 
degraded for larger $\lambda$ values due to over-emphasized loss values of the LWP 
task.
This tendency was consistent for both domains.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Performance of adapted models on various domains}

We regard a domain of unlabeled data used for training by our method as an adaptation 
domain.
We evaluated each adapted model (LWP-S or LWP-T) on other domains than the adaptation 
domain.
Table \ref{tab:atoe} shows the mean F1 score of three runs for each dataset.
The following was observed:
(1) as expected, the adapted models performed the best on the adaptation domains (except 
CTB) compared with the models adapted to other domains;
(2) the models adapted to any target domains performed better than the BL on the source 
domains (except CTB) and performed similarly to or better than the BL on the irrelevant 
domains,which were neither the source nor the target domains.
These results show that the proposed method can adapt to a target domain, while preventing 
performance degradation on source and other domains.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 8
\begin{table}[b]
\hangcaption{Performance of models adapted to adaptation domains (``adapt'')  on the test sets of evaluation domains (``eval'')}
\label{tab:atoe}
\vspace{-0.5\Cvs}
\input{04table08.tex}
\par\vspace{4pt}\small
Cells with gray background indicate the results on the same adaptation and evaluation domains.
\addtocounter{table}{-1}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Performance on unknown words}

We examined performance of the proposed method on unknown words, i.e., 
\pagebreak
words not occurring in the training set.
Table \ref{tab:oov_rec1} shows the mean recall of three runs of each method on 
all unknown words in each test set.
The proposed method, LWP-T, performed the best on 8 out of 9 domains and the improvements 
over the BL and the LF were +7.1 and +5.7 points on average.
The performance of the LWP-S was in between that of the BL and the LWP-T on those 8 domains.
The performance difference between the LWP-T and the LF suggests the importance of 
learning word information within the target domain's context for accurate 
recognition of unknown words.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 9
\begin{table}[t]
\caption{Recall of all unknown words in each test set}
\label{tab:oov_rec1}
\input{04table09.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Next, we evaluated the performance on high frequency unknown word types in the JPT, 
C-ZX, and FR test sets.
Table \ref{tab:oov_rec2} shows the mean recall of three runs of the BL, LF, or 
LWP-T on each unknown word type.
The proposed method recognized unknown words better than the baselines in most cases on 
JPT and C-ZX. %(8 or 10 out of 10 words).
This indicates that learning via the auxiliary labels in unlabeled data 
contributed to accurate recognition of these unknown words.
In contrast, both LF and LWP-T had degraded performance on FR.
This can be explained by the performance degradation on the words not contained 
in the lexicon used by both method, which corresponds to 5 of the top-10 
unknown words in FR.
This result suggests that the lexicon-based models are biased so that a character
sequence without positive signals is not recognized as a word.

To analyze what words were not correctly recognized, we investigate the lexicon 
coverage rate (LCR), which indicates the percentage of word types contained in 
the lexicon,\footnote{For example, given a lexicon 
$\mathcal{V}=\{ {\rm A}, {\rm B}, {\rm C} \}$, the LCR for a set of word types 
$\{ {\rm C}, {\rm D}, {\rm E}, {\rm F} \}$ is $1/4=25\%$.} for unknown word types.
Table \ref{tab:lcr} shows the LCR for unknown word types recognized with high recall 
($\geq 50\%$) and low recall ($<50\%$) by the proposed method.
The LCR was clearly lower for low recall words ($-50$ points on average) than high 
recall words.
This matches the intuition that words not covered by lexicons are difficult to 
recognize, since there are no explicit clues to segment such character sequences
as words.
A direct solution is to construct and use lexicons with larger coverage by an 
automatic word acquisition method and/or manual annotation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 10
\begin{table}[p]
\caption{Recall of top-10 frequent unknown word types}
\label{tab:oov_rec2}
\input{04table10.tex}
\vspace{4pt}\small
``In $\mathcal{L}_s \cup \mathcal{L}_t$'' indicates whether the word is in the lexicon. ``Freq'' indicates the frequency of each word type in the test set.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 11
\begin{table}[t]
\hangcaption{Lexicon coverage rate for unknown word types recognized with $\geq50\%$ and $<50\%$ recall by the proposed method LWP-T} 
\label{tab:lcr}
\input{04table11.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}
\label{sec:related_work}

For both Chinese and Japanese languages, word segmentation has been traditionally 
addressed by applying statistical learning algorithms, such as maximum entropy 
\cite{uchimoto2001,xue2003}, CRFs \cite{peng2004,kudo2004}, structured 
perceptron \cite{yzhang2007,yzhang2010}, and logistic regression 
\cite{neubig2011}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Neural Network Models for Word Segmentation}
Various neural network architectures have been explored for Chinese word segmentation
to reduce the burden of manual feature engineering.
Specifically, character-based neural models have been developed to model the task as 
sequence labeling.
Starting with work using feed-forward neural networks \cite{xzheng2013,mansur2013},
more sophisticated architectures have also been used 
as main components of word segmentation models to derive effective features automatically;
e.g., neural tensor networks \cite{pei2014},
gated recursive neural networks \cite{chen2015gated}, LSTMs 
\cite{chen2015long,ma2018}, CNNs \cite{chen2017feature,wang2017}, and SANs 
\cite{lgan2019}, 

Word-based neural models have also been proposed.
Typical models \cite{mzhang2016,cai2016,cai2017,yang2019} sequentially determine 
whether or not to segment each character on the basis of word-level features and 
segmentation history, while keeping multiple segmentation candidates by beam search 
decoding.

There has been less work applying neural models on Japanese word segmentation 
than for Chinese.
\citeA{morita2015} integrated an RNN language model into a statistical Japanese 
morphological analysis framework, which simultaneously segments a sentence into words 
and predicts word features, such as POS and lemma.
\citeA{kitagawa2018} applied a pure neural model based on an LSTM.
\fullciteA{higashiyama2019} proposed a BiLSTM-based model enhanced with word information 
using an attention mechanism.
\citeA{tolmachev2019} demonstrated that a BiLSTM or a SAN-based morphological analyzer 
relying only on character embeddings decreased the model size by more than 95\%
compared to traditional dictionary-based models while achieving competitive performance.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Use of Linguistic Resources for Word Segmentation}
It has been demonstrated that the use of linguistic resources, such as 
unlabeled/partially-labeled data and lexicons, has achieved robust performance for 
out-of-domain texts.
Well-known techniques using unlabeled data include self-training \cite{yliu2012}
and statistical features \cite{gwu2014,sudoh2014} such as accessor variety 
and branching entropy. %\cite{hzhao2008,wsun2011,}.
Punctuation and hyperlink information can be regarded as natural annotation.
Text with such information was viewed as partially-labeled data 
\cite{wjiang2013,lzhang2013,yliu2014}.
Distantly-supervised data generated from unlabeled data and lexicons was also used 
as partially-labeled data \cite{yliu2014,lzhao2018}.
Another well-known technique based on lexicons is a lexicon feature that indicates 
occurrence of lexicon entries \cite{neubig2011,mzhang2014,gwu2014}. 

As common practice in recent neural models, large unlabeled text has been used to 
pre-train character/word embeddings \cite{xzheng2013,chen2015long,mzhang2016}.
The work mainly focused on improvements of in-domain performance by using general 
texts such as news or Wikipedia.
On the other hand, \fullciteA{zhou2017} and \citeA{yye2019} proposed word 
segmentation-oriented training methods of character or word embeddings.
Both showed that their (pre-) trained embeddings learned from target domain text
contributed to performance improvements on the target domains.
\citeA{xwang2019} integrated auto-segmented label information by an unsupervised 
segmenter into a neural segmentation model to boost performance of in-domain word 
segmentation.

Some recent work explored neural models incorporating unlabeled data and/or lexicons 
in different manners and showed their improved performance for target domains such as 
scientific literature, novels, and social media.
\citeA{qzhang2018} proposed a character-based BiLSTM model integrated with discrete 
lexicon features and added a target lexicon when decoding text in target domains.
\citeA{lzhao2018} proposed a character-based BiLSTM model that made use of 
unlabeled and partially-labeled data in target domains.
They combined a segmentation model with a character-level language model learned from 
unlabeled data and trained the model with a modified loss function to handle 
partially-labeled data.
\citeA{jliu2019} proposed a character-based CNN model integrated with a regularization 
loss based on a lexicon so that the model's predictions for unlabeled sentences included
more words in the lexicon.
\citeA{lgan2019} proposed a character-based SAN model enhanced with word embeddings.
They used a target word-POS lexicon for a domain adaptation technique that used POS 
embeddings, instead of word embeddings, for domain-specific words whose embeddings were 
non-available.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

In this work, we proposed a novel method using unlabeled data and lexicons for 
cross-domain word segmentation.
To recognize unknown words not occurring in source domain training data, we 
incorporated lexical knowledge into a neural character-based segmenter as an auxiliary 
prediction task to identify word occurrences in unlabeled sentences.
We conducted domain adaptation experiments on Japanese and Chinese datasets with various
target domain test sets, including science and technology, recipes, and novels.
The experimental results demonstrated that our auxiliary task improved performance 
for target domains by 3.2 F1 points on average over the baseline BiLSTM segmenter,
while achieving similar or better performance for source and other domains.
Additionally, compared with existing Japanese and Chinese word segmenters,
our method achieved better or competitive performance.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

We would like to thank our anonymous reviewers for their constructive comments.
We used the Cookpad dataset provided by Cookpad Inc. via IDR Dataset Service 
of the National Institute of Informatics.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.5}
\bibliography{04refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Shohei Higashiyama}{
received a B.S degree in engineering and an M.S. degree in system informatics 
from Kobe University in 2012 and 2014, respectively.
He was a researcher at NEC Corporation from 2014 to 2019.
He is currently a Technical Researcher at the National Institute of Information 
and Communications Technology (NICT) and a Ph.D. student at Nara Institute of 
Science and Technology (NAIST).
His main research interests are natural language processing and machine learning.
}

\bioauthor[:]{Masao Utiyama}{
received a B.S. in Computer Science from the University of Tsukuba, Japan, in 1992,
an M.S. in Computer Science from the University of Tsukuba, Japan, in 1994,
a Ph.D. in Engineering from the University of Tsukuba, Japan, in 1997.
From 1997 to 1999, he was a Research Associate at Shinshu University, Japan.
He has been a member of the National Institute of Information and Communications 
Technology, Japan, since 1999, and is currently an Executive Researcher at NICT.
}

\bioauthor[:]{Yuji Matsumoto}{
is currently a team leader at RIKEN Center for Advanced Intelligence Project (AIP). 
He received his M.S. and Ph.D. degrees in information science from Kyoto University 
in 1979 and in 1990, respectively.
He joined Machine Inference Section of Electrotechnical Laboratory in 1979.  
He has been an academic visitor at Imperial College of Science and Technology, 
the deputy chief of First Laboratory at ICOT, an associate professor at Kyoto University, 
and a professor at Nara Institute of Science and Technology.  
His main research interests are natural language understanding and machine learning.  
He is an ACL fellow and a fellow of Information Processing Society of Japan.
}

\bioauthor[:]{Taro Watanabe}{received the B.E. and M.E. degrees in information science
from Kyoto Univ., Kyoto, Japan in 1994 and 1997, respectively, and obtained
the Master of Science degree in language and information technologies from
the School of Computer Science, Carnegie Mellon University in 2000.
In 2004, he received the Ph.D. in informatics from Kyoto Univ., Kyoto, Japan.
After working as a researcher at ATR, NTT and NICT, and as a software engineer 
at Google, he is a professor at Nara Institute of Science and Technology 
starting from 2020.
His research interests include natural language processing, machine learning 
and machine translation.
}

\bioauthor[:]{Eiichiro Sumita}{
received his Ph.D. in Engineering from Kyoto University in 1999,
and a master's and bachelor's in Computer Science from the University of 
Electro-Communications in 1982 and 1980, respectively.
He is now in the National Institute of Information and Communication Technology 
(NICT), its fellow and the associate director-general of Advanced Speech Translation 
Research and Development Promotion Center (ASTREC).
His research interests cover machine translation and e-learning.
He is a co-recipient of the Maejima Hisoka Prize in 2013, the Commendation for 
Science and Technology by the Minister of Education, Culture, Sports, Science and 
Technology, Prizes for Science and Technology in 2010, and the AAMT Nagao Award 
in 2007.
}

\end{biography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate




\end{document}

