\documentclass[english,tombow]{jnlp_3.3}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{url}
\usepackage{gentombow}
\settombowbleed{5mm}
\usepackage{url}
\usepackage{hangcaption_jnlp}
\usepackage{amssymb}
\usepackage{array}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%追加パッケージ
%\usepackage{graphicx}
%%%%\usepackage{plext} % 最初に読み込むこと
\usepackage{multirow}
%%%%\usepackage{amsfonts}
%%%%\usepackage{url}
%%%%\usepackage{subcaption}
%%%%\usepackage{color}
% booktab
\usepackage{booktabs}
%\usepackage[hang,small,bf]{caption}
%\usepackage[subrefformat=parens]{subcaption}
%\captionsetup{compatibility=false}
% latexにない数式の定義
\def\vector#1{\mbox{\boldmath $\mathrm{#1}$}} % \vector
\newcommand{\argmax}{\mathop\textrm{arg~max}\limits} % argmax
%%%%\newcommand{\argmin}{\mathop\textrm{arg~min}\limits} % argmin
%%%%\global\long\def\T#1{#1^{\top}} % 転置記号 (x^{\top})
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\Volume{28}
\Number{1}
\Month{March}
\Year{2021}
%%%%\Issuetitle{「　　　　」特集号}

\received{2020}{7}{30}
\revised{2020}{11}{20}
%%%%\rerevised{2006}{5}{6}
\accepted{2020}{12}{26}

\setcounter{page}{206}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Reflection-based Word Attribute Transfer}
\eauthor{Yoichi Ishibashi\affiref{Author_1} \and Katsuhito Sudoh\affiref{Author_1} \and Koichiro Yoshino\affiref{Author_1} \and Satoshi Nakamura\affiref{Author_1}} 
\eabstract{%
Word embeddings, which often represent analogic relations such as $\overrightarrow{king} - \overrightarrow{man} + \overrightarrow{woman} \approx \overrightarrow{queen}$, can be used to change an attribute of a word, including its gender. To transfer the gender attribute of \emph{king} to obtain \emph{queen} in this analogy, we subtract a difference vector $\overrightarrow{man} - \overrightarrow{woman}$ from \emph{king} based on the knowledge that \emph{king} is male. However, developing such knowledge is significantly costly for words and attributes. In this work, we propose a novel method for word attribute transfer based on reflection mapping without an analogy-based operation. Experimental results show that our proposed method can transfer the word attributes of the given words without changing the words that are invariant with respect to the target attributes.}

\ekeywords{Word, Word Embedding, Reflection, Word Attribute Transfer}

\headauthor{Ishibashi et al.}
\headtitle{Reflection-based Word Attribute Transfer}

\affilabel{Author_1}{}{Graduate School of Science and Technology, Nara Institute of Science and Technology}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Distributed representation \cite{hinton1984distributed} is a type of data representation that is trained on objectives to embed similar data samples into closed points in a vector space for capturing their similarities. 
In recent natural language processing, several studies have used neural networks. 
The distributed representation is compatible with neural networks because the representation can capture the features of language data and compress them into low-dimensional vectors. 
Word embedding methods handle word semantics in natural language processing \cite{DBLP:journals/corr/abs-1301-3781,DBLP:conf/nips/MikolovSCCD13,DBLP:conf/emnlp/PenningtonSM14,DBLP:conf/nips/NickelK17,DBLP:journals/corr/VilnisM14}. 
Word embedding models such as skip-gram with negative sampling (SGNS) \cite{DBLP:conf/nips/MikolovSCCD13} or global vectors for  word representation (GloVe) \cite{DBLP:conf/emnlp/PenningtonSM14} capture analogic relations such
as $\overrightarrow{king} - \overrightarrow{man} + \overrightarrow{woman} \approx \overrightarrow{queen}$. 
Previous works \cite{DBLP:conf/nips/LevyG14,DBLP:journals/tacl/AroraLLMR16,DBLP:conf/acl/GittensAM17,DBLP:conf/acl/EthayarajhDH19a,DBLP:conf/icml/AllenH19} offer theoretical explanations based on pointwise mutual information (PMI) \cite{DBLP:journals/coling/ChurchH90} for maintaining analogic relations in word vectors. 

These relations can be used to transfer a certain attribute of a word,
such as changing \emph{king} into \emph{queen} by transferring its gender.
This transfer can be applied to perform data augmentation; for example, rewriting \emph{He is a boy} to \emph{She is a girl}. 
It can be used to generate negative examples for natural language inference \cite{DBLP:conf/acl/HovyKSK18}. 
For example, in the Stanford Natural Language Inference (SNLI) corpus \cite{DBLP:conf/emnlp/BowmanAPM15}, negative examples can be generated by transferring a hypothesis sentence from entailment to contradiction. 
We tackled a novel task that changes a word by transferring certain attributes associated with the word, which is called \emph{word attribute transfer} (Fig.~1).

A naive way for word attribute transfer is to use a difference vector based on analogic relations, such as adding $\overrightarrow{woman} - \overrightarrow{man}$ to $\overrightarrow{king}$ to obtain $\overrightarrow{queen}$.
This requires explicit knowledge whether an input word is male or female.
We have to add a difference vector to a male word and subtract it from a female word for achieving gender transfer.
We also have to avoid changing words that are invariant with respect to gender attributes, such as \emph{is} and \emph{a} in the example above, as they are gender-invariant words.
Developing such knowledge is significantly costly for words and attributes in practice.
In this paper, we propose a novel framework for word attribute transfer based on \emph{reflection} that does not require explicit knowledge of the given words in its prediction.

The contributions of this work are twofold: (1) We propose a word attribute transfer method that obtains a vector with an inverted binary attribute without explicit knowledge.
(2) The proposed method demonstrates more accurate word attribute transfer for words that have target attributes than other baseline methods, while ensuring that the words that do not have target attributes are unchanged.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia9f1.pdf}
\end{center}
\caption{Examples of word attribute transfer}
\label{fig:concept}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Word Attribute Transfer Task}

In this task, we focus on modeling the binary attributes (e.g., male and female\footnote[1]{Gender-specific words are sometimes considered socially problematic. Here, we use this as an example based on the man-woman relation.}). 
Let $x$ denote a word and let $\vector{v}_x \in \mathbb{R}^n$ denote its $n$-dimensional vector representation.
We assume that $\vector{v}_x$ is learned in advance using an embedding model, such as a skip-gram.
In this task, we have two inputs, a word $x$ and vector $\vector{z} \in \mathbb{R}^n$, which represents a certain target attribute, and an output word $y$.
$y$ is the word obtained through the transfer of $x$ according to the target attribute specified by $\vector{z}$.
$y$ should be the same as the reference word $t$.
Note that $t$ is the same as $x$ when $x$ is invariant based on the target attribute.
In this paper, $\vector{z}$ is an $n$-dimensional vector embedded from a target attribute ID by using an embedding function of a deep learning framework. 
For example, given a set of attributes $\vector{\mathcal{Z}} = \{\mathrm{gender}, \mathrm{antonym}\}$, we assign different random vectors $\vector{z}_{\mathrm{gender}}$ for gender and $\vector{z}_{\mathrm{antonym}}$ for antonym.
Let $\mathcal{A}$ denote a set of triplets $(x, t, \vector{z})$, e.g., $(man, woman, \vector{z}_{\mathrm{gender}})\in \mathcal{A}_\mathrm{gender}$,
and $\mathcal{N}$ denote a set of invariant words for an attribute $\vector{z}$, e.g., $(person, \vector{z}_{\mathrm{gender}}) \in \mathcal{N}_\mathrm{gender}$. 
This task transfers an input word vector $\vector{v}_x$ to an output word vector $\vector{v}_y \in \mathbb{R}^n$ by using a transfer function $f_{\vector{z}_{\mathrm{attr}}}$ that inverts the attribute $\vector{z}_{\mathrm{attr}}$ of $\vector{v}_x$.
$\vector{v}_y$ is expected to be the same as its reference word vector $\vector{v}_t \in \mathbb{R}^n$.
This is denoted according to the following formula:
\begin{equation}
\vector{v}_{t} \approx \vector{v}_{y} = f_{\vector{z}} (\vector{v}_x).
\label{eq:task}
\end{equation}
The following properties must be satisfied: (1) attribute words $\{x | (x, t, \vector{z}) \in \mathcal{A}\}$ are transferred to their counterparts, and (2) invariant words $\{x | (x, \vector{z}) \in \mathcal{N}\}$ are not changed (are transferred back into themselves).
For instance, with $\vector{z}_{\mathrm{gender}}$, for a given input word \emph{man}, the gender attribute transfer $f_{\vector{z}_{\mathrm{gender}}} (\vector{v}_{man})$ should result in a vector close to $\vector{v}_{woman}$.
When given another input word \emph{person} as $x$, the result should be $\vector{v}_{person}$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Analogy-based Word Attribute Transfer}

Analogy is a general idea that can be used for word attribute transfer.
PMI-based word embedding methods, such as SGNS and GloVe, capture analogic relations, as shown in Eq. \ref{eq:an1} \cite{DBLP:conf/naacl/MikolovYZ13,DBLP:conf/conll/LevyG14,DBLP:conf/repeval/Linzen16}.
By rearranging Eq. \ref{eq:an1}, Eq. \ref{eq:an2} is obtained:
\begin{align}
\vector{v}_{queen} & \approx \vector{v}_{king} - \vector{v}_{man} + \vector{v}_{woman}, \label{eq:an1} \\
& \approx \vector{v}_{king} - (\vector{v}_{man} - \vector{v}_{woman} ). \label{eq:an2}
\end{align}
The analogy-based transfer function is
\begin{equation}
f_{\vector{z}}(\vector{v}_x) = 
\begin{cases}
\vector{v}_x - \vector{d} & \hspace{2mm} \mathrm{if} \hspace{3mm} x \in \mathcal{M} ,\\
\vector{v}_x + \vector{d} & \hspace{2mm} \mathrm{if} \hspace{3mm} x \in \mathcal{F},
\end{cases}
\label{eq:an3}
\end{equation}
where $\mathcal{M}$ is a set of words with a particular target attribute (e.g., male) and $\mathcal{F}$ is a set of words with an inverse attribute (e.g., female).
$\vector{d}$ is a difference vector, such as $\vector{v}_{man} - \vector{v}_{woman}$.
Eq. \ref{eq:an3} indicates that the operation changes depending on whether the input word $x$ belongs to $\mathcal{M}$ or $\mathcal{F}$.
However, to transfer the word attribute based on analogy, we require explicit knowledge such as the attribute value ($\mathcal{M}$, $\mathcal{F}$, or others) that is contained by the input word.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Reflection-based Word Attribute Transfer}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Ideal Transfer Mapping without Knowledge}

What is an ideal transfer function $f_{\vector{z}}$ for the word attribute transfer?
The following are the ideal natures of a transfer function:
\begin{align}
\forall (m, w, \vector{z}) \in \mathcal{A}, & \quad \vector{v}_m = f_{\vector{z}}(\vector{v}_w), \label{eq:inv1} \\
\forall (m, w, \vector{z}) \in \mathcal{A}, & \quad  \vector{v}_w = f_{\vector{z}}(\vector{v}_m), \label{eq:inv2} \\
\forall (u, \vector{z}) \in \mathcal{N}, & \quad  \vector{v}_u = f_{\vector{z}}(\vector{v}_u). \label{eq:inv_non}
\end{align}
The function $f_{\vector{z}}$ enables a word to be transferred without explicit knowledge
because the operation of $f_{\vector{z}}$ does not change depending on whether the input word belongs to $\mathcal{M}$ or $\mathcal{F}$.
By combining Eqs. \ref{eq:inv1}, \ref{eq:inv2} and \ref{eq:inv_non}, we obtain the following formulas: 
\begin{align}
\forall (m, w, \vector{z}) \in \mathcal{A}, & \quad \vector{v}_m = f_{\vector{z}}(\hspace{0.7mm}f_{\vector{z}}(\vector{v}_m)\hspace{0.7mm}), \label{eq:inv3} \\
\forall (m, w, \vector{z}) \in \mathcal{A}, & \quad \vector{v}_w = f_{\vector{z}}(\hspace{0.7mm}f_{\vector{z}}(\vector{v}_w)\hspace{0.7mm}), \label{eq:inv4} \\
\forall (u, \vector{z}) \in \mathcal{N}, & \quad \vector{v}_u = f_{\vector{z}}(\hspace{0.7mm}f_{\vector{z}}(\vector{v}_u)\hspace{0.7mm}). \label{eq:inv_non2}
\end{align}
Hence, the ideal transfer function is a mapping that becomes an identity mapping when we apply it twice for any $\vector{v}$.
Such a mapping is called \textit{involution} in geometry.
For example, $f \colon \vector{v} \mapsto - \vector{v}$ is an example of an involution.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Reflection} 

\textit{Reflection} $\mathrm{Ref}_{\vector{a},\vector{c}}$ is an ideal function because this mapping is an involution, as shown below:
\begin{equation}
\forall \vector{v}  \in \mathbb{R}^n, \quad \vector{v} = \mathrm{Ref}_{\vector{a},\vector{c}}(\hspace{0.7mm} \mathrm{Ref}_{\vector{a},\vector{c}}(\vector{v}) \hspace{0.7mm}).
\label{eq:doubleRef}
\end{equation}
Reflection reverses the location between two vectors in a Euclidean space through an affine hyperplane called a \emph{mirror}.
$\vector{a}$ and $\vector{c}$ are parameters that determine the mirror. 
$\vector{a} \in \mathbb{R}^n$ is a vector orthogonal to the mirror and $\vector{c} \in \mathbb{R}^n$ is a point through which the mirror passes.
Reflection is different from inverse mapping. 
When $m$ and $w$ are paired words, reflection can transfer $\vector{v}_m$ and $\vector{v}_w$ between each other with identical reflection mapping as shown in Eqs. \ref{eq:inv1} and \ref{eq:inv2}; however, an inverse mapping cannot perform this action.
Given a vector $\vector{v}$ in the Euclidean space $\mathbb{R}^n$, the formula for the reflection in the mirror is given by
\begin{equation}
\mathrm{Ref}_{\vector{a},\vector{c}}(\vector{v}) = \vector{v} - 2 \frac{(\vector{v} - \vector{c}) \cdot \vector{a}}{\vector{a} \cdot \vector{a}}\vector{a}.
\label{eq:ref}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[b]
\begin{center}
\includegraphics{28-1ia9f2.pdf}
\end{center}
\caption{Reflection-based word attribute transfer with a single mirror}
\label{fig:ref}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Proposed Method: Reflection-based Word Attribute Transfer}
\label{sec:proposed}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Reflection by a Single Mirror}
\label{non_pm}

We apply reflection to the word attribute transfer process.
We learn a mirror (hyperplane) in a pretrained embedding space using training word pairs with binary attribute $\vector{z}$ (Fig. \ref{fig:ref}).
Because the mirror is uniquely determined by two parameter vectors, $\vector{a}$ and $\vector{c}$, we estimate $\vector{a}$ and $\vector{c}$ from the target attribute $\vector{z}$ using fully connected multilayer perceptrons (MLPs):
\begin{align}
\vector{a} & = \mathrm{MLP}_{\theta_1}(\vector{z}), \label{eq:a} \\
\vector{c} & = \mathrm{MLP}_{\theta_2}(\vector{z}), \label{eq:c}
\end{align}
where $\theta$ is a set of trainable parameters of $\mathrm{MLP}_{\theta}$.
The transferred vector $\vector{v}_y$ is obtained by inverting the attribute $\vector{z}$ of $\vector{v}_x$ by reflection:
\begin{equation}
\vector{v}_y = \mathrm{Ref}_{\vector{a},\vector{c}}(\vector{v}_x).
\label{eq:y}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[b]
\begin{center}
\includegraphics{28-1ia9f3.pdf}
\end{center}
\caption{Reflection using parameterized mirrors} 
\label{fig:pm}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Reflection by Parameterized Mirrors}
\label{pm}

Reflection with a mirror according to Eqs. \ref{eq:a} and \ref{eq:c} assumes a single mirror that only depends on $\vector{z}$.
The previous discussion assumed pairs that share a stable pair, such as \emph{king} and \emph{queen}. 

However, as gender-variant words often do not come in pairs, gender is not sufficiently stable to be modeled by a single mirror.
For example, although \emph{actress} is exclusively feminine, \emph{actor} is clearly neutral in several cases.
Thus, \emph{actor} is not a masculine counterpart such as \emph{king}.
In fact, bias exists in gender words in the embedding space \cite{DBLP:conf/emnlp/ZhaoZLWC18,DBLP:conf/acl/KanekoB19}.
This phenomenon can occur not only with gender attributes but also with other attributes.
The assumption of a single mirror forces the mirror to be a hyperplane that goes through the midpoints for all word vector pairs.
However, the vector pair \emph{actor}-\emph{actress}, shown on the right in Fig. \ref{fig:pm}, cannot be transferred well as the single mirror (the green line) does not satisfy this constraint owing to the bias of the embedding space.
To solve this problem, we propose \textit{parameterized mirrors} based on the idea of using different mirrors for different words.
We define the mirror parameters $\vector{a}$ and $\vector{c}$ using the word vector $\vector{v}_x$ to be transferred in addition to the attribute vector $\vector{z}$:
\begin{align}
\vector{a} & = \mathrm{MLP}_{\theta_1}([\vector{z} ; \vector{v}_x]) \label{eq:a_pm}, \\
\vector{c} & = \mathrm{MLP}_{\theta_2}([\vector{z} ; \vector{v}_x]) \label{eq:c_pm},
\end{align}
where $[\cdot ; \cdot]$ indicates the vector concatenation in the column. 
\textit{Parameterized mirrors} are expected to work more flexibly on different words than a single mirror because \textit{parameterized mirrors} dynamically determine similar mirrors for similar words.
For instance, as shown in Fig. \ref{fig:pm}, let us assume that we learned the mirror (the blue line) that transfers $\vector{v}_{hero}$ to $\vector{v}_{heroine}$ in advance.
If the input word vector $\vector{v}_{actor}$ resembles $\vector{v}_{hero}$, a mirror that resembles the one for $\vector{v}_{hero}$ should be derived and used for the transfer.
Conversely, the reflection works as an identity mapping for a vector on the mirror (e.g., $\vector{v}_{person}$ in Fig. \ref{fig:pm}). 
That is, the proposed method assumes that invariant word vectors are located on the mirror.
Because we used a 300-dimensional embedded space in the experiment, we assume that the invariant word vector exists in a 299-dimensional subspace.

It should be noted that Eq. \ref{eq:doubleRef} may not hold for parameterized mirrors.
In the reflection with a single mirror, it is true that $\vector{v} = \mathrm{Ref}_{\vector{a},\vector{c}}(\hspace{0.7mm} \mathrm{Ref}_{\vector{a},\vector{c}}(\vector{v}))$.
However, this is not guaranteed with the $\vector{v}$-parameterized reflection $\mathrm{Ref}_{\vector{a_v},\vector{c_v}}(\vector{v})$.
This is because the mirror parameters $\vector{a_v}$ and $\vector{c_v}$ depend on an input word vector, as shown in Eqs. \ref{eq:a_pm} and \ref{eq:c_pm}.
Thus, we exclude this constraint and employ the constraints given by Eqs. \ref{eq:inv1}--\ref{eq:inv_non} for our loss function.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Weight Sharing}

In neural networks, weight sharing can reduce the number of trainable weights and often improve performance \cite{DBLP:conf/acl/XuYCW18,DBLP:conf/emnlp/LampleOCDR18}.
The mirror parameters $\vector{a}$ and $\vector{c}$ can be defined using a shared MLP as follows:
\begin{align}
\vector{o} & = \mathrm{MLP}_{\theta}([\vector{z} ; \vector{v}_x]) \label{eq:share1}, \\
\vector{a} & = \vector{W}_a \vector{o} \label{eq:share2}, \\
\vector{c} & = \vector{W}_c \vector{o} \label{eq:share3},
\end{align}
where $\theta$ indicates the shared weights.
$\vector{o} \in \mathbb{R}^m$ is an output vector of $\mathrm{MLP}_{\theta}$.
$\vector{W}_a \in \mathbb{R}^{n \times m}$ and $\vector{W}_c \in \mathbb{R}^{n \times m}$ are weight matrices corresponding to $\vector{a}$ and $\vector{c}$, respectively.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Loss Function}

The following properties must be satisfied in word attribute transfer: (1) words with attribute $\vector{z}$ are transferred and (2) words without it are not transferred.
Thus, loss $L(\Theta)$ is defined as:
\begin{align}
L(\Theta) = 
\frac{1}{|\mathcal{A}|} \sum_{(x,t,\vector{z}) \in \mathcal{A} }( \vector{v}_y - \vector{v}_t)^2 
+  \frac{1}{|\mathcal{N}|}  \sum_{(x,\vector{z}) \in \mathcal{N} }(\vector{v}_y - \vector{v}_x)^2, \label{eq:loss}, 
\end{align}
where $\Theta$ is a set of trainable parameters ($\Theta = \{\theta\}$ for weight sharing and $\Theta = \{\theta_1, \theta_2\}$ otherwise).
The first term draws the target word vector $\vector{v}_{t_i}$ closer to the corresponding transferred vector $\vector{v}_{y_i}$ and the second term prevents words that are invariant with respect to a target attribute from being moved by the transfer function. 
$\vector{v}_y$ is the output of a reflection (Eq. \ref{eq:y}).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiment}

We evaluated the performance of word attribute transfer using data with four different attributes. 
We used 300-dimensional word2vec\footnote[2]{\url{https://code.google.com/archive/p/word2vec/}} and GloVe\footnote[3]{\url{https://nlp.stanford.edu/projects/glove/}} models as the pretrained word embedding.
We used four different datasets of word pairs with four binary attributes: Male-Female (MF), Singular-Plural (SP), Capital-Country (CC), and Antonym (AN) (Table \ref{tab:dataset}).
These word pairs were collected from analogy test sets \cite{DBLP:journals/corr/abs-1301-3781,DBLP:conf/naacl/GladkovaDM16} and the Internet. 
Antonyms were obtained from the literature \cite{DBLP:conf/eacl/NguyenWV17}.
Their datasets were collected from WordNet \cite{DBLP:journals/cacm/Miller95} and Wordnik \footnote[4]{\url{https://www.wordnik.com/}}.
The original data by \citeA{DBLP:conf/eacl/NguyenWV17} contains synonyms; however, we excluded them and used only the antonyms.
We compared the models that train with attributes individually with the models that train with joint attributes.
The invariant word dataset $\mathcal{N}$ were constructed by random sampling from WordNet by excluding the attribute-variant words in the corresponding set $\mathcal{A}$.
We sampled the invariant words for the invariant portion of the training data by varying their occupancy, i.e., 0, 5, 10, 25, and 50\%, to investigate their effects on the tradeoffs between variant and invariant words.
We also chose 1,000 invariant words for the test ($|\mathcal{N}_{\mathrm{test}}|=$1,000). 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[b]
\input{09table01.tex}
\caption{Statistics of binary-attribute word datasets} 
\label{tab:dataset}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Evaluation Metrics} 
\label{eval}

We measured the accuracy and stability performances of the word attribute transfer.
The accuracy measures the number of input words in $\mathcal{A}_{\mathrm{test}}$ that were transferred correctly to the corresponding target words.
The stability score measures the number of words in $\mathcal{N}_{\mathrm{test}}$ that were not mapped to other words.
For example, in the MF transfer, given \emph{man}, the transfer is regarded as correct if \emph{woman} is the closest word to the transferred vector; otherwise, it is incorrect.
Given \emph{person}, the transfer is regarded as correct if \emph{person} is the closest word to the transferred vector; otherwise, it is incorrect.
The accuracy and stability scores are calculated using the following formula:
\begin{gather}
\delta (\vector{v}_y, t) = 
\begin{cases}
1  & \hspace{2mm} \mathrm{if} \hspace{3mm} \argmax_{k \in \mathcal{V}} (\mathrm{cos}(\vector{v}_y, \vector{v}_k)) = t \\
0  & \hspace{2mm} \mathrm{otherwise},
\end{cases}
\label{eq:delata1} \\
\mathrm{Accuracy} = \frac{1}{|\mathcal{A}_{\mathrm{test}}|} \sum_{(x, t, \vector{z}) \in \mathcal{A}_{\mathrm{test}}} \delta (\vector{v}_{y}, t), \label{eq:acc1} \\
\mathrm{Stability} =  \frac{1}{|\mathcal{N}_{\mathrm{test}}|} \sum_{(x, \vector{z}) \in \mathcal{N}_{\mathrm{test}}} \delta (\vector{v}_{y}, x),
\end{gather}
where $\mathcal{V}$ is the vocabulary of the word embedding model and $\mathrm{cos}(\vector{v}_y, \vector{v}_k)$ is the cosine similarity measure, which is defined as $\mathrm{cos}(\vector{v}_y, \vector{v}_k) = \frac{\vector{v}_y \cdot \vector{v}_k}{\| \vector{v}_y\| \|\vector{v}_k\|}$.

For the accuracy evaluation in the AN transfer, we used a different definition, as presented subsequently, to evaluate the accuracy because there are multiple possible candidates for the transfer in the AN dataset.
\begin{gather}
\delta_{\mathrm{AN}}  (\vector{v}_y, t) = 
\begin{cases}
1  & \hspace{2mm} \mathrm{if} \hspace{3mm} \argmax_{k \in \mathcal{V}} (\mathrm{cos}(\vector{v}_y, \vector{v}_k)) \in \mathcal{T}, \\
0  & \hspace{2mm} \mathrm{otherwise},
\end{cases}
\label{eq:delata2} \\
\mathrm{Accuracy}_\mathrm{AN} = \frac{1}{|\mathcal{A}_{\mathrm{test}}|} \sum_{(x, \mathcal{T}, \vector{z}) \in \mathcal{A}_{\mathrm{test}}} \delta_\mathrm{AN} (\vector{v}_{y}, \mathcal{T}),
\label{eq:acc2}
\end{gather}
where $\mathcal{T} = \{t_1, t_2, ..., t_3 \}$ is a set of target words of the input antonym word $x$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Methods and Configurations}

Because these datasets are significantly small, we added 300-dimensional Gaussian noise to every input vector during training to avoid overfitting, i.e., $\vector{v}_x + \vector{g}_{\sigma}$, where  $\vector{g}_{\sigma}$ is the Gaussian noise and $\sigma$ is the standard deviation of the Gaussian distribution. 
The reported results are presented for the best set of hyperparameters evaluated on the validation set for each model after a grid search on the following values: Adam \cite{DBLP:journals/corr/KingmaB14} learning rate $\alpha \in \{0.0001, 0.00015, 0.001, 0.0015\}$ (the other hyperparameters were the same as the original hyperparameters), $\sigma \in \{0.0, 0.05, 0.1, 0.15, 0.2\}$, and MLP inner hidden size $\in \{300, 500, 1500, 3000\}$.
Table \ref{tab:hymerparam} lists the best hyperparameters of the proposed method.
We did not use regularization methods such as dropout \cite{DBLP:journals/jmlr/SrivastavaHKSS14} or batch normalization \cite{DBLP:conf/icml/IoffeS15} because they did not show any improvement in our pilot test.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\input{09table02.tex}
\caption{Hyperparameters for reflection-based word attribute transfer}
\label{tab:hymerparam}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In training, the attribute and invariant word data were combined into one training dataset, where an invariant word $(x, \vector{z}) \in \mathcal{N}$ was represented as $(x, x, \vector{z}) \in \mathcal{N}$, similar to an attribute word $(x, t, \vector{z}) \in \mathcal{A}$. 
Thus, we could simply implement the loss function (Eq. \ref{eq:loss}) as follows: $L(\Theta) = \frac{1}{|\mathcal{A} \cap \mathcal{N}|} \sum_{(x,t,\vector{z}) \in \mathcal{A} \cap \mathcal{N} }(\vector{v}_{y} - \vector{v}_{t})^2$, where $\mathcal{A} \cap \mathcal{N}$ is a mini-batch of training data.


In our experiment, we compared our proposed method with the following baseline methods:
\begin{description}
\item[\textsc{Ref}]
This is a reflection-based word attribute transfer with a single mirror. We used a fully connected MLP with a rectified linear unit (ReLU) \cite{DBLP:journals/jmlr/GlorotBB11} to estimate $\vector{a}$ and $\vector{c}$.
\item[\textsc{Ref}$+$\textsc{PM}]
This is a reflection-based word attribute transfer with \textit{parameterized mirrors}. 
We used the same architecture of MLP as \textsc{Ref}. 
\item[\textsc{Ref}$+$\textsc{PM}$+$\textsc{Share}] 
This method consists of a reflection-based word attribute transfer with \textit{parameterized mirrors}.
We used the MLP with shared weights to estimate $\vector{a}$ and $\vector{c}$ (refer to Weight Sharing in Section \ref{sec:proposed}).
\item[MLP]
This is a fully connected MLP with ReLU: $\vector{v}_y = \mathrm{MLP} ([\vector{v}_x; \vector{z}])$. The highest accuracy models are a five-layer MLP with 1500 hidden units for SP, five-layer MLP with 3000 hidden units for AN, and three-layer MLP with 300 hidden units for the other datasets.
The optimal configurations were as follows: the learning rate for Adam $\alpha = 0.00015$ for all datasets; moreover, $\sigma = 0.05$ for AN and $\sigma = 0.1$ for the other datasets.
\item[TransE] 
The word attribute transfer task is similar to link prediction in which $(x, z, t)$ is replaced with $(\mathrm{head}, \mathrm{label}, \mathrm{tail})$.
In link prediction, given a set of triplets $(\mathrm{head}, \mathrm{label}, \mathrm{tail})$, the knowledge graph embedding model predicts the tail from the head and label.
We applied TransE \cite{DBLP:conf/nips/BordesUGWY13}, a baseline model for knowledge graph embeddings, to word attribute transfer.
We modified the model to input the word vector into the knowledge graph embedding model based on the following equations:
\begin{gather}
\vector{h} = \vector{W}_{\mathrm{head}} \vector{v}_x,\\
\vector{t} = \vector{W}_{\mathrm{tail}} \vector{v}_t,
\end{gather}
where $\vector{h} \in \mathbb{R}^k$ is a head vector and $\vector{t} \in \mathbb{R}^k$ is a tail vector.
$\vector{W}_{\mathrm{head} \in \mathbb{R}^{k \times n}}$ and $\vector{W}_{\mathrm{tail} \in \mathbb{R}^{k \times n}}$ are weight matrices corresponding to the head and tail, respectively.
The label vector $\vector{l}$ was embedded in the same way as the original TrasnE based on a set of relations $\{\mathrm{MF}, \mathrm{SP}, \mathrm{CC}, \mathrm{AN}\}$.
The optimal configurations were as follows: the latent dimension $k = 200$, learning rate $\lambda$ for stochastic gradient descent $\lambda = 1.0$, and margin $\gamma = 5.0$.
TransE was implemented using an open toolkit for knowledge embedding called OpenKE \cite{DBLP:conf/emnlp/HanCLLLSL18}.
In the evaluation, when calculating the accuracy and stability in Eqs. \ref{eq:delata1} and \ref{eq:delata2}, the score function of TransE was used instead of $\mathrm{cos}(\vector{v}_y, \vector{v}_k)$.
\item[\textsc{Diff}]
This method consists of analogy-based word attribute transfer with a difference vector, $\vector{d} = \vector{v}_m - \vector{v}_w$, where $m$ and $w$ are in the training data of $\mathcal{A}$. We chose the $\vector{d}$ that achieved the best accuracy in the validation data of $\mathcal{A}$.
We determined whether to add or subtract $\vector{d}$ to $\vector{v}_x$ based on explicit knowledge (Eq. \ref{eq:an3}). Here,
\textsc{Diff}$^+$ and \textsc{Diff}$^-$ transfer word attributes using a difference vector regardless of the explicit knowledge.  $^+$ and $^-$ add or subtract the difference vector to any input word vector.
\item[\textsc{MeanDiff}]
This method includes an analogy-based word attribute transfer with a mean difference vector $\bar{\vector{d}}$, where \\
$\bar{\vector{d}} = \frac{1}{|\mathcal{A}_{\mathrm{train}}|} \sum_{(m_i, w_i, \vector{z}) \in \mathcal{A}_{\mathrm{train}}} (\vector{v}_{m_i} - \vector{v}_{w_i})$.
We determined whether to add or subtract $\bar{\vector{d}}$ to $\vector{v}_x$ based on the explicit knowledge (Eq. \ref{eq:an3}).
\end{description}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[t]
\input{09table03.tex}
\hangcaption{Results of accuracy and stability scores. MF, SP, CC, and AN are datasets. Here, ``joint'' models are trained with joint attributes and ``individual'' models are trained with an individual attribute.}
\label{tab:score_table}
\vspace{-10pt}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Evaluation of Accuracy and Stability}

Table \ref{tab:score_table} lists the accuracy and stability results.
Because AN has a many-to-many relationship, a single difference vector cannot be obtained.
Therefore, the analogy methods (\textsc{Diff}, \textsc{Diff}$^{+-}$, \textsc{MeanDiff}, and \textsc{MeanDiff}$^{+-}$) were not applied to AN.
Different pretrained word embeddings by GloVe and word2vec provided similar results.
\textsc{Ref}$+$\textsc{PM} and \textsc{Ref}$+$\textsc{PM}$+$\textsc{Share} achieved the best accuracy among the methods that did not use explicit attribute knowledge.
For example, the accuracy of \textsc{Ref}$+$\textsc{PM} was 74\% for CC; however, the accuracy of MLP was 18\%.
For most attributes, our proposed methods outperformed the analogy-based transfer. 
Weight sharing did not significantly improve the performance of the proposed methods. 
The parameterized mirror improved the performance of a reflection-based transfer, although the learning was unstable (Fig.~\ref{fig:acc_plot}).
For stability, reflection-based transfers achieved superior stability scores, which exceeded 93\% in most cases.
We mixed all the attribute datasets and trained models.
The best model was the proposed method trained with an individual attribute dataset.
In the joint condition, the MLP demonstrated better performance than that in the individual attribute condition with the help of the larger training data.
The results show that our proposed methods transfer an input word if it has a target attribute and does not transfer an input word with better scores than the baseline methods, even though the proposed methods do not use knowledge of the input words.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 4
\begin{figure}[b]
\begin{center}
\includegraphics{28-1ia9f4.pdf}
\end{center}
\caption{Visualization of validation accuracy} 
\label{fig:acc_plot}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[b]
\input{09table04.tex}
\hangcaption{Accuracy of the top three nearest neighbors of TransE. A ``joint'' model is trained with joint attributes. An ``individual'' model is trained with an individual attribute.}
\label{tab:transe_top3}
\end{table} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In TransE, the accuracy was almost 0\% for all the attributes.\footnote[5]{We also experimented with ComplEx \cite{DBLP:conf/icml/TrouillonWRGB16}, which is a knowledge graph embedding model in a complex space; however, it is not described in this paper because it is not comparable because of its low accuracy.}
However, in the MF, SP, and CC relations, several reference words were within the top three nearest neighbors, as listed in Table \ref{tab:transe_top3}.
This result is similar not only in this task but also when learning with WN18 \cite{DBLP:conf/icml/TrouillonWRGB16}.
This is owing to the nature of TransE.
While considering AN, the accuracy in the three nearest neighbors was still low.
This can be explained by the poor performance of TransE for one-to-many or many-to-many relationships \cite{DBLP:conf/nips/BordesUGWY13}.



In the individual attribute condition, MLP worked poorly, especially in terms of stability for MF and CC, while it showed better transfer accuracy for AN than the proposed method.
We reviewed the training curves resulting from the MLP, which are shown in Fig.~\ref{fig:acc_plot} and \ref{fig:loss_plot}; however, they showed reasonable convergence.
This would be due to the training data size in the individual attribute condition, because MLP stability significantly improves in the joint condition.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 5
\begin{figure}[b]
\begin{center}
\includegraphics{28-1ia9f5.pdf}
\end{center}
\caption{Visualization of training loss} 
\label{fig:loss_plot}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We also investigated the tradeoff between transfer accuracy and stability by changing the size of the invariant words and the stability of the learning-based methods by conducting an additional experiment that varied $|\mathcal{N}_{\mathrm{train}}|$.
The large size of $\mathcal{N}_{\mathrm{train}}$ is expected to increase the stability; however, it may also decrease the accuracy.
The stability scores demonstrated by the MLP did not improve (Table \ref{tab:stb}) for MF and CC.
Conversely, the proposed methods achieved high stability scores with $|\mathcal{N}_{\mathrm{train}}|=5\%$ and maintained the accuracy.
We hypothesized that the high stability was owing to the distance between the word and its mirror.
If invariant words are distributed on the mirror, they will not be transferred. 
We investigated the distance between the input word vector $\vector{v}_x$ and its mirror (Fig.~\ref{fig:distance}).
The result showed that invariant words were close to the mirror and attribute words were distributed away from it.
If the distance between paired words is significantly small, the distance between the word and its mirror is also small.
Fig.~\ref{fig:distance_w2w} shows the distribution of the distance between the input $\vector{v}_x$ and the target word vector $\vector{v}_t$.
The distance between paired words for MF and SP is considerably smaller than that for CC and AN.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[t]
\input{09table05.tex}
\hangcaption{Relation between the size of $|\mathcal{N}_{\mathrm{train}}|$ and the stability of methods trained with an individual attribute}
\label{tab:stb}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 6
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia9f6.pdf}
\end{center}
\hangcaption{%
Distribution of distance between the input word vector and its mirror  $\frac{|(\vector{v}_x-\vector{c}) \cdot \vector{a}|}{\| \vector{a} \|}$ learned by \textsc{Ref}$+$\textsc{PM}. It can be observed that invariant words are close to the mirror and attribute words are distributed away from it.} 
\label{fig:distance}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 7
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia9f7.pdf}
\end{center}
\caption{Distribution of distance between the input word vector and the target word vector ${\|\vector{v}_x - \vector{v}_t\|}$}
\label{fig:distance_w2w} 
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Although analogy-based methods achieved high stability, their accuracy results were low.
In particular, the \textsc{MeanDiff}$^+$ and \textsc{MeanDiff}$^-$ did not change the original vector.
We hypothesized that the result can be attributed to the significantly small L2 norm of the mean difference vector $\bar{\vector{d}}$.
Table \ref{tab:diff_analysis} lists the relationship between the $\textsc{MeanDiff}$ performances and the L2 norm of the mean difference vector. 
The stability was high because the original vector was almost unchanged even if $\vector{d}$ was added or subtracted.
Conversely, when the L2 norm of $\bar{\vector{d}}$ was large, the accuracy became high as the difference vectors were similar to each other.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[b]
\input{09table06.tex}
\hangcaption{%
Analysis of difference vectors. \textbf{L2} is the L2 norm of the difference vector that the model used during the inference time ($\vector{d}$ for \textsc{Diff} and $\bar{\vector{d}}$ for \textsc{MeanDiff}). \textbf{cos} is the distribution of cosine similarities between the difference vector ($\vector{d}$ or $\bar{\vector{d}}$) and other difference vectors.}
\label{tab:diff_analysis}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 8
\begin{figure}[b]
\begin{center}
\includegraphics{28-1ia9f8.pdf}
\end{center}
\hangcaption{%
Two-dimensional principal component analysis projection of the $300$-dimensional mirror parameter $\vector{a}$. The mirror parameters were estimated by the proposed model (\textsc{Ref}$+$\textsc{PM}$+$\textsc{Share}) trained by each attribute.} 
\label{fig:pca}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\textsc{Diff}, \textsc{Diff}$^+$, and \textsc{Diff}$^-$ obtained high accuracy for CC and low accuracy for SP.
This is due to the use of a fixed difference vector $\vector{d}$ in Diff.
% 
We investigated the mean cosine similarity between the difference vector $\vector{d}$ and other difference vectors, i.e.,
$\mathrm{mean} = \frac{1}{|\mathcal{A}|} \sum_{(x, t, \vector{z}) \in \mathcal{A}} \mathrm{cos}(\vector{d}_{(x,t)}, \vector{d})$, where $\vector{d}_{(x,t)}$ is the difference vector of a word pair $(x, t)$ other than $\vector{d}$.
We found that the mean cosine similarity between SP words was almost 0\% in \textsc{Diff}$^-$, as listed in Table \ref{tab:diff_analysis}.
Thus, when we use a single difference vector, several SP words are transferred into inappropriate words.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Visualization of Parameterized Mirrors}

Fig. \ref{fig:pca} shows the principal component analysis (PCA) results of the mirror parameter $\vector{a}$ obtained for the test words.
We normalized the L2 norm of $\vector{a}$ to 1  ($\frac{\vector{a}}{\|\vector{a}\|}$).
We compared the PCA results with the results of the model trained with the joint attributes and the model trained with an individual attribute.
Similar results were obtained for both conditions.
Fig. \ref{fig:pca} suggests that the mirror parameters of the paired words are similar to each other and that those with an attribute form a cluster; words with the same attribute have similar mirror parameters, i.e., $\vector{a}$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 7
\begin{table}[b]
\vspace{-1\Cvs}
\input{09table07.tex}
\caption{Comparison of gender transfers. Each method transfers words in a sentence one by one.}
\label{tab:example1}
\end{table} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Transfer Example}

Table~\ref{tab:example1} lists the gender transfer results for a tiny example sentence. 
Here, the attribute transfer was applied to every word in the sentence $X = \{x_1, x_2, ...\}$.
Here, because words such as \emph{a} and \emph{.} are not in the vocabulary of word2vec, we omitted them from the inputs.
The MLP resulted in several incorrect transfers on gender-invariant words, e.g., \emph{the} became \emph{dear\_madam}, \emph{were} became \emph{laundresses}, \emph{boyfriend} became \emph{boyfriend}, and \emph{boy} became \emph{mother}.
Analogy-based transfers can transfer only in one direction.
\textsc{Diff}$^+$ could transfer if $x$ is female, e.g., it transferred from \emph{woman} to \emph{man}, but it could not transfer \emph{boy}.
Similarly, \textsc{Diff}$^-$ failed to transfer from female to male.
Moreover, as the stability of these methods was low, they resulted in erroneous transfers.
For example, in \textsc{Diff}$^-$, \emph{the} became \emph{Sir}.
\textsc{Ref}$+$\textsc{PM} could transfer only gender words without using explicit gender information. 
Thus, \emph{woman} was transferred to \emph{man} without the knowledge that \emph{woman} is a female word.
When the gender-invariant word \emph{married} was input, it was not changed by reflection, without the knowledge that \emph{married} has no gender attribute. 

From Table \ref{tab:example_ma}, it can be noted that words with different target attributes were transferred by each reflection-based transfer.
Thus, when \emph{daughter} was input for a MF transfer, it was transferred to \emph{son} and \emph{daughters} for SP transfer.
When \emph{Tokyo} was input for MF, SF, and AN tranfers, it was not transferred; however, it was transferred to \emph{Japan} in the CC transfer.
When \emph{stereo} was input for MF, SP, and CC transfers, it was not transferred; however, it was transferred to \emph{monaural} in the AN transfer.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 8
\begin{table}[b]
\input{09table08.tex}
\caption{Transfer of different attributes with the proposed method (\textsc{Ref}$+$\textsc{PM})}
\label{tab:example_ma}
\end{table} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 9
\begin{table}[b]
\input{09table09.tex}
\caption{Comparison of analogy scores}
\label{tab:analogy_score}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}

In our method, the performance of GloVe was better than that of word2vec.
Table \ref{tab:analogy_score} lists the scores of the Google analogy test set \cite{DBLP:journals/corr/abs-1301-3781}  for the different embedding methods, i.e., word2vec and GloVe.
From in Table \ref{tab:analogy_score}, it can be noted that the score of GloVe was higher than that of word2vec.
This indicates that the embedding space of GloVe worked better than that of word2vec in this task.
The performance of the word attribute transfer using reflection probably depends on the analogic space, because transferring will be easy if the pair of transferred words exists in a similar place in the analogic space.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 10
\begin{table}[b]
\input{09table10.tex}
\caption{Error analysis results}
\label{tab:error_word2vec}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Error Analysis}

We analyzed the attribute words that could not be transferred accurately by models trained with the individual attributes.
We categorized the failed output words into three error cases.
{\addtolength{\leftmargini}{1em}
\begin{itemize}
\item[\textbf{Case1}] The output word was the same as the input word ($y = x$). 
\item[\textbf{Case2}] The attribute of the input word was transferred but was incorrect. For example, in CC transfer, the transfer result was \emph{Beijing} when \emph{Japan} was given.
\item[\textbf{Case3}] Other types of errors.
\end{itemize}
}%
Table \ref{tab:error_word2vec} lists the results of the error analysis in word2vec.
The results show that most of the failures in the reflection-based transfer was in Case1.
We speculated that such unchanged attribute word pairs tended to be close to each other.
Fig. \ref{fig:distance_vs} shows the difference in the distance between the input word and the target word ${\|\vector{v}_x - \vector{v}_t\|}$ in the changed and unchanged attribute word pairs.
Contrary to this hypothesis, it was shown that there was no difference in the distance between the changed and unchanged pairs.
Table \ref{tab:error_example} lists examples of Case2 and Case3 in the proposed method.
For example, when given \emph{stepbrother} as a gender word, the proposed method (\textsc{Ref}$+$\textsc{PM}) did not output \emph{stepsister} but provided the result of \emph{stepmother}.
In MF and CC transfers, the maximum failures using MLP were observed in Case2 and Case3 errors, while the proposed methods demonstrated significant failure in the Case1 category.
This shows that the reflection-based transfer is more stable in MF and CC transfers than MLP. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 9
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia9f9.pdf}
\end{center}
%%%%\label{fig:distance_vs_SGNS}
%%%%\label{fig:distance_vs_GloVe}
\hangcaption{Distribution of the distance between the input word vector  $\vector{v}_x$ and the target word vector $\vector{v}_t$ (Comparisons between the changed and unchanged attribute words)}
\label{fig:distance_vs}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}

The embedded vectors obtained by SGNS \cite{DBLP:journals/corr/abs-1301-3781,DBLP:conf/nips/MikolovSCCD13} and GloVe \cite{DBLP:conf/emnlp/PenningtonSM14} have analogic relations.
The theory of analogic relations in word embeddings has been widely discussed \cite{DBLP:conf/nips/LevyG14,DBLP:journals/tacl/AroraLLMR16,DBLP:conf/acl/GittensAM17,DBLP:conf/acl/EthayarajhDH19a,DBLP:conf/icml/AllenH19,DBLP:conf/repeval/Linzen16}.
\citeA{DBLP:conf/nips/LevyG14} explain that SGNS factorizes a shifted PMI matrix.
\citeA{DBLP:conf/icml/AllenH19} and \citeA{DBLP:conf/acl/EthayarajhDH19a} argued that they proved the existence of such analogic relations without strong assumptions. 
In our work, we focused on the analogic relations in a word embedding space and propose a novel framework to obtain a transferred word vector with the target attribute. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 11
\begin{table}[t]
\input{09table11.tex}
\caption{Examples of Case2 and Case3 errors in the proposed method}
\label{tab:error_example}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Link prediction in knowledge graph embeddings can also be applied to word transfer tasks.
In knowledge graph embeddings \cite{DBLP:conf/nips/BordesUGWY13,DBLP:conf/icml/TrouillonWRGB16,DBLP:conf/icml/NickelTK11}, given a set of triplets $(\mathrm{head}, \mathrm{tail}, \mathrm{label})$, $\mathrm{tail}$ is predicted from $\mathrm{head}$ and $\mathrm{label}$
TransE \cite{DBLP:conf/nips/BordesUGWY13} is a knowledge graph embedding model.
TransE embeds entity and relation into an embedding space using a score function.
The score function models the analogy-like operation translating from the head entity to the tail entity according to the relation.
\citeA{DBLP:conf/icml/TrouillonWRGB16} proposed the knowledge graph embedding model based on complex values for link prediction.
Their knowledge graph embedding model is better suited for modeling a variety of binary relations, including symmetric and asymmetric relations.
Our task differs from link prediction in that when an invariant word for attribute $\vector{z}$ is entered, the model returns the input $x$.


The style transfer task presented in previous studies \cite{DBLP:conf/aaai/JainMAS19,DBLP:conf/nips/LogeswaranLB18} resembles ours.
In style transfer, the text style of the input sentences is changed.
For instance, \citeA{DBLP:conf/aaai/JainMAS19} transferred the style from formal to informal sentences.
\citeA{DBLP:conf/nips/LogeswaranLB18} transferred sentences by controlling attributes such as mood and tense.
These style transfer tasks use sentence pairs.
Our word attribute transfer task uses word pairs.
Style transfer changes sentence styles; however, our task changes the word attributes. 

\citeA{DBLP:conf/naacl/SoricutO15} studied morphological transformation based on character information. 
Our work aims for a more general attribute transfer, such as gender transfer and obtaining the antonym, and is not limited to morphological transformation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

This research aimed to transfer word binary attributes (e.g., gender) for applications such as data augmentation of a sentence.\footnote[6]{Our code and datasets are available at: \url{https://
github.com/ahclab/reflection}}
We can transfer word attributes using the analogy of word vectors; however, this process
requires explicit knowledge on whether the input word has the attribute or not.
However, this knowledge cannot be developed for various words and attributes in practice. 
The proposed method uses reflection-based mappings to transfer attribute-variant words into their counterparts while ensuring that attribute-invariant words are unchanged, without using attribute knowledge in the inference time.
The experimental results showed that the proposed method outperformed baseline methods in terms of transfer accuracy for attribute-variant words and stability for attribute-invariant words.
We speculated that the reason why the proposed method achieved significantly high stability was that invariant words were distributed in the mirrors.
We examined the distance between the input word vector and its mirror.
The result showed that invariant words were distributed near the mirror and attribute words were distributed away from the mirror.




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

We would like to thank the anonymous reviewers for their insightful comments and suggestions.
Part of this work was presented at the 58th Annual Meeting of the Association for
Computational Linguistics (ACL2020) Student Research Workshop (SRW) \cite{DBLP:conf/acl/IshibashiSYN20}.
This work was supported by JST CREST Grant Number JPMJCR1513.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{09refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Yoichi Ishibashi}{%
He is a Ph.D. student at the Nara Institute of Science and Technology.
He received his bachelor's degree in computer science from Kyoto Sangyo University in 2018 and his master's degree in engineering from Nara Institute of Science and Technology in 2020.
}

\bioauthor[:]{Katsuhito Sudoh}{%
He is an associate professor at the Nara Institute of Science and Technology.
He received his bachelor's degree in engineering in 2000 and his master's and Ph.D. degrees in informatics in 2002 and 2015, respectively, from Kyoto University.
He worked with the NTT Communication Science Laboratories from 2002 to 2017.
He currently works on machine translation and natural language processing.
He is a member of ACL, ANLP, IPSJ, and ASJ.
}

\bioauthor[:]{Koichiro Yoshino}{%
He received his B.A. degree in 2009 from Keio University,
M.S. degree in informatics in 2011, and Ph.D. degree in informatics in
2014 from Kyoto University. From 2014 to 2015, he was a
research fellow (PD) of Japan Society for Promotion of Science. From
2015, he has worked as an assistant professor in Nara Institute of Science and Technology (NAIST). He is currently working on areas of spoken and natural language processing, especially on spoken dialogue systems. Dr.
Koichiro Yoshino received the JSAI SIG Research Award in 2013 and ANLP
outstanding paper award in 2018. He is a member of IEEE, SIGDIAL, ACL,
IPSJ, RSJ, and ANLP.
}

\bioauthor[:]{Satoshi Nakamura}{%
He is a professor at the Nara Institute of Science and Technology, Team Leader of the Tourism Information Analytics Team, AIP Center, RIKEN, and Honorar professor at the Karlsruhe Institute of Technology, Germany. He received his B.S. from the Kyoto Institute of Technology in 1981 and a Ph.D. from the Kyoto University in 1992. He was the Director of ATR Spoken Language Communication Research Laboratories in the period 2000--2008 and Vice President of ATR in the period 2007--2008. He was the Director General of Keihanna Research Laboratories and Executive Director of Knowledge Creating Communication Research Center, National Institute of Information and Communications Technology in 2009--2010. He is currently the director of the Augmented Human Communication laboratory and a full professor at the Data Science Center and Information Science Division, Graduate School of Science and Technology, Nara Institute of Science and Technology. He was an Elected Board Member of the International Speech Communication Association, ISCA, in the period June 2011-2019, and IEEE Signal Processing Magazine Editorial Board member in 2012-2015, IEEE SPS Speech and Language Technical Committee Member in 2013-2015. He is an ATR Fellow, IPSJ Fellow, IEEE Fellow, and ISCA Fellow.
}
\end{biography}



\biodate




\end{document}
