\documentclass[english]{jnlp_3.3}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
\usepackage{multirow}
%%%%\usepackage{tabularx}
%%%%\usepackage{caption, subcaption}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{28}
\Number{4}
\Month{December}
\Year{2021}
%%%%\Issuetitle{「　　　　」特集号}

\received{2021}{2}{18}
\revised{2021}{7}{1}
%%%%\rerevised{2006}{5}{6}
\accepted{2021}{8}{4}

\setcounter{page}{1116}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Stylistically User-specific Response Generation}
\eauthor{Abdurrisyad Fikri\affiref{mirai} \and Hiroya Takamura\affiref{mirai}\affiref{aist} \and Manabu Okumura\affiref{mirai}} 
\eabstract{%
The ability to capture the conversation context is a necessity to build a good conversation model. However, a good model must also provide interesting and diverse responses to mimic actual human conversations. Given that different people can respond differently to the same utterance, we believe that using user-specific attributes can be useful for a conversation task. In this study, we attempt to drive the style of generated responses to resemble the style of real people using user-specific information. Our experiments show that our method applies to both seen and unseen users. Human evaluation also shows that our model outperforms the baselines in terms of relevance and style similarity.
}
\ekeywords{Response Generation, User-specific Style, Persona}

\headauthor{Fikri et al.}
\headtitle{Stylistically User-specific Response Generation}

\affilabel{mirai}{}{Department of Information and Communications Engineering, School of Engineering, Tokyo Institute of Technology}
\affilabel{aist}{}{National Institute of Advanced Industrial Science and Technology (AIST)}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\label{sec:intro}

Recently, dialog and conversation tasks have gained considerable attention \cite{li-EtAl:2016:P16-13,Zhou2017MechanismAwareNM,gao-etal-2019-structuring,zhang2020consistent,zheng2020stylized}. 
Because of the large amount of textual data available on the Internet, fully data-driven neural network models are now dominating dialog tasks. Starting with simple end-to-end sequence-to-sequence models, which simply generate responses given the contexts \cite{D11-1054,sordoni-EtAl:2015:NAACL-HLT,NIPS2014_5346}, some studies have introduced various approaches to integrate more information and features in an attempt to improve the quality of the responses. 

While mimicking actual human conversations is the goal, studies on conversation tasks frequently must focus on limited aspects because of their difficulty.
One particular problem with the conversation tasks that are repeatedly addressed is that neural conversation models tend to generate safe and general responses, e.g., ``I do not know'' or ``I am OK''\cite{sordoni-EtAl:2015:NAACL-HLT,li-EtAl:2016:P16-13}. This occurrence might be related to the fact that neural models are commonly trained to generalize the outputs.
While this type of response is not wrong, actual conversations involving humans would produce more interesting and non-monotonic responses. Hence, more information or features that can represent human behavior must be integrated to improve the quality of the responses. 
To address the interestingness and response diversity issues, some studies have focused on improving the style or response characteristics to resemble human-produced ones. One approach is to integrate user-specific information/features \cite{li-EtAl:2016:P16-13,bak-oh-2019-variational,wu-etal-2020-guiding}, or some specific persona characteristics \cite{chu-etal-2018-learning,Li_Jiang_Feng_Sprague_Zhou_Hoey_2020} to establish the association between the responses and the corresponding persona. Another approach is to transfer the specific style or latent information from additional texts to the responses \cite{W17-3541,zhang-etal-2018-personalizing,gao-etal-2019-structuring}. 

In this study, we intend to guide a model to learn to generate responses with a style that resembles a real person. We combine both user-specific identification features and user-specific style information in addition to conversation utterances. First, to obtain user-generated conversation data, we crawled Twitter as the source of the dataset. Second, we constructed conversation pairs using tweets and their corresponding replies. Third, we collected user identity (i.e., username) for the user-specific identification feature. Finally, we collected individual, single, non-reply tweets from each retrieved username and used them as user-specific style information.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[b]
\input{07table01.tex}
\caption{Sample responses from our proposed model involving four users.}
\label{tab:variation}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Because an input utterance can elicit various responses, evaluating outputs in response generation tasks is more challenging than evaluating outputs in other text generation tasks with finite expected outputs, such as machine translation. Therefore, in addition to standard automatic evaluation metrics, we rely on human evaluations to measure the performance of our model.
We measured the quality of the generated responses using the following criteria: \textit{fluency}, \textit{relevance}, and \textit{style similarity}. We also asked human judges' preferred responses.
The evaluation results show that our model can produce relevant responses that resemble the intended users' styles more than the baselines. Table \ref{tab:variation} contains some examples of generated responses from our model.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}
\label{sec:related_work}

Research on neural conversation models has been progressing, particularly following the adaptation of the sequence-to-sequence encoder-decoder framework of \citeA{NIPS2014_5346}. 
Some earlier studies used previous utterances to predict the next utterance \cite{Vinyals-Le:2015:ICML-Workshop,sordoni-EtAl:2015:NAACL-HLT,P15-1152}. 
However, because it has been demonstrated that a single input can elicit multiple responses \cite{P15-1152,N16-1014}, using only previous utterances as the conversation context may be insufficient to produce human-like natural responses.
Several approaches have been proposed to address this issue, either by incorporating specific mechanisms to alter the way responses are generated or by incorporating additional information aside from conversation utterances. 

One of the early attempts to modify the response to follow some specific ``styles'' using a specific mechanism was proposed by \citeA{miyazaki-etal-2015-automatic}. Using specific sentence-end expressions on a Japanese corpus to determine personal attributes, such as gender and age, their proposed model converted the conversation sentences to match the intended characteristic style. Another early attempt to convert dialog sentences to express more \textit{individuality} was proposed by \citeA{Mizukami2015}. Based on the statistical translation model, they combined paraphrasing and statistical language model techniques to transform sentences to include the characteristics of the target.
While \citeA{miyazaki-etal-2015-automatic} defined the specific attributes, a study by \citeA{Zhou2017MechanismAwareNM} assumed that there exists a latent feature that acts as a response mechanism that can be learned by the neural model. Using this feature, conditioned on the output of the input encoder, the model can generate diverse responses given the same utterance conditioned by different mechanisms. We argue that this work can be used to establish a foundation for associating a set of words with specific features. Another example of this approach is the work by \citeA{zhang2020consistent}, which attempted to diversify responses while maintaining the consistency of the user/speaker persona. Instead of using an additional specific feature to identify the speakers, they extracted the persona or conversation topic from previous utterances or conversation history.

We can classify the strategy for integrating additional information into three categories: (1) adding abstract information, such as user-identification features, (2) incorporating character classifications, and (3) adding readable text information.
In the first category, \citeA{li-EtAl:2016:P16-13} proposed a \textit{persona-based} encoder-decoder model, which uses a feature called \textit{speaker embeddings} that is based on the user's identity. The \textit{speaker embeddings} were learned together in the decoding phase with the intention of associating each speaker with specific words. Correspondingly, some ``nearby'' speakers might have similar words. 
Similar to \citeA{li-EtAl:2016:P16-13}, \citeA{wu-etal-2020-guiding} used similar input information while using a different variational generation approach to generate diverse responses. They used two regularization terms to guide the model to pay more attention to the user's hidden information and language preference. 
Another variational generation approach was proposed by \citeA{bak-oh-2019-variational}. Using identity features from both speakers and respondents, they adopted a variational hierarchical RNN architecture to personalize the responses.
A study by \citeA{sato-etal-2017-modeling} also added some specific discrete variables to represent conversational situations, such as involved users or conversation timing. These variables were considered additional inputs to the model.

In the second category, \citeA{Li_Jiang_Feng_Sprague_Zhou_Hoey_2020} introduced an approach to construct human-level attributes from movie character tropes and used them in a response selection task. They learned the language styles of movie characters associated with several traits, and then retrieved the suitable response associated with the same traits as the target character. A similar attempt to infer personal information in the form of movie character tropes was  previously proposed by \citeA{chu-etal-2018-learning}. They classified the character trope based on the dialog snippets using an attentive memory network.

In the third category, \citeA{D17-1228} proposed a method to steer the output style using small sets of sentences called \textit{scenting datasets}, where each set is based on one figure or persona. These sets consist of spoken-like sentences, such as speeches, movie subtitles, song lyrics. While there is evidence that this method can produce a different style for each \textit{scenting dataset}, because a model is trained to focus only on a style, it is incapable of producing multiple styles simultaneously. A similar approach was used by \citeA{zhang-etal-2018-personalizing} and \citeA{mazare-etal-2018-training}. Using a set of introductory sentences as the user's profile/persona information, they attempted to allow the model to learn and ``memorize'' the way a user speaks.

While \citeA{D17-1228} and \citeA{zhang-etal-2018-personalizing} used parallel conversation-styled texts to guide the model, \citeA{gao-etal-2019-structuring} proposed a structured shared latent space that enables the model to bridge conversation modeling and non-parallel style transfer using non-conversation texts. The work of \citeauthor{zheng2020stylized} \citeyear{zheng2020stylized} was also an attempt to capture the styles of unpaired non-conversational texts. They introduced an inverse dialog module that aims to produce a pseudo input utterance from a once-stylized response using a transformer-based encoder-decoder model \cite{NIPS2017_3f5ee243}, initialized with pretrained GPT weights \cite{radford2019language}. This pseudo input utterance would then be fed to the dialog module to help maintain the conversation context while stylizing the response.

With the increasing popularity of transformer-based models, \citeA{zhang-etal-2020-dialogpt} extended the text generation model GPT-2 \cite{radford2019language} to address the challenge of neural response generation. Inheriting the architecture of GPT-2, DialoGPT models a multi-turn dialog session as a long text and frames the generation task as language modeling. Although this model does not use user-specific information or mechanisms to capture user-specific response style, it is claimed to be able to capture the conversation style because it is pretrained with a large amount of data and contains multi-layer self-attentive mechanisms,. 

Our goal is similar to \citeA{Mizukami2015}, which aims to express more individuality rather than pre-defined personality characteristics.
In terms of input information, our work is similar to \citeA{li-EtAl:2016:P16-13} in using user-specific features to compute embeddings to help our model associate words to a specific user. 
However, we also incorporate a parallel readable text as style information for the corresponding user, which is similar to \citeA{zeng-etal-2019-automatic}, which uses a user's description and numeric features as additional information.
Using this approach, our model can deal with multiple user styles, producing diverse responses, even for unseen users in the training dataset. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Dataset}
\label{sec:dataset}

Because our target is to incorporate and emphasize the user-level speaking styles of actual humans, that is, the respondents, we need user identifications to be attributed to the conversation sentences. Therefore, we constructed our datasets from tweets collected using the Twitter API.  We constructed two types of datasets: \textit{conversation dataset} and \textit{user-info dataset}. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Conversation Dataset}

As the name implies, this dataset is designated to train the model to generate a conversation response given an input utterance. We crawled Twitter conversations and retrieved only tweets that satisfied the following conditions: (1) the tweets had to be reply tweets, that is, responses to other tweets, (2) the corresponding users have engaged in conversations with a minimum of three turns. We paired each reply with the tweet, named as \textit{response} and \textit{input utterance}, respectively. Then, we used the respondents' usernames as the user identification attribute; hence, \textit{user embeddings}. Note that to associate the user embeddings with their respective sentences, we constructed only user embeddings from this conversation dataset.

To improve data quality, we pre-processed the retrieved tweets to remove some noise, such as tweets with non-ASCII characters, duplications, and non-English tweets. To simplify the conversations, we also removed URLs, hashtags, and mentions from the tweets. 
We then filtered to include only sentences with a maximum of 15 words. The final conversation dataset consisted of 233,293 pairs of input utterances and responses from 207 human users. We limited the vocabulary tokens to 35,000 words, with utterances and responses averaging 8.35 and 8.3 tokens, respectively. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{User-info Dataset}
\label{ssec:user-info-dataset}

We constructed this user-info dataset from individual tweets (i.e., non-reply tweets) from the same users in the \textit{conversation dataset}.
There are two designated purposes for this dataset: (1) to be complementary style information for available users in the \textit{conversation dataset}, and (2) as a means to handle unseen users, that is, non-existing users in the training phase. The usage of this dataset is explained in more detail in Section \ref{ssec:attention_model}.

As the number of available single tweets from each user varies greatly, 135 tweets on average with 395 tweets are the highest, we intend to normalize and extract the important words only. To extract the selected words, we compared several scoring methods: word frequency (selecting the most frequent words), TF-IDF \cite{SparckJones:1988:SIT:106765.106782}, and pointwise mutual information (PMI) \cite{church-hanks-1990-word}. We treated all the words in the retrieved tweets as the input for every method, and then extracted the top \textit{N} words for each method. Hence, we begin with three sets of \textit{N} words. We then deployed these sets in separate preliminary experiments and qualitatively evaluated the results. We asked two fluent English speakers to compare the quality of the generated responses using the three sets of words. We provided three sets of samples of the generated responses and asked them to evaluate \textit{fluency} and \textit{relevance} to the input message. Based on their evaluation, we decided to choose TF-IDF as the scoring method to extract important words for our \textit{user-info dataset}. 
We limited the maximum number of each user's words to 50 and regarded this set of words as the user's style characteristic. From all users in the \textit{conversation dataset}, approximately 15\% of users did not have this set of words.
While we collected the sentences from the users in the \textit{conversation dataset}, we also used this dataset independent of the corresponding users in a ``mix-and-match'' setting. For example, we used the \textit{user-info dataset} from \textit{user A} to enhance the style of \textit{user B}. More details are provided in Section \ref{ssec:attention_model}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Approach}
\label{sec:approach}

As described in Section \ref{sec:related_work}, there are various approaches for integrating styles into generated or selected responses. We require our model to be able to use both user embeddings (user identification features) and user-info embeddings (users' frequent choice of words) as style-guiding features. 
Considering the effectiveness of RNN and the sequence-to-sequence approach in conversation tasks \cite{li-EtAl:2016:P16-13,sato-etal-2017-modeling,bak-oh-2019-variational}, we decided to adopt the same approach as our main architecture. To accommodate the model to learn the style, we decided to follow \citeA{li-EtAl:2016:P16-13} and \citeA{zeng-etal-2019-automatic} to use the attention mechanism as a strategy to guide the response generation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Encoder-Decoder Model}
\label{ssec:encoder-decoder-model}

We base our dialog module on the encoder-decoder model using an end-to-end sequence-to-sequence approach. Given the input sequence $X = (x_1, x_2, ..., x_{n_X})$, the model aims to produce the output sequence $Y = (y_1, y_2, ..., y_{n_Y})$ as a generated response. We adopt the long short-term memory (LSTM)\cite{Hochreiter:1997:LSM:1246443.1246450} for the encoder to compute the representation of the input sequence. Then, we keep the encoder hidden state $\bar{h}_s$ to feed into the decoder unit. We use the attention mechanism \cite{D15-1166} in the decoding phase to pay attention to the input utterance and user-specific style information. Generally, the decoding process can be interpreted using the following equation:
\begin{align}
p(y_t|y_{<t},x) &= \text{softmax}(W_s\tilde{h}_t),\\
\tilde{h}_t &= \tanh(W_c[c_t;h_t]),\\ \label{eq:attn_hidden}
h_t &= LSTM(y_{t-1},h_{t-1}),\\
c_t &= \sum_{s=1}^{S} a_{t}(s)\bar{h}_s,\\
a_{t}(s) &= \text{softmax}(h_t^\top W_a\bar{h}_s),\\\nonumber
&= \frac{\exp(h_t^\top W_a\bar{h}_s)}{\sum_{s'}\exp(h_t^\top W_a\bar{h}_{s'})} 
\end{align}
where $y_t$ denotes the response token at time step $t$. The objective is to learn its probability distribution given the preceding response tokens $y_{<t}$ and previous utterance $x$. The model measures the alignment scores $a_t(s)$ between the source-hidden states $\bar{h}_s$ and the current encoder output $h_t$. Then, we calculate the current context $c_t$ by weight-averaging the alignment scores $a_t(s)$ and the source-hidden states $\bar{h}_s$. Please note that we also calculate the alignment scores for the user-specific style information, which will be explained in the next section. Then, we concatenate all contexts and apply the $tanh$ function to obtain $\tilde{h}_t$ to feed into the $softmax$ function to produce the response token distribution. $W_a$, $W_c$, and $W_a$ denote the decoder learned parameters. 

Following \citeauthor{D15-1166} \citeyear{D15-1166}, we also apply an input-feeding approach to capture the previous alignment of the model. Input feeding is performed by concatenating $\tilde{h}_t$ to the decoder input in the next time step. For both the encoder and decoder, we employed two-layer LSTM architectures.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Response Generation with Attention to Speaker Information}
\label{ssec:generation-with-attention}

As our intention is to capture the characteristics of the users, that is, the persons \mbox{who respond} to the given utterance, we argue that using user-related information would be useful to achieve that objective. 
In contrast to \citeA{Li_Jiang_Feng_Sprague_Zhou_Hoey_2020}, which classified movie characters into several associated traits, we wanted the model to learn the style at the user level. Additionally, because we adopt LSTM in our model, we argue that using a simpler mechanism would be more effective.
Following the approach from the persona-based model by \citeA{li-EtAl:2016:P16-13}, we aim to pair conversation utterances with corresponding users-identity features and train them together. However, as this approach can only accommodate users present in the training data, we intend to overcome this limitation by adding a small dataset for each user to serve as another characteristic feature. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{User-specific Information}
\label{sec:user-specific-info}

In this study, we define two types of user-specific information: \textit{user embeddings} and \textit{user-info embeddings}. \textit{User embedings} are derived from usernames in the training data, whereas \textit{user-info embeddings} are derived from separate collections of words that are frequently used by the corresponding users in the conversation dataset. 

Following the setup described in Section \ref{ssec:encoder-decoder-model}, let $I_{train}$ denote the set of users (respondents) in the training data, $K_{word}$ the dimensions of word embeddings, and $K_{user}$ denote the dimensions of user embeddings. We convert each input sequence $X$ into embeddings of size $K_{word}$. Then, we define a user identity embedding $u_i$ with size $K_{user}$ for each user $i \in I_{train}$. The user embedding $u_i$ is shared with all conversations involving user $i$.

The second type of user-specific information involves a collection of users' selected words. To capture the characteristics, especially the speaking style, of each user, we argue that we must define a feature or a set of information that can capture it. Because ``speaking style'' is an abstract concept, we define our user-specific style as a set of selected words that can represent each user's behavior in common/daily conversation.

Let $I$ denote the set of users. Note that $I_{train}$ is a subset of $I$. Several sentences were collected for each user in $I$. From this collection of sentences, we extracted $N$ words to represent the characteristics of the user. 
As described in Section \ref{ssec:user-info-dataset}, we extracted the selected words using the TF-IDF method. Each of these words was then converted to an embedding of dimension $K_{word}$; hence, \textit{user-info embeddings}.
Note that as we extracted the selected words for each user in $I$, we have the corresponding style information not only for the user in $I_{train}$ but also for all users in $I$ if available. Therefore, our \textit{user-info embeddings} can also be used for users who are not found in the training data.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Attentional Conversation Model}
\label{ssec:attention_model}

Our attentional LSTM model takes three features as input: \textit{input word embeddings},  \textit{user embeddings},\textit{ user-info embeddings}, and \textit{input word embeddings} are used in the encoding phase, and \textit{user embeddings} and \textit{user-info embeddings} are used in the decoding phase. Because our model also incorporates the input-feeding approach, the input for the decoding phase is the concatenation of the output of the previous decoding time step $y_{t-1}$, user embedding $u_i$, and input-feeding $\tilde{h}_{t-1}$. The user-info embeddings will later be used as input for the additional attention mechanism. Hence, the decoding process can be described as follows:
\begin{equation}
\label{eq:dec_input}
h_t = LSTM([y_{t-1};u_i;\tilde{h}_{t-1}],h_{t-1}).
\end{equation}
The user-info embeddings are constructed from the collection of the top $N$ ranked words uttered by the intended users, where the users are not necessarily present in the training data. Using the same embeddings as the input word embeddings, we compose $P_i = \{p_{i_1}, ..., p_{i_N}\}$ ($\forall k,\ p_{i_k} \in \mathbb{R}^{K_{word}}$) as user-info embeddings for user $i$.

The model is trained to attend not only to the input source, that is, the hidden states of the encoder, but also to the user-info embeddings. Therefore, because this model uses two contexts, we must adjust Equation \ref{eq:attn_hidden} as follows:
\begin{equation}
\label{eq:attn_hidden_double}
\tilde{h}_t=\tanh(W_c[c_t^{(X)};c_t^{(P)};h_t]),
\end{equation}
where $c^{(X)}_t$ and $c^{(P)}_t$ are the context for the input source and user-info embeddings, respectively. 
This proposed model is illustrated in Figure \ref{fig:model}.
We can see in the Figure that the co-attended user-info embeddings are fed directly to the attention module. This allows us to inject any user's \textit{user-info embeddings} as the pair of \textit{user embeddings} in the ``plug-and-play'' mechanism, even if the user does not exist in the training data.

In terms of using co-attention beside the main attention on the input source, our model is similar to \citeA{zeng-etal-2019-automatic}, which applied co-attention to the user's self-description sentence. However, our model differs in terms of what to attend to and how we treat user-level features. We argue that adding attention to a user-related word collection would be more effective than a self-description sentence, which sometimes has no correlation with the identity of the user.
In terms of the user-identifying feature, we use \textit{user embeddings} as the feature and concatenate these embeddings to decoder inputs every time step. In comparison, they used embeddings from the user's numeric features, such as age and gender, and applied a gated memory module on the embeddings before combining them with other decoder inputs. 
\pagebreak
We postulate that including the \textit{user embeddings} in each time step of the decoding phase would allow the model to learn more easily from user-level styling.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[t]
\begin{center}
\includegraphics{28-4ia7f1.pdf}
\end{center}
\hangcaption{%
Overview of our neural conversation model with attention to user-specific information. We used a two-layer LSTM for both the encoder and decoder. The attention layer sources hidden states $\bar{h}_{s'}$ and user-info embeddings $P_i$ for user $i$. The user embeddings $u_i$ are concatenated to the decoder input at every step.}
\label{fig:model}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiment}
\label{sec:experiment}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Data}

We used both the \textit{conversation dataset} and the \textit{user-info dataset}, as described in Section \ref{sec:dataset}. For validation,  we used 1,000 pairs of utterances and responses from six users from the \textit{conversation dataset} alongside the \textit{user-info dataset} with an average of 47 tokens for each user. For the test phase, we extracted different sets of tweet sentences from 12 random users in the \textit{conversation dataset}, resulting in 100 pairs of utterances and responses. To test the learned model against unseen users, we simply replaced the users with new users who did not exist in the training data, who were then concerted to an unknown (\textit{UNK}) token.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Implementation}
We implemented our model using PyTorch\footnote{\url{https://pytorch.org/}} framework. Both the encoder and decoder employed a two-layer LSTM. The hyperparameter details are as follows:
\begin{itemize}
\item Each LSTM layer contains 300 hidden units.
\item Embedding size is set to 300.
\item Network parameters are initialized with uniform distribution $[-0.05,0.05]$.
\item Training batch size is set to 128.
\item Learning rate for the encoder is set to 0.0001, multiplied by 2.5 for the decoder.
\item Dropout rate is set to 0.1.
\item Vocabulary size is 35,000.
\end{itemize}

We trained the model using the Adam optimizer \cite{Kingma2014AdamAM} using different learning rates between the encoder and decoder. We conducted several procedures to determine the training stop conditions. We observed a decrease in the loss $H_{y'}(y):=-\sum_i y'_i \log(y_i)$. We set the epoch to 50 and then we set the early stop condition to when the loss difference was less than 7\% for a long period. As we were cautious in relying on numbers only, we also asked two fluent English speakers to evaluate the trained model quality by evaluating a small set of generated responses. Finally, the training was stopped at the 47th epoch. We also limited the maximum length of an utterance to 15 words per sentence. The training was run on a single Titan X GPU for approximately three days.

The input utterance and user-info embeddings were initialized with GloVe embeddings \linebreak
\cite{pennington-socher-manning:2014:EMNLP2014}. We replaced words not in the vocabulary with the $UNK$ tokens. The same treatment was applied to unseen users in user embeddings. We set the $UNK$ token embedding to a vector of all zeroes at the initial stage. To select the token prediction, we applied the greedy approach.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Baseline and Comparison Models}
We adopted the persona-based model of \citeA{li-EtAl:2016:P16-13} to serve as the benchmark for our model. Their model uses a user-identification attribute, called \textit{speaker embeddings}, in the decoding phase to allow the model to associate words that often appear together with some specific users. Therefore, user-specific associated words can be regarded as a user's style.

Our model is similar to their model in terms of using user embeddings during the decoding phase. However, as mentioned in Section \ref{sec:dataset}, user embeddings cannot cover unseen users. Our model overcomes this issue by using user-info embeddings. The decoder input of both our model and their model can be represented by Equation \ref{eq:dec_input}. Because the baseline model does not have user-info embeddings, our model's attentional hidden $\tilde{h}_t$ is different from theirs. The attentional hidden of the baseline model is the same as Equation \ref{eq:attn_hidden}, while our model is represented by Equation \ref{eq:attn_hidden_double}.

We prepared a variant of our proposed model using unseen (\textit{UNK}) users for user embeddings. The rationale for this setting is to investigate whether our model could generate better responses against our baseline handicap. We used only this variant during the evaluation phase.
We also compare our model with the standard sequence-to-sequence (seq2seq) model without user and user-info embeddings. This baseline can also work as an ablation study to verify whether additional user-specific features can improve the generation in general, particularly in real-world conversations.
A summary of the baseline comparison models is presented in Table \ref{tab:baseline_models}. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\input{07table02.tex}
\caption{Comparison models on the effectiveness of the proposed user-specific information features.}
\label{tab:baseline_models}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

As the transformer-based \cite{NIPS2017_3f5ee243} models are gaining popularity, we argue that it would be useful to compare our model's outputs with the transformer-based models. For this evaluation, we adopted DialoGPT \cite{zhang-etal-2020-dialogpt} as the comparison model. DialoGPT is a large neural response generation model that extends GPT-2 \cite{radford2019language} to the conversation data from Reddit. We fine-tuned DialoGPT on our conversation dataset and generated responses using the same set of conversation pairs used in the previous stages. Note that the DialoGPT model does not use user-specific information features. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Evaluation Setup}
\label{sec:evaluation_setup}

Compared with other generation tasks that have finite ground truth, such as machine translation, a conversation task can have multiple correct outputs given the same input. 
Therefore, an evaluation metric that can compare an output prediction to multiple references, such as BLEU \cite{Papineni:2002:BMA:1073083.1073135}, is preferable. 
BLEU, which was originally designed to evaluate machine translation tasks, count the number of overlapping words or n-grams between the output and references.

Despite the common use of BLEU in evaluating conversation tasks \cite{sordoni-EtAl:2015:NAACL-HLT,li-EtAl:2016:P16-13,zheng2020stylized}, according to \citeA{D16-1230:Liu_HowNotTo}, the BLEU score is weakly correlated with human judgments of dialog systems.
To overcome this issue, several studies have proposed new metrics for evaluating open-domain dialog systems \cite{lowe-etal-2017-towards,ghazarian-etal-2019-better,yuma-etal-2020-ubleu}. However, these new metrics require learning models to measure the scores. 
Therefore, we decided to use BLEU and perplexity as the means to compare the model performance. We also included \textbf{\textit{Distinct}} (\textit{dist-1} and \textit{dist-2}) scores to represent the diversity of the generated responses, calculated by averaging the number of n-grams by the total generated tokens \cite{li-EtAl:2016:P16-13}.

In addition to the automatic evaluation metrics, we provide a human evaluation of our model outputs. 
We hired human evaluators from the Amazon Mechanical Turk service to evaluate the quality of the generated responses. The evaluators were asked to evaluate their responses using the following criteria:

\begin{itemize}
\item \textbf{\textit{fluency}} or \textbf{\textit{naturalness}}: Whether the response could be produced by a (English speaking) human.
\item \textbf{\textit{relevance}} or \textbf{\textit{adequacy}}: Whether the response could be accepted as a suitable answer or contains useful information regarding the input utterance.
\item \textbf{\textit{style}}: Whether the response could be produced by the same person if some profile information was provided.
\end{itemize}
In addition to measuring the \textbf{\textit{style}}, we wanted to ascertain that the model could produce fluent sentences while maintaining relevance to the conversation context. Hence, the \textit{\textbf{fluency}} and \textbf{\textit{relevance}} criteria.

Because evaluating \textbf{\textit{style}} is supposed to be more difficult compared to other criteria, we conducted the evaluation in two stages: 1) assessing \textbf{\textit{fluency}} and \textbf{\textit{relevance}} simultaneously, and 2) evaluating \textbf{\textit{style}} similarity.
For these stages, we prepared a separate set to produce model-generated responses. We randomly selected 12 users from the conversation dataset and retrieved the tweets that they replied to as input utterances. We collected 100 tweets by retrieving five to ten tweets from each user. Each pair of input utterances and their responses were evaluated by ten judges.

In the first stage, we provided human evaluators with pairs of input utterances and the respective responses from each baseline model. Because we have four baseline models, each input utterance is paired with four response alternatives. We used a three-point Likert scale labeled {\textit\{bad, enough, good\}}, which was later converted to \{$-1$, 0, +1\}, and asked the evaluators to score every response alternative in terms of \textbf{\textit{fluency}} and \textbf{\textit{relevance}}.

For the second stage, because we wanted to investigate the effect of user-specific information, we only evaluated models that use user-specific feature(s), that is, we omitted vanilla Seq2Seq in this stage. 
We provided the evaluators with Twitter user bio, that is, a user's short biography or profile information that commonly contains keywords, and some sample tweets from the respective users.
We asked the evaluators to compare the style of the generated responses to the sample tweets of each user. We also required the evaluators to consider the user's bio when judging the responses. An example of the user information provided is shown in Figure \ref{fig:bio_sample}.
As we must see which model can generate responses with more resemblance to the user-specific style, the evaluators were required to rank the response alternatives from each model. We used five-score ranks with one as the most resembling style and five as the least resembling one; tie scores were permitted.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{28-4ia7f2.pdf}
\end{center}
\hangcaption{Example of a user's Twitter bio and sample tweets used in style evaluation. We censored any mention of other accounts.}
\label{fig:bio_sample}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

To evaluate and compare the generated responses from both our models and the fine-tuned DialoGPT, we recruited three volunteers, who evaluated the same 100 pairs of conversations.
We asked the evaluators to compare the two response alternatives from our model and DialogGPT, and decide which was preferable given the input utterance. We randomly assigned the two response alternatives as \textit{Response 1} or \textit{Response 2} to help the evaluators stay objective. We then asked the evaluators to choose from options \{Response 1, Response 2, Neither\}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results and Discussion}
\label{sec:results}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Automatic and Human Evaluation}

The summary of the evaluation is shown in Table \ref{tab:result_percent}.
In automatic evaluation, we find that our model $\mathrm{User + Info}$, which is fed with seen users and user-info embeddings, can achieve a higher diversity of the generated words, as shown by the dist-1 and dist-2 scores. It is also interesting that our model, which was fed with unseen users in the testing phase, $\mathrm{UNK + Info}$, achieved the highest BLEU score compared to other settings. We argue that a qualitative evaluation is required to help understand the models' performance.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[b]
\input{07table03.tex}
\hangcaption{%
Automatic (BLEU, perplexity, distinct) and Human evaluation results for \textit{fluency} and \textit{relevance}, presented as raw score percentages. Our $\mathrm{UNK+Info}$ model with unseen users gains 26.5\% more for fluency and 36.5\% more for relevance compared with baseline.}
\label{tab:result_percent}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

For human evaluation, we first evaluated all models in Table \ref{tab:baseline_models} in terms of \textit{fluency} and \textit{relevance}. For both criteria, the evaluators assigned a score from three choices: bad, enough, and good. To compare the models, we first counted the number of each score label for each model received, which we call the \textit{raw scores}. 
According to these scores, for both criteria, $\mathrm{UNK + Info}$ (our model with unseen users) received the highest \textit{good} score, followed by the $\mathrm{Seq2Seq}$ model. The UNK + Info group gained 26.5\% more fluency points compared to the baseline. To calculate this gain, we simply compared the percentage obtained by UNK + Info (67.3\%) against UserOnly (53.2\%).

To determine the overall quality of response generation, in terms of \textit{fluency} and \textit{relevance}, we also must include the \textit{ sufficient } scores from both criteria. We combined the \textit{enough} and \textit{good} scores as \textit{acceptable}. According to the \textit{acceptable} score, 
%%%%as shown in Table \ref{tab:result_acceptable}, 
    as shown in Table \ref{tab:result_acceptableX}a, 
the Seq2Seq model achieved the highest score in terms of \textit{fluency}, gaining 0.87\% more points than our UNK + Info model. As the gain is arguably small, we consider our model good enough in terms of \textit{fluency}.

In terms of \textit{relevance}, Table \ref{tab:result_percent} shows that all models received higher \textit{bad} scores than their \textit{good} scores. We argue that this occurrence implies that \textit{relevance} is more difficult to achieve than \textit{fluency}. However, our UNK + Info achieved the highest \textit{acceptable} score for this criterion, gaining 36.5\% more than the baseline.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[t]
\input{07table04.tex}
\hangcaption{%
Results of \textit{fluency} and \textit{relevance} criteria of four alternative models. From the table on the left (a), we can see that the vanilla Seq2Seq model achieved the highest acceptable response on \textit{fluency}. However, from the table on the right (b), on average, UNK + Info achieved the highest scores on both criteria.}
\label{tab:result_acceptableX}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We also calculated the average scores by converting the \{bad, enough, good\} scores to \{$-1$, 0, 1\} for each model; 
%%%%the results are shown in Table \ref{tab:result_avg}. 
    the results are shown in Table \ref{tab:result_acceptableX}b. 
According to these average scores, our UNK + Info tops in both the \textit{fluency} and \textit{relevance} criteria. Using one-way ANOVA as the significance test, we confirmed that our main model (User + Info) was significantly better than the baseline (UserOnly) in the relevance criteria.


In the second stage, we measured the style similarity between the generated responses and the corresponding user's sample tweets. As in the first stage, each of the 100 input-response pairs was evaluated by ten human evaluators, resulting in 1,000 samples, from which we removed some results that did not show consistency, for example, results with identical responses with different rank scores. As described in Section \ref{sec:evaluation_setup}, because our intent was to investigate the influence of user-specific information, we evaluated only three models in this stage. 
%%%%The results are presented in Table \ref{tab:result_style}.
    The results are presented in Table \ref{tab:result_styleX}a.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[p]
\input{07table05.tex}
\hangcaption{%
Table on the left (a) shows that UNK + Info achieved better style compared to the baselines with user-specific information. In the table on the right (b), we further compared the outputs from UNK + Info and DialoGPT in the preferable measure. We argue that the low agreement among the evaluators (0.2068 on Fleiss' Kappa) shows a similar quality between the two response alternatives. }
\label{tab:result_styleX}
\vspace*{1\Cvs}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[p]
\input{07table06.tex}
\hangcaption{Examples of responses from different users generated by our model, using known users and their user-info respectively, and its variant model, using unseen users and the same user-infos.}
\label{tab:response_example}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

As we asked the evaluators to assign a rank score on a scale of 1 to 5 (smaller number is better), 
%%%%according to Table \ref{tab:result_style}, 
    according to Table \ref{tab:result_styleX}a, 
the average scores were relatively positioned in the middle rank, that is, around rank three. This suggests that, in general, all models only generate ``good enough'' responses in terms of style. Nevertheless, both of our models achieved a higher rank compared to baseline, and the Friedman test confirmed that UNK + Info is significantly better than the baseline. Some examples generated by our models are listed in Table \ref{tab:response_example}.

Because we noticed that our UNK + Info model performed better than other models, we further compared the responses generated by UNK + Info with those generated by DialoGPT \cite{zhang-etal-2020-dialogpt}. 
Following the setup described in Section \ref{sec:evaluation_setup}, we recruited three volunteers who evaluated 100 pairs of conversations. The volunteers were asked to choose the preferred response from the two alternatives generated by our model and DialoGPT.
%%%%The results are shown in Table \ref{tab:dialogpt_compare}. 
    The results are shown in Table \ref{tab:result_styleX}b. 

Note that DialoGPT's outputs were selected more times at 40.67\% than our UNK + Info model at 35.67\%. Considering that DialoGPT's capacity and model size are larger than those of our model, we argue that our achievement is acceptable. Additionally, we measured the inter-annotator agreement among the evaluators by calculating the Fleiss' Kappa \cite{fleiss1971mns} of the results, resulting in a kappa score of 0.2068. This low score indicates a slight agreement among evaluators \cite{Landis77}. We argue that the slight agreement could indicate a similar quality between our model and DialoGPT, which makes the evaluation relatively difficult.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Discussion}

Our main intention is to incorporate the characteristics of each individual user into the machine-generated responses. We specifically attempted to incorporate more user-specific information to emphasize the response styles of different users. Therefore, we conducted experiments to incorporate additional user-specific information, and the evaluation we performed showed that the human evaluators recognized a better change in response style.

Furthermore, one aspect that distinguishes our model from others is the application of external resources in addition to user-specific features. Generally, if a conversation model is trained to detect user-specific or persona-specific traits, such features should be included in the training. However, our external resource (user-info dataset) can be used even without the corresponding users in the training. Our work can also serve as evidence of the usability of using external resources for a response generation model with a simple architecture.

An interesting finding is that the UNK + Info model, which is fed with unseen users, achieved better scores than our User + Info model on the style preference of human judges. Despite achieving lower scores on the distinct evaluation metric, which is considered to represent diversity, this UNK + Info model also achieved the highest BLEU score compared to the others. Although these scores could imply that BLEU correlates negatively with response diversity, we would need more experiments and analysis to achieve this conclusion. However, we argue that these scores imply that style preferability is not always aligned with response diversity.

Another point to analyze is that using both user-specific information, i.e., seen \textit{user embeddings} and \textit{user-info embeddings}, might overwhelm the response generation, resulting in overly ``stylized'' outputs. Through manual observation, we assume that a model with more injected information can be too focused on the styles and lose some relevance to the input utterance. However, the baseline, with less information, still received lower scores on automatic and human evaluations. We postulate that our architecture with co-attention to external information could generate a better response in terms of style without losing relevance or other expected quality.

We also evaluated the human preference between the generated responses from the UNK + Info model and DialoGPT.
Considering that DialoGPT is bigger than our model, it is not surprising that DialoGPT acquired a higher score in this evaluation. However, given the low agreement between the evaluators, we can assume that even for a large transformer-based model, the open-domain response generation task with real-world conversations is a difficult task. Therefore, we argue that the performance of our model is acceptable.

In conclusion, the problem persists in styling-generated responses. Regardless of the results being better than the baseline, generating fluent and relevant responses with an expected style is challenging. 
Nevertheless, we expect that with the popularity of the transformer-based model and other large pre-trained methodologies, incorporating a specific and intended style into the generated responses would be possible in the near future.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion and Future Work}
\label{sec:conclusion}

In this study, we conducted experiments to address the response diversity issue, particularly in response style.
We used user-specific information to drive the generated responses to resemble the real user's utterances. 
We considered usernames and user-info dataset as user-specific information.

The evaluation through human judgment showed that the outputs of our model were better than the baseline overall, especially the variant with unseen users. Although our model has a simple architecture and is small, it can produce responses with acceptable quality. We believe that our experiments can serve as evidence that, even with limited size, a simple and intuitive architecture can improve the response quality. 

The remaining challenge is how to properly incorporate and emphasize response style without damaging the context or its relevance. We believe that with recent advances in style transfer research and the advancement of deep learning model architecture in general, a good compromise between controlling style and preserving conversation context can be achieved soon.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

Part of this work was presented at the 11th International Natural Language Generation Conference from 5th to 8th of November, 2018 in Tilburg, The Netherlands \cite{fikri-etal-2018-stylistically}. The authors would like to thank the volunteers for their help in evaluating this work. This work was partially supported by JST PRESTO (Grant Number JPMJPR1655).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{07refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Abdurrisyad Fikri}{%
Abdurrisyad Fikri received B.Sc. in informatics from Bandung Institute of Technology in 2012. He worked as a Web Application Developer from 2013 to 2016 at Imani Prima, Indonesia. He received M. Eng. from Tokyo Institute of Technology in 2018. He is currently a Ph.D. student of Department of Information and Communications Engineering, Tokyo Institute of Technology. His current research interests are dialogue models and multi-modal natural language processing.
}

\bioauthor[:]{Hiroya Takamura}{%
Hiroya Takamura
received his Ph.D. from Nara Institute of Science and Technology.
He worked as a professor at Tokyo Institute of Technology, and currently is a research team leader at AI Research Center of Advanced Industrial Science and Technology.
His current research interests include natural language processing.
}

\bioauthor[:]{Manabu Okumura}{%
Manabu Okumura was born in 1962. He received B.E., M.E. and Dr.~Eng.\ from Tokyo Institute of Technology in 1984, 1986 and 1989 respectively. He was an assistant at the Department of Computer Science, Tokyo Institute of Technology from 1989 to 1992, and an associate professor at the School of Information Science, Japan Advance Institute of Science and Technology from 1992 to 2000. He is currently a professor at Institute of Innovative Research, Tokyo Institute of Technology. His current research interests include natural language processing, especially text summarization, computer assisted language learning, sentiment analysis, and text data mining.
}
\end{biography}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate




\end{document}

