\documentclass[english,other]{jnlp_3.6}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%\usepackage{newfloat}
%%%%\usepackage{listings}
%%%%\usepackage{color}
%%%%\usepackage{soul}
%%%%\usepackage{multirow}
%%%%\usepackage{makecell}
%%%%\renewcommand\UrlFont{\rmfamily}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{30}
\Number{1}
\Month{March}
\Year{2023}
%%%%\Issuetitle{Special Issue: XXXXXXXX}

\setcounter{page}{243}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{Society Column}
\etitle{StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning}
\eauthor{Hong Chen\affiref{1}\affiref{3} \and Duc Minh Vo\affiref{1} \and Hiroya Takamura\affiref{2}\affiref{3} \and \\
	Yusuke Miyao\affiref{1}\affiref{3} \and Hideki Nakayama\affiref{1}\affiref{3}} 

\headauthor{Chen et al.}
\headtitle{StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning}

\affilabel{1}{}{The University of Tokyo}
\affilabel{2}{}{Tokyo Institute of Technology}
\affilabel{3}{}{National Institute of Advanced Industrial Science and Technology, Japan}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Even for humans, evaluating story quality is a challenging task. Although many literature criteria have been proposed, the most straightforward way is to count how many readers like the story which is referred as to human preference.
Bearing it in mind, story writing community usually uses upvote count as a story quality criterion.
Existing methods which use 
\textit{referenced metrics} (e.g., BLEU~\cite{papineni2002bleu}, METEOR~\cite{banerjee2005meteor}, ROUGE~\cite{lin2004rouge}) and \textit{unreferenced metrics} (e.g., UNION~\cite{guan2020union}, MANPLTS~\cite{ghazarianplot}), deviate from human preference.
On the contrary, we aim to explicitly evaluate a story, introducing a human preference-liked system consisting of three subtasks: Ranking, Rating and Reasoning.
This study addresses a novel task StoryER, that consists of preference score prediction, aspect rating and comment generation.
We introduce a new dataset for StoryER task and create benchmarks to promote the story evaluation research. 
Comprehensive experiments and intensive analysis indicate our preference score prediction outperforms previous metrics, and more accurately reflects human preference. Aspect rating and comment generation also helps in the evaluation and provide explanations. Moreover, we point out the remaining challenges under various scenarios in the hope that facilitates future research.  


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{StoryER}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Task Definition} 

Given a story $\mathbf{s}$, the task is to output a set $\left \{ p_s, \mathbf{a}^{\rm c}, \mathbf{a}^{\rm r}, \mathbf{c} \right \}$ where $p_s$ denotes the preference score of the story $\mathbf{s}$, which is used for comparing story quality.
For more explicit explanation, we further output confidence scores
$\mathbf{a}^{\rm c} = \left \{ a^{\rm c}_{k}\right \}^{K}_{k=1}$, aspect ratings
$\mathbf{a}^{\rm r} = \left \{ a^{\rm r}_{k}\right \}^{K}_{k=1}$, and comments
$\mathbf{c} = \left \{ c_{k}\right \}^{K}_{k=1}$
 for $K$ aspects ($K=10$ in our experiments), respectively.
Confidence scores $\mathbf{a}^{\rm c}$ reflect the likelihood of utilizing the specific aspects as measures, as some aspects (e.g., horror) are not applicable in some stories (e.g., comic story).
Aspect ratings $a^{\rm r}_{k}$ are considered as the scores of each aspect.
Comments $\mathbf{c}$ demonstrate the reason that the reader upvotes/downvotes the story, producing a more explicit explanation for the aspect rating.
We assume $\sum_{k=1}^{K}a^{\rm c}_{k}=1$ for aspect confidence, and $a^{\rm r}_{k} \in [0, 1]$ for aspect rating, which is calculated separately during the training.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Learning a Story Evaluator}

Following~\cite{ghazarianplot}, we use Longformer-Encoder-Decoder (LED)~\cite{beltagylongformer} to produce a preference score, as well as ratings and comments for the pre-defined aspects. As shown in Figure~\ref{fig:model}, we encode the story $\mathbf{s}$, and use its feature on the special token (i.e., [CLS]) to predict the preference score $p_s$, aspect confidence $\mathbf{a}^{\rm c}$ and rating $\mathbf{a}^{\rm r}$ by additional layers. For generating comments, we concatenate the story with aspect category name with a special token (i.e., {\textless}sep{\textgreater}), and send it into the same encoder. The decoder outputs the comment $\mathbf{c}$ that implies the performance of the story on the given aspect. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[b]
\vspace{-0.5\Cvs}
\begin{center}
\includegraphics{30-1ia25f1.pdf}
\end{center}
\hangcaption{Overview of our model. The encoder (left) predicts the preference score, aspect confidence, and aspect rating. The decoder (right) generates the comment for each aspect.}
\label{fig:model}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Task 1: Preference Score Prediction (Ranking).}
Our model learns to predict the preference score by ranking two stories from the same prompt. 
As shown in Figure~\ref{fig:model}, we use the feature of [CLS] in the story, following a linear layer with sigmoid activation and finally turning it into a scalar score. 
We take Margin Ranking Loss to enlarge the margin gap $m$ of the scores between stories with high and low upvotes:
where $\mathbf{W}_{\rm p_s}$ denotes a linear layer for the feature of the story $\mathbf{v}_{s}$. $\sigma(\cdot)$ is the sigmoid activation function. $s_{high}$ and $s_{low}$ represent the highly-upvoted and lowly-upvoted stories.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Task 2: Aspect Confidence and Rating Prediction (Rating). }
We adopt two additional linear layers on the same feature $\mathbf{v}_s$ used in the story ranking. 
One is with learnable parameters $\mathbf{W}_{{\rm a}^{\rm c}}$, outputting confidence scores $\mathbf{a}^{\rm c} = \text{softmax}(\mathbf{W}_{\rm a^{\rm c}} \mathbf{v}_s)$.
We calculate the multi-class cross-entropy loss for the aspect confidence.
$\mathbf{y}_{\rm a^{\rm c}}[k] = 1$ if the $k$-th aspect is selected, otherwise $\mathbf{y}_{\rm a^{\rm c}}[k] = 0$.
For aspect rating, binary cross-entropy loss is calculated separately for each selected aspects.
$M_s$ denotes the set of aspects that are selected for story $\mathbf{s}$. 
$\mathbf{y}_{\rm a^{\rm r}}[k]$ denotes the normalized rating score for the $k$-th aspect. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Task 3: Comment Generation (Reasoning).}
The comments are generated conditioned on the aspect $\mathbf{a}$ and the story $\mathbf{s}$.
We input the concatenation of the aspect category name, special token, story, and train the LED under Maximum Likelihood Estimation (MLE) with the comment as target:
where the $\mathbf{c}_t$ denotes the $t$-th token in the comment.
To train the model, all of the above losses are added up.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[t]
\resizebox{\textwidth}{!}{%
\begin{tabular}{l|cccccc}
\hline
                      & \#comment & \#comment* & rate (1--5) & rate* (1--5) & comment\_len & comment\_len* \\
\hline
\textbf{Structure} &  &   &    & &  &        \\
\quad opening/beginning     & 3615         & 5617          & 2.53       & 3.15        & 30.20        & 32.44         \\
\quad middle/twist/flow/conflict     & 3967         & 5971          & 2.24       & 2.78        & 30.59        & 31.63         \\
\quad ending                & 5610         & 7615          & 2.13       & 2.49        & 31.48        & 31.59   \\
\hline
\textbf{Writing Style} &  &   &    & &  &        \\
\quad character shaping     & 5101         & 7102          & 2.21       & 2.53        & 31.57        & 34.23         \\
\quad scene description     & 4168         & 6172          & 2.18       & 2.53        & 31.75        & 39.30         \\
\hline
\textbf{Type} &  &   &    & &  &        \\
\quad heartwarming/touching (Romance)    & 426          & 1866          & 2.99       & 4.39        & 32.05        & 32.64         \\
\quad sad/crying/tragedy (Tragedy)       & 462          & 1680          & 3.12       & 3.93        & 30.85        & 34.67         \\
\quad horror/scary (Horror)          & 815          & 1985          & 2.49       & 3.61        & 30.92        & 33.24         \\
\quad funny/hilarious/laugh (Comedy) & 1153         & 3156          & 3.25       & 3.96        & 30.04        & 30.91         \\
\quad novelty/good idea/brilliant (Fiction)            & 2782         & 4784          & 2.49       & 3.26        & 32.51        & 32.70         \\\hline
\bf{Overall}               & 28099        & 45948         & 2.56       & 3.26        & 31.20        & 33.33   \\\hline     
\end{tabular}
}%
\hangcaption{Data statistics in 46k aspect rating and reasoning data. * denotes the data statistics after data augmentation. We list the number of comments with rating scores (2nd and 3rd columns), averaged rating scores (4th and 5th columns) and averaged word count (6th and 7th columns).}
\label{tab:comment_statistics}
\end{table}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Dataset}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{100k Story Ranking Data.}
We first collect 193,842 stories prior to 03/2020 from WP\footnote{\url{https://huggingface.co/datasets/rewardsignal/reddit_writing_prompts}} along with their prompt, the number of upvotes and uncategorized comments.
We remove the stories updated from 12/2019 to 03/2020, since newly-updated stories usually have few upvotes regardless of whether they are good or bad.
Then, we exclusively keep stories with the word count between 200 and 800.
Finally, we pick two stories from the same prompt, one highly upvoted (i.e., upvotes $\geq$ 50~\footnote{We notice that some stories that receive upvotes $\geq$ 50 can be listed in \url{/r/bestofWritingPrompts/}}) and one lowly upvoted (i.e., upvotes $\leq$ 0), resulting in a total of 63,929 unique stories and 116,971 story pairs.
We split the story pairs based on the prompts into training, validation and testing, to ensure that each division receives a unique set of prompts. 
Table 1 shows the data statistics of our dataset.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{30-1ia25f2.pdf}
\end{center}
\hangcaption{The Writing Prompt Dataset with metadata (left) contains prompt, story, upvotes, and comments from readers. Our dataset collection pipeline (right) shows the template for data collection. We ask the workers to select 3--5 aspects, score each aspect 1--5 from poor to good and leave the comments that shows the reason for the score they rated.}
\label{fig:dataset}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{46k Aspect Rating and Reasoning Data.} \label{sec:46k}
Apart from the preference score, we require our model to provide ratings and comments on pre-defined aspects to aid in the explanation of the predicted preference score. The dataset collection template is shown in Figure \ref{fig:dataset}. We show the upvote counts to the workers, and ask them to find positive reasons for highly-upvoted stories and negative reasons for lowly-upvoted stories.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results}

We evaluate the predicted preference scores obtained by all compared methods on 100k Story Ranking test data. 
\textbf{Pairwise Ranking Accuracy (Acc)} is calculated as the percentage of the story with higher upvotes getting a higher score than the one with lower upvotes.
We also compute the \textbf{averaged score gap (Dis)} between two stories in pairs.
Table~\ref{tab:main_result} (Human (Ranking)) indicates that existing methods on preference-aware story evaluation on human-written stories are close to random selection (i.e., $\text{Acc}=0.5$, $\text{Dis}=0$).
In contrast, our method can successfully compare two stories and achieve an acceptable score gap between two stories.

We also calculate the correlation between our predicted preference scores and human judgment for stories. 
We use the correlation metrics Spearman ($\rho$)~\cite{zar1972significance} and Kendall
($\tau$)~\cite{schaeffer1956concerning}, which are known to be beneficial in estimating monotonic associations for not normally distributed and ranked scores.
We collect and annotate both human-written, i.e., \textbf{WP$_{200}$} (stories from WritingPrompts), \textbf{SCARY$_{200}$} (stories from scary stories) and machine-generated stories, i.e., \textbf{PREF$_{200}$} (stories from LED generated stories), \textbf{COH$_{200}$} (human manipulated stories) as our test data.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\resizebox{\textwidth}{!}{%
\begin{tabular}{l|cc|cc|cc|cc|cc}
\hline
 &   \multicolumn{6}{c|}{\textbf{Human Written Story}} & \multicolumn{4}{c}{\textbf{Machine Generated Story}} \\
\cline{2-11}
Methods & \multicolumn{2}{c|}{Ranking} &
  \multicolumn{2}{c|}{WP$_{200}$} &
  \multicolumn{2}{c|}{SCARY$_{200}$} &
  \multicolumn{2}{c|}{PREF$_{200}$} &
  \multicolumn{2}{c}{COH$_{200}$} \\
 & Acc(\%) & Dis    & $\rho$ & $\tau$ & $\rho$ & $\tau$ & $\rho$          & $\tau$          & $\rho$ & $\tau$ \\ \hline
PPL        & 61.80   & ---      & 0.181* & 0.130* & 0.409* & 0.300*& $-0.090$ & $-0.067$  & $-0.698$*         & $-0.452$*         \\
ft-PPL     & 61.85   & ---    & 0.168  & 0.120  & 0.449* & 0.328*& $-0.098$ & $-0.074$  & $-0.690$*         & $-0.443$*         \\
Ruber-bert & 39.16   & $-0.032$ & $-0.019$ & $-0.014$ & 0.032  & 0.026& $-0.140$ & $-0.100$  & $-0.141$          & $-0.092$           \\
UNION      & 48.74   & 0.003  & 0.001  & 0.001  & 0.156* & 0.114 & $-0.068$ & $-0.045$  & 0.188*          & 0.132*           \\
MANPLTS    & 53.08   & 0.016  & 0.124  & 0.107  & $-0.070$ & $-0.061$ &   0.164  & 0.130      &   0.729*          & 0.498*  \\
Ours &
  \textbf{73.93} &
  \textbf{0.228} &
  \textbf{0.583*} &
  \textbf{0.422*} &
  \textbf{0.578*} &
  \textbf{0.420*} &
   \textbf{0.343*} &
  \textbf{0.234*}&
  0.194* &
  0.132* 
 \\ \hline
\end{tabular}
}%
\hangcaption{Evaluation on preference score prediction. Compared with previous works, our predict scores more correctly match the human judgement. We conduct hypothesis test, and * denotes that $p \leq 0.01$. }
\label{tab:main_result}
\vspace{-1\Cvs}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Table~\ref{tab:main_result} depicts the correlation between human and automatic evaluation metrics on preference (WP$_{200}$, SCARY$_{200}$ and PREF$_{200}$).
We see that our method outperforms previous methods by a large margin on both human-written and machine-generated stories in terms of human preference. 
Not surprisingly, in Table~\ref{tab:main_result} (COH$_{200}$), MANPLTS on story coherence evaluation is against our model, as coherence issue does not frequently happen in our training data (i.e., human-written stories). 
We notice that preference-aware judgments (PREF$_{200}$) and coherence-based judgments (COH$_{200}$) are distinct. Metrics that perform well in terms of coherence may perform poorly in terms of preference, and vice versa.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion} 

With the development of story generation, we believe that preference-aware story evaluation will be the mainstream research when machine-generated stories do not suffer from serious coherence problems. Further studies on our dataset can also be conducted to reveal the point that influence the readers to upvote the stories.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

This paper is supported by Institute of AI and Beyond of the University of Tokyo, JSPS KAKENHI Grant Numbers JP19H04166, JP22H00540, 22K17947 and also based on results obtained from a project JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
For experiments, computational resources of AI Bridging Cloud Infrastructure (ABCI) provided by National Institute of Advanced Industrial Science and Technology (AIST) was used.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.7}
\bibliography{25refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Hong Chen}{PhD, The University of Tokyo. Research Intern, AIRC.}

\bioauthor[:]{Duc Minh Vo}{PostDoc, the University of Tokyo.}

\bioauthor[:]{Hiroya Takamura}{Professor, Tokyo Institute of Technology. Team leader, AIRC.}

\bioauthor[:]{Yusuke Miyao}{Professor, The University of Tokyo.}

\bioauthor[:]{Hideki Nakayama}{Professor, The University of Tokyo.}

\end{biography}


\end{document}


