\documentclass[english,tombow]{jnlp_3.2}
\usepackage{jnlpbbl_1.4}
\usepackage[dvipdfm]{graphicx,xcolor}
\usepackage{gentombow}
\usepackage{amsmath}
\settombowbleed{5mm}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
%%%%\makeatletter
%%%%\def\@{\spacefactor3000\space}
%%%%\makeatother

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%\usepackage{times}
%%%%\usepackage{amsmath}
\usepackage{amssymb}
%%%%\usepackage{booktabs}
%%%%\usepackage{subcaption}
\usepackage{multirow}
\usepackage{tikz}
%%%%\usepackage{xcolor}
\usepackage{url}
\usepackage[linesnumbered,ruled]{algorithm2e}
%%%%\newcommand\BibTeX{B\textsc{ib}\TeX}
%%%%\renewcommand{\arraystretch}{1.2}
%%%%\newcommand{\red}[1]{{\color[HTML]{d62728}\textbf{#1}}}
%%%%\newcommand{\orange}[1]{{\color[HTML]{ff7f0e}\textbf{#1}}}
    \newcommand{\orange}[1]{{\color[cmyk]{0,0.6,0.9,0}\textbf{#1}}}
%%%%\newcommand{\blue}[1]{{\color[HTML]{1f77b4}\textbf{#1}}}
    \newcommand{\blue}[1]{{\color[cmyk]{1,0.5,0,0}\textbf{#1}}}
%%%%\newcommand{\green}[1]{{\color[HTML]{2ca02c}\textbf{#1}}}
    \newcommand{\green}[1]{{\color[cmyk]{1,0,1,0}\textbf{#1}}}
%%%%\newcommand{\gray}[1]{{\color[HTML]{808080}\textbf{#1}}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\renewcommand\UrlFont{\rmfamily}
\renewcommand{\refname}{References}
\allowdisplaybreaks
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\Volume{27}
\Number{3}
\Month{September}
\Year{2020}
%%%%\Issuetitle{Special Issue: XXXXXXXX}

\received{2020}{2}{1}
\revised{2020}{5}{1}
%%%%\rerevised{2006}{5}{6}
\accepted{2020}{6}{17}

\setcounter{page}{599}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Learning to Select, Track, and Generate for Data-to-Text}
\eauthor{Hayate Iso\affiref{naist} \and Yui Uehara\affiref{aist} \and Tatsuya Ishigaki\affiref{titech}\affiref{aist} \and Hiroshi Noji\affiref{aist} \and Eiji Aramaki\affiref{naist}\affiref{aist} \and\\ Ichiro Kobayashi\affiref{ocha}\affiref{aist} \and Yusuke Miyao\affiref{ut}\affiref{aist} \and Naoaki Okazaki\affiref{titech}\affiref{aist} \and \\
	Hiroya Takamura\affiref{titech}\affiref{aist}} 
\eabstract{%
We propose a data-to-text generation model with two modules, one for tracking and the other for text generation. Our tracking module selects and keeps track of salient information and remembers which record has been mentioned. Our generation module generates a summary conditioned on the state of tracking module. In addition, we also explore the effectiveness of the writer information for generation. Experimental results show that our model outperforms existing models in all evaluation metrics even without writer information. Incorporating writer information further improves the performance, contributing to content planning and surface realization.
}
\ekeywords{Natural Language Generation, Data-to-Text Generation, Entity Tracking}

\headauthor{Iso et al.}
\headtitle{Learning to Select, Track, and Generate for Data-to-Text}

\affilabel{naist}{}{Nara Institute of Science and Technology}
\affilabel{aist}{}{Artificial Intelligence Research Center, AIST}
\affilabel{titech}{}{Tokyo Institute of Technology}
\affilabel{ocha}{}{Ochanomizu University}
\affilabel{ut}{}{The University of Tokyo}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Advances in sensor and data storage technologies have rapidly increased the amount of data produced in various fields such as weather, finance, and sports.
To address the information overload caused by the massive amounts of data, data-to-text generation technology, which expresses the contents of data in natural language, is becoming more important \cite{barzilay2005collective}.
Recently, neural methods can generate high-quality short summaries especially from small pieces of data \cite{liu2018table}. 

Despite this success, it remains challenging to generate a high-quality long summary from data \cite{wiseman2017challenges}.
One of the difficulties is that the input data is too large for a naive model to find its salient part, i.e., to determine which part of the data should be mentioned. 
In addition, the salient part moves as the summary explains the data.
For example, when generating a summary of a basketball game (Table~\ref{tab:example} (b)) from the box score (Table~\ref{tab:example} (a)), 
the input contains numerous data records about the game: e.g., \textit{Jordan Clarkson scored 18 points}.
Existing models often refer to the same data record multiple times \cite{puduppully2019data}. However, existing models may mention an incorrect data record, e.g., \textit{Kawhi Leonard added 19 points}: the summary should mention \textit{LaMarcus Aldridge}, who scored 19 points.
Thus, we need a model that finds salient parts, tracks transitions of salient parts,
and expresses information faithful to the input.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[t]
\hangcaption{Example of input and output data: task defines box score (1a) used for input and summary document of  game (1b) used as output}
\label{tab:example}
\input{05table01.tex}
\par\vspace{4pt}\small
Extracted entities are shown in bold face. Extracted values are shown in green.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In this paper, we propose a novel data-to-text generation model with two modules, one for saliency tracking and another for text generation.
The tracking module keeps track of saliency in the input data so as when the module detects a saliency transition, the tracking module selects a new data record\footnote{We use `data record' and `relation' interchangeably.} and updates the state of the tracking module.
The text generation module generates a document conditioned on the current tracking state.
The model proposed here could be considered to be a neural network-based variant of the end-to-end data-to-text generation model that selects the input information and keeps decision histories while generating the report \cite{angeli-etal-2010-simple}.
The major difference from the traditional model is to make use of the recurrent neural network.
Our model employs the recurrent neural network-based saliency tracking module for maintaining the decision histories as continuous vectors instead of explicitly keeping all decision history.
In addition, some writer-specific patterns and characteristics i.e., how data records are selected to be mentioned; and how data records are expressed as text, e.g., the order of data records and the word usages. In addition. writer information was added to the model.

The experimental results demonstrate that, even without writer information, this model achieves the best performance among all previous models across all evaluation metrics with a 91.98\% precision of relation generation, 43.31 F1 score of content selection, 21.56\% normalized Damerau-Levenshtein Distance (DLD) of content ordering, and 15.74 of \textsc{BLEU} score. It was also confirmed that adding writer information further improves the performance.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Data-to-Text Generation}

Data-to-text generation is a task that generates descriptions from structured or non-structured data including sports commentary \cite{tanaka1998reactive,chen2008learning,taniguchi2019generating}, weather forecasts \cite{liang2009learning,mei2016talk}, biographical text from the ``infobox'' in Wikipedia \cite{lebret2016neural,sha2018order,liu2018table} and market comments on stock prices \cite{murakami2017learning,aoki2018generating}.

Neural generation methods have become the mainstream approach for data-to-text generation.
The encoder-decoder framework \cite{cho2014learning,sutskever2014sequence} with the attention \cite{bahdanau2015neural,luong2015effective} and copy mechanism \cite{gu2016incorporating,gulcehre2016pointing} has been successfully applied to data-to-text tasks.
However, neural generation methods sometimes yield fluent but inadequate descriptions \cite{tu2017context}.
In data-to-text generation, descriptions inconsistent with the input data are highly problematic.

Recently, \citeA{wiseman2017challenges} introduced the \textsc{RotoWire} dataset, which contains multi-sentence summaries of basketball games with box-score (Table~\ref{tab:example}). This dataset requires the selection of a salient subset of data records for generating descriptions.
They also proposed automatic evaluation metrics for
measuring the informativeness of generated summaries.
Their dataset and metrics have also been used by other researchers \cite{nie2018operation,li2018point,puduppully2019data,puduppully2019entity,rebuffel2020hierarchical}.

\citeA{puduppully2019data} proposed a two-stage method 
that first predicts the sequence of data records to be mentioned
and then generates a summary conditioned on the
predicted sequences. 
Their idea is similar to the one proposed here in that the both consider a sequence of data records as content planning.
However, the model proposed here differs from theirs in this one uses a recurrent neural network for saliency tracking, and that the decoder dynamically chooses a data record to be mentioned without fixing a sequence of data records.

\citeA{puduppully2019entity} simultaneously proposed an entity-centric method for data-to-text generation tasks, which keeps track of each entity's state at every time step.
The advantage of this approach is that the model does not require a random variable annotation.
In contrast to their approach, our model has intermediate discrete actions that should make it easier to interpret the generation behavior of the model.
In addition, our model shows empirically better performance on both \textsc{RotoWire-Modified} and \textsc{RotoWire-FG}, especially for the content selection performance.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Memory Modules}

The memory network can be used to maintain and update representations of salient information \cite{weston2015memory,sukhbaatar2015end,graves2016hybrid,henaff2017tracking}.
This module is often used in natural language understanding to keep track of the entity state \cite{kobayashi2016dynamic,hoang2018entity,bosselut2018simulating}.

Recently, entity tracking has become popular when trying to generate coherent text \cite{kiddon2016globally,ji2017dynamic,yang2017reference,clark2018neural}.
\citeA{kiddon2016globally} proposed a neural checklist model that updates predefined item states.
Furthermore, \citeA{ji2017dynamic} proposed an entity representation for the language model that updated entity tracking states when the entity was introduced. Notably, their method selects the salient entity state whreas our model extends this entity tracking module for data-to-text generation tasks.
The entity tracking module selects the salient entity and appropriate attribute in each timestep, updates their states, and generates coherent summaries from the selected data record.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Data}
\label{section:data}

Through careful examination, it was found that in the original dataset \textsc{RotoWire}, some NBA games had two documents, one of which was sometimes in the training data and the other was in the test or validation data. Such documents are similar to each other, though not identical. To make this dataset more reliable as an experimental dataset, a new version was created.

The script provided by \citeA{wiseman2017challenges} was run, which was used for crawling the \textsc{RotoWire} website for NBA game summaries. The script collected approximately 78\% of the documents in the original dataset; the remaining documents disappeared. The box-scores associated with the collected documents were also collected. It was observed that some of the box-scores were modified compared to the original \textsc{RotoWire} dataset.

The dataset contains 3,738 instances (i.e., pairs of a document and box-scores). However, the four shortest documents were not summaries; they were, for example, an announcement about the postponement of a match. Thus 4 instances like these were deleted and the set was left with 3,734 instances.
We followed the dataset split by \citeA{wiseman2017challenges} to split our dataset into training, development, and test data. 
Finally, the sizes of our training, development, test dataset are respectively  2705, 532, and 497. On average, each summary has 384 tokens and 644 data records.
Each match had only one summary in the dataset. Information on the writer fof each document was also collected and the dataset contained 31 different writers.
The most prolific writer in the dataset wrote 607 documents. There were also writers who wrote less than ten documents; on average, each writer wrote 117 documents.
The statistics of the number of relations and the length of summary for each writer were also calculated. As shown in Figure~\ref{fig:box}, each writer had a remarkable tendency for the number of retrieved relations and the length of the summary.
This new dataset is called \textsc{RotoWire-Modified}.\footnote{For information about the dataset, please follow this link: \url{https://github.com/aistairc/rotowire-modified}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[t]
\begin{center}
\includegraphics{27-3ia5f1.eps}
\end{center}
\hangcaption{Bar chart for the number of summaries for each writer, and boxplots for the number of relations and the length of summary for each writer. Horizontal axes are sorted by the median number of relations retrieved by the information extractor.}
\label{fig:box}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Saliency-Aware Text Generation}

At the core of this model is a neural language model with a hidden state $\boldsymbol{h}^{\textsc{LM}} \in \mathbb{R}^h$ to generate a summary $y_{1:T} = (y_1, \dots, y_T)$ given a set of data records $\boldsymbol{x}$, where $h$ is the size of hidden states.
This model has another hidden state $\boldsymbol{h}^{\textsc{Ent}} \in \mathbb{R}^h$, which is used to remember the data records that have been referred to.
$\boldsymbol{h}^{\textsc{Ent}}$ is also used to update $\boldsymbol{h}^{\textsc{LM}}$, meaning that the referred data records affect the text generation. 

Our model decides whether to refer to $\boldsymbol{x}$, which data record $r\in \boldsymbol{x}$ to be mentioned, and how to express a number. The selected data record is used to update $\boldsymbol{h}^{\textsc{Ent}}$.
%
Formally, we use the four variables:
\begin{enumerate}
    \item $ Z_t $: binary variable that determines whether the model refers to input $\boldsymbol{x}$ at time step $t$ ($ Z_t = 1 $).
    \item $ E_t $: At each time step $t$, this variable indicates the salient entity (e.g., \textsc{Hawks}, \textsc{LeBron James}).
    \item $ A_t $: At each time step $t$, this variable indicates the salient attribute to be mentioned (e.g., \textsc{Pts}).
    \item $ N_t $: If attribute $A_t$ of the salient entity $E_t$ is a numeric attribute, this variable determines if a value in the data records should be 
    output in Arabic numerals (e.g., 50)~($N_t = 1$) or in English words (e.g., five)~($N_t = 0$).
\end{enumerate}

To keep track of the salient entity, our model predicts these random variables at each time step $t$ through its summary generation process. A running example of our model is shown in Table~\ref{tab:annotate} and full algorithm is described in Algorithm \ref{alg}.
For a concise description, we omit the condition for each probability notation. $<$\textsc{SoD}$>$ and $<$\textsc{EoD}$>$ represent ``start of the document'' and ``end of the document,'' respectively.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\caption{A running example of our model's generation process}
\label{tab:annotate}
\input{05table02.tex}
\vspace{4pt}\small
At every time step $t$, the model predicts each random variable. The model firstly determines whether to refer to data records ($Z_{t} = 1$) or not ($Z_{t} = 0$). If random variable $Z_t = 1$, model selects entity $E_t$, its attribute $A_t$ and binary variables $N_t$ if needed. For example, at $t = 202$, the model predicts random variable $Z_{202} = 1$ and then selects the entity \textsc{\textbf{Jabari Parker}} and its attribute \textsc{\textbf{Player Pts}}. Given these values, the model outputs token $\mathbf{15}$ from selected data record.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


In the following subsections, we explain how to initialize the model, predict these random variables, and generate a summary. The illustration of our proposed model is shown in Figure~\ref{fig:model}.

Before explaining our method, we describe our notation. Let $\mathcal{E}$ and $\mathcal{A}$ denote the sets of entities and attributes, respectively.
Each record $r \in \boldsymbol{x}$ consists of entity $e\in\mathcal{E}$, attribute $a\in\mathcal{A}$, and its value $\boldsymbol{x}[e, a]$, and is therefore represented as $r = (e, a, \boldsymbol{x}[e, a])$.
For example, the box-score in Table~\ref{tab:example} has a record $r$ such that $e = \textsc{Anthony Davis}, a = \textsc{Pts},$ and $\boldsymbol{x}[e, a] = 20$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% algo. 1
\begin{algorithm}[t]
\scriptsize
\caption{Generation process}
\label{alg}
\input{05algo01.tex}
\end{algorithm}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Initialization}
\label{sec:init}

Let $\boldsymbol{r}\in \mathbb{R}^{d}$ denote the embedding of data record $r \in \boldsymbol{x}$, where $d$ is the dimension of embeddings.
Let $\bar{\boldsymbol{e}}$ denote the embedding of entity $e$. Note that $\bar{\boldsymbol{e}}$ depends on the set of data records, i.e., it depends on the game.
We also use $\boldsymbol{e}$ for static embedding of entity $e$, which, on the other hand, does not depend on the game.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{27-3ia5f2.eps}
\end{center}
\hangcaption{Illustrations of generation process. It consists of hidden states of language modeling, entity tracking, and dynamic entity representations as external memory.}
\label{fig:model}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Given the embedding of entity $\boldsymbol{e}$, attribute $\boldsymbol{a}$, and its value $\boldsymbol{v}$, we use the concatenation layer to combine the information from these vectors to produce the embedding of each data record $(e,a,v)$, denoted as $\boldsymbol{r}_{e,a,v}$ as follows:
\begin{equation}
    \boldsymbol{r}_{e,a,v} = \tanh\left(\boldsymbol{e} + \boldsymbol{a} + \boldsymbol{v} + \boldsymbol{b}^{\textsc{R}}\right),
    \label{init}
\end{equation}
where $\boldsymbol{b}^{\textsc{R}} \in \mathbb{R}^d$ denotes a bias vector.\footnote{We also concatenate the embedding vectors that represents whether the entity is in home or away team.}

We obtain a game-dependent entity embedding $\bar{\boldsymbol{e}}$ in the set of data records $\boldsymbol{x}$,
by summing all the data-record embeddings transformed by a matrix:
\pagebreak
\begin{equation}
    \bar{\boldsymbol{e}} = \tanh\left(\sum_{a\in \mathcal{A}} \boldsymbol{W}^{\textsc{A}}_{a}\boldsymbol{r}_{e, a, \boldsymbol{x}[e,a]} + \boldsymbol{b}_a^A\right),
\end{equation}
where $\boldsymbol{W}^{\textsc{A}}_{a} \in \mathbb{R}^{d\times d}$ is a weight matrix, and $\boldsymbol{b}^{\textsc{A}}_{a}\in \mathbb{R}^d$ is a bias vector for attribute $a$, respectively.
Since $\bar{\boldsymbol{e}}$ depends on the game as above, $\bar{\boldsymbol{e}}$ is supposed to represent how entity $e$ played in the game.

To initialize the hidden state of each module, we use embeddings of $<$\textsc{SoD}$>$ for $\boldsymbol{h}^{\textsc{LM}}$ and averaged embeddings of $\bar{\boldsymbol{e}}$ for $\boldsymbol{h}^{\textsc{ENT}}$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Saliency Transition}
\label{subsec:saliency}

Generally, the saliency of text changes during text generation.
In our work, we suppose that the saliency is represented as the entity and its attribute being talked about. We therefore propose a model that refers to a data record at each timepoint, and transits to another as text goes.

To determine whether to transit to another data record or not at time $t$, the model calculates the following probability:
\begin{equation}
    p(Z_t = 1 \mid \boldsymbol{h}_{t-1}^{\textsc{LM}}, \boldsymbol{h}_{t-1}^{\textsc{Ent}}) = \sigma(\boldsymbol{w}_{z}^{\top}(\boldsymbol{h}_{t-1}^{\textsc{LM}} \oplus \boldsymbol{h}_{t-1}^{\textsc{Ent}}) + b_{z}),
\end{equation}
where $\boldsymbol{w}_z \in \mathbb{R}^{2d}$ is a weight vector, $\oplus$ indicates the concatenation of vectors, and $\sigma (\cdot) $ is the sigmoid function.
If $p(Z_t = 1 \mid \boldsymbol{h}_{t-1}^{\textsc{LM}}, \boldsymbol{h}_{t-1}^{\textsc{Ent}})$ is high, the model transits to another data record.

When the model decides to transit to another, the model then determines which entity and attribute to refer to, and generates the next word (Section~\ref{sec:ent}).
On the other hand, if the model decides not to transit to another, the model generates the next word without updating the tracking states $\boldsymbol{h}^{\textsc{Ent}}_t = \boldsymbol{h}^{\textsc{Ent}}_{t-1}$ (Section~\ref{sec:out}).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Selection and Tracking}
\label{sec:ent}

When the model refers to a data record ($Z_t = 1$), it selects an entity and its attribute. It also tracks the saliency by putting the information about the selected entity and attribute into the memory vector $\boldsymbol{h}^{\textsc{Ent}}$.
The model begins to select the subject entity and update the memory states if the subject entity will change. 

Specifically, the model first calculates the probability of selecting an entity  $e_t$ that is referred at time step $t$:
\begin{align}
    &p(E_t = e \mid  \boldsymbol{h}_{t-1}^{\textsc{LM}}, \boldsymbol{h}_{t-1}^{\textsc{Ent}})\nonumber\\
    \propto
    &\begin{cases}
        \exp\left({\boldsymbol{h}^{\textsc{Ent}\top}_s\boldsymbol{W}^{\textsc{Old}}\boldsymbol{h}_{t-1}^{\textsc{LM}}}\right) & \text{if } e \in \mathcal{E}_{t-1}\\
        \exp\left({\bar{\boldsymbol{e}}^{\top}\boldsymbol{W}^{\textsc{New}}\boldsymbol{h}_{t-1}^{\textsc{LM}}}\right) & \text{otherwise}
    \end{cases},
\end{align}
where $\boldsymbol{W}^{\textsc{Old}} \in \mathbb{R}^{h\times h}$ and $\boldsymbol{W}^{\textsc{new}} \in \mathbb{R}^{d\times h}$ are weight matrices,
$\mathcal{E}_{t-1}$ is the set of entities that have already been referred to by time step $t$,
and $s$ is defined as $ s = {\max \{s: s \le t - 1, e = e_s \}} $, which indicates the time step when this entity was last mentioned.

The model selects the most probable entity as the next salient entity and updates the set of entities that appeared ($\mathcal{E}_t = \mathcal{E}_{t - 1} \cup \{e_t\}$).

If the salient entity changes $(e_t \not= e_{t - 1})$, the model updates the hidden state of the tracking model $\boldsymbol{h}^{\textsc{Ent}}$ with a recurrent neural network with a gated recurrent unit~(\textsc{Gru}) \cite{chung2014empirical}:
\begin{equation}
     \boldsymbol{h}_{t}^{\textsc{Ent}'} 
    = 
    \begin{cases}
    \boldsymbol{h}_{t-1}^{\textsc{Ent}} \hspace{2.8cm}\text{ if } e_t = e_{t-1}\\
    \textsc{Gru}^{\textsc{E}}(\bar{\boldsymbol{e}}, \boldsymbol{h}_{t-1}^{\textsc{Ent}}) \hspace{0.4cm}  \text{ else if } e_t \not \in \mathcal{E}_{t-1}\\
    \textsc{Gru}^{\textsc{E}}(\boldsymbol{W}^\textsc{S}\boldsymbol{h}^{\textsc{Ent}} + \boldsymbol{b}^{\textsc{S}}, \boldsymbol{h}_{t-1}^{\textsc{Ent}}) \hspace{0.2cm} \text{  otherwise,}
    \end{cases}
\label{eqn:entity_tracker_update}
\end{equation}
where $\boldsymbol{h}_t^{\textsc{Ent}'}$ is the intermediate entity tracker representation, $\boldsymbol{W}^{\textsc{S}} \in \mathbb{R}^{d\times h}$ is a weight matrix, and $\boldsymbol{b}^{\textsc{S}}\in\mathbb{R}^{d}$ is a bias vector.
Note that if the selected entity at time step $t$, $e_t$, is identical to the previously selected entity $e_{t-1}$, the hidden state of the tracking model is not updated.
If the selected entity $e_t$ is new ($e_t \not \in \mathcal{E}_{t-1}$), the hidden state of the tracking model is updated with the embedding $\bar{\boldsymbol{e}}$ of entity $e_t$ as input.
In contrast, if entity $e_t$ has already appeared in the past ($e_t \in \mathcal{E}_{t-1}$) but is not identical to the previous one $(e_t \not= e_{t-1})$, we use  $\boldsymbol{h}_s^{\textsc{Ent}}$ (i.e., the memory state when this entity last appeared) 
to fully exploit the local history of this entity.
In other words, the entity tracker contains the {\it memory states} of all the selected entities, instead of having explicitly separate memory modules for the entities as in the model proposed by \citeA{puduppully2019entity}.


Given the updated hidden state of the tracking model $\boldsymbol{h}_{t}^{\textsc{Ent}'}$, we next select the attribute of the salient entity at time $t$, $a_t$, by the following probability:
\begin{gather}
    p(A_t = a \mid e_t,\boldsymbol{h}_{t-1}^{\textsc{LM}}, \boldsymbol{h}_{t}^{\textsc{Ent}'}) \\
    \propto \exp\left(\boldsymbol{r}_{e_t, a, \boldsymbol{x}[e_t,a]}^{\top}\boldsymbol{W}^{\textsc{Attr}}  (\boldsymbol{h}_{t-1}^{\textsc{LM}} \oplus \boldsymbol{h}_{t}^{\textsc{Ent}'})\right),\nonumber
\end{gather}
where $\boldsymbol{W}^{\textsc{Attr}} \in \mathbb{R}^{d\times 2h}$ is a weight matrix.
After selecting $a_t$, i.e., the most probable attribute of the salient entity, the tracking model updates the memory state $\boldsymbol{h}_{t}^{\textsc{Ent}}$ with the embedding of the data record $\boldsymbol{r}_{e_t, a_t, \boldsymbol{x}[e_t, a_t]}$ introduced in Section~\ref{sec:init}:
\begin{equation}
    \boldsymbol{h}_{t}^{\textsc{Ent}} = \textsc{Gru}^{\textsc{A}}(\boldsymbol{r}_{e_t, a_t, \boldsymbol{x}[e_t,a_t]}, \boldsymbol{h}_{t}^{\textsc{Ent}'}).
\end{equation}
Note that we used different \textsc{GRU} units ($\textsc{GRU}^{\textsc{E}}$ and $\textsc{GRU}^{\textsc{A}}$) for updating the tracking model because they accept the different type of embeddings for the input.

Once the entity is referred to the model, we use the hidden states of lastly mentioned time step for each entity as the hidden states of the entity.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Summary Generation}
\label{sec:out}

Given the two hidden states, one for language model $\boldsymbol{h}_{t-1}^{\textsc{LM}}$ and the other for tracking model $\boldsymbol{h}_{t}^{\textsc{Ent}}$, the model generates the next word $y_t$.
We also incorporate a copy mechanism that copies the value of the salient data record $\boldsymbol{x}[e_t, a_t]$.

If the model refers to a data record ($Z_t = 1$), it directly copies the value of the data record $\boldsymbol{x}[e_t, a_t]$.
However, the values of numerical attributes 
can be expressed in at least two different manners: Arabic numerals (e.g., {\it 14}) and English words (e.g., {\it fourteen}).
One simple solution would be to follow the basic English rule that uses English words for single digits and Arabic numerals for the others. However, in the real descriptions, we found many irregular usages that do not conform to the basic English rule above.
Thus, we decide which one to use by the following probability:
\begin{equation}
    p(N_t = 1 \mid \boldsymbol{h}_{t-1}^{\textsc{LM}}, \boldsymbol{h}_{t}^{\textsc{Ent}}) = \sigma(\boldsymbol{w}^{\textsc{N}\top}(\boldsymbol{h}_{t-1}^{\textsc{LM}}\oplus \boldsymbol{h}_{t}^{\textsc{Ent}}) + b^{\textsc{N}}),
\end{equation}
where $\boldsymbol{w}^{\textsc{N}} \in \mathbb{R}^{2h}$ is a weight vector.
The model then updates the hidden states of the language model:
\begin{equation}
     \boldsymbol{h}_t'= \tanh\left(\boldsymbol{W}^{\textsc{H}}(\boldsymbol{h}_{t-1}^{\textsc{LM}} \oplus\boldsymbol{h}_{t}^{\textsc{Ent}}) + \boldsymbol{b}^{\textsc{H}}\right),\label{eq:hidden}
\end{equation}
where $\boldsymbol{W}^{\textsc{H}} \in \mathbb{R}^{h \times 2h}$ is a weight matrix, and $\boldsymbol{b}^{\textsc{H}} \in \mathbb{R}^{h}$ is a bias vector.

If the model does not refer to a data record ($Z_t = 0$), it predicts the next word $y_t$ via a probability over words conditioned on the context vector $\boldsymbol{h}_t'$:
\begin{equation}
    p(Y_t \mid \boldsymbol{h}_t') = \text{softmax}(\boldsymbol{W}^{\textsc{Y}} \boldsymbol{h}_t'),\label{eq:prob}
\end{equation}
where $\boldsymbol{W}^{\textsc{Y}}\in \mathbb{R}^{|\mathcal{V}|\times h}$ is a weight matrix and $|\mathcal{V}|$ is the size of the vocabulary.

Subsequently, the hidden state of language model $\boldsymbol{h}^{\textsc{LM}}$ is updated:
\begin{equation}
    \boldsymbol{h}_t^{\textsc{LM}} = \textsc{LSTM}(\boldsymbol{y}_{t}\oplus\boldsymbol{h}_t', \boldsymbol{h}_{t-1}^{\textsc{LM}}),
\end{equation}
where $\boldsymbol{y}_t \in \mathbb{R}^{d}$ is the embedding of the word generated at time step $t$.\footnote{%
In our initial experiment, we observed a word repetition problem when the tracking model is not updated during generating each sentence. To avoid this problem, we also update the tracking model with special trainable vectors $\boldsymbol{v}_{\textsc{REFRESH}}$ to refresh these states after our model generates a period: $\boldsymbol{h}_t^{\textsc{Ent}} = \textsc{Gru}^{A}(\boldsymbol{v}_{\textsc{Refresh}}, \boldsymbol{h}_t^{\textsc{Ent}})$}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Incorporating Writer Information}
\label{subsec:writer}

For the document-level data-to-text generation task, we hypothesize that each writer would select the different entity information from the table and describe the summary in their writing style.
To test our hypothesis, we also incorporate the information about the writer into our model to study how external style information affects the generation.
Specifically, 
instead of using Equation (\ref{eq:hidden}), 
we concatenate the embedding $\boldsymbol{w}$ of a writer
to $\boldsymbol{h}_{t-1}^{\textsc{LM}}\oplus\boldsymbol{h}_{t}^{\textsc{Ent}}$
to construct context vector $\boldsymbol{h}_t'$:
\begin{equation}
    \boldsymbol{h}_t'= \tanh\left(\boldsymbol{W}'^{\textsc{H}}(\boldsymbol{h}_{t-1}^{\textsc{LM}} \oplus\boldsymbol{h}_{t}^{\textsc{Ent}}) + \boldsymbol{w}+ \boldsymbol{b}'^{\textsc{H}}\right),
 \end{equation}
Since this new context vector $\boldsymbol{h}_t'$ is used for calculating the probability over words in Equation~(\ref{eq:prob}), the writer information will directly affect word generation, which is regarded as surface realization in terms of traditional text generation. Simultaneously, context vector $\boldsymbol{h}_t'$ enhanced with the writer information is used to obtain $\boldsymbol{h}_t^{\textsc{LM}}$, which is the hidden state of the language model and is further used to select the salient entity and attribute, as mentioned in Sections~\ref{subsec:saliency} and \ref{sec:ent}. Therefore, in our model, the writer information affects both surface realization and content planning.
 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Learning Objective}

We apply fully supervised training that maximizes the following log-likelihood:
\begin{align*}
 &\log p(Y_{1:T}, Z_{1:T}, E_{1:T}, A_{1:T}, N_{1:T} \mid \boldsymbol{x})\\
 & \quad =\sum_{t=1}^T\log p(Z_t = z_t \mid \boldsymbol{h}_{t-1}^{\textsc{LM}}, \boldsymbol{h}_{t-1}^{\textsc{Ent}})~\\
 & \qquad + \sum_{t:Z_t = 1}\log p(E_t = e_t \mid \boldsymbol{h}_{t-1}^{\textsc{LM}}, \boldsymbol{h}_{t-1}^{\textsc{Ent}})\\
 & \qquad + \sum_{t:Z_t = 1}\log p(A_t = a_t \mid e_t,  \boldsymbol{h}_{t-1}^{\textsc{LM}}, \boldsymbol{h}_{t}^{\textsc{Ent}'})\\
 & \qquad + \sum_{t:Z_t = 1, a_t \text{is num\_attr}} \log p(N_t = n_t \mid  \boldsymbol{h}_{t-1}^{\textsc{LM}}, \boldsymbol{h}_{t}^{\textsc{Ent}})\\
 & \qquad + \sum_{t: Z_t = 0}\log p(Y_{t} = y_t \mid \boldsymbol{h}_t')
\end{align*}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Experimental Settings}

We used \textsc{RotoWire-Modified} as the dataset for our experiments, which we explained in Section~\ref{section:data}.
The training, development, and test data respectively contained 
2,705, 532, and 497 games.

We also used recently introduced dataset, \textsc{RotoWire-FG} \cite{wang-2019-revisiting}, which is enlarged, enriched, and purified dataset from the original \textsc{RotoWire} dataset.
The biggest difference with the \textsc{RotoWire-Modified} is whether to include the unsupported sentence by the box-scores. 
In the \textsc{RotoWire-FG} dataset, they convert all of the numbers written in English words into the Arabic numbers.
Thus, we don't have to introduce the random variable $N_t$ to determine the surface realization of the selected number for this dataset.
We used the official split for training, development, and test data respectively contained games.

Since we take a supervised training approach, we need the annotations of the random variables (i.e., $Z_t$, $E_t$, $A_t$, and $N_t$) in the training data, as shown in Table~\ref{tab:annotate}.
Instead of simple lexical matching with $r \in \boldsymbol{x}$, which is prone to errors in the annotation, 
% %While the annotation could be obtained by a rule-based approach (e.g., lexical matching with $r \in \boldsymbol{x}$), it is possible to lead the annotation errors. Thus, 
we use the information extraction systems.
For \textsc{RotoWire-Modified} dataset, we used the information extraction model provided by \citeA{wiseman2017challenges} and for \textsc{RotoWire-FG} dataset, we used the hard-coded information retrieval system provided by \citeA{wang-2019-revisiting}.
These annotation performances directly affect the model performance.
Although the model-based system is trained on noisy rule-based annotations, 
we conjecture that it is more robust to errors because it is trained to minimize the marginalized loss function for ambiguous relations.
However, the model only considers sentence-level relation extractions so that the model always fails to extract relations across the sentence.

We set the dimensions of the embeddings to 128, and those of the hidden state of RNN to 512 and all of parameters are initialized with the Xavier initialization \cite{glorot2010understanding}.
We set the maximum number of epochs to 30, and choose the model with the highest \textsc{Bleu} score on the development data.
The initial learning rate is 2e-3 and AMSGrad is also used for automatically adjusting the learning rate \cite{reddi2018convergence}.
Our implementation uses DyNet \cite{neubig2017dynet}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Models to be Compared}

We compare our model\footnote{Our code is available from \url{https://github.com/aistairc/sports-reporter}}
against the following baseline models. 
\begin{itemize}
    \setlength{\itemsep}{0pt}
    \item \textsc{ED-CC}: the model consists of an attention-based encoder-decoder model with conditional copy mechanism used by \citeA{wiseman2017challenges}.
    \item \textsc{Hierarchical}: the model consists of the encoder with hierarchical attention proposed by \citeA{rebuffel2020hierarchical}
    \item \textsc{NCP}: the model first predicts the sequence of data records and then generates a summary conditioned on the predicted sequences proposed by \citeA{puduppully2019data}.
    \item \textsc{ENT}: the model is simultaneously proposed with ours, which extends the encoder-decoder model with memory modules proposed by \citeA{puduppully2019entity}.
\end{itemize}
\textsc{ED-CC} model refers to all data records every timestep, while \textsc{NCP} model refers to a subset of all data records, which is predicted in the first stage.
Unlike these models, our model uses one memory vector $\boldsymbol{h}^{\textsc{Ent}}_{t}$ that tracks the history of the data records, during generation.
We retrained the baselines on our new dataset.
We also present the performance of \textsc{Gold} and \textsc{Templates} summaries. The \textsc{Gold} summary is exactly identical with the reference summary and each \textsc{Templates} summary is generated in the same manner as \citeA{wiseman2017challenges}.

In the latter half of our experiments, 
we examine the effect of adding information about writers.
In addition to our model enhanced with writer information,
we also add writer information to the \textsc{NCP} model by \citeA{puduppully2019data}.
Their method consists of two stages corresponding to content planning and surface realization.
Therefore, by incorporating writer information to each of the two stages, we can clearly see which part of the model to which the writer information contributes to. 
For \textsc{NCP} model, we attach the writer information in the following three ways: 
\begin{enumerate}
    \item concatenating writer embedding $\boldsymbol{w}$ with the input vector for LSTM in the content planning decoder (stage 1);
    \item concatenating writer embedding $\boldsymbol{w}$ with the input vector for LSTM in the text generator (stage 2);
    \item using both 1 and 2 above.
\end{enumerate}
For more details about each decoding stage, readers can refer to \citeA{puduppully2019data}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Evaluation Metrics}

As evaluation metrics, we use BLEU score \cite{papineni2002bleu} and the extractive metrics proposed by \citeA{wiseman2017challenges}, i.e., relation generation (RG), content selection (CS), and content ordering (CO) as evaluation metrics. 
The extractive metrics measure how well the relations extracted from the generated summary match the correct relations:\footnote{%
The model for extracting relation tuples was trained on tuples made from the entity (e.g., team name, city name, player name) and attribute value (e.g., ``Lakers'', ``92'') extracted from the summaries, and the corresponding attributes (e.g., ``\textsc{Team Name}'', ``\textsc{Pts}'') found in the box- or line-score. The precision and the recall of this extraction model are respectively 93.4\% and 75.0\% in the test data.}

\begin{itemize}
    \item[-] RG: 
	the ratio of the correct relations out of all the extracted relations, where correct relations are relations found in the input data records $\boldsymbol{x}$. The average number of extracted relations is also reported.
	\item[-] CS: precision and recall of the relations extracted from the generated summary against those from the reference summary.
	\item[-] CO: edit distance measured with normalized Damerau-Levenshtein Distance (DLD) between the sequences of relations extracted from the generated and reference summary. 
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results and Discussions}

We first focus on the quality of tracking model and entity representation in Sections~\ref{subsec:result_tracking} to
\ref{subsec:result_output}, where we use the model without writer information.
We examine the effect of writer information in Section~\ref{subsec:result_writer}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Saliency Tracking-Based Model}
\label{subsec:result_tracking}

As shown in Table~\ref{tab:result}, our model outperforms all baselines across all evaluation metrics on \textsc{RotoWire-Modified}.
While the most similar model, \textsc{ENT}, shows the similar performance, our model performs noticeable improvement on F1 score of the content selection.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[t]
\caption{Experimental result on the \textsc{RotoWire-Modified}}
\label{tab:result}
\input{05table03.tex}
\vspace{4pt}\small
Each metric evaluates whether important information (CS) is described accurately (RG) and in correct order (CO).
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We also show the result on the \textsc{RotoWire-FG} dataset in Table~\ref{tab:result-fg}.
Our model outperforms the relation generation and the content selection scores and shows the second-highest performance on the \textsc{Bleu} score.
The \textsc{NCP}+TR model, however, clearly shows the best performance on the content ordering and \textsc{Bleu} score.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[t]
\caption{Experimental result on the \textsc{RotoWire-FG}}
\label{tab:result-fg}
\input{05table04.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

These two experimental results show the following properties.
(1) The entity-centric models, which are the \textsc{ENT} model and our proposed model, perform almost always better than the baselines under the noisy data-to-text setting (\textsc{RotoWire-Modified}).
(2) The explicit annotation, where the \textsc{NCP} model and our proposed model have, is particularly useful under the clean data-to-text setting (\textsc{RotoWire-FG}).
In other words, our proposed model could be seen as picking the best characteristics of both the \textsc{ENT} and the \textsc{NCP} models.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Qualitative Analysis of Entity Embedding}
\label{subsec:result_entity}

Our model has the game-dependent entity embedding $\bar{\boldsymbol{e}}$, which depends on the box score for each game in addition to static entity embedding $\boldsymbol{e}$.
Now we analyze the difference of these two types of embeddings.

We present a two-dimensional visualizations of both embeddings produced using PCA \cite{pearson1901liii}.
As shown in Figure~\ref{fig:static_ent}, which is the visualization of static entity embedding $\boldsymbol{e}$, the top-ranked players are closely located.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[b]
\begin{center}
\includegraphics{27-3ia5f3.eps}
\end{center}
\hangcaption{%
Illustrations of static entity embeddings $\boldsymbol{e}$. Players with colored letters are listed in the ranking top 100 players for the 2016-17 NBA season. Only {\it LeBron James} is in red and the other players in top 100 are in blue. Top-ranked players have similar representations of $\boldsymbol{e}$.}
\label{fig:static_ent}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 4
\begin{figure}[b]
\begin{center}
\includegraphics{27-3ia5f4.eps}
\end{center}
\hangcaption{%
Illustrations of game-dependent entity embedding $\bar{e}$. Both left and right figures are for \textit{Cleveland Cavaliers} vs. \textit{Detroit Pistons}, on different dates. LeBron James is in red letters. Entities with orange symbols appeared only in the reference summary. Entities with blue symbols appeared only in the generated summary. Entities with green symbols appeared in both the reference and the generated summary. The others are with red symbols. $\Box$ represents player who scored in the double digits, and $\Diamond$ represents player who recorded double-double. Players with $\triangle$ did not participate in the game. $\circ$ represents other players.}
\label{fig:ebar}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We also present the visualizations of game-dependent entity embeddings $\bar{\boldsymbol{e}}$ in Figure~\ref{fig:ebar}.
Although we did not carry out feature engineering specific to the NBA (e.g., whether a player scored double digits or not)\footnote{%
In the NBA, a player who accumulates a \textit{double}-digit score in one of five categories (points, rebounds, assists, steals, and blocked shots) in a game, is regarded as a good player. If a player had a double in two of those five categories, it is referred to as \textit{double-double}.} for representing the game-dependent entity embedding $\bar{\boldsymbol{e}}$, the embeddings of the players who performed well for each game have similar representations.
In addition, the change in embeddings of the same player was observed depending on the box-scores for each game.
For instance, \textit{LeBron James} recorded a double-double in a game on April 22, 2016.
For this game, his embedding is located close to the embedding of \textit{Kevin Love}, who also scored a double-double.
However, he did not participate in the game on December 26, 2016. His embedding for this game became closer to those of other players who also did not participate.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Duplicate Ratios of Extracted Relations}
\label{subsec:result_duplicate}

As \citeA{puduppully2019data} pointed out, 
a generated summary may mention the same relation multiple times. Such duplicated relations are not favorable in terms of the brevity of text.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 6 --> 5
\begin{figure}[b]
\begin{center}
\includegraphics{27-3ia5f6.eps}
\end{center}
\hangcaption{%
Ratios of generated summaries with duplicate mention of relations. Each label represents number of duplicated relations within each document. While \protect\citeA{wiseman2017challenges}'s model exhibited 36.0\% duplication and \protect\citeA{puduppully2019data}'s model exhibited 15.8\%, our model exhibited only 4.2\%.}
\label{fig:ratio}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Figure~\ref{fig:ratio} shows the ratios of the generated summaries with duplicate mentions of relations in the development data.
While the models by \citeA{wiseman2017challenges} and \citeA{puduppully2019data} respectively showed 36.0\% and 15.8\% as duplicate ratios, our model exhibited 4.2\%.
This suggests that our model dramatically suppressed generation of redundant relations.
We speculate that the tracking model successfully memorized which input records have been selected in $\boldsymbol{h}_s^{\textsc{Ent}}$.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Qualitative Analysis of Output Examples}
\label{subsec:result_output}

Figure~\ref{fig:example_summary} shows the generated examples from validation inputs with \citeA{puduppully2019data}'s model and our model.
Whereas both generations seem to be fluent, the summary of \citeA{puduppully2019data}'s model includes erroneous relations colored in orange.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Fig. 7 --> 6
\begin{figure}[b]
\begin{center}
\includegraphics{27-3ia5f7.eps}
\end{center}
\hangcaption{%
Example summaries generated with \protect\citeA{puduppully2019data}'s model (left) and our model (right). Names in bold face are salient entities. Blue numbers are correct relations derived from input data records but are not observed in reference summary. Orange numbers are incorrect relations. Green numbers are correct relations mentioned in reference summary.}
\label{fig:example_summary}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Specifically, the description about \textsc{Derrick Rose}'s relations, ``15 points, four assists, three rounds and one steal in 33 minutes.'', is also used for other entities (e.g., \textsc{John Henson} and \textsc{Willy Hernagomez}).
This is because \citeA{puduppully2019data}'s model has no tracking module unlike our model, which mitigates redundant references and therefore rarely contains erroneous relations.

However, when complicated expressions such as parallel structures are used
our model also generates erroneous relations 
as illustrated by the underlined sentences describing the two players who scored the same points.
For example, ``11-point efforts'' is correct for \textsc{Courtney Lee} but not for \textsc{Derrick Rose}.
As a future study, it is necessary to develop a method that can handle such complicated relations.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 5 -->7
\begin{figure}[t]
\begin{center}
\includegraphics{27-3ia5f5.eps}
\end{center}
%%%%\label{fig:ent}
%%%%\label{fig:attr}
\caption{Probability sequences of selecting entities and attributes.}
\label{fig:seq_prob}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We also show the sequences of probabilities of selecting an entity $p(E_t = e_t)$ and an attribute $p(A_t = a_t)$ in Figure~\ref{fig:seq_prob}.
The model first refers to overall information of the game, and then starts to look for salient players in the game.
Once the model selects a player, it consistently selects the same player until it finishes mentioning the important information of this player. After finishing mentioning them, the model almost never refers to that entity.
This suggests that the model successfully learns to guide the consistent entity reference, and mitigate the saliency of the entity mentioned so far.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Use of Writer Information}
\label{subsec:result_writer}

We first look at the results of an extension of \citeA{puduppully2019data}'s model with writer information $\boldsymbol{w}$
in Table~\ref{tab:author}.
By adding $\boldsymbol{w}$ to content planning (stage 1), the method obtained improvements in CS (37.16 to 45.58), CO (16.67 to 21.26), and BLEU score (14.82 to 18.01).
By adding $\boldsymbol{w}$ to the component for surface realization (stage 2), the method obtained an improvement in BLEU score (14.82 to 16.57), while the effects on the other metrics were not very significant.
By adding $\boldsymbol{w}$ to both stages, the method scored the highest BLEU, while the other metrics were not very different from those obtained by adding $\boldsymbol{w}$ to stage 1.
This result suggests that writer information contributes to both content planning and surface realization when it is properly used, and improvements of content planning lead to much better performance in surface realization.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[t]
\caption{Effects of writer information}
\label{tab:author}
\input{05table05.tex}
\vspace{4pt}\small
$\boldsymbol{w}$ indicates that \textsc{Writer} embeddings are used. Numbers in bold are the largest among the variants of each method.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Our model showed improvements in all metrics and showed the best performance by incorporating writer information $\boldsymbol{w}$.
As discussed in Section~\ref{subsec:writer},
$\boldsymbol{w}$ is supposed to affect both content planning and surface realization. Our experimental result is consistent with the discussion. 


We show generated examples of the \textsc{Proposed} model with and without writer information in Table~\ref{tab:example_writer}. The summary generated by the \textsc{Proposed} model with writer information shares more similar vocabulary usage with the \textsc{Gold} summary than the summary generated by the \textsc{Proposed} model without writer information.

In addition to the text style, the writer information is useful for choosing similar relations with the \textsc{Gold} summary.
Although the \textsc{Proposed} model without writer information mentions a bigger number of relations by generating longer summary, the \textsc{Proposed} model with writer information mentions a similar number of relations with the \textsc{Gold} summary.
Specifically, the \textsc{Gold} summary contains the 31 relations and summaries generated by the \textsc{Proposed} model with or without writer information contain 46, 34 relations, and shares 20, 24 correct relations with relations in the \textsc{Gold} summary respectively.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[p]
\caption{Example summaries and their \textsc{Bleu} scores on the sample of the development set}
\label{tab:example_writer}
\input{05table06.tex}
\vspace{4pt}\small
Blue numbers are correct relations derived from input data records but are not observed in reference summary. Orange numbers are incorrect relations. Green numbers are correct relations mentioned in reference summary.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

In this research, we proposed a new data-to-text model that produces a summary text while tracking the salient information.
As a result, our model outperformed the existing models in all evaluation measures on the \textsc{RotoWire-Modified} dataset.
We also explored the effects of incorporating writer information to data-to-text models.
With writer information, our model successfully generated highest quality summaries that scored 20.62 points of \textsc{BLEU} score.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment
This paper is an extension of our manuscript presented by the 57th Annual Meeting of the Association for Computational Linguistics \cite{iso2019learning} with more detailed analysis and experiments on new dataset.
This paper is based on results obtained from a
project commissioned by the New Energy and
Industrial Technology Development Organization
(NEDO), JST PRESTO (Grant Number JPMJPR1655), and AIST-Tokyo Tech Real World Big-Data Computation Open Innovation Laboratory (RWBC-OIL).
The authors would like to thank Enago (\url{https://www.enago.jp/}) for the English language review.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.5}
\bibliography{05refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Hayate Iso}{%
received his B.S. degree from Osaka University in 2016 in Mathematical Science and M.S. degree from Nara Institute of Science and Technology (NAIST) in 2017, in information science.
He is currently a Ph.D. student at NAIST. His research interests include natural language generation and social media analysis.
}

\bioauthor[:]{Yui Uehara}{%
received her M.S. degree from Japan Advanced Institute of Science and Technology in 2019.
Her current research interests are computational music research and natural language processing.
}

\bioauthor[:]{Tatsuya Ishigaki}{%
received his Ph.D. degree from Tokyo Institute of Technology, and is currently a postdoctoral researcher at Artificial Intelligence Research Center, AIST.
}

\bioauthor[:]{Hiroshi Noji}{
received his Ph.D. from the Graduate University for Advanced Studies. He was an assistant professor at Nara Institute of Science and Technology from 2016 to 2018, and is currently a researcher at Artificial Intelligence Research Center, AIST. His current research interests are syntactic parsing as well as modeling of human language processing such as language acquisition.
}

\bioauthor[:]{Eiji Aramaki}{%
received his Ph.D. from the University of Tokyo in 2005, and is currently an associate professor at Nara Institute of Science and Technology.
His current research interest is in natural language processing for the medical field. 
}

\bioauthor[:]{Ichiro Kobayashi}{%
received his Ph.D. from Tokyo Institute of Technology, and is currently a professor at Ochanomizu University.
His current research interests are in language processing and intelligent information processing.
}

\bioauthor[:]{Yusuke Miyao}{%
received his Ph.D. from the University of Tokyo in 2006, and is currently a professor at the University of Tokyo.
His current research interests are syntactic parsing and its applications.
}

\bioauthor[:]{Naoaki Okazaki}{%
received his Ph.D. from the University of Tokyo in 2007, and is currently a professor at Tokyo Institute of Technology.
His current research interests are natural language processing, text mining, and machine learning.
}

\bioauthor[:]{Hiroya Takamura}{%
received his Ph.D. from Nara Institute of Science and Technology, and is currently a professor at Tokyo Institute of Technology and a team leader at Knowledge and Information Research Team of Advanced Industrial Science and Technology.
His current research interest is computational linguistics.
}

\end{biography}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate




\end{document}

