\documentclass[english,other]{jnlp_3.5}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%\usepackage{latexsym}
\usepackage{paralist}
%%%%\usepackage{booktabs}
%%%%\usepackage{graphicx}
%%%%\usepackage{makecell}
%%%%\usepackage{amssymb}
%%%%\usepackage{verbatim}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{29}
\Number{4}
\Month{December}
\Year{2022}
%%%%\Issuetitle{Special Issue: XXXXXXXX}


\setcounter{page}{1284}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{Society Column}
\etitle{Constructing Timeline Summarization as a Heterogeneous Graph Attention Network}
\eauthor{Jingyi You\affiref{Author_1}}

\headauthor{You}
\headtitle{Constructing Timeline Summarization as a Heterogeneous Graph Attention Network}

\affilabel{Author_1}{}{Tokyo Institute of Technology}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}\label{sec:intro}

Timeline summarization (TLS) aims to extract sentences that describe an evolutionary story from a massive amount of web articles with respect to a specific topic in chronological order.
TLS has drawn much attention in recent years \cite{chen2019learning,martschat2018temporally,you2021abstractive,ghalandari2020examining,yu2021multitimeline} since it releases people from the burdensome manual creation of summaries and gives readers faster but comprehensive access to track events from many aspects, such as start and end, causality, and the main protagonists involved.

Most studies on TLS seek ways to combine two individual subtasks: date selection and event detection.
Depending on different strategies for them, current methods are generally divided into three categories \cite{ghalandari2020examining}:
\begin{inparaenum}[1)]
    \item {\em direct summarization} approaches \cite{chieu2004query,tran2013predicting,martschat2018temporally,duan2020comparative} directly identify topic-related sentences from a collection of news articles to form a timeline;
    \item {\em date-wise summarization} methods \cite{wang2016low,ghalandari2020examining,li2021automated,quatra2021summarize} first select salient dates then construct a timeline for each date individually with sentences of the highest score; and
    \item {\em event detection} algorithms \cite{steen2019abstractive,duan2020comparative,yu2021multitimeline} detect events by clustering sentences from multi-timeline news articles then identify several of the most important events and summarize them separately.
\end{inparaenum}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Motivation}

Our work is inspired by several unsolved issues.
First, current TLS methods mainly adopt statistical hand-designed features to represent dates, e.g., the number of published articles and topic-related sentences in a specific time duration \cite{yu2021multitimeline,ghalandari2020examining}, and employ sentence-BERT \cite{reimers2019sentence} and other pre-defined unchangeable representations for sentences.
The low-level or unlearnable representations tend to ignore the semantic and temporal information interaction between sentences and dates, which significantly degrades the performance of downstream tasks.
Secondly, traditional approaches focus on either date selection or event detection.
Although excellent date selection algorithms can pinpoint accurate timeline dates, they usually extract topic-irrelevant sentences.
While event detection algorithms are capable of avoiding redundant summaries by various clustering strategies, they sometimes capture wrong timeline dates.

To circumvent the above dilemma, we propose to jointly learn date/sentence representations and event detection-based sentence clustering in a heterogeneous graph attention network (HAN) for TLS (Heter\textsc{Tls}).
Specifically, we construct a heterogeneous graph with dates, words, and sentences as semantic units to solve the first problem.
In this graph, words act as a bridge between dates and sentences, enabling date nodes to learn different granularities (word- and sentence-level) of semantic information and sentence representations to be complemented with a date-related intra- and cross-sentence message.
As for the second issue, semi-supervised date prediction and event detection-based clustering are integrated into an overall objective, where labeled dates guide and facilitate sentence clustering, and sentence-level clustering information indicating main events improves the accuracy of unlabeled date prediction.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Workflow of HeterTLS}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[t]
\begin{center}
\includegraphics{29-4ia23f1.pdf}
\end{center}
\caption{Illustration of Heter\textsc{Tls}.}
\label{overview}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Given a collection of news documents $\mathcal{D}$ within $\mathcal{T}$ dates, TLS involves
\begin{inparaenum}[1)]
    \item predicting a sequence of date labels $\{y_{1},\cdots,y_{\mathcal{T}}|y_{i}\in\{0,1\}\}$, where $y_{t}=1$ represents the $t$-th date included in the timeline; and
    \item ranking and extracting sentences from candidates for each selected date.
\end{inparaenum}
Given a graph $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$ with $\mathcal{V}=V_{d} \cup V_{w} \cup V_{s}$ and $\mathcal{E}=E_{w-d} \cup E_{w-s}$, where $V_{d}$, $V_{w}$, and $V_{s}$ respectively denote a node set for dates, words, and sentences and $E_{w-d}$ and $E_{w-s}$ are a set of undirected edges between word-date and word-sentence.



Figure~\ref{overview} exhibits an overview of Heter\textsc{Tls}, which consists of three main components: (a) {\em graph constructor and initializer}, (b) {\em heterogeneous graph encoder with sentence clustering constraint}, and (c) {\em timeline summary extractor}.
We first construct heterogeneous network for date, sentence, and word nodes with two initialization strategies. We then extract meta-paths and iteratively update node representations via HAN under nuclear norm constraint on sentence nodes. Finally, we predict unlabeled date nodes and extract sentences from candidate clusters.

We initialize the $j$-th sentence node by concatenating its local n-gram feature $p_{j}$ and sentence-level global feature $q_{j}$ as $X_{s_{j}}=[p_{j};q_{j}]$.
$p_{j}$ is captured by a convolutional neural network (CNN) \cite{lecun1998gradient} with different kernel sizes, and $q_{j}$ is gripped by a bidirectional long short-term memory (Bi-LSTM) \cite{hochreiter1997long}.
Date nodes take the average-pooling of their connected sentences as initialization for both aforementioned strategies.
To leverage the saliency of each word in different sentences or dates, we propose term frequency-inverse sentence frequency (TF-ISF) and term frequency-inverse date frequency (TF-I\textsc{Date}F) weights to initialize edges in $E_{w-s}$ and $E_{d-w}$.
Specifically, TF is the number of occurrences of $w_{i}$ in $s_{j}$ or $d_{t}$, and ISF/I\textsc{Date}F is determined by dividing the total number of sentences or dates in $\mathcal{D}$ by the number of sentences or dates containing $w_{i}$.


Node representations are updated by hierarchical heterogeneous graph attention layers, where the node-level attention layer ensures information propagation and aggregation in a single meta-path, while the semantic-level one is committed to merging messages from multiple meta-paths.
Afterwards, the semantic-level attention layer fuses all the meta-path information for the $i$-th node.
To avoid gradient vanishing after certain iterations, a residual connection and position-wise feed-forward network layer are added after the semantic-level attention layer.
We alternately update each type of node to realize information propagation and aggregation.
Detecting main events from $\mathcal{D}$ can effectively reduce the redundancy when generating summaries.
Considering that the multiplicity of eigenvalue $0$ of the normalized Laplacian matrix of $H_{s}$ is equal to the number of clusters in $H_{s}$, we obtain $k$ clusters in $H_{s}$ by reorganizing its columns or rows and converting it into a block-diagonal form with $k$ blocks, as shown in Figure~\ref{overview} (c).
Finally, in Heter\textsc{Tls}, date selection and sentence clustering-based event detection are jointly trained to combine their advantages into an overall objective: $\mathcal{L}=\mathcal{L}_{\mathit{classify}}+\lambda \mathcal{L}_{\mathit{cluster}}$,
where $\mathcal{L}_{\mathit{classify}}$ minimizes the cross-entropy over all labeled date nodes between the ground-truth during training, $\mathcal{L}_{\mathit{cluster}}= \|H_{s}\|_{*}$, and $\lambda$ serves as a weighted coefficient to balance $\mathcal{L}_{\mathit{classify}}$ with $\mathcal{L}_{\mathit{cluster}}$.

With $l$ selected dates and their corresponding representations $\{\textbf{h}_{d_{1}},\cdots,\textbf{h}_{d_{l}}\}$, we represent the $k$ clusters as $\{\textbf{h}_{c_{1}},\cdots,\textbf{h}_{c_{k}}\}$ by averaging sentence representations inside that cluster.
As shown in Figure~\ref{overview} (c), candidate clusters for the $t$-th selected date are determined by calculating the cosine similarity between the date representation with all cluster representations as $cos(\textbf{h}_{d_{t}},\textbf{h}_{c_{j}}) (j\in \{1,\cdots, k\}$).
Finally, we apply CENTROID-OPT \cite{gholipour2017revisiting} as a sentence ranking algorithm within a cluster and summarize each date individually by selecting one sentence per cluster with the highest ranking score.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

We addressed several fundamental problems concerning TLS and proposed a joint learning model called HeterTLS, which trains a HAN by utilizing clustering structure learning-based event detection.
Due to the accurate node initialization and only measuring the similarity between date and cluster representations, our model achieved the most efficient training speed, which also indicates the feasibility of our model being applied in the real world.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{23refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Jingyi You}{%
received her M.S. degree in Information and Communications Engineering from Tokyo Institute of Technology in 2020, and a bachelor's in Computer Science and Technology from Dalian University of Technology in 2018.
She is currently a Ph.D. student at Tokyo Institute of Technology.
}


\end{biography}




\end{document}



