\documentclass[tombow,english]{jnlp_2.3}
\usepackage{jnlpbbl_1.3}
\usepackage[dvipdfm]{graphicx}
\usepackage{gentombow}
\usepackage{amsmath}
\settombowbleed{5mm}
\usepackage{url}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
%%%%\makeatletter
%%%%\def\@{\spacefactor3000\space}
%%%%\makeatother
\renewcommand\UrlFont{\rmfamily}

\makeatletter
\renewcommand{\@citeauthor}[1]{%
    \if@filesw\immediate\write\@auxout{\string\citation{#1}}\fi%
    \edef\@citeP{}%
    \@for\@citeb:=#1\do{% %%%% revised
    \@ifundefined{b@\@citeb}%
     {\expandafter\def\csname b@\@citeb\endcsname{?}%
      \expandafter\def\csname Y@\@citeb\endcsname{?}%
      \@warning{Citation `\@citeb' on page \thepage\space undefined}%
     }%
     {\@ifundefined{flag@\@citeb}%
      {\global\expandafter\def\csname flag@\@citeb\endcsname{DUMMY}%
       \def\BCA##1##2{##1}}%
      {\def\BCA##1##2{##2}}%
     }%
    \edef\B@my@dummy{\csname b@\@citeb\endcsname}%
    \ifx\@citeP\B@my@dummy%
    \else%
        \if@F@cite\@F@citefalse\else{,\ }\fi%
        \def\BBA{\BBAB}%
	\csname b@\@citeb\endcsname%\ \BBOP\csname Y@\@citeb\endcsname%
    \fi%
    \edef\@citeP{\csname b@\@citeb\endcsname}%
    }% %%%% revised
    \@F@citetrue}
\makeatother

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%% author's definition
%%%%%\usepackage[dvipdfmx]{graphicx}
%%%%%\usepackage{amsmath}
%%%%%%%%% Packages added by the author
%%%%%\usepackage{appendix}
%%%%%\usepackage{booktabs}   % Tables format
%%%%%\usepackage{adjustbox}  % Tables format
\usepackage{tabularx}   % Tables format
%%%%%\usepackage{xurl}       % URL format
\usepackage{lingmacros} % Examples format
%%%%%\newcolumntype{L}[1]{>{\raggedright\arraybackslash}m{#1}}   % Table 5 format
%%%%%\newcolumntype{N}{@{}m{0pt}@{}}                             % Table 5 format 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\Volume{27}
\Number{2}
\Month{June}
\Year{2020}

\received{2019}{11}{1}
\revised{2020}{2}{7}
%%%%\rerevised{2006}{5}{6}
\accepted{2020}{3}{13}
%%%%\Issuetitle{Special Issue: XXXXXXXX}

\setcounter{page}{383}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Empirical Exploration of the Challenges in Temporal Relation Extraction from Clinical Text}
\eauthor{Diana Galvan-Sosa\affiref{Author_1} \and Koji Matsuda\affiref{Author_2}\affiref{Author_1} \and Naoaki Okazaki\affiref{Author_3} \and Kentaro Inui\affiref{Author_1}\affiref{Author_2}}
\eabstract{%
Time is an important concept in human-cognition, fundamental to a wide range of reasoning tasks in the clinical domain. Results of the Clinical TempEval 2016 challenge, a set of shared tasks that evaluate temporal information extraction systems in the clinical domain, indicate that current state-of-the-art systems do well in solving event and time expression identification but perform poorly in temporal relation extraction. This study aims to identify and analyze the reason(s) for this uneven performance. It adapts a general domain tree-based bidirectional long short-term memory recurrent neural network model for semantic relation extraction to the task of temporal relation extraction in the clinical domain, and tests the system in a binary and multi-class classification setting by experimenting with general and in-domain word embeddings. Its results outperform the best Clinical TempEval 2016 system and the current state-of-the-art model. However, there is still a significant gap between the system and human performance. Consequently, this study delivers a deep analysis of the results, identifying a high incidence of nouns as events and class overlapping as posing major challenges in this task. 
}
\ekeywords{Clinical Text, Natural Language Processing, Temporal Relation Extraction, Aspect, Nominal Events}

\headauthor{Galvan-Sosa et al.}
\headtitle{Challenges in Temporal Relation Extraction from Clinical Text}

\affilabel{Author_1}{}{Graduate School of Information Sciences, Tohoku University}
\affilabel{Author_2}{}{RIKEN Center for Advanced Intelligence Project}
\affilabel{Author_3}{}{School of Computing, Tokyo Institute of Technology}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Figure 1
\begin{figure}[b]
\begin{center}
\includegraphics{27-2ia10f1.eps}
\end{center}
\caption{Example temporal relation annotation with and without using narrative containers.}
\label{fig:thyme}
\end{figure}
%%%%%&%%%%%%%%%%%%%%%%%%%%%%%%

Human reasoning has to do with time. High-level cognition concepts, such as \textit{duration} and \textit{sequence}, influence the structure of human interaction with the external world. Temporal reasoning is a fundamental ability not only in humans but also in intelligent systems. In Natural Language Processing (NLP), Temporal Information Extraction (TIE) is an active research area where the ultimate goal is to be able to represent the development of a story over time. This is key to text processing tasks including question answering \cite{uzzaman2012evaluating} and text summarization \cite{jung2011building}, and it follows the traditional pipeline of named entity recognition and relation extraction separately. In a temporal context, entities are typically classified as either events or time expressions and temporal relations describe how their time intervals interact, assuming a linear model of time.
Besides reasoning, choosing an accurate representation of time is challenging. In language, events are typically conceptualized as something that occurs, and they all have some duration. In the clinical domain, events can range from procedures to diseases to diagnoses, or to anything that the patient experiences. For simplicity, a point-based temporal logic is typically used to associate two periods of time. For example, given an event \textsc{a} (``surgery'') and a time expression \textsc{b} (``tomorrow''), where \textsc{a} precedes \textsc{b}, we can infer a temporal relation BEFORE between \textsc{a} and \textsc{b}. Intuitively, we can also say that \textsc{b} comes AFTER \textsc{a}. The main problem with this temporal logic is that several temporal relations, which are not necessarily relevant to the reason about the situation described, can be identified within a text. Narrative containers are defined by \citeA{Pustejovsky2011} as an effort to reduce the scope of temporal relations between pairs of events and time expressions. 
As illustrated in Figure~\ref{fig:thyme}, narrative containers can be thought of as temporal buckets in which an event or a series of events may fall \cite{Styler2014}. They help visualize the temporal relations within a text and facilitate the identification of other temporal relation types. Until now, the only corpus annotated with this schema is limited to clinical texts.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Clinical TempEval Challenges}

Research on TIE has been instigated by Temporal Evaluation (TempEval) shared tasks that are focused on processing news article documents \cite{Verhagen2007,Verhagen2010,Uzzaman2013}. However, in recent years, due to the high role of temporal reasoning in the interpretation of clinical narratives, the target domain has been shifted to the clinical domain. The resulting Clinical TempEval challenges \cite{Bethard2015,Bethard2016,Bethard2017} evaluate systems on temporal information extraction from clinical notes and pathology reports from colon cancer patients, defining a series of sub-tasks that aim to identify temporal entities (\textsc{event}s and \textsc{timex3}: time expressions) and the temporal relations (\textsc{tlink}) between them. Participating systems can choose to use raw text as input (phase 1) or they can use raw text with \textsc{event} and \textsc{timex3} annotations (phase 2), in which case their task is to only identify temporal relations. The temporal relation extraction track is further divided into two sub-tasks: (1) the identification of relations between events and the document creation time and (2) the identification of narrative container relations (\textsc{tlink:contains}) between a directed pair $(e_1,e_2)$. In this case, $e_1$ and $e_2$ are entities of either \textsc{event} or \textsc{timex3} type. Clinical TempEval 2017 \cite{Bethard2017} introduced a new aspect to the challenge, which still maintains the aforementioned sub-tasks but sets a new goal---to explore how well the systems trained in one medical domain perform on data from another. Such systems are trained on colon cancer data but are instead tested on brain cancer data. 

Results of the systems participating in Clinical TempEval 2016 suggest that they perform well on time-entity identification tasks. Nevertheless, temporal relation extraction has proven to be the most difficult task. UTHealth \cite{Lee2016}, the best ranked system in Clinical TempEval 2016, showed a significant gap of 0.25 when compared to human performance,\footnote{There are two scores for human performance: inter-annotator agreement and annotator-adjudicator agreement. We consider ann-adj as the upper bound performance since the models are trained on the adjudicated data, not on the individual annotator data \cite{Bethard2015,Bethard2016,Bethard2017}} even when gold-standard entity annotations were provided. The improved task performance of recent works by \citeA{Lin2016} and \citeA{Leeuwenberg2017} further enhanced the credibility of UTHealth's results, but the gap with respect to humans is still around 0.21. Regardless of the increase in the annotation agreement of temporal relations by relying on narrative containers, there is a consensus within the research community regarding the difficulties experienced in TIE. However, the reasons behind the skewed results between entity and temporal relation predictions still remain unclear. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Previous Work}

Until Clinical TempEval 2016, classic machine learning algorithms for classification such as conditional random fields, support vector machines (SVM) and logistic regression with a variety of features (e.g., lexical, syntactic, and  morphological) were the predominant approach to TIE \cite{Bethard2015,Bethard2016}. In fact, the best performance was achieved by the UTHealth team \cite{Lee2016} using an end-to-end system based on a linear and structural Hidden Markov Model (HMM)-SVM. Only a few teams tried a neural based method, including recurrent neural networks-based (RNN) models \cite{Fries2016} and convolutional neural networks-based (CNN) models \cite{Chikka2016,Li2016}. Furthermore, among those teams, only \citeA{Chikka2016} participated in the \textsc{contains} identification task, being around 0.30 below UTHealth's top performance.
 
Recent works by \citeA{Lin2016}, \citeA{Dligach2017} and \citeA{Leeuwenberg2017} followed the settings of Clinical TempEval 2016 but they did not participate in the competition. Even though \citeA{Leeuwenberg2017} developed a new state-of-the-art model for temporal relation extraction, their results are still below human performance. Moreover, none of the aforementioned works provide a detailed discussion of \textit{why} the current performance is so low and \textit{how} the results on temporal relation extraction can be improved, save for Leeuwenberg and Moens, who in their first attempt on tackling this task on Clinical TempEval 2016 \cite{Leeuwenberg2016}, identified false negatives as their major problem. 

Rather than a model's architecture or a dataset size, we believe that the complexity of temporal representation in natural language is likely to be the main cause of the low performance on temporal relation. \textit{Tense} and \textit{aspect} are the two grammatical means of expressing the notion of time in English, but little has been discussed about the latter in clinical texts.  Furthermore, the focus of previous work on temporal relation extraction is set on narrative containers, relegating the identification of other temporal relations to a second place. However, we believe that the key is to look at the whole set of temporal types to achieve the ultimate goal of developing systems that can reason about time to automatically create a timeline of a patient's health care.

This study contributes to the current understanding of how temporal relations work in the clinical domain. It begins by illustrating how we adapt a general domain neural model for semantic relation extraction to temporal relation extraction from clinical text. It then analyzes the adopted system's overall performance, including the identification of narrative containers. Next, it discusses two major problems encountered when working with the narrative container's annotation schema, ending with a discussion on the necessary efforts needed to further improve the performance of the current state-of-the-art temporal relation extraction systems to perform on par with humans in terms of efficiency in completing the same tasks.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Methods}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{From Relation Extraction to Temporal Relation Extraction}

To determine the challenges in temporal relation extraction from clinical text, this study adapts a general domain relation extraction model. In NLP literature, the term ``relation extraction'' is a short form for ``semantic relation extraction'', which is the existing association between the meaning of words, phrases, or sentences. Time concepts, such as duration and sequence, are embedded in a word's meaning (e.g., ``bleeding'' is an action that usually lasts for a moment and comes after another action like ``cutting''). Therefore, semantic relations and temporal relations naturally overlap.

Relation extraction is a well-studied task in NLP, where besides semantics, word sequence structures---such as recurrent neural networks (RNN) and linguistic features like the path of target words in the dependency tree---have shown to be effective \cite{xu2015semantic}. There is already a relation extraction model that integrates all of these elements---the end-to-end tree-based bidirectional long short-term memory-RNN model of \citeA{Miwa2016}. Due to its availability and state-of-the-art performance, we chose this model over \citeauthor{Leeuwenberg2017}'s \citeyear{Leeuwenberg2017} system. Moreover, we aim to take advantage of word sequence and dependency tree structures to further improve the performance on the Clinical TempEval relation extraction task. 
Given a sentence, \citeauthor{Miwa2016}'s \citeyear{Miwa2016} three-layer model (i.e., embedding, sequence and dependency layers) jointly identifies entities and the relations between them. The model receives a sentence and an annotation file with a pair of terms as input and outputs the predicted relation type and directionality of the terms: $(t_1,t_2)$ if $t_1$ is the source and $t_2$ the target, and $(t_2,t_1)$ otherwise. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Experimental Settings}
\label{sec:expset}

Similar to Clinical TempEval 2016, we used the THYME corpus \cite{Styler2014}, a dataset of 600 clinical notes and pathology reports from colon cancer patients at the Mayo Clinic. The corpus is annotated at the document level and the identified entities are given a set of attributes depending on their type (i.e., \textit{DocTimeRel}, \textit{Type}, \textit{Polarity}, \textit{Degree}, \textit{Contextual Modality}, and \textit{Contextual Aspect} for \textsc{event}s and \textit{Class} for \textsc{timex3}). Temporal relation annotations specify source and target entities along with one of the following \textsc{tlink} types: \textsc{before}, \textsc{begins-on}, \textsc{contains}, \textsc{ends-on}, and \textsc{overlap}. Considering \citeauthor{Miwa2016}'s \citeyear{Miwa2016} processes one sentence at a time, data was pre-processed to get sentence-level annotations. Any two \textsc{event/timex3} can be a candidate pair. Therefore, all the entities in a sentence were used in generating all pair permutations as candidates. Pairs that did not have any temporal relations were then labeled as \textsc{none} (see Appendix~\ref{app:annotations}). The frequency of \textsc{tlink}s in the THYME corpus is higher than the relations in the SemEval-2010 Task 8 dataset \cite{Hendrickx2010}, on which \citeauthor{Miwa2016}'s \citeyear{Miwa2016} model was tested for relation classification. For this reason, we did not consider it necessary to extend the set of \textsc{tlink}s to its transitive closure for data augmentation (i.e.,  A \textsc{contains} B $\land $ B \textsc{contains} C $\rightarrow$ A \textsc{contains} C). Table~\ref{dataset1} and Table~\ref{dataset2} detail the resulting datasets.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Table 1 and 2
\begin{table}[b]
\setlength{\captionwidth}{189pt}
\noindent
\begin{minipage}[t]{189pt}
\hangcaption{Label distribution of pre-processed dataset for binary classification}
\label{dataset1}
\input{10table01.tex}
\end{minipage}
\hfill
\begin{minipage}[t]{189pt}
\hangcaption{Label distribution of pre-processed dataset for multi-class classification}
\label{dataset2}
\input{10table02.tex}
\end{minipage}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In addition to the model's default Wikipedia word embeddings, we trained word vectors of 200 dimensions using word2vec \cite{Mikolov2013} on a subset of PubMed2014.\footnote{\url{ https://www.nlm.nih.gov/databases/download/pubmed_medline.html}} PubMed2014 has 10,969,353 abstracts from 1,118,934 different journals. From those, we selected 634,813 abstracts from 38,677 journals related to Oncology and Gastroenterology. The MIMIC II clinical corpus \cite{Saeed2011} is closer in genre to the THYME dataset, but due to its nature, one must get an application approval for its use. For simplicity purposes, we instead chose PubMed. 
Next, we conducted four experiments at the intra-sentential level. The first experiment followed the Clinical TempEval 2016, focusing only on the identification of the \textsc{contains} type. The remaining experiments included all of the five annotated \textsc{tlink}s. Further details of each of the experiments are given below:
\renewcommand{\theenumi}{\Roman{enumi}}
\begin{enumerate}
\item \textbf{TLINK:CONTAINS binary classification:} In order to obtain results comparable to \citeA{Lee2016}, the best ranked system in Clinical TempEval 2016, we only considered \textsc{tlink:contains} instances. The model chooses between \textsc{contains} and \textsc{none} relations.
\item \textbf{Multi-class classification with Wikipedia word embeddings:} To test the model in a real-world setting (i.e., a document that not only includes \textsc{contains} relations), we added the remaining pairs in the gold standard that have any of the other \textsc{tlink} types to the train and test sets. 
\item \textbf{Multi-class classification with PubMed word embeddings:} In addition to the previous setting (II), we used word embeddings trained on a subset of PubMed instead of the default word vectors trained on Wikipedia.
\item \textbf{Multi-class classification with PubMed word embeddings and filtered negative examples:} Two extra difficulties of temporal relations are to ascertain whether an \textsc{event} happened or not in a clinical context, and evaluate whether the said \textsc{event} actually relates to the patient. For this reason, the THYME corpus differentiates ``real'' (\textit{Contextual Modality:} \textsc{actual} or \textsc{hedged}) from ``non-real'' (\textit{Contextual Modality:} \textsc{hypothetical} or \textsc{generic}) events. Real events cannot be related to non-real events. Therefore, in addition to the previous setting (III), we experimented removing a candidate pair whenever the $e_1$ contextual modality value\footnote{Note that entity attributes introduced at the beginning of this section were only used for pre-processing, and not as features in our model.} was \textsc{actual} or \textsc{hedged} and $e_2$  had \textsc{hypothetical} or \textsc{generic} modality, and vice versa.
\end{enumerate}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results}

The evaluations were performed using the official Clinical TempEval scorer.\footnote{\url{http://alt.qcri.org/semeval2016/task12/index.php?id=software}} Table~\ref{comparable} shows performance on the \textsc{contains} identification task as a binary classification problem. The first row shows the top performance in Clinical TempEval 2016, while the second row is a result outside of the competition. We obtained an F1 score of 0.633, outperforming both UTHealth and \citeA{Lin2016}. Our model shows a high precision, but a lower recall than UTHealth; this can be attributed to the \textsc{none} relations prevalent in the dataset. Despite the recorded improvement, it is not possible to compare our system's performance with the current state-of-the-art set by \citeauthor{Leeuwenberg2017}'s \citeyear{Leeuwenberg2017}, which was obtained using a multi-class classification approach.
Table~\ref{results-table} reports our experimental results with the three multi-class classification settings presented in Section~\ref{sec:expset}. Switching from binary classification to multi-class classification, we observe a significant drop in precision and a lower F1 score. This is expected because the classifier now has more \textsc{tlink}s as options to choose from. Despite this change, our model outperforms both UTHealth and the state-of-the-art model in terms of the F1 score of \textsc{contains}. Using PubMed word embeddings yielded the best F1 score for \textsc{ends-on} and \textsc{contains}, and down-sampling negative examples on this setting improved the F1 score of \textsc{before}, \textsc{ends-on} and \textsc{overlap}. More details on the impact of using in-domain word embeddings and the FNE strategy are provided in Appendix~\ref{app:results}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%% Table 3
\begin{table}[t]
\hangcaption{Performance of systems and humans on identifying \textsc{contains} relations. Our results come from five different random seeds}
\label{comparable}
\input{10table03.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Table 4 
\begin{table}[t]
\hangcaption{Results of the three multi-class classification experiments and Leeuwenberg and Moens's (2017) Structured Perceptron (SP) best results on the THYME test set}
\label{results-table}
\input{10table04.tex}
\vspace{4pt}\small
The SP results were reproduced from the original paper. The results come from five different random seeds. FNE -- filtered negative examples.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussions}

Since this study did not change the architecture of \citeauthor{Miwa2016}'s \citeyear{Miwa2016} model, 
\pagebreak
the reader can, therefore, consult their study \cite{Miwa2016} for a detailed discussion on the system's performance on relation extraction. This section complements their discussion, which focuses on the linguistic characteristics of the dataset (clinical and temporal) that harms the system's performance. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Figure 2
\begin{figure}[b]
\begin{center}
\includegraphics{27-2ia10f2.eps}
\end{center}
\hangcaption{Confusion matrix of our multi-class classification model with PubMed word embeddings on the dev set.}
\label{fig:matrix}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Error Analysis}

Our error analysis focused on one-fourth of our experiments. Systems participating in the Clinical TempEval narrative container identification task only received credit for a pair of entities that they correctly identified the source, target, and the \textsc{contains} relation between them. Given this setting, we understand that even when using manual event and time annotations, the challenge is not only to predict the \textsc{tlink} type but also the correct directionality of the entities. Therefore, part of our analysis aims to ascertain whether type classification or directionality identification is the most difficult task or if they are both equally problematic for the model. For this reason, we designed the confusion matrix using \citeauthor{Miwa2016}'s \citeyear{Miwa2016} output instead of the Clinical TempEval 2016 script output. The confusion matrix on Figure~\ref{fig:matrix} shows the results on the development set. 
Overall, due to the high number of negative instances, most of the false positives fall into the $None(e_1, e_2)$ category. This type of relation is the reason why the system shows high precision. Apart from this, we can identify the performance on \textsc{overlap} as our system's main problem. The accuracy in both $Overlap(e_1, e_2)$ and $Overlap(e_2, e_1)$ is considerably low, with the latter being the lowest among all types (with 0.021). Not even the performance on $BeginsOn(e_2, e_1)$---with 0.14---is as low as $Overlap(e_2, e_1)$, although they have a similar number of instances (430 and 557, respectively). $Overlap(e_1, e_2)$, with 0.14, is comparable to $BeginsOn(e_2, e_1)$, despite having four times more instances (1,831 vs. 430).  This explains why we focused our error analysis on \textsc{overlap}.
From Figure 2, we can observe that $Overlap(e_1, e_2)$ is usually predicted as $Contains(e_1, e_2)$ and $Overlap(e_2, e_1)$ is predicted as $Contains(e_2, e_1)$. In both cases, the directionality of the entities was correct but the system failed to identify the appropriate temporal relation. For $Overlap(e_1, e_2)$, there were 112 sentences misclassified as $Contains(e_1, e_2)$, while in $Overlap(e_2, e_1)$ there were 32 $Contains(e_2, e_1)$ misclassifications. \textsc{event}-\textsc{event} pairs were the predominant type of pair in the former while \textsc{timex3}-\textsc{event} was for the latter, with 101 and 25 instances, respectively. We took all of the aforementioned misclassified sentences for supplementary examination and discuss the reason(s) for these errors in the following section. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Temporal Relations and Aspectual Classes}

Before proceeding further, it is important to understand the definition of \textsc{overlap} and \textsc{contains}. Both temporal relations are closely related because they encompass the notion of two things happening at the same time. However, \textsc{contains} relations imply that the contained event (i.e., the target) occurs entirely within the temporal bounds of the event it is contained within (i.e., the source) while \textsc{overlap} relations are those where containment is not entirely sure. Also, since $e_1$ \textsc{overlap} $e_2$ means the same as $e_2$ \textsc{overlap} $e_1$, \textsc{overlap} is the only symmetrical \textsc{tlink} type. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Time representation: Interval algebra and linguistics}

Strictly speaking, every entity occupies time. An entity's time interval is crucial in understanding its temporal relation with respect to another entity, especially in the case of \textsc{contains} and \textsc{overlap} relations where the end point of the target is key in determining whether there is complete containment or not. The temporal relations used by the THYME project rely on \citeauthor{Allen1990}'s \citeyear{Allen1990} interval algebra, a precise way of expressing time periods using clear start and end points. By comparing those, we can easily indicate the position of two events on the timeline. However, the concept of time is widely discussed across disciplines and Allen's representation is just one among many others. In linguistics, the expression of time is understood because of two important grammatical systems: \textbf{\textit{tense}} and \textbf{\textit{aspect}}.
\textbf{Tense} is used to locate the time of an event being talked about with respect to the time at which the speaker utters the sentence (i.e., speech time), while \textbf{aspect} is used to describe how a speaker views the contour of a situation (i.e., as beginning, continuation, or completion), independent of which position in time this situation occupies \cite{Li2000,klein2013time}. Therefore, when discussing a situation such as a patient having a surgery, we would use \textbf{past tense} if the surgery happened before speech time (\textit{The patient had a surgery}), \textbf{future tense} if the surgery is about to happen (\textit{The patient will have a surgery}), and \textbf{present tense} if the time of the surgery overlaps with the speech time (\textit{The patient has a surgery}). Aspect, on the other hand, gives information about the surgery. The past tense in \textit{The patient had a surgery} not only locates the surgery event before the speech time, but also conveys that the surgery was completed. This leads us to an important characteristic of tense and aspect---the boundaries between them are often not clear-cut \cite{Li2000}. The linguistic forms that express each of these notions tend to grammaticize into other categories \cite{bybee1994evolution}. In the case of English, the past tense form indicates past tense and perfective aspect (i.e., when a situation is reported in its entirety) simultaneously.

Having identified the functions of tense and aspect, we now focus on the latter. The study of aspect is commonly divided into the \textit{grammatical aspect} (also known as viewpoint aspect \cite{smith1983theory}) and the \textbf{\textit{lexical aspect}} (also known as the situational aspect).\footnote{For our discussion, we will use the term ``aspect'' to refer to lexical aspect.} Since temporal relations between events in the THYME project are thought in terms of their \textbf{start and endpoints}, the definition of the lexical aspect, which designates the internal temporal organization of the situation described by a verb, is particularly important to us \cite{klein2013time}. One of the best known and widely accepted aspect classification is that of Vendler, which distinguishes four categories of the inherent semantics of verb and verb phrases: \textit{activities}, \textit{accomplishments}, \textit{achievements}, and \textit{states} \cite{vendler1957verbs}. Figure~\ref{fig:vendler} presents Vendler's classification using \citeauthor{Andersen1990}'s \citeyear{Andersen1990} schematization. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Figure 3
\begin{figure}[b]
\begin{center}
\includegraphics{27-2ia10f3.eps}
\end{center}
\hangcaption{Vendler's four-way classification. Arrows represent an indefinite time interval, solid lines indicate a homogeneous duration, and dashed lines indicate a dynamic duration. An X is used to represent a situation's natural end point.  Abbreviations: C -- Clear and NC -- Not Clear.}
\label{fig:vendler}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Figure 4
\begin{figure}[b]
\begin{center}
\includegraphics{27-2ia10f4.eps}
\end{center}
\hangcaption{Allen and Vendler's interval representation of \textsc{overlap} and \textsc{contains} relations. A$-$/B$-$ and A$+$/B$+$ represent the start and end of an event, respectively. Filled-dots represent clear start points while an empty-dot represent a not-clear start point.}
\label{fig:timelines}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Table 5
\begin{table}[t]
\caption{Sample of the analyzed misclassified sentences by our system}
\label{sentences}
\input{10table05.tex}
\par\vspace{4pt}\small
$e_1$ and $e_2$ are shown in \textbf{bold} and \textit{italics}, respectively.
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Aspectual classification of temporal entities}
\label{subsec:aspect}

We expect that by categorizing the source and target entities of a relation as one of Vendler's types, the underlying reasoning for the \textsc{tlink} classification will be simplified. For example, categories with \textbf{no clear end points} (such as \textit{activities} and \textit{states}) are more likely to overlap with those with \textbf{clear end points} (such as \textit{accomplishments} and \textit{achievements}). Figure~\ref{fig:timelines} illustrates an \textsc{overlap} and \textsc{contains} relations using Allen and Vendler's representation of time periods. 
While analyzing \textsc{overlap} relations that were mistaken for \textsc{contains}, we realized that only a few events were verbs. \textsc{event}s in sentences 1, 3 and 9 in Table~\ref{sentences} are some examples of this (``invades,'' ``seeking,'' and ``moving''). This pointed out the necessity of discriminating between \textbf{verbal and non-verbal} events to understand how they are temporally related. Our observations suggest that when recognizing an entity semantic type (e.g., sign or symptoms, diseases, and procedures); it is imperative to consider the action associated to it. Therefore, procedures such as colonoscopy, biopsy, pathology, and surgery have to be \textit{performed} (a dynamic verb with a natural end point---an \textit{accomplishment}). Diseases such as adenocarcinoma and appendicitis are \textit{present}, they exist, and consequently, they fall into the \textit{state} category. This is also the case for signs or symptoms like nausea, fever, or discomfort. Following this line of reasoning, it is easier to differentiate an \textsc{overlap} relation from \textsc{contains} in sentence 5 because we understand that nausea was present during the performance of the dialysis, but there is no enough information as regards to whether the nausea is still present or not. In other words, its end point is unclear. In the case of \textsc{timex3}-\textsc{event} pairs like those in sentences 8 to 10 in Table~\ref{sentences}, the nature of the \textsc{overlap} relation between the entities is due to the ambiguity of the time expressions combined with actions that we perceive as ongoing. For example, in sentence 9, the action of moving is an \textit{activity} that is done indeterminably throughout the day as \textit{multiple times a day} imply. On the other hand, in sentence 7, there is a time expression with a definite time interval overlapping the patient's state of being hospitalized.
\citeA{Styler2014} point out that several entities and other non-events are often interpreted in terms of their associated \textbf{eventive properties}. However, their discussion differs from ours in that they focus on how these properties define entities such as medications or disorders as an \textsc{event}, rather than how the implicit interpretation of their eventuality (\textit{taking a medication} or \textit{having a disorder}) is necessary to relate two entities from a temporal perspective. They also introduce the \textbf{``contextual aspect,''} which is one of \textsc{event} attributes, but their definition does not relate to the one used in linguistics. The contextual aspect attribute allows one of the three values, \textsc{n/a}, \textsc{novel}, and \textsc{intermittent}, but as explained in the THYME guidelines, the \textsc{n/a} value simply represents an \textsc{event} as neither \textsc{novel} (i.e., new on the patient's timeline) nor \textsc{intermittent} (i.e., when there may be a series of smaller events within a single \textsc{event}). The \textsc{intermittent} value can be useful in identifying an \textit{activity} or an \textit{accomplishment}, but as shown in Figure~\ref{fig:aspect_values}, just a small portion of \textsc{event}s were annotated with a value different from the default one. Moreover, \textbf{aspect is a property of verbs}, and our analysis insinuates that it is more common to find nouns as events. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Figure 5
\begin{figure}[b]
\begin{center}
\includegraphics{27-2ia10f5.eps}
\end{center}
\caption{\textsc{contextual aspect} attribute values by set.}
\label{fig:aspect_values}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The temporally locating of two events on a timeline requires a high level of reasoning that even humans can turn into a complicated task. All the aforementioned inferences for differentiating between two of the most frequent and most similar \textsc{tlink} types (\textsc{contains} and \textsc{overlap}) were done by heavily relying on the \textbf{internal constituency} of an event. Leveraging on aspectual type for temporal relation extraction is a promising approach that was explored by \citeA{Costa2012} on TempEval data, and our analysis implies that clinical data can also profit from it. However, this approach is limited since aspect is a property of verbs. 

So far, we have been able to identify a \textbf{high similarity} of \textsc{contains} and \textsc{overlap} relations as one of the reasons why these two types of \textsc{tlink} are easily confused by our system, which did not pose much difficulties in identifying other \textsc{tlink} types with a similar number of instances. This differs from what \citeA{Styler2014} report for the annotator disagreement, which they say comes from different opinions about whether any two \textsc{event}s require an explicit \textsc{tlink} between them or an inferred one, \textit{rather than what type of \textsc{tlink} it would be (e.g., \textsc{before} vs. \textsc{contains})}. Our observations suggest that the main problem is not the amount of data available, but rather how temporal properties are encoded in language. The next section elaborates this point.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Temporality of Nominal Events}

To deepen our understanding on the complexity of the temporal relation extraction task, we divided all \textsc{overlap} and \textsc{contains} false negatives into the four possible pair types: \textsc{event}-\textsc{event}, \textsc{timex3}-\textsc{timex3}, \textsc{event}-\textsc{timex3}, and \textsc{timex3}-\textsc{event}. As shown in Table~\ref{nominal_fq} (left), a significant amount of \textsc{overlap} and \textsc{contains} links were \textsc{event}-\textsc{event} relations. Therefore, we looked further into this type of pairs, discriminating between verb (V) and non-verb (NV) events. Table~\ref{nominal_fq} (right) shows the results in more detail.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Table 6
\begin{table}[b]
\hangcaption{Distribution of misclassified \textsc{contains} and \textsc{overlap} pairs by type of \textsc{tlink} (left) and \textsc{event} type (right)}
\label{nominal_fq}
\input{10table06.tex}
\par\vspace{4pt}\small
Abbreviations: E--\textsc{event}, T--\textsc{timex3}, V--Verb and NV--Non-Verb
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

As mentioned by \citeA{Pustejovsky2011} and further discussed in \citeA{Styler2014}, \textsc{event}-\textsc{event} pairings are a complex and vital component, particularly in clinical narratives, where doctors rely on shared domain knowledge and it is essential to read ``between the lines.'' The distribution of verb/non-verb entities in Table~\ref{nominal_fq} (right) indicates that most \textsc{event}-\textsc{event} misclassified pairings were either of NV-NV type or included a NV entity. This finding is of prime relevance to temporal reasoning since temporality is naturally encoded in verbs, expressing actions or events, while nouns are usually the person or thing doing or receiving that action (i.e., subject or object). Without a verb, the semantics of nouns hardly give a notion of time. Consider the following sentences:
\vspace{0.5\Cvs}
\enumsentence{
\textbf{Tumor} \textit{invades} into the muscularis propria.
\label{ex:sent1}
}
\enumsentence{
Resected cecal \textbf{adenocarcinoma} with \textit{resection} of liver metastasis.
\label{ex:sent2}
}
\vspace{0.5\Cvs}
In sentence~\ref{ex:sent1}, the NV entity ``tumor'' is annotated to overlap with the V entity ``invades.'' Similarly, in sentence~\ref{ex:sent2} ``adenocarcinoma'' is annotated to overlap with ``resection.'' To understand how the time interval of these \textsc{event} entities overlap, we inevitably look for an associated action to picture their duration, but the sole definition of \textit{tumor} or \textit{adenocarcinoma} does not provide us with that information. Clearly, it is not until we attribute these two nouns the property of \textit{being present} (i.e., to exist) that we can think about a \textit{state}, which has no inherent endpoint. This forced reasoning is not straightforward, even for humans. This depicts the fact that it could be even harder for computers to process. 

Considering that verbs can be nominalized, we looked for nominalizations in our system's misclassified sentences. We observed some NV entities such as ``consultation,'' ``diagnosis,'' ``discharge,'' ``examination,'' and ``resection,'' which derive from the verbs ``to consult,'' ``to diagnose,'' ``to discharge,'' ``to examine,'' and ``to resect.'' However, it was more common to find NV entities like ``cancer,'' ``diabetes,'' ``history,'' ``anesthesia,'' and ``dialysis,'' just to name a few. These entities are a good example of non-events being interpreted in terms of their (implicit) associated action. The THYME project defines an \textsc{event} as \textit{anything} relevant to the clinical timeline. This interpretation is broader than the one originally defined by \citeA{pustejovsky2005specification}, where the term \textsc{event} considers anything that \textit{happens or occurs}, and is generally expressed by means of tensed or untensed verbs, nominalizations, adjectives, predicative clauses, or prepositional phrases.  Consequently, the THYME project definition allows non-events, such as medications or disorders, to be annotated as \textsc{event}s. While \citeA{Styler2014} mentioned this, they did not show the frequency of nouns.

As we saw in previous examples, the time intervals of NV entities are more difficult to conceptualize, while V entities, such as ``removed'' or ``improving,'' have their time properties morphologically encoded. Therefore, regardless of the low number of V-V relations, temporal information from verb predicates usually have more explicit hints; NV entities are more challenging and require more careful examination.

In Section~\ref{subsec:aspect}, we pointed out the high similarity of \textsc{overlap} and \textsc{contains} as one of the challenges of the temporal relation extraction task. Here we conclude that the high frequency of NV entities and the complexity of noun-noun relations is likely to be another reason why our system and previous works lag behind human performance. Not even the model of \citeA{Miwa2016}, which was designed to extract noun-noun relations, was able to handle the \textsc{tlink}s in Clinical TempEval. As was noted earlier, the semantics of nouns are not enough to give the notion of an \textsc{event} duration. This directly affects our system's performance.

We already introduced Vendler's aspectual classification and discussed how it helps to separate two extremely similar \textsc{tlink}s. Unfortunately, this is not compatible with nominal predicates. In order to be able to use Vendler's topology in the clinical domain, the currently implicit associated actions to nominal \textsc{event}s would have to be manually made explicit. Assuming that once identified, the verbs were classified as an \textit{activity}, \textit{accomplishment}, \textit{achievement}, or \textit{state}, this information can be used as a feature vector by our model. Alternatively, verb/non-verb entities distinction of \textsc{event}s is the first step that can alleviate the incompatibility of aspect with nominals, and positively influence the temporal relation extraction task. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Precision and Recall Imbalance}

All the experiments introduced in Section~\ref{sec:expset} outperformed the best Clinical TempEval 2016 system and the state-of-the-art model as shown in Table~\ref{comparable} and Table~\ref{results-table}. However, in all the settings, our model showed high performance, save for a recall lower than all of the systems presented in Table~\ref{comparable}. Previously, we attributed  this to the high number of negative instances in our dataset. As seen in Table~\ref{results-table}, filtering some of the negative instances resulted in a slight increase in recall, but we were still unable to reach a better balance with precision. Moreover, our best recall score was still below the one achieved by \citeA{Lee2016} and \citeA{Lin2016}. Consequently, we present an additional set of experiments to complement the multi-classification results in Table~\ref{results-table}; i.e., training the model without the \textsc{none} class.\footnote{We did not remove the \textsc{none} class from the test set.} These results are shown in Table~\ref{class-none}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Table 7
\begin{table}[b]
\caption{Results of our multi-class classification experiments on the THYME test set}
\label{class-none}
\input{10table07.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We can observe that the overall performance of our model remains the same. The classification of \textsc{contains} remains the highest, followed by \textsc{before}. The performance of the \textsc{ends-on} and \textsc{overlap} varies between the third and fourth best: \textsc{ends-on} is the third best under the \textit{Wikipedia without None} and \textit{PubMed word embeddings}, while \textsc{overlap} is under the \textit{Wikipedia word embeddings} and \textit{PubMed without None}. \textsc{begins-on} showed the lowest performance. Removing the \textsc{none} class resulted in a better balance between precision and recall, increasing the latter for all classes. Consequently, the F1 score increased as well. Despite this change, the \textsc{overlap} still showed the second lowest performance. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Class oversampling}

At the beginning of these experiments, we assumed that the dataset size was not one of the underlying reasons of the low performance witnessed in previous works. Our error analysis indicated that low performance can be attributed to two reasons: the high similarity of \textsc{overlap} with \textsc{contains} and the temporality of nominal events. Since we are using a neural model, it is natural to think that the more instances we have, the better the classification. 
We explored this assumption using four new experimental settings:
\begin{enumerate}
\item \textbf{Oversampling OVERLAP:} Given that \textsc{overlap} has one of the lowest performance, even though it is the second most frequent class in the dataset, we balance \textsc{overlap} with \textsc{contains} by random oversampling instances in the train set (ratio 1:1).
\item \textbf{Oversampling OVERLAP (without None):} Same setting as (I). The \textsc{none} class is removed for training but we maintain it for purposes of testing.
\item \textbf{Oversampling all the minority classes:} For better comparison, we balance all the minority classes (\textsc{before}, \textsc{begins-on}, \textsc{ends-on}, and \textsc{overlap}) with \textsc{contains} by random oversampling instances in the train set (ratio 1:1).
\item \textbf{Oversampling all the minority classes (without None):} Same setting as (III). The \textsc{none} class is removed for training but we maintain it for purposes of testing.
\end{enumerate}

The results are shown using default Wikipedia word embeddings and PubMed word embeddings in Table~\ref{oversampling}. Once again, we observe that the overall performance of our model remained the same. In the best case scenario, \textsc{overlap} has the third best performance. However, it continues to show similar performance with the remaining minority classes. Similar to our first set of experiments, training with the \textsc{none} class results in high precision but low recall. In line with the results of Table~\ref{class-none}, by removing the \textsc{none} class from the training set we get a better balance. Both oversampling techniques resulted in close Macro-F1 scores whenever we removed the \textsc{none} class from training and changed the default Wikipedia word embeddings for PubMed word embeddings (0.449 vs. 0.449 and 0.468 vs. 0.465, respectively). A McNemar's test \cite{mcnemar1947note} on these results yields a p-value of 0.077 for \textit{Wikipedia without None} and \textit{PubMed without None} oversampling \textsc{overlap} and a p-value of 0.466 when oversampling all minority classes. This is not significant at the 0.05 alpha level. The reason might be the same as why in-domain embeddings had a limited improvement in Section~\ref{sec:expset} multi-class classification experiments (see Appendix~\ref{app:emb}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Table 8
\begin{table}[t]
\caption{Results of our multi-class classification experiments on the THYME test set}
\label{oversampling}
\input{10table08.tex}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusions}

Clinical language processing represents a special challenge to NLP systems. The structure of clinical texts range from telegraphic constructions to long utterances describing a patient's condition or a suggested diagnosis. The high use of domain knowledge to infer temporal relations between events does not make this task any easier. A doctor naturally interprets adenocarcinoma (a type of cancer) as an abnormal, uncontrolled and \textit{progressive} growth of tissue, which temporally speaking is and should be thought as an ongoing process unless explicitly qualified (\textit{``We resected the adenocarcinoma, and since margins were clear, \underline{we can say it is gone}''}). This is a non-trivial task for a computer even when relying on context information. 

There have been several attempts on tackling temporal relation extraction from clinical text, mostly led by the Clinical TempEval challenges. However, the results are still far from human performance and there is little information about the underlying reasons. This encouraged our work to adapt a state-of-the-art system and do a detailed error analysis, which pointed out that one of the major challenges is how to handle the eventive properties of nominals---the predominant type of events on the most frequent type of pairs (\textsc{event}-\textsc{event}).

Existing knowledge bases, such as the Unified Medical Language System's (UMLS) Metathesaurus help to classify entities into semantic types like \textit{Therapeutic or Preventive procedure}, \textit{Sign or Symptom} or \textit{Disease or Syndrome}. However, the associated events and actions cannot be found in this or any other knowledge base. Therefore, we hypothesize that a resource containing aspectual information of the actions associated to common nominals, such as procedures or diseases, can further improve temporal relation extraction in the clinical domain. Since this will require manual annotation effort from annotators with linguistic and clinical knowledge, we first plan to analyze further \textsc{event}-\textsc{event} relations by differentiating events as verbal and non-verbal events.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

This work was supported by JST CREST Grant Number JPMJCR1513, Japan. We would like to thank the anonymous reviewers for their insightful comments and suggestions.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.5}
\bibliography{10refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\appendix

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Adaptation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Sentence-level Annotations}
\label{app:annotations}

We used the Clinical Language Annotation, Modeling and Processing (CLAMP) toolkit\footnote{\url{http://clinicalnlptool.com/index.php}} for tokenization and sentence boundary detection. We then matched all entities' spans from the gold standard with the sentence offsets on the CLAMP output to identify those within the same sentence.  Therefore, we created new annotations containing a pair of words, their offsets in the sentence, the temporal relation between them marked on the gold standard, and the directionality of the arguments. Example~\ref{ex:annotation} shows an annotation of the \textsc{tlink}---\textit{\textsc{contains}(lifelong, nonsmoker)} in the sentence \textit{``He is a lifelong nonsmoker.''} Note that no entity type (i.e., \textsc{event} or \textsc{timex3}) or any of its associated attributes are included. Our code for generating these sentence-level annotations is available at \url{https://github.com/cl-tohoku/thyme-sentence-annotations}.\footnote{Note that it will be necessary to request access to the THYME corpus. Please follow the instructions on the THYME wiki: \url{https://clear.colorado.edu/TemporalWiki/index.php/Main_Page#Getting_access_to_the_THYME_corpus_and_gold_standard_annotations}} 
%%%% Example 1
\enumsentence{
\small
\begin{tabular}{lll}
& \\
T1 & Term 8 16 & lifelong\\
T2 & Term 17 26 &  nonsmoker\\
R1 & ContainsSource-ContainsTarget & Arg1:T1 Arg2:T2 \\
& \\
\end{tabular}
\label{ex:annotation}
}
%%%%
The THYME corpus does not identify instances where two entities have none of the \textsc{tlink} relations. Hence, we define a \textsc{none} label and apply it as follows: since any two \textsc{event/timex3} can be a candidate pair, we take all entities in a sentence to generate all pair permutations as candidates. Pairs that do not have any temporal relation are then labeled as \textsc{none}. Therefore, for entities $e_1$, $e_2$, and $e_3$ in a sentence where $e_1$ \textsc{contains} $e_2$, pair ($e_1$, $e_2$) is considered as a positive instance while the resulting candidate pairs from our procedure ($e_1$, $e_3$), ($e_2$, $e_1$), ($e_2$, $e_3$), ($e_3$, $e_1$), and ($e_3$, $e_2$) are considered as negative instances. Due to the large number of negative instances produced (1:3 ratio of positive to negative examples), for a sentence with entities $e_4$, $e_5$, and $e_6$ with a \textsc{tlink}: $e_5$ \textsc{before} $e_6$---(or any \textsc{tlink} but \textsc{contains})---no negative instances were generated.
\subsection{Implementation and Training}
This study used \citeauthor{Miwa2016}'s \citeyear{Miwa2016} implementation, available at \url{https://github.com/tticoin/LSTM-ER}.
It followed their training settings, updating the model parameters (including weights, biases, and embeddings) by backpropagation through time \cite{werbos1990backpropagation} and Adam \cite{kingma2014adam} with gradient clipping, parameter averaging, and L2-regularization. Dropout \cite{srivastava2014dropout} was applied to the embedding layer and to the final hidden layers for relation classification. The hyper-parameters used were their default hyper-parameters for the SemEval-2010 Task 8: initial learning rate (1e-6), regularization parameter (1e-6), input dropout probability (0.5), output dropout probability (0.3), gradient clipping size (1), and number of epochs (63). 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Multi-Classification Performance}
\label{app:results}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{In-Domain Word Embeddings}
\label{app:emb}

Once we verified the adopted model gave competitive results on the narrative container identification task, we focused on increasing the system's recall. We, therefore changed the default word representations trained on Wikipedia for in-domain word embeddings. Word representation depends on the words in context, and since the clinical domain is a specific field with different vocabulary from those used in the general domain, we expected the model to benefit from a resource like PubMed. However, our results suggest that this does not have a significant impact on most \textsc{tlink}s. Only \textsc{begins-on} and \textsc{ends-on} recall considerably improved. This limited improvement can be attributed to the data size. The subset of PubMed abstracts used to train our in-domain word embeddings is smaller than the Wikipedia dump on which \citeauthor{Miwa2016}'s \citeyear{Miwa2016} default word embeddings were trained. A possible method of improving word representation in a temporal-aware context is to rely on transfer learning. For this purpose, pre-training BERT \cite{devlin2018bert} on general domain temporal data and fine-tuning on the Clinical TempEval task could lead to interesting results. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Down-Sampling Negative Examples}

We still witnessed an imbalance between precision and recall despite the fact that we increased recall by using in-domain word embeddings. Moreover, our results are still below UTHealth's recall score (highest on \textsc{contains} identification task). By filtering \textsc{event} pairs as described in Section~\ref{sec:expset} experimental setting (IV), the \textsc{none} class was reduced by 10\%. This further improved the recall for most \textsc{tlink}s except for \textsc{ends-on}. A McNemar's test on the results of \textit{PubMed word emb} and \textit{PubMed word emb + FNE} yields a p-value of 0.006, which is significant at the 0.05 level. This means that by filtering negative examples, the model's proportion of errors decreases with respect to \textit{PubMed word emb}. 



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Diana Galvan-Sosa}{%
Received the bachelor's degree in computer science and technology from the Monterrey Institute of Technology and Higher Education, Mexico, in 2010, and the M.Sc degree in information science from Tohoku University, Japan, in 2018, where she is currently pursuing the Ph.D. degree in information science. Her research interests include natural language processing, especially knowledge acquisition for machine reading comprehension. 
}

\bioauthor[:]{Koji Matsuda}{%
Received his bachelor's degree in engineering from Toyohashi Institute of Technology in 2006, and his M.E. degree in engineering from the Tokyo Institute of technology in 2012. He is currently a Ph.D. candidate at the Graduate School of Information Sciences, Tohoku University. He also works as Technical Staff in RIKEN since 2018. His research interests include natural language processing, especially automatic knowledge base construction and relation extraction.
}

\bioauthor[:]{Naoaki Okazaki}{%
Professor at Department of Computer Science, School of Computing, Tokyo Institute of Technology. He obtained his Ph.D. from Graduate School of Information Science and Technology, University of Tokyo in 2007. He worked as a post-doctoral researcher in University of Tokyo (in 2007--2011) and as an associate professor in Tohoku University (in 2011--2017). He is also a visiting research scholar of the Artificial Intelligence Research Center (AIRC), AIST. His research interests include natural language processing, text mining, and machine learning.
}

\bioauthor[:]{Inui Kentaro}{%
Professor of the Graduate School of Information Sciences at Tohoku University, where he is head of the Natural Language Processing Lab. He also leads the Natural Language Understanding Team at the RIKEN Center for the Advanced Intelligence Project. His research areas include natural language processing and artificial intelligence. He currently serves as Vice-chairperson of Association for Natural Language Processing, Member of Science Council of Japan, and Director of NPO FactCheck Initiative Japan.
}
\end{biography}

\biodate



\end{document}

