\documentclass[english,tombow]{jnlp_3.3}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{gentombow}
\settombowbleed{5mm}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
\usepackage{bm}
%%%%\usepackage{subcaption}
%%%%\usepackage{amsfonts}
%%%%\usepackage{url}
\usepackage{booktabs}
%%%%\usepackage{float}
%%%%\usepackage{color}
%%%%\usepackage{xcolor}
%%%%\usepackage{soul}
%%%%% \usepackage{natbib}
%%%%\usepackage[round]{natbib}
%%%%\bibliographystyle{abbrvnat}
%%%%% \setcitestyle{authoryear,open={((},close={))}}
%%%%\usepackage{makecell}
\newcommand{\tabincell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}
\renewcommand\vec[1]{\overrightarrow{#1}}
\newcommand\cev[1]{\overleftarrow{#1}}
%%%%\newcommand*{\ho}[1]{\textcolor{magenta}{Ouchi {\bf #1}}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\Volume{28}
\Number{1}
\Month{March}
\Year{2021}
%%%%\Issuetitle{「　　　　」特集号}


\received{2020}{8}{1}
\revised{2020}{11}{8}
%%%%\rerevised{2006}{5}{6}
\accepted{2020}{12}{21}

\setcounter{page}{183}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Data Augmentation by Rubrics for Short Answer Grading}
\eauthor{Tianqi Wang\affiref{Author_1}\affiref{Author_2} \and Hiroaki Funayama\affiref{Author_1}\affiref{Author_2} \and Hiroki Ouchi\affiref{Author_2} \and Kentaro Inui\affiref{Author_1}\affiref{Author_2}}
\eabstract{%
Short Answer Grading (SAG) is the task of scoring students' answers for applications such as examinations or e-learning. Most of the existing SAG systems predict scores based only on the answers, and critical evaluation criteria such as rubrics are ignored, which plays a crucial role in evaluating answers in real-world situations. In this paper, we propose a semi-supervised method to train a neural SAG model. We extract keyphrases that are highly related to answers scores from rubrics. Weights to words of answers are calculated as attention labels instead of manually annotated attention labels, based on span-wise alignments between answers and keyphrases. Only answers with highly weighed words are used as attention supervision. We evaluate the proposed model on two analytical assessment tasks of \textit{analytic score prediction} and \textit{justification identification}. Analytic score prediction is the task of predicting the score of a given answer for a prompt, and Justification identification involves identifying a justification cue in a given student answer for each analytic score. Our experimental results demonstrate that both performance of grading and justification identification is improved by integrating attention semi-supervised training, especially in a low-resource setting.
}
\ekeywords{SAG, Rubrics, Semi-Supervision, Short Answer Grading, Short Answer Scoring, Data Augmentation}

\headauthor{Wang et al.}
\headtitle{Data Augmentation by Rubrics for Short Answer Grading}

\affilabel{Author_1}{}{Tohoku University}
\affilabel{Author_2}{}{RIKEN Center for Advanced Intelligence Project}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\label{sec:intro}

In a pedagogical setting, assignments (e.g., examinations, homework, etc.) are crucial for assessing students' knowledge.
Students are generally required to answer questions in relatively short texts.
Educators can have a clear idea of how well the students are learning from their answers.
Simultaneously, such assignments allow students to detect and correct their errors and misunderstandings \cite{sychev2020automatic}.
Figure~\ref{fig:prompt_ans} shows a typical example
where students are asked to explain a sentence that appears in an essay to test their reading comprehension.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[p]
\begin{center}
\includegraphics{28-1ia8f1.pdf}
\end{center}
\hangcaption{A typical example of a prompt to a provided essay to test the reading comprehension of students. An example of as student's answer is also shown.}
\label{fig:prompt_ans}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[p]
\begin{center}
\includegraphics{28-1ia8f2.pdf}
\end{center}
\hangcaption{%
An example of rubrics explaining the rules for scoring.
There are four items labeled from A to D to score.
Key elements are defined by rubrics with some keyphrases (in quote marks) for each item,
and answers get points if it contains any key element.
Answers are scored by each item, and the final score is calculated as the sum of item scores.
}
\label{fig:rubrics}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

For grading assignments, each question, henceforth \textit{prompt}, is provided along with scoring rubrics.
Educators typically assess student answers following the scoring rubrics.
Figure~\ref{fig:rubrics} gives an example.
The rubrics explain in detail the rules to grade each item.
Student answers are scored on four different criteria, labeled from A to D in the reference answer.
The holistic score of an answer is assumed to be the sum of all the item scores.

One of the crucial ingredients of rubrics is the key elements it provides.
\textbf{Key elements} are concepts or information defined by the rubrics
that an answer needs to contain to receive points.
Rubrics provide some keywords or keyphrases as examples of the key elements.
For instance, according to the rubrics shown in Figure~\ref{fig:rubrics},
the phrase 西洋では(\textit{in the western world}) is a keyphrase to define a key element for item A,
thus answers will receive 2 points if they contain a similar phrase.
At the same time, rubrics provide some examples of incorrect keyphrases that a student answer may contain.
(e.g., 0 points for the phrase 外国(\textit{foreign countries}) for item A).

Due to the diversity of open-ended questions in answer scoring,
educators' workload can be excruciatingly time-consuming,
especially for online courses with a large number of students.
The task of Short Answer Grading (SAG) has been proposed to assist educators with grading.
SAG is the task of estimating scores of short-text answers written as an answer to a given prompt,
on the basis of whether the answer satisfies the rubrics prepared by a human in advance \cite{mohler2011learning,funayama2020preventing,mizumoto2019analytic}.
The SAG systems play a central role in providing stable and sustainable scoring in repeated
and large-scale examinations and (online) self-study learning support systems \cite{attali2006automated,mizumoto2019analytic,shermis2010automated,leacock2003c,burrows2015eras}.
Two analytical assessment tasks of SAG are formalized in \cite{mizumoto2019analytic}:
i) \textit{analytic score prediction} and ii) \textit{justification identification}.
Analytic score prediction gives scores to answers,
and justification identification identifies answer segments that contribute to the scoring,
showing the reasons for the corresponding prediction of analytic scores.
We discuss the details of task settings in Section~\ref{sec:task}.

SAG has been studied mainly with machine learning-based approaches.
The task is considered as inducing a regression model from a given set of manually scored sample answers (e.g., training instances).
As observed in a variety of other NLP tasks,
recently proposed neural models have been yielding strong results \cite{riordan2017investigating,mizumoto2019analytic}.

SAG is generally a low-resource task, that the scarcity of training data is an issue for SAG tasks.
\cite{heilman2015impact} investigate the importance of the size of training data on non-neural SAG models with discrete features.
Recently proposed neural models for SAG are trained on each prompt independently \cite{riordan2017investigating,mizumoto2019analytic}.
Thus, the lack of data is a crucial issue for neural models as well.
For the two assessment tasks we introduced earlier, annotation of both scores and justification cues are needed to train the neural models.
The workload of annotation can be hefty, especially for the annotation of justification cues.
\cite{mizumoto2019analytic} developed a dataset with annotated analytic scores and an attention supervised model,
and the experimental results show that the performance is improved by attention supervision.
However, the model is trained on each prompt,
thus it is necessary to prepare annotated data as attention supervision for each prompt,
leading to a high cost even in low resource settings.
To solve this issue, we introduce rubrics to SAG model instead of using annotated justification cues,
in order to reduce the cost of the annotation of justification cures.

This paper is the first study that explores how to incorporate rubric information into neural SAG models in order to reduce the required amount of training data.
We extract keyphrases that are highly related to the scores of answers from rubrics,
and calculate the weights of the answers' words based on span-wise alignments between answers and the extracted keyphrases.
We then propose a semi-supervised method to train a neural SAG model.
Weights to highly weighed words are used as attention supervisory signals for semi-supervised learning,
instead of manually annotated data.

Our experimental results demonstrate that the performance of SAG is improved by the training data augmented with pseudo attention supervision,
especially in a low-resource setting.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Task and Evaluation}
\label{sec:task}

We evaluate the proposal model on two analytical assessment tasks of SAG formalized in \cite{mizumoto2019analytic}:
i) \textit{analytic score prediction} and ii) \textit{justification identification}.

\textbf{Analytic score prediction} is the task of predicting the score of a given answer for a prompt.
Given a number of answer texts such as those shown in Figure~\ref{fig:prompt_ans},
an SAG model is expected to output either scores \cite{mohler2011learning}
or labels (e.g. \textit{correct}, \textit{natural}, \textit{incorrect}, etc.) \cite{dzikovska2013semeval}.
Given a student answer that consists of $n$ tokens, $t^{ans}_{1:n}=(t^{ans}_1, t^{ans}_2, ..., t^{ans}_n)$,
the goal is to predict the analytic score $s\in \mathbb R$.

As shown in Figure~\ref{fig:rubrics}, an answer gains analytic scores for each item (e.g. (A)$\sim$(D))
following corresponding analytic criterion (\textit{Try one's best} for 3 points, e.g.).
In this paper, we train the model on each item independently to predict the analytic scores.
To evaluate the precision of analytic score prediction,
we use quadratic weighted kappa (QWK) \cite{cohen1968weighted},
which is commonly used in the SAG literature.

\textbf{Justification identification} involves identifying a justification cue in
a given student answer for each item as interpretation to the prediction of the analytic score.
A justification cue is a segment of an answer that contributes to the analytic score.
An example of justification cues for item D is shown in Figure~\ref{fig:justification}.
As the manually annotated justification cue,
the phrase 言葉を尽くして(\textit{try their best}) in the answer refers to 言葉を尽くして(\textit{try one's best}) in the rubric,
and 自分の考えを伝える(\textit{convey their ideas}) refers to （自分の考えを）伝えようとする(\textit{convey one's thoughts}).
Formally, given a student answer $t^{ans}_{1:n} = (t^{ans}_1, t^{ans}_2, ..., t^{ans}_n)$,
the goal is to identify the segment $p_{a:b}=(t^{ans}_a, t^{ans}_{a+1},..., t^{ans}_b)$,
where $1 \le a \le b \le n$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia8f3.pdf}
\end{center}
\hangcaption{%
An example of justification cues in a student answer for item D.
There are seven keyphrases defined in the rubric.
Tokens in green are manually annotated justification cues.
Tokens in yellow are justification cues generated by answers and key elements.
A deeper color represents higher relatedness of the justification cue to the key elements.
}
\label{fig:justification}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In this paper, to evaluate performance on justification identification,
we predict justification cues depending on the predicted attention,
and calculate F1 score following the method in \cite{mizumoto2019analytic} for evaluation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}
\label{sec:related}

Many existing SAG studies have mainly focused on exploring better representations of answers and similarity
measures between student answers and reference answers.
A wide variety of methods have been explored so far,
ranging from Latent Semantic Analysis (LSA) \cite{mohler2011learning}, edit distance-based similarity,
and knowledge-based similarity using WordNet \cite{pedersen2004wordnet,magooda2016vector}
to word embedding-based similarity \cite{sultan2016fast}.
It is reported in \cite{riordan2017investigating} that neural network-based feature representation learning \cite{taghipour2016neural} is effective for SAG.

In contrast to the popularity of learning answer representations,
the use of rubric information for SAG has been received little attention so far.
In \cite{Sakaguchi:15:NAACL}, the authors compute similarities, such as BLEU \cite{papineni2002bleu},
between an answer and each key element in a rubric,
and use them as features in a support vector regression (SVR) model.
Text patterns are generated from reference answers and rubrics in \cite{ramachandran2015identifying},
and the results demonstrate that the automatically generated pattern outperforms the manually generated regexp pattern.

Simultaneously, data augmentation techniques have been developed to achieve better performance with a limited dataset.
\cite{mintz2009distant} assume that if two entities participate in a relation,
all sentences that mention these two entities express that relation.
The assumption is relaxed by \cite{riedel2010modeling} as expressed-at-least-once assumption,
saying that if two entities participate in a relation,
at least one sentence that mentions these two entities might express that relation.
To augment the dataset for the relation classification task,
\cite{ye2019distant} found all sentences containing those entities for each pair of entities that appear in some Freebase relation in a large unlabeled corpus extract textual features to train the classifier.
\cite{wei2019eda} proposed an Easy Data Augmentation (EDA) method.
EDA consists of four operations: synonym replacement, random insertion, random swap, and random deletion.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Neural baseline model}
\label{sec:neural base}

We take the simple but effective neural model \cite{taghipour2016neural,riordan2017investigating}
as the baseline of the neural model in this paper, as is shown in Figure~\ref{fig:key_idea}(a).
Given an answer of $n$ tokens, $t^{ans}_{1:n} = (t^{ans}_1, t^{ans}_2, \dots, t^{ans}_n)$,
the embedding layer outputs an embedding vector $\bm{t}_i^{ans} \in \mathbb{R}^{d^\mathrm{token}}$
for each token $t^{ans}_i \in t^{ans}_{1:n}$.
Taking a sequence of these vectors $\bm{t}^{ans}_{1:n} = (\bm{t}_1^{ans}, \bm{t}_2^{ans}, \dots, \bm{t}_n^{ans})$ as input,
the Bi-LSTM layer then produces a contextualized vector $\bm{h}_i = [\vec{\bm{h}_i};\cev{\bm{h}_i}]$
by concatenating the forward hidden state of the Bi-LSTM $\vec{\bm{h}_i} \in \mathbb{R}^{d^\mathrm{lstm}}$
and the backward one $\cev{\bm{h}_i} \in \mathbb{R}^{d^\mathrm{lstm}}$.
The attention layer then assigns an attentional weight $\alpha_i$ to each contextualized vector $\bm{h}_i$ as follows:
\begin{align}
\nonumber e_i & = \bm{h}^\top_i \bm{W} \bm{m} ,\\
\label{eq:attention}\alpha_i & = \frac{\exp(e_i)}{\sum^n_{k=1} \exp(e_k)} ,
\end{align}
where $\bm{W} \in \mathbb{R}^{2d^\mathrm{lstm} \times d}$ is trainable parameter matrix
and $\bm{m} \in \mathbb{R}^{d}$ is a trainable parameter vector.
Then the regression layer computes an analytic score $s$ by the weighted sum:
\begin{align}
\nonumber \bm{f} &= \sum_{i=1}^n \alpha_i \bm{h}_i,\\
\nonumber s &= \operatorname{sigmoid}(\bm{w}^\top \bm f + b),
\end{align}
where $\bm{w} \in \mathbb{R}^{d}$ is a trainable parameter vector and $b \in \mathbb{R}$ is a trainable bias parameter.
Owing to the sigmoid function, each analytic score takes a value from 0 to 1.
Analytic scores are rescaled back to the original score range before evaluation on QWK.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 4
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia8f4.pdf}
\end{center}
\hangcaption{%
The architecture of baseline models and the proposed method.
$t^{ans}_i$ and $t^{key}_i$ are the $i^{th}$ word of answer and keyphrases.
For the attention-unsupervised baseline model, only analytic scores are passed as supervisory signals.
Manually annotated justification cues are used as attention supervisory signal for the attention-supervised model,
and manually annotated data is used for our proposed model.
We search for justification cues automatically based on matching between answers and keyphrases,
and the backpropagation of attention happens only when justification cues are found in the training data.
}
\label{fig:key_idea}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The attention mechanism (Equation~\ref{eq:attention}) is used not only for analytic score prediction
but also for justification identification \cite{mizumoto2019analytic}.
Specifically, based on the attention values $\alpha_i$ for each token $t^{ans}_i$ in Equation~\ref{eq:attention},
we extract a set of token indices as justification cues $\mathcal{C}$:
\begin{gather*}
\alpha_{max}=\max_{i=1, \dots, n}\alpha_i, \\
\mathcal C = \{i\in[1, n]|\alpha_{max} - \alpha_i < \gamma\},
\end{gather*}
where $\gamma$ is a hyperparameter selected by using the development set.

To train the model, the mean squared error (MSE) is used as the loss function to minimize:
\begin{equation}
\nonumber l=\frac{1}{N}\sum_{j=1}^N(s^\mathrm{gold}_j-s_j)^2
\end{equation}
where $s^\mathrm{gold}_j$ and $s_j$ are the gold and predicted scores for the $j$-th answer, respectively.
$N$ is the number of the answers in the training set.

For justification cue identification,
\cite{mizumoto2019analytic} provides a dataset with manual annotations for ground truth justification cues,
and the model can be trained by using them as attention supervision, as shown in Figure~\ref{fig:key_idea}b.
The loss is calculated as follows:
\begin{equation} \label{eq:loss}
l=\frac{1}{N}\sum_{j=1}^N(s^\mathrm{gold}_j-s_j)^2 + \frac{1}{Nn}\sum_{j=1}^N\sum_{i=1}^n(\alpha^\mathrm{gold}_{j,i} - \alpha_{j,i})^2
\end{equation}
where $\alpha^\mathrm{gold}_{j,i}$ and $\alpha_{j,i}$ are the manually annotated attention
and predicted attention for the $i$-th token in the $j$-th answer,
and $n$ is the number of tokens in each answer.
We take this attention supervised method as a reference for comparison on performance in this paper.

In \cite{mizumoto2019analytic}, the SAG model is designed to compute the sum of analytic scores for each item as the holistic score for a given answer.
The performance of those models is evaluated in terms of the accuracy of predicted holistic scores.
In this study, however, we focus on the performance of predicting \emph{item-wise} analytic scores (rather than holistic scores)
to closely analyze the effects of using rubrics, which are provided for each item.
We, therefore, train models and evaluate their performance for each item separately.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed Method}
\label{sec:prop}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Key idea}
\label{sec:key_idea}

\cite{mizumoto2019analytic} reported an improvement in the performance of analytic score prediction from an attention supervised model.
The annotations for ground-truth justification cues $\alpha^\mathrm{gold}$ are used as supervisory signals to learn the attention mechanism.
For example, in Figure~\ref{fig:key_idea} (b), we have the supervisory signals for attention simply based on the annotated justification cues.
Attention to tokens of justification cues (言葉を尽くして自分の考えを伝える(\textit{try their best to convey their ideas})) is set to 1,
and attention to other tokens is set to 0.

Instead of manual annotation of justification cues,
we consider using the rubrics to automatically identify which span of a given answer should be attended.
If this identification can be carried out reasonably accurately,
the identified cues can be used as pseudo supervisory signals of attention to train the attention mechanism.

The idea is illustrated in Figure~\ref{fig:key_idea} (c).
For each input answer $t^{ans}$ from a training set,
we first search for the justification cues by matching every candidate span in $t^{ans}$ with a keyphrase $t^k$ from the rubric.
In the figure, the yellow shades depict the weights of spans of the identified justification cues.
If a span that is likely to be a justification cue is identified,
we then use it as a pseudo supervisory signal to train the attention prediction component.
Of course, such a span may not be identified because, for example,
(i) the list of keyphrases in the rubric may not exhaust all the potential variety of justification cues in answers,
or (ii) a given answer may not contain any contributive keyphrase.
In such a case, we do not use that instance for supervising attention prediction
but use only its gold score to train the overall model as in the attention-unsupervised baseline model (Figure~\ref{fig:key_idea} (a)).

Notice that the rubrics also provide some zero points keyphrases such as 「外国」は不可(`Foreigner' is invalid).
Even though those zero points phrases do not contribute any points to an answer,
we consider them as cues for mistakes that are easily made by students,
and feed into the SAG model as part of supervision to train the model to be able to distinguish mistakes.
However, because very little training data includes zero points keyphrases,
especially in low resource settings where the size of training data is limited,
no difference was observed by introducing zero points keyphrases during experiments.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 5
\begin{figure}[b]
\begin{center}
\includegraphics{28-1ia8f5.pdf}
\end{center}
\hangcaption{%
Span-wise matching between answers and keyphrases.
Given an answer of $n$ tokens $t^{ans}_{1:n} = (t^{ans}_1, \dots, t^{ans}_n)$,
span $p_{a:b}=(t^{ans}_a,...,t^{ans}_b)$ is a subsequence of the answer.
Weights of $p_{a:b}$ are then calculated as $w_{a:b}$ based on similarity between $p_{a:b}$ and keyphrase $k$.
The span $p^{k}$ with the highest similarity $w^{k}$ is considered as a justification cue for each keyphrase $k$.
We attach $w^{k}$ to the justification cue as attention value.
If a token is contained by more than one justification cue, we take the maximum value as the attention value.
In this example, （他人を）説得する is the keyphrase.
By matching each span of the answer to the keyphrase,
the best-matched span is identified as 他人を同意させる,
and the attention is set to 0.44 according to the similarity.
}
\label{fig:matching}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Data augmentation based on span-wise matching}
\label{sec:att_gen}

A more concrete example for data augmentation is given in Figure~\ref{fig:matching}.
First, answers and rubrics are tokenized by MeCab \cite{kudo2006mecab},
and keyphrases, such as （他人を）説得する, are extracted from the given rubrics.
We then search for justification cues by matching a given answer to the keyphrases.
Weights of answer tokens are calculated based on span-wise matching between answers and keyphrases as illustrated in the figure.

Given an answer of $n$ tokens $t^{ans}_{1:n} = (t^{ans}_1, \dots, t^{ans}_n)$,
we first enumerate all possible spans:
$\mathcal{P} = \{ _{a:b} | 1 \leq a \leq b \leq n\}$,
where each span $p_{a:b} = (t^{ans}_a, \dots, t^{ans}_b)$ is a subsequence of the tokens in the answer.
Each keyphrase is a sequence of $m$ tokens: $t^k_{1:m} = (t^k_1, \dots, t^k_m)$.
The similarity score $w_{a:b}$ between a keyphrase $t^k_{1:m}$ and each span $p_{a:b}$
is calculated by some similarity (distance) measure (e.g., Levenshtein distance): $w_{a:b} = \mathrm{sim}(t^k_{1:m}, p_{a:b})$.
Based on the similarities, we define the matching score $w^k$ to indicate
how well the keyphrase $t^k_{1:m}$ matches (or is entailed in) the answer.
Specifically, we define $w^k$ as the highest similarity score between the keyphrase and each span
$p_{a:b}$: $w^{k} = \max_{p_{a:b} \in \mathcal{P}} w_{a:b}$.
Also, we extract the span with the highest similarity as the pseudo justification cue:
$p^{k} = \mathrm{argmax}_{p_{a:b} \in \mathcal{P}} w_{a:b}$.

Here, based on the pseudo justification cues (spans) $p^{k}$,
we define a pseudo supervisory signal $\alpha^\mathrm{pseudo}_i$ of each token $t^{ans}_i$ in an answer $t^{ans}_{1:n}$ for learning the attention mechanism,
instead of the ground-truth $\alpha^\mathrm{gold}_i$ (Equation~\ref{eq:loss}).
Specifically, if a token $t^{ans}_i$ is contained in a span $p^{k} = (t^k_a, \dots, t^k_b)$,
we use the similarity score $w^{k}$ as the pseudo supervisory signal:
\begin{gather}\nonumber
\mathcal{W}_i = \{w^k | t^{ans}_i \in p^{k} | k \in (k_1, \dots, k_K) \}, \\
\label{eq:pseudo-attention}
\alpha^\mathrm{pseudo}_i = \begin{cases}
\max_{w^k \in \mathcal{W}_i} w^k,& \mathrm{if} \:\: |\mathcal{W}_i| \neq 0\\
0
\end{cases}
\end{gather}
In the former equation, we judge whether a token $t^{ans}_i$ is contained in a span $p^{k}$,
which is extracted by using each keyphrase $k \in (k_1, \dots, k_K)$,
and if so, we extract the similarity score $w^k$ associated with the span $p^{k}$.
A token may be contained in multiple spans,
thus we pick the highest one of the similarities $w^k \in \mathcal{W}_i$ in the latter equation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Semi-supervised learning with pseudo labels}
\label{sec:proposed}

We propose an attention semi-supervised method for SAG.
Only attention to justification cues similar to keyphrases will be used as pseudo signals for attention.
Given an answer $a$, if the maximum value of attention to justification cues is not high enough,
the answer will not be used for attention-supervised learning.
A binary label $\eta$ is attached to each answer $a$ to prepare training data for semi-supervised learning,
\[
\eta_j =\begin{cases}
1,&\operatorname{max}(\bm{\alpha}^\mathrm{pseudo}_j) \ge T\\
0,&\operatorname{max}(\bm{\alpha}^\mathrm{pseudo}_j) < T\end{cases}
\]
where $\bm{\alpha}^\mathrm{pseudo}_j = (\alpha_{j,1}, \cdots, \alpha_{j,n})$ is a set of the pseudo attention signals for the tokens
$(t^{ans}_{j,1}, \dots, t^{ans}_{j,n})$ in an answer $a_j$ generated by Equation~\ref{eq:pseudo-attention} and threshold $T$ is a hyperparameter.
We consider justification cues are matched in answers where $\eta=1$.

The loss function is designed as follows:
\[
l=\frac{1}{N}\sum_{j=1}^N(s^\mathrm{gold}_j-s_j)^2 + \frac{1}{Nn}\sum_{j=1}^N\sum_{i=1}^n(\eta_j\alpha^\mathrm{pseudo}_{j,i} - \eta_j\alpha_{j,i})^2
\]
where $N$ is the number of answers, and $n$ is the number of tokens in each answers.

Note that answer $a_j$ is considered containing no justification cues if $\eta_j=0$,
and the corresponding loss of attention is always 0.
In this way, only answers with a significant match to keyphrases are used for attention-supervised learning.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}
\label{sec:experiments}

Our research question is how much benefit we can gain from the proposed method by introducing rubrics to SAG models.
There are three targets we want to explore through experiments to answer this question:
i) what is the best settings of proposed attention semi-supervised training,
ii) what is the best way to import rubric to SAG models,
and iii) how much benefit we can obtain from the proposed semi-supervised method.
To address these questions, we first run the proposed model in a low-resource setting with various hyperparameters to find the best settings.
Then we propose four simple rubric-based models and compare the performance with our model.
Finally, we compare the proposed model with the attention unsupervised baseline model to show the improvement.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Settings}
\label{sec:settings}

We applied our proposed method on the dataset provided by \cite{mizumoto2019analytic}
\footnote{\url{https://aip-nlu.gitlab.io/projects/sas-j}}.
In total, the dataset includes 6 prompts (\textit{Q1}$\sim$\textit{Q6}),
with 1600 answers as training data, 250 answers as development data, and 250 answers as test data for each prompt.
Similar to the example shown in Figure~\ref{fig:rubrics}, answers are scored by multiple items,
and justification cues are manually annotated for each answer,
where the corresponding tokens are annotated to 1 while the other tokens are annotated to 0.
Three of the prompts (\textit{Q1}, \textit{Q2}, \textit{Q3}) each contain 4 items,
and the remaining prompts (\textit{Q4}, \textit{Q5}, \textit{Q6}) each contain three items, thus equating to 21 items to grade.

We pre-train 100-dimensional Word2Vec word embeddings \cite{mikolov2013distributed} on Japanese Wikipedia data to initialize the word embedding layer.
Little gain is obtained from fine-tuning the embedding layer according to \cite{riordan2017investigating};
thus we freeze the embedding layer during the training step in our experiments
to reduce the number of parameters to learn in low-resource settings.
The Bi-LSTM layer's dimension is set to $d^{lstm}=250$, and the dropout probability is set to 0.5.
The parameter $d$ for $\bm W$ and $\bm m$ of the attention layer is set to 100.
$\bm{W}$ and $\bm{m}$ are initialized randomly from normal distribution, with the standard deviation set to 0.01.

We train our model on each item, optimized with the Adam \cite{kingma2014adam} optimizer,
and the learning rate is set to 0.001.
Experiments are repeated for 5 times with different random seed from 0 to 4 for initialization,
and the average results across all random seeds are used as the final results.
To explore the performance in low-resource settings,
we train our model on various sizes of training data,
ranging from 48 to 1600 (3\%, 6\%, 12.5\%, 25\%, 50\%, 100\%).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Rubric-based baselines}
\label{sec:baselines}

To explore different ways to introduce rubric information to the SAG model,
we proposed three simple baseline models that score answers with rubric information,
and apply regular expression (regexp) matching instead of span-wise matching to our proposed method for comparison.

Rule-based model on top of regexp matching \textbf{(RGEXP-RULE)}:
As described in Figure~\ref{fig:rubrics}, an answer is scored by steps (D(1) and D(2) for item D).
On each step, corresponding points are assigned to the answer if the answer contains any required keywords,
and the sum of scores assigned in each step is output as the final score for the answer.
Following this rule, we first create regular expressions for keywords based on the rubrics.
For example, regex for `哲学(の営み)は' is constructed as `哲学(の営み)?は'.
Then we identify keywords in the given answer by regexp matching for each step,
and output the sum of corresponding scores of matched keywords in each step as the prediction of answer score.
Only the keywords with the highest score will be selected if multiple keywords are matched in any step.

Rule-based model on top of span-wise matching \textbf{(SPAN-RULE)}:
This method works similarly to \textbf{RGEXP-RULE} introduced above,
but a span-wise matching method based on edit-distance is used instead of regexp matching.
The details of the span-wise matching method are introduced in Section~\ref{sec:results}.
Note that there is a hyperparameter $T$ for span-wise matching.
We apply different values of $T$ from 0.1 to 1.0, and take the value leading to the highest mean QWK over all prompts.

Neural model with rubrics \textbf{(CONCAT-RGRS):}
As demonstrated in Figure~\ref{fig:neural_rubric_baseline}, we introduce rubric information to the neural model.
A number of keyphrases are extracted from rubrics.
We encode the answer and key elements by an encoder consisting of a word embedding layer,
a BiLSTM layer and an attention layer, with settings introduced in Section~\ref{sec:settings}.
The encoded answer $\bm{a}$ and encoded keyphrases $\bm{k}_1, \bm{k}_2, ..., \bm{k}_K$ are concatenated to $\bm{f}$,
and feed into a regression layer to predict the score of the given answer.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 6
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia8f6.pdf}
\end{center}
\caption{Neural model with rubrics.}
\label{fig:neural_rubric_baseline}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Our method with regexp matching \textbf{(RGEXP-RGRS)}:
Instead of proposed span-wise matching based on edit distance,
we apply regexp matching between answers and keyphrases to generate pseudo attention supervisory signals.
Then the model is trained with the augmented training data,
and with the same settings as introduced in Section \ref{sec:settings}.

The results are discussed in Section \ref{sec:rubric_baselines_performance}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results and Discussion}
\label{sec:results}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Settings of proposed model}

We generate pseudo supervisory signals for attention semi-supervised training
following the method introduced in Section \ref{sec:prop}.
To simplify the process, we calculate the weights of span $p_{a:b}$ based on
Levenshtein edit distance \cite{levenshtein1966binary} between $p_{a:b}$ and keyphrase $t^k_{1:m}$:
\[
w_{a:b}=1-\frac{\operatorname{edit\_distance}(p_{a:b}, t^k_{1:m})}{\operatorname{max}(b-a+1, m)}, 0\le w_{a:b}\le 1
\]
We limit the length of spans to $0\le b - a< 3$ to avoid lengthy spans,
and set $w_{a:b}$ to 0 if $w_{a:b}<T$ to avoid influence from low-weighted tokens,
as is shown in Figure~\ref{fig:att_thres}.

The size of training data as attention supervision is controlled by a threshold $T$.
With a low value of $T$, spans will be identified as justification cues
even if the similarity to keyphrases is low,
and most of the training data will be used as pseudo attention supervision.
On the other side, with a higher value of $T$,
very limited training data containing justification cues
that are similar enough to keyphrases will be used as attention supervision.

We compute MSE between predicted attention and attention supervision
as the loss of attention as we introduced in Section~\ref{sec:proposed},
thus we binarize labeled attention $\bm{\alpha}^\mathrm{pseudo}$ to $\bm{\beta}^\mathrm{pseudo}$
in order to enhance the effectiveness of attention supervision.
\[
\beta_{j,i}=\begin{cases}1, & \alpha_{j,i} > T\\0, & otherwise\end{cases}
\]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 7
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia8f7.pdf}
\end{center}
%%%%\label{fig:att0.0}
%%%%\label{fig:att0.5}
%%%%\label{fig:att0.9}
\hangcaption{%
Influence of $T$ to pseudo supervisory signal on attention.
Tokens marked by a deeper color are closer to key elements provided in rubrics.
With greater values of $T$, only tokens similar enough to keyphrases are used as supervisory signals.
}
\label{fig:att_thres}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

To demonstrate the influence of value $T$, we manually set $T$ to 0, 0.25, 0.5 and 0.9.
Figure~\ref{fig:semi_data} shows statistical information on answers containing justification cues.
The blue bars indicate the size of training data containing justification cues ($\eta=1$),
and the red polylines represent the MSE between $\bm{\beta}^\mathrm{pseudo}$ and manually annotated gold attention.
Note that MSE tends to be higher for lower $T$ values
because spans unrelated to key elements are considered as justification cues.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 8
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia8f8.pdf}
\end{center}
\hangcaption{%
Statistics of training data as attention supervision generated with various value of threshold $T$. The size of data is indicated by the blue bars. The red polylines show MSE of pseudo supervisory signals to manually annotated attention.
}
\label{fig:semi_data}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The performance of the proposed model in low-resource settings (3\% training data)
with various values of $T$ is shown in Figure~\ref{fig:performance_semi_low_res}.
Both performance with $\bm{\alpha}^\mathrm{pseudo}$ and $\bm{\beta}^\mathrm{pseudo}$ are shown for comparison.
The blue bars on left represent performance with $\bm{\alpha}^\mathrm{pseudo}$, the red bars on right represent performance with $\bm{\beta}^\mathrm{pseudo}$. We also show the performance of the ATT-UNSUP model and ATT-SUP model for reference.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 9
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia8f9.pdf}
\end{center}
%%%%\label{fig:semi_qwk_low_res}
%%%%\label{fig:semi_f1_low_res}
\hangcaption{%
Performance of the proposed model in a low-resource setting, where training size is set to 3\%, with various values of threshold parameter $T$ from 0.0 to 0.9. The blue bars on left represent performance with generated pseudo attention ($\bm{\alpha}^\mathrm{pseudo}$), and the red bars on right represent performance with the binary pseudo attention ($\bm \beta^\mathrm{pseudo}$). The performance of the ATT-UNSUP model and ATT-SUP model are for comparison.
}
\label{fig:performance_semi_low_res}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The results demonstrate that the neural SAG model benefits from attention semi-supervised training.
Both QWK for analytic score prediction and F1 score for justification identification
are improved compared to the attention unsupervised baseline model,
even though the pseudo supervisory signal for attention is not necessarily correct compared to manually annotated data.

The results also demonstrate that the performance of the proposed model changes with the values of $T$,
and performance with binary attention is more sensitive to $T$.
With a proper value of $T$, the binary pseudo supervisory signal for attention leads to better improvement in performance.
We select the value of $T$ for each prompt by following steps.
We run the proposed model in low-resource settings (3\% training data) with $\bm{\beta}^\mathrm{pseudo}$
and repeat the experiments with various values of $T$.
Values of $T$ leading to the best QWK on the development data for each prompt are selected,
listed in Table~\ref{tab:besth_threshold},
and will be used as the hyperparameter for other scales of training data.

We discuss the performance of proposed model in the following section.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[b]
\input{08table01.tex}
\caption{Best threshold T seleted based on development data.}
\label{tab:besth_threshold}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Performance of proposed model}

We compare the proposed method to the attention unsupervised baseline model (ATT-UNSUP).
Because the attention supervised model(ATT-SUP) is trained with manually annotated justification cues,
we show the performance for reference.
The difference in training data between the models is listed in Table~\ref{tab:settings},
and the mean results over all of the prompts are shown in Figure~\ref{fig:performance_semi_att_bin}.
The figure indicates that the baseline models achieved comparable performance to the reports from \cite{mizumoto2019analytic},
and the attention supervision improves performance on score prediction, especially in low-resource settings.
Performance on justification identification is also improved over all the training sizes.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[b]
\input{08table02.tex}
\caption{Training data of each model.}
\label{tab:settings}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Compared to the ATT-UNSUP model,
the performance is improved by semi-supervised learning with pseudo supervisory signals for attention,
especially in low-resource settings.
It is also important to mention that we achieved comparable performance to the baseline with a larger size of training data,
indicating that our proposed semi-supervised method does not harm the performance when a large amount of training data is available.
At the same time, F1 for justification identification is also improved through various training sizes,
%%%%as shown in Figure~\ref{fig:semi_f1_att_bin},
    as shown in Figure~\ref{fig:performance_semi_att_bin}(b),
indicating better interpretability of our proposed method on the scoring results.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 10
\begin{figure}[b]
\begin{center}
\includegraphics{28-1ia8f10.pdf}
\end{center}
%%%%\label{fig:semi_qwk_att_bin}
%%%%\label{fig:semi_f1_att_bin}
\hangcaption{%
PPerformance of the proposed model on various size of training data, shown in orange lines. The model is trained with proper hyperparameters. The ATT-UNSUP stands for attention the unsupervised baseline model, and the ATT-SUP stands for the attention supervised baseline model.
}
\label{fig:performance_semi_att_bin}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Still, ATT-SUP performs best overall, especially on justification identification.
One of the reason is the justification cues used as pseudo supervisory signals
are identified by a simple span-wise matching method based on edit-distance.
However, SPAN-RGRS model achieves a comparable performance to ATT-SUP,
without any requirement on manually annotation of justification cues.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[b]
\input{08table03.tex}
\hangcaption{Performance in low-resource settings. ATT-UNSUP is the attention unsupervised baseline model, and SPAN-RGRS is our proposed model.}
\label{tab:perform_prompts}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Performance trained with 3\% training data on each prompt is listed in Table~\ref{tab:perform_prompts}.
ATT-UNSUP represents the performance of the attention unsupervised baseline model,
and SPAN-RGRS represents the performance of our proposed model.
The table shows that the proposed model outperforms the ATT-UNSUP model for almost all of the prompts.
Note that the performance of both the baseline model and the proposed method vary with items
because of the diversity of prompts and answers.
The benefit we obtain from the proposed method also varies with items for the same reason.

For instance, the rubric for prompt Q1/A is simple that answers containing 西洋 or ヨーロッパ get 2 points.
Compared to other prompts, more benefit is obtained from our proposed method,
and the performance is close to the attention supervised baseline model.
Two reasons are leading to these results.
First, because the keyphrases included by the rubrics are simple and the number of keyphrases is limited,
the scoring rules are easy to learn.
Second, for the same reason, the generated pseudo attention signals are more reliable than other prompts.
The rubric for prompt Q1/D is more complicated, as is shown in Figure \ref{fig:rubrics}.
The answers are graded by two steps, and keyphrases are longer and more diverse.
Hence the improvement from our proposed method is limited.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Comparison against rubric-based baselines}
\label{sec:rubric_baselines_performance}

We compare the performance to rubric-based baseline models introduced in Section \ref{sec:baselines},
and the mean performances over all prompts in low resource settings (3\% training data) is listed in Table \ref{tab:rubric_settings}.
The performance on each prompt is shown in Figure \ref{fig:rubric_baseline_prompts}.
The prompts are sorted by the performance of our proposed SPAN-RGRS model.
The attention unsupervised neural baseline model ATT-UNSUP is included for reference.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[b]
\input{08table04.tex}
\caption{Performance of rubric-based models in low resource settings (3\%, 48 instances)}
\label{tab:rubric_settings}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 11
\begin{figure}[t]
\begin{center}
\includegraphics{28-1ia8f11.pdf}
\end{center}
\caption{Performance of rubric-based models on each prompt.}
\label{fig:rubric_baseline_prompts}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Firstly, note that CONCAT-RGRS performs close to ATT-UNSUP.
It demonstrates that it is vital to find a proper way to introduce rubric information to SAG models.
Directly encoding rubrics into features does not help improve the performance.
Secondly, SPAN-RULE performs much better than RGEXP-RULE,
indicating that the proposed span-wise matching method works better than regexp matching.
Similarly, SPAN-RGRS performs better than RGEXP-RGRS for the same reason.
Note that compared to RGEXP-RGRS, SPAN-RGRS model does not need manually designed regexp for matching,
but still achieves a better result compared to RGEXP-RGRS.
Furthermore, Figure~\ref{fig:rubric_baseline_prompts} shows that
both RGEXP-RULE and SPAN-RULE achieve outstanding performance for some specific cases (Q1/A, e.g.),
where the rubrics are simple, and keyphrases are easy to match to answers.
However, for prompts with complicated rubrics, keyphrases are more challenging to extract,
and answers are more diverse to match keyphrases, leading to a very low QWK.
As a typical example, the SPAN-RGRS model works well on Q6/B,
but the REGEXP-RULE model achieves a very deficient performance.
One of the rubrics for Q6/B is 幸福に比べて真実が軽視されていることがわかればよい
(\textit{The answer is acceptable if ``the truth is disregard compared to happiness'' can be entailed}).
Because of the open rubric, answers to Q6/B are more diverse than other prompts.
As a result, it is more difficult to identify correct justification cues by regular expression matching.
Thus most of the answers are scored to 0 by REGEXP-RULE model.
On the other side, the proposed span-wise matching method is more tolerant of justification cues.
Moreover, because we train an attention layer and a regression layer for the REGEXP-RGRS model to predict scores,
with not only justification cues but also answer score pairs,
the model gives relatively acceptable scoring results,
even for the cases when it fails to identify the correct justification cues. 
Hence, neural models are more stable than rule-based models.

\section{Conclusion}
Annotation of scores and justification cues are needed to train neural models for SAG.
Particularly, the annotation on justification cues brings a heavy workload to human annotators.
Considering that attention supervision is required for each prompt,
the preparation of training data is still time-consuming, even for low-source settings.

To reduce the workload on annotation,
we proposed a method to augment training data with rubrics,
and generate pseudo attention supervision instead of manually annotated data.
We identify justification cues by matching answers to keyphrases provided by rubrics.
Pseudo supervisory signals for attention are created based on matching scores,
and only justification cues that are significantly matched are used for attention supervision.

The proposed method improves the performance of SAG on both analytic score prediction and justification identification tasks,
compared to the attention unsupervised baseline model, especially in low-resource settings.
The performance is comparable to the attention supervised model,
but no additional work is required to prepare training data for attention supervision.

There is still a gap between the proposed method and the attention supervised model for some prompts.
Considering the pseudo attention as supervisory data is generated simply
based on the edit distance between answers and key elements,
a carefully constructed method to generate attention can further benefit.
We will explore this problem in our future work.




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{08refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Tianqi Wang}{%
received a B.S degree in information security from Harbin Institute of Technology in 2012, and an M.S. degree in system informatics from Tohoku University 2016, respectively. He is currently a Ph.D. student at Graduate School of Information Science, Tohoku University. His main research interests are natural language processing and machine learning.}

\bioauthor[:]{Hiroaki Funayama}{%
is a first-year master's student at the Graduate School of Information Sciences, Tohoku University. He received a B.S. degree in engineering from Tohoku University, Japan, in 2020. His research interests include natural language processing, especially in automatic scoring and other educational applications of natural language processing.}

\bioauthor[:]{Hiroki Ouchi}{%
received his M.S. and Ph.D. degrees from Nara Institute of Science and Technology, in 2015 and in 2018, respectively. Currently, he is a postdoctoral researcher at RIKEN Center for Advanced Intelligence Project (AIP). His research interests cover natural language processing and machine learning.}

\bioauthor[:]{Kentaro Inui}{%
is a Professor of the Graduate School of Information Sciences at Tohoku University, where he is head of the Natural Language Processing Lab. He also leads the Natural Language Understanding Team at the RIKEN Center for the Advanced Intelligence Project. He currently serves as Vice-chairperson of Association for Natural Language Processing, Member of Science Council of Japan, and Director of NPO FactCheck Initiative Japan.}

\end{biography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate


\end{document}

