\documentclass[english,tombow]{jnlp_2.2}

\usepackage{jnlpbbl_1.3}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{array}
\usepackage{footnote}
\makesavenoteenv{tabular}
\makesavenoteenv{table}
\usepackage{gb4e}\noautomath


\Volume{26}
\Number{2}
\Month{June}
\Year{2019}

\received{2018}{11}{1}
\revised{2019}{2}{7}
\accepted{2019}{3}{18}

\setcounter{page}{329}

\esubject{Paper}
\etitle{Annotating a Driving Experience Corpus with Behavior and Subjectivity}
\eauthor{Ritsuko Iwai\affiref{Author_1}\affiref{Author_2} \and Daisuke Kawahara\affiref{Author_1} \and Takatsune Kumada\affiref{Author_1}\affiref{Author_2} \and Sadao Kurohashi\affiref{Author_1}} 
\eabstract{
To communicate with humans in a human-like manner, systems need to understand behavior and psychological states in situations of human-machine interactions, such as in the cases of autonomous driving and nursing robots. We focus on driving situations as they are part of our daily lives and concern safety. To develop such systems, a corpus annotated with behavior and subjectivity in driving situations is necessary. In this study, subjectivity includes emotions, polarity, sentiments, human judgments, perceptions, and cognitions. We construct a driving experience corpus (DEC) (261 blog articles, 8,080 sentences) with four manually annotated tags. First, we annotate spans with driving experience tags (DE). Then, three tags, other's behavior (OB), self-behavior (SB), and subjectivity (SJ), are annotated within DE spans. In addition to describing the guidelines, we present corpus specifications, agreement between annotators, and three major difficulties during the development: the extended self, important information, and voice in mind. Automatic annotation experiments were conducted on the DEC using Conditional Random Fields-based methods. On the test set, the F-scores were about .55 for both OB and SB and approximately. 75 for SJ, respectively. We provide error analysis that reveals difficulties in interpreting nominatives and differentiating behavior from subjectivity.
}
\ekeywords{Driving Experience Corpus, Behavior, Subjectivity, Annotation, Conditional Random Fields, Bi-directional Long Short-Term Memory Conditional Random Fields}

\headauthor{Iwai et al.}
\headtitle{Annotating a Driving Experience Corpus with Behavior and Subjectivity}

\affilabel{Author_1}{}{Graduate School of Informatics, Kyoto University}
\affilabel{Author_2}{}{RIKEN BSI-TOYOTA Collaboration Center, RIKEN}


\begin{document}

\maketitle

\section{Introduction}

Given the drastic improvements in the field of machine learning, agents, and robots, systems share or play a role in tasks that used to be fully controlled by humans. Many companies are, for example, competing to realize automatic driving and are already test-driving on roads \cite{ref54}. However, it is critical that systems understand individual human differences and predict their behavior, especially when a miss may bring about a serious fatal accident. Systems must catch the attention of drivers and make them take the appropriate action. Thus, it is important for systems to communicate with their users in accordance with their individual differences. 
    Tapus, \c{T}\u{a}pu\c{s} and Matari\'{c} (2008)\nocite{ref47}, for example, 
demonstrated that participants were more engaged in rehabilitation exercises with robots that behaved and gave messages in a way that reflected the personalities of the participants. We regard it as important that a system communicates with its user via a language based on his/her personality to bridge the inconsistencies between the system's requirements and the user's attention and actions by understanding his/her subjectivity.

Personality is defined as an ``abstraction used to describe and explain the coherent patterning over time and space of affects, cognitions, desires, and the resulting behaviors that an individual experiences and expresses'' \cite[p. 70]{ref34}. We consciously or unconsciously evaluate others' personalities, predict their next behaviors or utterances, and sometimes adjust our communication style as a result. The definition of personality suggests two important things that a human or a system needs to understand in human communication: behaviors and their psychological states. In other words, behavior and subjectivity are the critical determinants. Thus, having knowledge of human personality, subjectivity and related behaviors are quite critical to understanding and predicting drivers' behaviors and psychological states. Such knowledge will be important for this system's implementation to choose or generate natural language/messages in driving situations and to allow systems to communicate with users in the future. A driver with high anxiety, for example, may be more upset about another driver that they consider to be following too closely (even if the distance is generally enough). If systems know that inter-car distance causes anxiety, and the driver tends to be anxious, systems can show a message such as ``You may be a little bit upset about the car distance, but it is well-managed by the systems.'' This is, however, not necessary for other drivers, and may even be inappropriate for impatient drivers. Another driver might easily lose concentration while driving and unconsciously drive slowly. If systems know that driving too slow causes the person in the following car to become irritated, systems can generate a message such as ``You may be driving a little bit too slow, which might irritate the following car,'' and the driver can concentrate on the situation again. In order to present these kinds of messages, systems need knowledge about their own or others' driving behaviors. More importantly, the knowledge is related to the driver's viewpoint, not social norms or manners, which reflect on actual human experiences.

For this purpose, it is necessary to collect driving behaviors and subjective reactions regardless of social norms, desirability, or ethics. We decided to use textual data derived from blog articles rather than conversational data in actual/experimental driving settings or newspapers for three reasons. First, our focus is on the drivers' own points of view. Formal texts such as newspapers are not adequate because they are written by professional journalists or authors to deliver facts or events as objectively as possible. Secondly, blog articles include more information about authors' actual driving behaviors regardless of social norms and desirability. In blogs, some authors describe, for example, their own dangerous driving, such as malicious tailgating. No drivers who participate in car driving experiments drive in such a manner if they know that all information is recorded. It is also not acceptable for researchers to allow participants such socially undesirable and highly accidental driving. Finally, blog articles reflect a more genuine subjectivity of authors' own behaviors than evaluations by drivers in driving experiments or annotators. The anonymity of blog articles supports authors' genuine subjectivity as they did not write articles for studies. One blog author, for example, \textit{positively} and \textit{proudly} described his malicious tailgating that stopped the target car, although the author's family was in the car, and the family attempted to stop the driver many times.

In summary, a corpus was designed, which satisfies the following three characteristics:

\vspace{0.5\Cvs}
\begin{itemize}
\item
Daily and informal expressions under actual driving experiences,
\item
Subjectivity (opinions, perceptions, cogitations, thinking, and emotions), and
\item
Driving behaviors.
\end{itemize}
\vspace{0.5\Cvs}

For the first step, we constructed a Driving Experience Corpus (DEC) in Japanese, which consists of 261 blog articles (8,080 sentences). The two sentences in Figure 1 show an annotation example from the actual DEC. Figure 2 depicts the process of corpus construction.

\begin{figure}[b]
\begin{center}
\includegraphics{26-2ia3f1.eps}
\end{center}
\caption{An example of an annotation}
\end{figure}
\begin{figure}[b]
\begin{center}
\includegraphics{26-2ia3f2.eps}
\end{center}
\caption{Process of annotation}
\end{figure}

This paper is organized as follows. Section 2 reviews previous studies relevant to our approach. Section 3 describes the guidelines of annotation, including definitions of each tag and difficult cases to annotate. Section 4 describes the annotation with corpus statistics and inter-annotator analysis. Section 5 gives a detailed description of experiments, along with the results, error analysis, and discussion. Finally, Section 6 presents the conclusions of our approach and future work.


\section{Related Work}

In this section, we introduce related work in four domains; (1) traffic, transportation, and driving, (2) experience mining and habitual behavior, (3) personality, and (4) annotation and corpus.


\subsection{Traffic, Transportation and Driving}

To our best knowledge, previous NLP studies related to driving have mainly focused only on driving actions, traffic rules, and ontologies. \citeA{ref45} constructed a car license test corpus for textual entailment. With traffic regulation texts, \citeA{ref15} proposed transportation terminology recognition for semi-automatic traffic ontology expansion to build necessary knowledge on traffic, such as traffic regulations and transportation manners necessary for traveling. \citeA{ref44} converted questions into SPARQL queries for traffic ontology. \citeA{ref46} constructed a traffic ontology-based Q and A dataset. These datasets, however, are not publicly available. Furthermore, human-centered or human experiential foci are not highlighted.


\subsection{Experience Mining and Habitual Behavior}

Experience mining and habitual behavior extraction are similar to our approach in that they also focus on events and human behavior, but their scopes are much broader than ours. Inui and his colleagues suggested extracting events and human behaviors by identifying the topic object, experiencer, event expression, event type and semantic type or orientation if applicable, factuality, and source pointer \cite<e.g.,>{ref7}. \citeA{ref17} defined ``an experience'' as having three constituents, (1) situation (time and space), (2) behavior (action and object), and (3) subjectivity (evaluation and emotion), and extracted experiences by structuralizing texts in blog entries in terms of time, space, action, object, evaluation, and emotions. \citeA{ref42} proposed extracting habitual behavior using Latent Dirichlect Allocation and Point-Mutual Information, and evaluated the method using texts about communication behaviors in question and answering websites. Later, \citeA{ref43} applied the method to extract habitual behaviors related to transportation.

Different from the previous studies as discussed in the previous section, all of these studies concern more daily and unstructured texts, and the information extracted by their methods reflect humans' daily behaviors. Their approaches, however, are too broad in comparison with our interests. The range of topics is not restricted to driving. In addition, our interest in behaviors targets a shorter span, which does not include habits.


\subsection{Personality}

Personality is the important internal framework that we use when we communicate with \linebreak
others. Psychological studies have suggested that personality is the universal framework to \linebreak
describe and understand individual differences in behaviors and psychologies \cite<e.g., >{ref1,ref3,ref4,ref21,ref22,ref35}. In Japan, many personality psychologists have demonstrated that such an internal framework is held in the Japanese population as well \cite<e.g.,>{ref14,ref23,ref24,ref30,ref38,ref39,ref53}.

In the field of Natural Language Processing, there are several studies that predict the personalities of authors from SNS (Social Network Services) texts both in English \cite<e.g.,>{ref2,ref31,ref32,ref36} and in Japanese \cite<e.g.,>{ref27,ref28}. These studies, however, focus on the detection of authors' personalities. Our goal is to associate behaviors, subjectivity, and personalities in the future.


\subsection{Annotation and Corpus}

Regarding subjectivity, subjective analysis is one important and active domain. Subjectivity in this analysis refers to ``information about any attitudes, beliefs, emotions, opinions, evaluations, and sentiment expressed in texts'' \cite[p. 817]{ref41}. The previous studies of subjectivity analysis include emotions \cite<e.g.,>{ref19,ref48,ref49}, opinions \cite<e.g.,>{ref12}, and sentiments \cite<e.g.,>{ref5,ref26}.

Regarding annotated corpora in this domain, four Japanese corpora have been constructed: (1) the University of Tsukuba Corpus Tagged with Polarity Evaluation (polarity per sentence),\footnote{http://www.nlp.mibel.cs.tsukuba.ac.jp/\textasciitilde inui/SA/corpus/readme-euc.txt} (2) the ACP corpus, (3) the Kyoto University and NTT Blog Corpus, and (4) the Phrase Polarity Corpus. The Tsukuba Corpus consists of 4,309 sentences in the travel services domain with the polarity per sentence. The ACP corpus was automatically constructed \cite{ref13}. This corpus is large (126,610 sentences) but lacks human filtering. \citeA{ref5} constructed the Kyoto University and NTT Blog Corpus of 249 blog articles (4,186 sentences) with sentiment information. 81 university students wrote these blog articles using the following four themes: sight-seeing in Kyoto, cellphones, sports, or gourmet food. \citeA{ref26} constructed a phrase-sentiment review corpus (59,758 phrases) with all manually annotated polarity information from the Tsukuba Corpus. These corpora do not focus on driving, thinking, and cognition.

Summarizing the four previous sections, the purpose of our study is different from the existing studies in the following three perspectives:

\vspace{0.5\Cvs}
\begin{itemize}
\item
It includes informal and daily expressions.
\item
It incorporates various subjectivity expressions, including emotions, polarity, sentiments, human judgments, perceptions, and cognitions.
\item
It includes driving experiences.
\end{itemize}
\vspace{0.5\Cvs}

\noindent
The DEC is a new corpus with manually annotated tags in very specific settings. Thus, it is expected that the corpus will contribute to new points of view in the field of traffic, transportation, and driving, as well as personality inference from behaviors.


\section{Guidelines}
\vspace{-0.5\Cvs}

The guidelines were constructed by an annotation team. The members consisted of the first author, three experienced annotators, and the second author as a supervisor. With 11 driving experiential blog articles, we repeated the annotations, discussions, and revisions of the guidelines. First, we introduce definitions of each tag, indicating examples. Next, we present difficult cases and our solutions.


\subsection{Definitions of Tags}

\begin{table}[b]
\caption{Definition of the tags}
\input{03table01.tex}
\end{table}

In this project, we focus on behaviors and subjectivity, especially when driving a car. To this end, we first tag a car driving experience and annotate the experiences with the behavior and subjectivity tags. We prepared for two types of tags, which have different annotation purposes. One is a wide-range tag labeled ``Driving Experience'' (DE) to determine ranges of the annotation in blog articles, which usually spans multiple sentences. The other type is a short-range tag basically within a sentence, which includes four tags labeled ``Others' Behavior'' (OB), ``Self-Behavior'' (SB), ``Subjectivity'' (SJ) and ``Together'' (TG). These tags are basically used on a clause-basis with several exceptions in SJ. The definitions are presented in Table 1. We explain the tags in the following sections with examples.


\subsubsection{Driving Experience Tag}

First, it is important to define the range of driving experiences. To exclude texts that are unrelated to car driving experiences, we prepared the DE tag. The DE tag is essentially different from the other tags as it is used to determine ranges where behavior and subjectivity tags are annotated. In other words, the DE tag is necessary in order for a blog article to be included in the DEC. The first three and last three sentences of example (1) are not related to the author's driving experience, and the author only described her car accident in the lines in-between. Those lines were given the DE tag.

\begin{exe}
\ex 
はい、純ちゃんです。\\
Hi, this is Jun-chan.\\
今朝起きたら雨が降った後がありました。\\
When I woke up, I found marks made by rain drops.\\
もっと寒かったら雪になっていたのかなぁ〜と思いました\\
It would have snowed if it was even colder.\\
私・・・\\
I\textellipsis \\
14日に事故って・・\\
I caused a car accident on the 14th\textellipsis \\
次の日、会社に行くまでの短い距離で恐怖を感じました。\\
The next day, I felt fear at a short distance on the way to the office.\\
-[omit]-\\
車つぶすくらいの事故にあった人は、乗るのに恐怖を感じるよね。\\
Those who are in an accident about crashing a car must feel a fear of driving.\\
玉子ボーロ\\
Egg bolo\\
お客さんからの差し入れ\\
Sent from a customer\\
なつかしぃ〜\\
How nostalgic.
\end{exe}

It is, however, often difficult to clearly differentiate the DE from the other information, especially when an author writes something related to driving but not really related to behavior or subjectivity, as seen in example (2).

\begin{exe}
\ex 
車線変更する時はちゃんと確認しましょう。\\
Let's check properly when changing lanes.\\
タイトル通りなんですけどね。\\
It's as the title, though.\\
いやまぁウィンカーだけが問題じゃないんですけど今回の話は。\\
Well, winker is not the only problem for the story this time.
\end{exe}

These three lines are related to the driving experience in that they refer to it, but unrelated in that they are not related to behavior and subjectivity. The guidelines define annotating as wide as possible---in this case, it says to include them in the DE tag---but the strict segmentation is not really the center of our interest.


\subsubsection{Behavior and Subjectivity Tags}

The behavior and subjectivity tags are attached only within each DE range. In our guidelines, ``behavior'' refers to an observable behavior. People who are present in the same scene can ``hear,'' ``listen to,'' ``see,'' or ``look'' in the same way as others do, so it is considered that they can be recognized as having something in common. Behaviors are divided into two tags according to who does the behavior, the author or others, because the delineation is very important. The system needs to differentiate its user and others both in terms of safety and communication. The accidental risk may be notably high if the system fails to deliver the message accurately.

Meanwhile, psychological reactions or mental states such as thinking and feeling are all regarded as subjectivity. The behavior and subjectivity tags are basically annotated within one sentence. All the tagged units, however, change from a word to a clause or to phrases, depending purely on semantics. If one sentence constitutes only one clause, it is simple, as seen in examples (3), (4), and (5).

\begin{exe}
\ex 
今日、初めて駒沢通りを走りました。(SB)\\
I drove the Komazawa Street for my first time.

\ex 
あのバイクは、ブレーキを踏まなかった (OB)\\
That bike did not step braking.

\ex 
対向車にヒヤリとした。(SJ) \\
I was thrilled by the oncoming car.
\end{exe}

However, sentences in blog articles are often not well-structured, and it may be difficult to clearly determine a span, as shown in Figure 3. The sentence consists of several clauses and some subjectivity within phrases. The first clause ``桜田門交差点手前の下り坂を下っていたとき'' is relatively simple: it is the behavior of the author. The next clause ``前を行くタクシーが客を拾おうとしたのか急ブレーキをかけたと思ったら車線変更気味に加速したので'' is, however, complicated by its subjectivity. This clause is, broadly speaking, talking about others' behavior in that it refers to what a taxi driver did, although ``the preceding taxi'' is literally ``a taxi,'' not a person. That is why, at first, the clause was annotated as OB. However, ``客を拾おうとしたのか'' is just an inference of the author. ``と思ったら'' clearly indicates the author's thought, so it is labeled as ``SJ.'' ``車線変更気味に'' is also part of the author's understanding. These three parts all have ``inference/thought'' words such as ``のか,'' ``と思ったら,'' and ``気味に.'' The next clause, ``ぴ＠も慌ててブレーキをかけたのですけど,'' is SB as a whole clause. ``慌てて,'' however, is ``with panic,'' which indicates the author's mental state, so the phrase was tagged with SJ. The last clause from 後ろから走ってきた\textellipsis いきなり落車しました。'' is the behavior of a third party, neither the author nor the taxi driver. This clause, again, includes the author's inference in between ``タクシーがバンパーにハスったのか,'' and this part was separated and annotated with SJ.


\begin{figure}[t]
\begin{center}
\includegraphics{26-2ia3f3.eps}
\end{center}
\caption{An example of one sentence with different units of annotation}
\end{figure}

\subsubsection{Together Tag}

The TG tag is used only when phrases or sentences are behaviors but annotators cannot differentiate either ``author'' or ``others'' as nominatives. In example (6), the author and a car passed each other, and thus it is apparently behavior. The passing occurred simultaneously, but the author and the driver drove their cars independently.

\begin{exe}
\ex 
お互い何のダメージもなくすれ違った \\
We passed each other without any damage.
\end{exe}


\subsection{Difficult Cases}

In the discussion process, we found three difficulties: (1) the extended self or we-ness, (2) important information, and (3) voice in mind.

\textbf{Extended self or we-ness:} Different from usual situations, it is sometimes quite difficult to differentiate ``me'' and ``you/other people'' when there are multiple people in a car.

In example (7), the author was NOT driving the car. She was just sitting in the passenger seat in the car while her husband was driving.

\begin{exe}
\ex 
今朝子供たちを幼稚園に送った後、夫の運転で走っていました。\\
After sending our children to kindergarten this morning, my husband was driving. \\
青になり、交差点を渡ったら・・・突然シグナルもださず、 ・・・ 、私たちの走ってるレーンに車線変更してきて、左前方こすられました。 \\
When we crossed over the intersection, \textellipsis a car suddenly did not send a signal, \textellipsis , changed the lane where we were running and scraped the left front of our car. \\
私、助手席座っていたのですが、キャーと思わず叫んでた。\\
I, sitting in the passenger seat, was screaming in spite of myself.
\end{exe}

Although she did not have any control over the car, they crashed due to no choice or fault of her own. Thus, we regarded this extended self as self-behavior. On the other hand, ``screaming'' in the last sentence is only what the author did, which was originally defined as self-behavior.

\textbf{Important information:} There were many cases in which the annotators ignored several important pieces of information related to driving behaviors. Typically, hesitation is related to time and weather conditions that appear in the very beginning of the experiences, far from the driving behavior. We know they often affect driving behaviors.

Another typical example includes important incidents or information that affects the preceding or subsequent tags, even though such information, of course, appears in very close sentences.

\begin{exe}
\ex 
さらに、水温計がみるみる上昇していた。\\
The water temperature gauge was climbing up even more. \\
６０キロくらいまで落ちたところで、ハザードをつけて路肩に寄せるしかなかった。\\
When my speed fell to about 60 km/hour, I had to put on a hazard light and pull over.
\end{exe}

The water temperature gauge increasing is just a fact and not a subjective behavior. However, it specifically led the author to put on the hazard lights and stop the car.

We decided to solve this issue only when such important information that affects behaviors or subjectivity occurred just before or after the main tags. This related information is easily connectable with the main behavior and subjectivity, while the distant information makes it difficult to determine to what extent it influences behavior and subjectivity. For the example sentences, we annotated the first sentence about the water temperature gauge with the same tag as the following sentence, which is SB.

\textbf{Voice in mind:} The texts in parentheses (utterances) were mostly included only when they had nominatives and/or verbs as behaviors. The parenthesis in example (5), however, does not appear with either nominatives or verbs. ``Embarrassing\textellipsis '' is part of the author's subjectivity. Expressions similar to the example often appear in the blog writings, so we annotated such expressions with the subjectivity tag as a voice in mind.

\begin{exe}
\ex 
今ではマジェ＆スポスタでタラタラとツーリングを楽しんでいるｍｊですがガキの頃には当時流行のレーサーレプリカで、粋がって峠を攻めたものでした（お恥ずかしい…）\\
Now mj/the author enjoys touring with Maje/Majesty {\&} Sposuta/Sportster, but I used to be a little boy going around mountain passes at high speeds with a stylish race replica at the time (embarrassing\textellipsis ).
\end{exe}


\section{Annotation}

\subsection{Corpus Annotation}

Manual annotation was performed by three annotators using the brat annotation tool \linebreak
\cite{ref40}. We prepared the driving-related corpus using 10 million blog articles that were filtered using driving-related words. The 250 blog articles that include both actual daily driving scenes and the authors' subjective views about the driving scenes were manually selected from the driving behavior corpus \cite{ref25} and the driving behavior and subjectivity corpus \cite{ref11}. Driving under special circumstances was not included, such as driving in driving schools or racing circuits. With the 250 driving-related blog articles from the driving-related corpora, the annotators started the actual annotations. For the first round, the 250 articles were divided into three piles, and each annotator completed their part. For the second round, the second annotator reviewed the annotated texts and corrected annotation errors if necessary. If there was a disagreement, they were discussed. This process resulted in a total of 261 annotated articles (11 were those used for creating the guidelines).


\subsection{Statistics}

Table 2 summarizes the DEC and annotated tags. The mean of the DE tags is 1.2, meaning that at least one or sometimes more DE tags were included per blog article. The DE tags include various events and their experiences. Some blogs included more experiences. The SJ tags were the most common among behaviors and subjectivity. Driving-related blog articles are, thus, considered very subjective, which is what we originally intended. The TG tags appeared at quite a low frequency, and the models did not sufficiently learn this tag in any experimental settings. Thus, we did not report the TG tag accuracy after this statistical section in this study.


\begin{table}[t]
\caption{Corpus statistics ($\text{N} = 261$ blog articles)}
\input{03table02.tex}
\end{table}


\subsection{Inter-Annotator Agreement}

To evaluate the guidelines, two annotators annotated the same five articles independently. Referencing \citeA{ref16}, we calculated $_{cu}\alpha $ and $_{(k)u}\alpha $ for each of the evaluated blog articles for OB, SB, and SJ with the alpha reliabilities for utilizing a continuum version 1.2.\footnote{https://mathet.users.greyc.fr/agreement/} \citeA{ref16} suggested new alphas to evaluate inter-rater agreement in the continuum data; they criticized Cohen's $\kappa $ or weighted $\kappa $, Scott's $\pi $ (pi), and Krippendorff's $\alpha $ in that they evaluate the pre-defined segmentations. In our task, there is no clear or objective segmentation. Therefore, we used $_{cu}\alpha $ to evaluate inter-annotator agreement regarding all the categorizations and $_{(k)u}\alpha $ to evaluate each tag in one blog article (evaluated blog: EvalBlog) (Table 3). The means of $_{cu}\alpha $ and $_{(k)u}\alpha $ are substantial. Thus, the guidelines are properly constructed to allow for consistent annotations.

We did not calculate the scores of DE. The tag is different from the other tags, and it is rather difficult to determine the range of where to annotate the behaviors and subjectivity tags. The differences are due to ambiguity in segmenting driving experiences, as discussed in section 3.1.1 when reviewing the discrepancies between the two annotators.

\begin{table}[t]
\caption{Inter-annotator agreements and the number of tags}
\input{03table03.tex}
\end{table}

\section{Automatic Annotation Experiments}

In this section, 
we report on the automatic annotation experiments that were conducted to examine to what extent it is possible to learn to identify human driving experiences, behavior, and subjectivity. First, we applied CRF and BiLSTM-CRF individually to the development data to choose the best model for each and compared them. Next, we applied the best model to the test dataset for the final evaluation. Finally, we present discussion and error analysis.


\subsection{Classification Method}

We used Conditional Random Fields (CRF) \cite{ref18} and Bidirectional Long Short-Term-Memory (BiLSTM-CRF) \cite<e.g.,>{ref6,ref20,ref33} to conduct the automatic annotation experiments using our annotated corpus. For CRF experiments, CRF$++$ version 0.561\footnote{https://taku910.github.io/crfpp} was used. For the BiLSTM-CRF experiments, we used the BiLSTM-CNN-CRF Implementation for Sequence Tagging\footnote{https://github.com/UKPLab/emnlp2017-bilstm-cnn-crf} by \citeA{ref33}, which uses Keras and TensorFlow. The BIO tagging scheme is used in both classifications.


\subsection{CRF Experiments}

\subsubsection{Feature Representations}

CRF$++$ is a sequential tagger that requires a template file that specifies the combinations of features. The basic features that are fed to the CRF are the sets of word identities and parts of speech (POSs). In addition to these basic features, we prepared the following five features.

\textbf{Dependency Relations (DPR):} Dependency relations are critical information in Japanese. If a word is not the last one within its phrase, it is given its following word. If a word is the last one in its phrase, it is given the content word in its parent phrase (Figure 4).

\begin{figure}[b]
\begin{center}
\includegraphics{26-2ia3f4.eps}
\end{center}
\caption{An example of the dependency feature}
\end{figure}

\textbf{Driving Behavior Words (DBW):} We used 836 verbs and verbal nouns collected from the three resources \cite{ref11}; (1) hand-writing about driving experiences, (2) driving-related words \cite{ref25}, and (3) words that are classified as transportation in the dictionary in the KNP.\footnote{http://nlp.ist.i.kyoto-u.ac.jp/index.php?KNP} The hand-writings were about thrilling driving experiences and difficult driving operations by those who participated in psychological experiments at two national research institutes\footnote{The National Institute of Advanced Industrial Science and Technology, and RIKEN} in Japan from 2012 to 2015 (399 drivers and 250 older---over 60 years old---drivers). We eliminated the words that were not related to driving such as 特注 `special order' by human evaluation. This evaluation resulted in the final 584 DBWs. The words include, for example, 運転 `drive,' 右折 `turning right,' 左折 `turning left,' カーブ `curve,' 駐車 `vehicle-parking,' 注意 `attention,' and 苛々 `irritation.'

\textbf{Driving Behavior Phrases (DBP):} Using the DBWs, we extracted the driving behavior predicate phrases from three resources \cite{ref11}; (1) the same writings as mentioned above, (2) two driving license textbooks for rules and driving techniques \cite{ref50,ref51}, and (3) driving-related blogs \cite{ref25}. For (1) and (2), we collected all of the predicates using a predicate-argument structure analysis with the KNP. For (3), we collected the predicates whose predicate or cases included DBWs. This approach resulted in 7,931 total phrases after human evaluations. The list includes, for example, ハンドルを切る `turn a steering wheel' and ブレーキを踏む `step on the brake.'

\textbf{Personality words (1) (PW1):} We extended 109 personality-related words \cite{ref8,ref52} to 487 words using the word2vec model that is trained with 200 million web-crawled Japanese sentences. These extended personality words are flagged only when zero anaphora resolution results indicate that the nominatives are human when using the Japanese dependency and case structure analyzer KNP.

\textbf{Personality words (2) (PW2):} We extended 142 Japanese personality-related words \cite{ref8,ref9} to 665 words using the same word2vec model as PW1. 395 words were selected based on human evaluations, and they more directly express personality traits such as 心配 `anxious' and 同調 `agreeable.' Conversely, PW1 includes more suggestive ones, such as 文化 `culture' and 美術 `arts.' These words are flagged only when zero anaphora resolution results indicate that the nominatives are human when using KNP.


\subsubsection{Experimental Setup}

We divided the dataset of 261 articles into training, development, and test data and conducted experiments. First, we used all of the features together for the training. After that step, we determined the parameters using the development data.


\begin{table}[b]
\caption{The F-scores of cross validation results}
\input{03table04.tex}
\end{table}

\subsubsection{Cross Validation}

To remove the uncertainty of a data split and to identify the best feature combination, 5-fold and 10-fold cross validations were applied to training and development data with different sets of features (Table 4). Table 4 indicates the F-scores of the five combined features and the four combined features that dropped one feature from each set. ``△'' represents elimination; △DBW, for example, means that four out of the five features are used (all those except DBW). The F-scores in both 5-folds and 10-folds, however, caused only slight differences (DE: .820〜.833, OB: .436〜.464, SB: .510〜.535, SJ: .703〜.709 for 10-folds and DE: .813〜.821, OB: .444〜.454, SB: .510〜.527, SJ: .711〜.714 for 5-folds). Therefore, we decided to use all of the features for further automatic annotation in the CRF.

\begin{table}[b]
\caption{F-scores of the development data ($\text{N} = 209$ blog articles)}
\input{03table05.tex}
\end{table}

\subsubsection{Parameters}

We adjusted two major parameters, the regularization algorithm and the hyperparameter C, to find the best model. Using the development data, we tested all of the combinations of the two parameters (regularization algorithm: L1/L2 and hyperparameter: 0.001/0.01/0.1/1/10). The cutoff threshold of features was not adjusted because our datasets were not very large. We compared the performances and chose L1 and set C to 1 because the combination resulted in the best performances regarding all the tags except DE tags (Table 5).


\subsubsection{Results of the Development Data}

In our first experiment, we compared the F-scores of each tag using the development data while increasing the training data step by step from 50 articles to 209 articles (Figure 5). For DE and SJ, the F-scores are notably good within the 50 articles (.761 for DE and .715 for SJ), and the scores increased slightly (.770 for DE and .750 for SJ). For the behavioral tags, the F-scores improved from .344 to .506 for OB and from .445 to .578 for SB. We integrated OBs and SBs into one tag, integrated behavior (BH), and BH performed much better than OB and SB, respectively (from .538 to .661). This suggests that the CRF model has learned the behavioral texts but finds it difficult to achieve the correct nominatives.

\begin{figure}[t]
\begin{center}
\includegraphics{26-2ia3f5.eps}
\end{center}
\caption{Learning curves of the development data}
\end{figure}

\subsection{BiLSTM-CRF Experiments}

\subsubsection{Experimental Setup}

We used the same dataset as that in the CRF experiments. The basic features fed to the BiLSTM-CRF were the sets of word identities and POSs.

\begin{table}[b]
\vspace*{-0.25\Cvs}
\caption{Evaluation designs}
\input{03table06.tex}
\vspace*{-0.5\Cvs}
\end{table}

\subsubsection{Optimizers and Parameters}

Table 6 indicates the designs and hyperparameters that we used for the experiment. 
\pagebreak
We selected Adam as the optimizer in the experiment based on the results of pre-experiments in comparison with the other optimizers, such as SGD.


\subsubsection{Results of the Development Data}

In the experiment, we conducted BiLSTM-CRF step by step while increasing the training data step by step from 50 articles to 209 articles as we did in the CRF experiments. In each step, we repeated the training five times and summarized the F-scores in the means and standard deviations for each tag using the development data (Figure 5). The error bars indicate $\pm 1$ standard deviations. The learning curves are quite similar to those in the CRF. For DE and SJ, the means of the F-scores are notably good within the 50 articles (.773 for DE and .724 for SJ), and the scores increased slightly (.783 for DE and .753 for SJ). For the behavioral tags, the means of the F-scores improved from .423 to .562 for OB and from .436 to .518 for SB. We integrated OBs and SBs into one tag again, integrated behavior (BH), and BH performed much better than OB and SB, respectively (from .626 to .693). This suggests that even the BiLSTM-CRF models have learned the behavioral texts but find it difficult to achieve the correct nominatives.

\begin{table}[t]
\caption{Results of the development data for Adam}
\input{03table07.tex}
\end{table}

Table 7 indicates all the means and standard deviations of the Precision, Recall, and F-scores using 209 blog articles.


\subsection{Results of the Test Data}

Table 8 indicates the F-scores of the CRF model and the means of the F-scores of the BiLSTM-CRF model of the development and test data, applying the models trained with the 209 blog articles. The scores, however, do not show a big difference between the two models. The test performance also indicated similar learning patterns to those seen in the development data. In the test dataset, however, the CRF model performed slightly better than the BiLSTM-CRF model did.

\begin{table}[t]
\caption{F-scores of the development and test data}
\input{03table08.tex}
\end{table}

\begin{figure}[b]
\begin{center}
\includegraphics{26-2ia3f6.eps}
\end{center}
\caption{Categorization of the twenty errors}
\end{figure}

\subsection{Discussion and Error Analysis}

We reviewed the results of our CRF model of the development data because learning results did not show a big difference between CRF and BiLSTM-CRF. We chose 20 errors randomly and reviewed and grouped them into three types---(1) nominative interpretation, (2) ambiguity of behavior, and (3) others---and two subtypes for nominative interpretation and ambiguity of behavior (Figure 6). Nominative interpretation appeared the most, and ambiguity of behavior and other errors followed. In the following sections, we describe each type with actual examples. The underlined phrases are the errors that we refer to and discuss. In the parenthesis that follows each example, the results of the annotator (Gold as the gold standard) and the CRF model (System) are indicated.


\subsubsection{Nominative Interpretation}

The most errors appeared because the model did not correctly understand nominatives. When humans see and understand any driving behavior, they regard them as extended others or self-behaviors; we usually expect that someone intentionally controls and maneuvers vehicles. Thus, vehicles are often not just ``objects'' but sometimes regarded as human. This is well-learned, as example (10) indicates.

\begin{exe}
\ex 
そして\underline {追い越し車線に入ると、私の前の軽自動車が加速開始。}(Gold: OB, System: OB) \\
And, \underline {\hbox spread24pt{when entering the overtaking lane, the subcompact car preceding me started}} \linebreak\underline{accelerating}.
\end{exe}

The car itself is not a human, and it does not behave itself, but humans assume a driver. Both human annotators and the model classified this sentence as behavior and further as OB. In this case, the model also succeeded in inferring the nominative.

However, it often failed to determine ``who.'' These nominative interpretation errors were further divided into subtypes, explicit and implicit. We explain each subtype as follows.

\textbf{Explicit:} The errors were categorized as explicit when the errors included clear personal pronouns, as in examples (11) and (12).

\begin{exe}
\ex 
\underline {僕の真横をかすめた瞬間に}僕は急ブレーキで停止 (Gold: OB, System: SB) \\
\underline {\hbox spread14pt{At the moment when (a car) passed so close just beside me}}, I suddenly braked and \linebreak stopped.

\ex 
\underline {今日の帰りも前に遅い車が走っていたんですけど}、その車は上記の通り5〜6台先 (Gold: OB, System: SB) \\
\underline {Today a slow car was running ahead on the way back}, but that car is 5--6 ahead as stated above.
\end{exe}

The system categorized both examples into SB instead of OB, although both texts include ``I/my/me'' information. In (11) the nominative is omitted, but humans infer ``another person / a car'' as the nominative because ``just beside me'' indicates someone or something else. In (12) ``a slow car'' is obviously not the author. With such explicit information, the system fails to determine the nominative of the behaviors.

\textbf{Implicit:} The implicit errors are the phrases that require understanding the contexts. In example (13), we inserted ``I'' based on our understanding of the whole blog article, which is omitted in the Japanese text. Given only this sentence, we are not able to judge whether the phrase is SB or OB.

\begin{exe}
\ex 
\underline {車（レジアスエース）で左レーンに車線変更したら}、Aピラーの死角にカローラワゴンが居てぶつかりそうになった。(Gold: SB, System: OB) \\
\underline {When (I) changed the lane to the left lane with a car (Regius Ace)}, one Corolla wagon \linebreak was in the blind spot of the A pillar and it began to collide.
\end{exe}

Unexpectedly, most of nominative interpretation errors were explicit. This suggests that the system still finds it difficult to interpret ``who'' in unstructured texts in spite of explicit cues. Interpreting or inferring nominatives correctly is actually an important issue in the field of Natural Language Processing in Japanese.


\subsubsection{Ambiguity of Behavior}

We define ambiguity of behavior as the errors that are difficult to clearly cut as one behavior, such as habitual behavior and momentary behavior. Here we introduce the five errors.

\textbf{Habits:} We categorized three errors into habits out of the five ambiguities of behavior errors (examples (14), (15), and (16)). Human annotators labeled all of these examples as SJ, while the system labeled them SB. The phrases in (14) refer to the author's habitual behaviors when he used to go to a cram school; the phrases in (15) refer to the author's everyday commuting; the phrases in (16) refer to the author's past decision about riding a bicycle. Although such expressions include behavioral parts, it is difficult to determine one clear experience. The guidelines instruct these types of phrases to be tagged as subjectivity. It is, however, not possible for the system to learn the rule.

\begin{exe}
\ex 
\underline {塾へ行く時は住宅街の狭い道を通っていたのでほぼ歩道を走行しており}、特に危険は無かった（と思う）。(Gold: SJ, System: SB) \\
\underline {When I went to a cram school, I had to run on a sidewalk almost because I was going} \linebreak\underline{through a narrow street in a residential area}, and there was no particular danger (I think).

\ex 
\underline {毎日通勤で片道２５ｋｍ１時間も車で走っていると}時々バカに会う(Gold SJ, System: SB) \\
\underline {Commuting to work by car with 25 km for one hour every day}, I sometimes see fools.

\ex 
それでも気付かなかったのだから、当然、恐く感じ、\underline {その日から自転車乗車中にはウォーク}\linebreak\underline{マンを使用しなくなりました}。（Gold: SJ, System: SB）\\
Having never noticed, naturally, I was horrified; \underline {from that day on, I have not used a} \linebreak\underline{Walkman while riding a bicycle}.
\end{exe}

\textbf{Moments:} We categorized two out of the five ambiguities of behavior errors as moments (examples (17) and (18)). These phrases refer to scenes where the author or a driver was trying to do something at a particular moment. We regarded the author or the person as actually doing the action, so the guidelines instruct these types of phrases to be annotated with behavior tags in accordance with their subjects. However, such rules were not replicable in training.

\begin{exe}
\ex 
\underline {それでも抜こうとするので}、救急車も安全のため何度も ブレーキ・・・ (Gold: OB, System: SJ) \\
Still (the car) \underline {trying to pull it out}, the ambulance also repeated braking for safety \textellipsis 

\ex 
\underline {そこで歩道に出ようとした瞬間、}後ろから来た車が横を通っていったのだけどそれに全く気付かなかった。(Gold: SB, System: SJ)\\
\underline {At the moment I tried to go out on the sidewalk there}, the car from behind went just beside (me), but I did not notice it at all.
\end{exe}


\subsubsection{Others}

In this section, we individually introduce six errors that we found impossible to group into one independent category or distribute to the above two types.

Annotators did not tag either behaviors or subjectivity in the underlined phrases of example (19), while the system attributed them as SB. This is, however, difficult; it may be hearsay or a reason to do the behavior (``rode my bike as much as possible''). It is no wonder, then, that the system failed to get the same answer as the Gold.

\begin{exe}
\ex 
\underline {日曜から梅雨入りとのこと}、頑張って走り回ってました。(Gold: Null,\footnote{``Null'' indicates that no tag is annotated.} System: SB) \\
\underline {From Sunday (it is heard) rainy season would start}, so I rode my bike as much as possible.
\end{exe}

It seems quite difficult to infer causes of the errors in examples (20--24). Here, we quickly review the errors and explain why each tag is annotated as a gold standard. Example (20) demonstrates ``voice in mind,'' as introduced in section 3.2. Annotators, thus, annotated it as SJ in accordance with the guidelines. Moreover, it is the author's intention to control, not demonstrate a behavior. The system, however, tagged it as SB. Example (21) describes the author's behavior; the system did not annotate any behaviors or subjectivity. In example (22) the author observed that a couple of cars wanted to get by the other cars, and the annotators were not sure if any actual behavior was found to convince the annotators. The behavior is defined as observable from others, so the subjectivity was annotated. Example (23) is relatively simple. It is one behavior by others, which is quite similar to the correct example (10). Example (24) simply describes a road configuration; it does not include behavior, subjectivity, or important information that affects the preceding or following behavior. No behavior or subjectivity tag was annotated regarding the expression in the Gold.

\begin{exe}
\ex 
回転しそうに（\underline {後続車に追突したくない}）なったのでハンドル操作で制御 (Gold: SJ, System: SB) \\
Since I wanted to turn (\underline {I did not want to collide with the following cars}), I controlled (my car) by steering the wheel.

\ex 
朝5時半に起床、6時に家を出て、\underline {まずは西加茂広域農道を全域走破}。(Gold: SB, System: Null) \\
I got up at 5:30 in the morning, left my house at 6 o'clock, and \underline {first rode the bike through} \linebreak\underline{the whole area of the Nishikamo Wide Farm Road}.

\ex 
私の後にも数台\underline {抜きたい雰囲気丸出しな車が}ズラリ。 (Gold: SJ, System: OB) \\
Behind me, there was an array of a couple of cars \underline {with an atmosphere that they wanted to} \linebreak\underline{get by other cars}.

\ex 
細い道で大きい車がつっこんできました (Gold: OB, System: SJ) \\
\underline {A big car ran into a narrow street}.

\ex 
\underline {１車線がしばらく続き、大洲市に入る頃に２車線になる}。（Gold: Null, System: SB） \\
\underline {A one-lane road continues for a while, and it becomes a two-lane road when it enters} \linebreak\underline{Ozu city}.
\end{exe}


\subsubsection{Discussion}

The results of the CRF and the examples indicate the system has learned to identify behaviors and subjectivity. With our experimental settings and corpus size, however, the performances were not so different between the two classifiers. The corpus size may not be large enough for BiLSTM-CRF to perform well, although the size of the DEC (261 blog articles, 8,080 sentences) is larger than other human-annotated corpora, such as the Tsukuba Corpus and the Kyoto University and the NTT blog corpus. Most importantly, it is still difficult for the system to determine nominatives. Although we have added some driving- and personality-related knowledge in CRF, context and knowledge are still not enough; annotators read and understand texts with other information as a whole regarding the experiences and situations.

Zero anaphora resolution is an active topic in NLP in Japanese \cite<e.g.,>{ref37}. With improvements in zero anaphora resolution, we expect to apply such results to the CRF models by adding these as one of the features, and to the BiLSTM-CRF models by adding the identified nominatives in the blog sentences in the future.

While improvement in zero anaphora resolution is expected to reduce nominative errors, ambiguity of behavior may be improved if the guidelines are revised. The errors, such as moment and habitual behavior, could be improved if the guidelines were revised. We are, however, not planning to revise our guidelines and re-annotate the corpus for two reasons. First, such differentiation is not our research interest. Second, it is not really clear whether such differentiation is the best as the idea came about only based on the error analysis. It is difficult to determine what subsets of behavior are the most accurate.


\section{Conclusions and Future Work}

We constructed a Driving Experience Corpus by annotating behaviors and subjectivity. With the tagged textual data, we constructed a model that sought to extract driving scenes, behaviors as actors or observers, and their subjectivity as humans recognize them, which will help the system to have the knowledge to communicate with its users in a more informal and human-like manner. The annotation scheme and guidelines reflect the ideas and knowledge from psychology to differentiate behavior from subjectivity by judging whether described events/states are observable. High inter-annotator agreement ensured that the guidelines were applied consistently.

The discrepancies in the process of constructing the guidelines are essential characteristics in Japanese blogs or SNS texts. Thus, our guidelines are also useful to those who prepare blogs and SNS texts in Japanese.

It is important that the DEC targets driving experiences in Japanese, but the issues underlying our focus on behavior and subjectivity are universal and can apply to any other language. The accurate interpretation of nominatives is highly important in the context of the driving environment that entails accidental risks that sometimes lead to injuries and/or death. Furthermore, understanding drivers' behaviors and subjectivity helps the system evaluate their personality. Accurate personality evaluation enhances people's feelings of being understood \cite{ref29}. Drivers' feelings of being understood by the system enhance the reliability and trustworthiness of the system and improve the likelihood of following the system's messages or cautions. We are now planning to apply the trained model to the larger Driving Behavior and Subjectivity Corpus \cite{ref11} and elucidate associations between behavior and subjectivity. The corpus and guidelines are expected to be usable for research purposes in the future.


\acknowledgment

Part of this study was presented at the 32nd Conference of Pacific Asia Language, Information and Computation from the 1st to the 3rd of December 2018 in Hong Kong \cite{ref10}. In addition, part of this study will be presented at the 25th Annual Meeting of Natural Language Processing from the 12th to the 15th of March 2019 in Nagoya, Japan \cite{ref11}.

We thank Toyota Nagoya Education Center Inc. for allowing us to use textual data from the Driving Textbooks for the study. We also thank those who participated in the psychological experiments.

We thank Manami Ishikawa, Marika Horiuchi, and Natsuki Nikaido for their careful annotation of the DEC. We also thank Kento Hasegawa, Hana Kikuchi, Ryo Amimoto, Tatsuya Hayashi, Aya Sato, Ayumi Sato, Akiko Sasanuma, Riko Uekusa, and Emi Yokoyama for their support during the project.


\bibliographystyle{jnlpbbl_1.5}
\begin{thebibliography}{}

\bibitem[\protect\BCAY{Costa \BBA\ \mbox{McCrae}}{Costa \BBA\
  \mbox{McCrae}}{1992}]{ref1}
Costa, P.~T.\BBACOMMA\ \BBA\ \mbox{McCrae}, R.~R. \BBOP 1992\BBCP.
\newblock {\Bem Manual for the Revised NEO Personality Inventory (NEO-PI-R) and
  NEO Five-Factor Inventory (NEO-FFI)}.
\newblock Psychological Assessment Resources.

\bibitem[\protect\BCAY{Golbeck, Robles, Edmondson, \BBA\ Turner}{Golbeck
  et~al.}{2011}]{ref2}
Golbeck, J., Robles, C., Edmondson, M., \BBA\ Turner, K. \BBOP 2011\BBCP.
\newblock \BBOQ Predicting Personality from Twitter.\BBCQ\
\newblock In {\Bem Proceedings of 2011 IEEE 3rd International Conference on
  Privacy, Security, Risk and Trust and 2011 IEEE 3rd International Conference
  on Social Computing}, \mbox{\BPGS\ 149--156}.

\bibitem[\protect\BCAY{Goldberg}{Goldberg}{1992}]{ref3}
Goldberg, L.~R. \BBOP 1992\BBCP.
\newblock \BBOQ The Development of Markers for the Big-Five Factor
  Structure.\BBCQ\
\newblock {\Bem Psychological Assessment}, {\Bbf 4}  (1), \mbox{\BPGS\ 26--42}.

\bibitem[\protect\BCAY{Goldberg, Johnson, Eber, Hogan, Ashton, Cloninger, \BBA\
  Gough}{Goldberg et~al.}{2006}]{ref4}
Goldberg, L.~R., Johnson, J.~A., Eber, H.~W., Hogan, R., Ashton, M.~C.,
  Cloninger, C.~R., \BBA\ Gough, H.~G. \BBOP 2006\BBCP.
\newblock \BBOQ The International Personality Item Pool and the Future of
  Public-Domain Personality Measures.\BBCQ\
\newblock {\Bem Journal of Research in Personality}, {\Bbf 40}  (1),
  \mbox{\BPGS\ 84--96}.

\bibitem[\protect\BCAY{Hashimoto, Kurohashi, Kawahara, Shinzato, \BBA\
  Nagata}{Hashimoto et~al.}{2011}]{ref5}
Hashimoto, C., Kurohashi, S., Kawahara, D., Shinzato, K., \BBA\ Nagata, M.
  \BBOP 2011\BBCP.
\newblock \BBOQ Construction of a Blog Corpus with Syntactic Anaphoric and
  Sentiment Annotations.\BBCQ\
\newblock {\Bem Japanese Journal of Natural Language Processing}, {\Bbf 18}
  (2), \mbox{\BPGS\ 175--201}.

\bibitem[\protect\BCAY{Huang, Xu, \BBA\ Yu}{Huang et~al.}{2015}]{ref6}
Huang, Z., Xu, W., \BBA\ Yu, K. \BBOP 2015\BBCP.
\newblock \BBOQ Bidirectional LSTM-CRF Models for Sequence Tagging.\BBCQ\
\newblock In {\Bem arXiv preprint arXiv:1508.01991}.

\bibitem[\protect\BCAY{Inui, Abe, Hara, Morita, Sao, Eguchi, et~al.}{Inui
  et~al.}{2008}]{ref7}
Inui, K., Abe, S., Hara, K., Morita, H., Sao, C., Eguchi, M., et~al. \BBOP
  2008\BBCP.
\newblock \BBOQ Experience Mining: Building a Large-Scale Database of Personal
  Experiences and Opinions from Web Documents.\BBCQ\
\newblock In {\Bem Proceedings of the 2008 IEEE/WIC/ACM International
  Conference on Web Intelligence and Intelligent Agent Technology, IEEE},
  \mbox{\BPGS\ 314--321}.

\bibitem[\protect\BCAY{Iwai, Kawahara, Kumada, \BBA\ Kurohashi}{Iwai
  et~al.}{2018}]{ref10}
Iwai, R., Kawahara, D., Kumada, T., \BBA\ Kurohashi, S. \BBOP 2018\BBCP.
\newblock \BBOQ Annotating a Driving Experience Corpus with Behavior and
  Subjectivity.\BBCQ\
\newblock In {\Bem Proceedings of the 32nd Pacific Asia Conference on Language,
  Information and Computation (PACLIC32)}.
\newblock to appear. Hong Kong.

\bibitem[\protect\BCAY{Iwai, Kumada, Kawahara, \BBA\ Kurohashi}{Iwai
  et~al.}{2017}]{ref8}
Iwai, R., Kumada, T., Kawahara, D., \BBA\ Kurohashi, S. \BBOP 2017\BBCP.
\newblock \BBOQ Translating Big-Five Personality Constructs from English to
  Japanese, Using Statistical Machine Translation.\BBCQ\
\newblock {\Bem Poster presented at the 18th Annual Meeting of Society for
  Personality and Social Psychology}.

\bibitem[\protect\BCAY{Iwai, Kumada, Kawahara, \BBA\ Kurohashi}{Iwai
  et~al.}{2018}]{ref9}
Iwai, R., Kumada, T., Kawahara, D., \BBA\ Kurohashi, S. \BBOP 2018\BBCP.
\newblock \BBOQ Automatic Conceptual Translation Approach: Construction and
  Validation of a Japanese Five-Factor Personality Questionnaire Based on
  Statistical Machine Translation.\BBCQ\
\newblock {\Bem Poster presented at the 19th Annual Meeting of Society for
  Personality and Social Psychology}.

\bibitem[\protect\BCAY{Iwai, Kumada, Takahashi, Kawahara, \BBA\ Kurohashi}{Iwai
  et~al.}{2019}]{ref11}
Iwai, R., Kumada, T., Takahashi, N., Kawahara, D., \BBA\ Kurohashi, S. \BBOP
  2019\BBCP.
\newblock \BBOQ Development of a Driving-Related Dictionary that Includes
  Psychological Expressions.\BBCQ\
\newblock In {\Bem Proceedings of the 25th Annual Meeting of Natural Language
  Processing}, \mbox{\BPGS\ 1201--1204}.

\bibitem[\protect\BCAY{Jakob \BBA\ Gurevych}{Jakob \BBA\
  Gurevych}{2008}]{ref12}
Jakob, N.\BBACOMMA\ \BBA\ Gurevych, I. \BBOP 2008\BBCP.
\newblock \BBOQ Extracting Opinion Targets in a Single-and Cross-Domain Setting
  with Conditional Random Fields.\BBCQ\
\newblock In {\Bem Proceedings of the 2010 Conference on Empirical Methods in
  Natural Language Processing}, \mbox{\BPGS\ 1035--1045}.

\bibitem[\protect\BCAY{Kaji \BBA\ Kitsuregawa}{Kaji \BBA\
  Kitsuregawa}{2006}]{ref13}
Kaji, N.\BBACOMMA\ \BBA\ Kitsuregawa, M. \BBOP 2006\BBCP.
\newblock \BBOQ Automatic Construction of Polarity-Tagged Corpus from HTML
  Documents.\BBCQ\
\newblock In {\Bem Proceedings of the 21st International Conference on
  Computational Linguistics}, \mbox{\BPGS\ 452--459}.

\bibitem[\protect\BCAY{Kashiwagi, Tsuji, Fujishima, \BBA\ Yamada}{Kashiwagi
  et~al.}{2005}]{ref14}
Kashiwagi, S., Tsuji, H., Fujishima, Y., \BBA\ Yamada, N. \BBOP 2005\BBCP.
\newblock \BBOQ Re-evaluation of Tsuji's Psycholexical Study in Terms of the
  Big Five Personality Factors.\BBCQ\
\newblock {\Bem The Japanese Journal of Psychology}, {\Bbf 76}  (4),
  \mbox{\BPGS\ 368--374}.

\bibitem[\protect\BCAY{Kawabe, Miwa, \BBA\ Sasaki}{Kawabe et~al.}{2015}]{ref15}
Kawabe, K., Miwa, M., \BBA\ Sasaki, Y. \BBOP 2015\BBCP.
\newblock \BBOQ Transportation Terminology Recognition for Semi-Automatic
  Traffic Ontology Expansion.\BBCQ\
\newblock In {\Bem Proceedings of the 21st Annual Meeting of Natural Language
  Processing Association}, \mbox{\BPGS\ 135--138}.

\bibitem[\protect\BCAY{Krippendorff, Mathet, Bouvry, \BBA\
  Widl{\"{o}}cher}{Krippendorff et~al.}{2015}]{ref16}
Krippendorff, K., Mathet, Y., Bouvry, S., \BBA\ Widl{\"{o}}cher, A. \BBOP
  2015\BBCP.
\newblock \BBOQ On the Reliability of Unitizing Textual Continua: Further
  Developments.\BBCQ\
\newblock {\Bem Quality \& Quantity}, {\Bbf 50}  (6), \mbox{\BPGS\ 2347--2364}.

\bibitem[\protect\BCAY{Kurashima, Fujishima, \BBA\ Okuda}{Kurashima
  et~al.}{2009}]{ref17}
Kurashima, T., Fujishima, K., \BBA\ Okuda, H. \BBOP 2009\BBCP.
\newblock \BBOQ Mining Experiences from Large-Scale Blog Entries.\BBCQ\
\newblock {\Bem The IEICE Transactions on Information and Systems}, {\Bbf
  J92-D}  (3), \mbox{\BPGS\ 301--310}.

\bibitem[\protect\BCAY{Lafferty, McCallum, \BBA\ Pereira}{Lafferty
  et~al.}{2001}]{ref18}
Lafferty, J., McCallum, A., \BBA\ Pereira, F. \BBOP 2001\BBCP.
\newblock \BBOQ Conditional Random Fields: Probabilistic Models for Segmenting
  and Labeling Sequence Data.\BBCQ\
\newblock In {\Bem Proceedings of the 18th International Conference on Machine
  Learning}, \mbox{\BPGS\ 282--289}.

\bibitem[\protect\BCAY{Laskowski \BBA\ Burger}{Laskowski \BBA\
  Burger}{2006}]{ref19}
Laskowski, K.\BBACOMMA\ \BBA\ Burger, S. \BBOP 2006\BBCP.
\newblock \BBOQ Annotation and Analysis of Emotionally Relevant Behavior in the
  ISL Meeting Corpus.\BBCQ\
\newblock In {\Bem Proceedings of the 5th International Conference on Language
  Resources and Evaluation}, \mbox{\BPGS\ 1111--1116}.

\bibitem[\protect\BCAY{Ma \BBA\ Hovy}{Ma \BBA\ Hovy}{2016}]{ref20}
Ma, X.\BBACOMMA\ \BBA\ Hovy, E. \BBOP 2016\BBCP.
\newblock \BBOQ End-to-End Sequence Labeling via Bi-directional
  LSTM-CNNs-CRF.\BBCQ\
\newblock In {\Bem Proceedings of the 54th Annual Meeting of the Association
  for Computational Linguistics, \textup{Volume 1: Long Papers}}, \mbox{\BPGS\
  1064--1074}.

\bibitem[\protect\BCAY{McCrae, Terracciano, \BBA\ {\mbox{Members} of the
  Personality Profiles of Cultures Project}}{McCrae et~al.}{2005}]{ref22}
McCrae, R.~R., Terracciano, A., \BBA\ {\mbox{Members} of the Personality
  Profiles of Cultures Project} \BBOP 2005\BBCP.
\newblock \BBOQ Universal Features of Personality Traits from the Observer's
  Perspective: Data from 50 Cultures.\BBCQ\
\newblock {\Bem Journal of Personality and Social Psychology}, {\Bbf 88}  (3),
  \mbox{\BPGS\ 547--561}.

\bibitem[\protect\BCAY{McCrae, Zonderman, Costa, Bond, \BBA\ Paunonen}{McCrae
  et~al.}{1996}]{ref21}
McCrae, R.~R., Zonderman, A.~B., Costa, P.~T., Bond, M.~H., \BBA\ Paunonen,
  S.~V. \BBOP 1996\BBCP.
\newblock \BBOQ Evaluating Replicability of Factors in the Revised NEO
  Personality Inventory: Confirmatory Factor Analysis versus Procrustes
  Rotation.\BBCQ\
\newblock {\Bem Journal of Personality and Social Psychology}, {\Bbf 70}  (3),
  \mbox{\BPGS\ 552--566}.

\bibitem[\protect\BCAY{Murakami}{Murakami}{2002}]{ref23}
Murakami, Y. \BBOP 2002\BBCP.
\newblock \BBOQ A Collection of Basic Personality Trait Words.\BBCQ\
\newblock {\Bem The Japanese Journal of Personality}, {\Bbf 11}  (1),
  \mbox{\BPGS\ 35--49}.

\bibitem[\protect\BCAY{Murakami}{Murakami}{2003}]{ref24}
Murakami, Y. \BBOP 2003\BBCP.
\newblock \BBOQ Big Five and Psychometric Conditions for Their Extraction in
  Japanese.\BBCQ\
\newblock {\Bem The Japanese Journal of Personality}, {\Bbf 11}  (2),
  \mbox{\BPGS\ 70--85}.

\bibitem[\protect\BCAY{Nakagawa}{Nakagawa}{2017}]{ref25}
Nakagawa, H. \BBOP 2017\BBCP.
\newblock {\Bem Automatic Construction of Driving Behavior Corpus, Using
  Driving-Related Words Acquired from Corpus}.
\newblock Senior's Thesis, Kyoto University.

\bibitem[\protect\BCAY{Nakazawa, Ikeda, Yamada, Yoshimura, Suzuki, \BBA\
  Komachi}{Nakazawa et~al.}{2018}]{ref26}
Nakazawa, N., Ikeda, K., Yamada, M., Yoshimura, R., Suzuki, Y., \BBA\ Komachi,
  M. \BBOP 2018\BBCP.
\newblock \BBOQ Japanese Phrase-Unit Corpus with Polarity Extracted from Review
  Documents.\BBCQ\
\newblock In {\Bem Proceedings of the 24th Annual Meeting of Natural Language
  Processing Association}, \mbox{\BPGS\ 781--784}.

\bibitem[\protect\BCAY{Nasukawa \BBA\ Kamijo}{Nasukawa \BBA\
  Kamijo}{2017}]{ref28}
Nasukawa, T.\BBACOMMA\ \BBA\ Kamijo, K. \BBOP 2017\BBCP.
\newblock \BBOQ Estimating Authors' Personality in Japanese.\BBCQ\
\newblock In {\Bem Proceedings of the 23rd Annual Meeting of Natural Language
  Processing Association}, \mbox{\BPGS\ 807--810}.

\bibitem[\protect\BCAY{Nasukawa, Kamijo, Yamamoto, \BBA\ Kitamura}{Nasukawa
  et~al.}{2016}]{ref27}
Nasukawa, T., Kamijo, K., Yamamoto, M., \BBA\ Kitamura, H. \BBOP 2016\BBCP.
\newblock \BBOQ Investigation of Linguistic Features for Personality Estimation
  of Author in Japanese.\BBCQ\
\newblock In {\Bem Proceedings of the 22nd Annual Meeting of Natural Language
  Processing Association}, \mbox{\BPGS\ 1181--1141}.

\bibitem[\protect\BCAY{Oishi, Schiller, \BBA\ Gross}{Oishi
  et~al.}{2012}]{ref29}
Oishi, S., Schiller, J., \BBA\ Gross, E.~B. \BBOP 2012\BBCP.
\newblock \BBOQ Felt Understanding and Misunderstanding Affect the Perception
  of Pain, Slant, and Distance.\BBCQ\
\newblock {\Bem Social Psychological and Personality Science}, {\Bbf 4}  (3),
  \mbox{\BPGS\ 259--266}.

\bibitem[\protect\BCAY{Oshio, Abe, Cutrone, \BBA\ Gosling}{Oshio
  et~al.}{2014}]{ref30}
Oshio, A., Abe, S., Cutrone, P., \BBA\ Gosling, S.~D. \BBOP 2014\BBCP.
\newblock \BBOQ Further Validity of the Japanese Version of the Ten Item
  Personality Inventory (TIPI-J).\BBCQ\
\newblock {\Bem Journal of Individual Differences}, {\Bbf 35}  (4),
  \mbox{\BPGS\ 236--244}.

\bibitem[\protect\BCAY{Park, Schwartz, Eichstaedt, Kern, Kosinski, Stillwell,
  Ungar, \BBA\ Seligman}{Park et~al.}{2015}]{ref31}
Park, G., Schwartz, H.~A., Eichstaedt, J.~C., Kern, M.~L., Kosinski, M.,
  Stillwell, D.~J., Ungar, L.~H., \BBA\ Seligman, M.~E.~P. \BBOP 2015\BBCP.
\newblock \BBOQ Automatic Personality Assessment through Social Media
  Language.\BBCQ\
\newblock {\Bem Journal of Personality and Social Psychology}, {\Bbf 108}  (6),
  \mbox{\BPGS\ 934--952}.

\bibitem[\protect\BCAY{Plank \BBA\ Hovy}{Plank \BBA\ Hovy}{2015}]{ref32}
Plank, B.\BBACOMMA\ \BBA\ Hovy, D. \BBOP 2015\BBCP.
\newblock \BBOQ Personality Traits on Twitter -or- How to Get 1,500 Personality
  Tests in a Week.\BBCQ\
\newblock In {\Bem Proceedings of the 6th Workshop on Computational Approaches
  to Subjectivity, Sentiment and Social Media Analysis}, \mbox{\BPGS\ 92--98}.

\bibitem[\protect\BCAY{Reimers \BBA\ Gurevych}{Reimers \BBA\
  Gurevych}{2017}]{ref33}
Reimers, N.\BBACOMMA\ \BBA\ Gurevych, I. \BBOP 2017\BBCP.
\newblock \BBOQ Reporting Score Distributions Makes a Difference: Performance
  Study of LSTM-networks for Sequence Tagging.\BBCQ\
\newblock In {\Bem Proceedings of the 2017 Conference on Empirical Methods in
  Natural Language Processing}, \mbox{\BPGS\ 338--348}.

\bibitem[\protect\BCAY{Revelle \BBA\ Condon}{Revelle \BBA\
  Condon}{2015}]{ref34}
Revelle, W.\BBACOMMA\ \BBA\ Condon, D.~M. \BBOP 2015\BBCP.
\newblock \BBOQ A Model for Personality at Three Levels.\BBCQ\
\newblock {\Bem Journal of Research in Personality}, {\Bbf 56}, \mbox{\BPGS\
  70--81}.

\bibitem[\protect\BCAY{Saucier \BBA\ Goldberg}{Saucier \BBA\
  Goldberg}{2001}]{ref35}
Saucier, G.\BBACOMMA\ \BBA\ Goldberg, L.~R. \BBOP 2001\BBCP.
\newblock \BBOQ Lexical Studies of Indigenous Personality Factors: Premises,
  Products, and Prospects.\BBCQ\
\newblock {\Bem Journal of Personality}, {\Bbf 69}  (6), \mbox{\BPGS\
  847--879}.

\bibitem[\protect\BCAY{Schwartz, Eichstaedt, Kern, Dziurzynski, \mbox{Ramones},
  Agrawal, Shah, Kosinski, Stillwell, Seligman, \BBA\ Ungar}{Schwartz
  et~al.}{2013}]{ref36}
Schwartz, H.~A., Eichstaedt, J.~C., Kern, M.~L., Dziurzynski, L.,
  \mbox{Ramones}, S.~M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D.,
  Seligman, M.~E.~P., \BBA\ Ungar, L.~H. \BBOP 2013\BBCP.
\newblock \BBOQ Personality, Gender, and Age in the Language of Social Media:
  The Open-Vocabulary Approach.\BBCQ\
\newblock {\Bem PLoS ONE}, {\Bbf 8}  (9), \mbox{\BPG\ e73791}.

\bibitem[\protect\BCAY{Shibata \BBA\ Kurohashi}{Shibata \BBA\
  Kurohashi}{2018}]{ref37}
Shibata, T.\BBACOMMA\ \BBA\ Kurohashi, S. \BBOP 2018\BBCP.
\newblock \BBOQ Entity-Centric Joint Modeling of Japanese Coreference
  Resolution and Predicate Argument Structure Analysis.\BBCQ\
\newblock In {\Bem Proceedings of the 56th Annual Meeting of the Association
  for Computational Linguistics (ACL2018)}, \mbox{\BPGS\ 579--589}.

\bibitem[\protect\BCAY{Shimonaka, Nakazato, Gondo, \BBA\ Takayama}{Shimonaka
  et~al.}{1998}]{ref38}
Shimonaka, J., Nakazato, K., Gondo, K., \BBA\ Takayama, M. \BBOP 1998\BBCP.
\newblock \BBOQ Construction and Factorial Validation of the Japanese
  NEO-PI-R.\BBCQ\
\newblock {\Bem The Japanese Journal of Personality Psychology}, {\Bbf 6}  (2),
  \mbox{\BPGS\ 138--147}.

\bibitem[\protect\BCAY{Shimonaka, Nakazato, Gondo, \BBA\ Takayama}{Shimonaka
  et~al.}{1999}]{ref39}
Shimonaka, J., Nakazato, K., Gondo, K., \BBA\ Takayama, M. \BBOP 1999\BBCP.
\newblock {\Bem Revised NEO-Personality Inventory (NEO-PI-R) and NEO
  Five-Factor Inventory (NEO-FFI) Manual for the Japanese Version}.
\newblock Tokyo: Tokyo Shinri.

\bibitem[\protect\BCAY{Stenetorp, Pyysalo, Topi{\'{c}}, Ohta, Ananiadou, \BBA\
  Tsujii}{Stenetorp et~al.}{2012}]{ref40}
Stenetorp, P., Pyysalo, S., Topi{\'{c}}, G., Ohta, T., Ananiadou, S., \BBA\
  Tsujii, J. \BBOP 2012\BBCP.
\newblock \BBOQ BRAT: A Web-Based Tool for NLP-Assisted Text Annotation.\BBCQ\
\newblock In {\Bem Proceedings of the Demonstrations at the 13th Conference of
  the European Chapter of the Association for Computational Linguistics},
  \mbox{\BPGS\ 102--107}.

\bibitem[\protect\BCAY{Stoyanov \BBA\ Cardie}{Stoyanov \BBA\
  Cardie}{2008}]{ref41}
Stoyanov, V.\BBACOMMA\ \BBA\ Cardie, C. \BBOP 2008\BBCP.
\newblock \BBOQ Topic Identification for Fine-Grained Opinion Analysis.\BBCQ\
\newblock In {\Bem Proceedings of the 22nd International Conference on
  Computational Linguistics}, \mbox{\BPGS\ 817--824}.

\bibitem[\protect\BCAY{Suzuki \BBA\ Tsuda}{Suzuki \BBA\ Tsuda}{2013}]{ref42}
Suzuki, N.\BBACOMMA\ \BBA\ Tsuda, K. \BBOP 2013\BBCP.
\newblock \BBOQ An Effective Method for Habitual Behavior Extraction from the
  Internet.\BBCQ\
\newblock {\Bem Procedia Computer Science}, {\Bbf 22}, \mbox{\BPGS\ 599--605}.

\bibitem[\protect\BCAY{Suzuki \BBA\ Tsuda}{Suzuki \BBA\ Tsuda}{2014}]{ref43}
Suzuki, N.\BBACOMMA\ \BBA\ Tsuda, K. \BBOP 2014\BBCP.
\newblock \BBOQ Evaluation of Habitual Behavior Extraction Methods with
  Topics.\BBCQ\
\newblock In {\Bem Proceedings of the 28th Annual Conference of the Japanese
  Society for Artificial Intelligence}, \mbox{\BPGS\ 1--3}.

\bibitem[\protect\BCAY{Suzuki, Miwa, \BBA\ Sasaki}{Suzuki et~al.}{2015}]{ref44}
Suzuki, R., Miwa, M., \BBA\ Sasaki, Y. \BBOP 2015\BBCP.
\newblock \BBOQ SPARQL Query Conversion of Question Text for Traffic
  Ontology.\BBCQ\
\newblock In {\Bem Proceedings of the 21st Annual Meeting of Natural Language
  Processing Association}, \mbox{\BPGS\ 171--174}.

\bibitem[\protect\BCAY{Taira, Tanaka, \BBA\ Nagata}{Taira et~al.}{2014}]{ref45}
Taira, H., Tanaka, T., \BBA\ Nagata, M. \BBOP 2014\BBCP.
\newblock \BBOQ Construction of Car License Test Corpus for Textual
  Entailment.\BBCQ\
\newblock In {\Bem Proceedings of the 28th Annual Meeting of the Japanese
  Society for Artificial Intelligence}, \mbox{\BPGS\ 1--3}.

\bibitem[\protect\BCAY{Takayama, Miwa, \BBA\ Sasaki}{Takayama
  et~al.}{2017}]{ref46}
Takayama, J., Miwa, M., \BBA\ Sasaki, Y. \BBOP 2017\BBCP.
\newblock \BBOQ Constructing a Traffic Ontology-Based QA Dataset.\BBCQ\
\newblock In {\Bem Proceedings of the 23rd Annual Meeting of Natural Language
  Processing Association}, \mbox{\BPGS\ 691--694}.

\bibitem[\protect\BCAY{Tapus, {\c{T}}{\u{a}}pu{\c{s}}, \BBA\
  Matari{\'{c}}}{Tapus et~al.}{2008}]{ref47}
Tapus, A., {\c{T}}{\u{a}}pu{\c{s}}, C., \BBA\ Matari{\'{c}}, M.~J. \BBOP
  2008\BBCP.
\newblock \BBOQ User---Robot Personality Matching and Assistive Robot Behavior
  Adaptation for Post-Stroke Rehabilitation Therapy.\BBCQ\
\newblock {\Bem Intelligent Service Robotics}, {\Bbf 1}  (2), \mbox{\BPGS\
  169--183}.

\bibitem[\protect\BCAY{Tokuhisa, Inui, \BBA\ Matsumoto}{Tokuhisa
  et~al.}{2008}]{ref48}
Tokuhisa, R., Inui, K., \BBA\ Matsumoto, Y. \BBOP 2008\BBCP.
\newblock \BBOQ Emotion Classification Using Massive Examples Extracted from
  the Web.\BBCQ\
\newblock In {\Bem Proceedings of the 22nd International Conference on
  Computational Linguistics, {\upshape \textbf{1}}}, \mbox{\BPGS\ 881--888}.

\bibitem[\protect\BCAY{Tokuhisa, Inui, \BBA\ Matsumoto}{Tokuhisa
  et~al.}{2009}]{ref49}
Tokuhisa, R., Inui, K., \BBA\ Matsumoto, Y. \BBOP 2009\BBCP.
\newblock \BBOQ Emotion Classification: Using Massive Examples Extracted from
  the Web.\BBCQ\
\newblock {\Bem Japanese Journal of Information Processing}, {\Bbf 50}  (4),
  \mbox{\BPGS\ 1365--1374}.

\bibitem[\protect\BCAY{{Toyota Nagoya Education Center Inc.}}{{Toyota Nagoya
  Education Center Inc.}}{1998}]{ref50}
{Toyota Nagoya Education Center Inc.} \BBOP 1998\BBCP.
\newblock {\Bem Driving Textbook: Rules and Knowledge}.

\bibitem[\protect\BCAY{{Toyota Nagoya Education Center Inc.}}{{Toyota Nagoya
  Education Center Inc.}}{2004}]{ref51}
{Toyota Nagoya Education Center Inc.} \BBOP 2004\BBCP.
\newblock {\Bem Driving Textbook: Driving Techniques}.

\bibitem[\protect\BCAY{Ueda, Kawahara, Kurohashi, Iwai, Iseki, \BBA\
  Kumada}{Ueda et~al.}{2016}]{ref52}
Ueda, S., Kawahara, D., Kurohashi, S., Iwai, R., Iseki, R., \BBA\ Kumada, T.
  \BBOP 2016\BBCP.
\newblock \BBOQ Automatic Translation of English Personality Adjectives into
  Japanese.\BBCQ\
\newblock In {\Bem Proceedings of the 22nd Annual Meeting of Natural Language
  Processing Association}, \mbox{\BPGS\ 282--285}.

\bibitem[\protect\BCAY{Wada}{Wada}{1996}]{ref53}
Wada, S. \BBOP 1996\BBCP.
\newblock \BBOQ Construction of the Big Five Scales of Personality Trait Terms
  and Concurrent Validity with NPI.\BBCQ\
\newblock {\Bem The Japanese Journal of Psychology}, {\Bbf 67}  (1),
  \mbox{\BPGS\ 61--67}.

\bibitem[\protect\BCAY{Welch \BBA\ Behrmann}{Welch \BBA\
  Behrmann}{2018}]{ref54}
Welch, D.\BBACOMMA\ \BBA\ Behrmann, E. \BBOP 2018\BBCP.
\newblock \BBOQ \mbox{Who's Winning the Self-Driving Car Race?: A Scorecard}
  Breaking Down Everyone from Alphabet's Waymo to Zoox (2018 May 7).\BBCQ\
\newblock {\Bem Bloomberg}.
\newblock Retrieved December 24, 2018, from https://www.bloomberg.com/.

\end{thebibliography}


\clearpage
\begin{biography}

\bioauthor[:]{Ritsuko Iwai}{
Ritsuko Iwai received her B.A. in Arts Science from International Christian University in 2001 and M.S. in Environmental Psychology from University of Surrey in 2002. After working at a company, she received M.A. in Education from International Christian University in 2013. She is currently a Ph.D. candidate in Informatics and a program-specific researcher at the Graduate School of Informatics at Kyoto University. She is interested in applying natural language processing to psychological research, in particular, personality and human behavior.}

\bioauthor[:]{Takatsune Kumada}{
Takatsune Kumada received his B.A., M.A., and Ph.D. in Psychology from University of Tsukuba in 1986, 1988 and 1991, respectively. He is currently a professor of the Graduate School of Informatics at Kyoto University. His research interests include visual cognition, executive control and human intelligence.}

\bioauthor[:]{Daisuke Kawahara}{
Daisuke Kawahara received his B.S. and M.S. in \mbox{Electronic} Science and Engineering from Kyoto University in 1997 and 1999, respectively. He obtained his Ph.D. in Informatics from Kyoto University in 2005. He is currently an associate professor at the Graduate School of Informatics at \linebreak Kyoto University. His research interests center on natural language processing, particularly knowledge acquisition and text understanding.}

\bioauthor[:]{Sadao Kurohashi}{
Sadao Kurohashi received the B.S., M.S., and Ph.D. in Electrical Engineering from Kyoto University in 1989, 1991 and 1994, respectively. He has been a visiting researcher of IRCS, University of Pennsylvania in 1994. He is currently a professor of the Graduate School of Informatics at Kyoto University. His research interests include natural language processing, knowledge acquisition/representation, and information retrieval. He received the 10th anniversary best paper award from Journal of Natural Language Processing in 2004, 2009 Funai IT promotion award, and 2009 IBM faculty away.}

\end{biography}

\biodate



\end{document}
