


\documentstyle[epsf,nlpbbl]{jnlp_e}

\setcounter{page}{21}
\setcounter{巻数}{2}
\setcounter{号数}{3}
\setcounter{年}{1995}
\setcounter{月}{7}
\受付{August}{9}{1995}
\再受付{October}{11}{1995}
\採録{December}{14}{1995}

\setcounter{secnumdepth}{2}

\title{}
\author{}
\jkeywords{}

\etitle{Use of Multiple Documents as Evidence\\ with Decreased Adding\\ in a Japanese Question-answering System}

\eauthor{Masaki Murata\affiref{NICT}   \and
  Masao Utiyama\affiref{NICT}        \and
        Hitoshi Isahara\affiref{NICT}}

\headauthor{Murata,~M.~et~al.}
\headtitle{Use of Multiple Documents in a Japanese Question-answering System}

\affilabel{NICT}
          {National Institute of Information and Communications Technology, Independent Administrative Institution, 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0289, Japan}
          {National Institute of Information and Communications Technology, Independent Administrative Institution, 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0289, Japan}

\eabstract{
We propose a new method of using 
multiple documents as evidence with decreased adding
to improve the performance of question-answering systems. 
Sometimes, the answer to a question may be found in multiple documents. 
In such cases, using multiple documents to predict answers
may generate better answers than using a single document. 
Our method therefore uses information from multiple documents, 
adding the scores of candidate answers 
extracted from various documents. 
However, because simply adding the scores can degrade the performance of 
question-answering systems, 
we add the scores with progressively decreasing weights to reduce the negative effect of simple adding. 
We carried out experiments using the Question-Answering Challenge (QAC) test collection.
The results showed that our method produced a statistically significant improvement, 
with the degree of improvement ranging from 0.05 to 0.14.
These results, and the fact that our method is simple and easy to use, 
indicate its potential feasibility and utility in question-answering systems. 
Experiments comparing our decreased adding method with
several previously proposed methods that use multiple documents
showed that our method was more effective than these other methods. 
}

\ekeywords{Question Answering, Multiple Documents, Decreased Adding, Combined Method}

\begin{document}

\maketitle

\section{Introduction}

Question-answering systems are designed 
to produce the correct answer to a question given as input. 
For example, when the question, ``What is the capital of Japan?'', is the input, 
a question-answering system might retrieve a document containing sentences like 
``Tokyo is Japan's capital and the country's largest and most important city. Tokyo is
also one of Japan's 47 prefectures.'' 
from online text, such as a web site, a newspaper article, or an
encyclopedia. The system then outputs ``Tokyo'' as the correct answer. 
Question-answering systems are likely to become increasingly important as a 
convenient alternative to systems designed for information retrieval, and as a 
basic component of future artificial intelligence systems. 
This field is attracting increasing research interest 
and several 
interesting studies on question-answering systems have been published recently \cite{MURAX,IBM_QA2001,Harabagiu_tois,Sasaki_yutaka_ieice,takahashi_ieice_qa}. 
Evaluated conferences, or contests, 
based on question-answering systems have been held in both the U.S.A. and Japan. 
In the U.S.A., a question answering track at the Text REtrieval Conference (TREC) was held \cite{trec8qa}, while
a similar conference called 
the Question-Answering Challenge (QAC) was held in Japan \cite{qac_hp}.
At evaluation conferences, which are aimed at improving the performance of question-answering systems, 
researchers uses the systems they have developed 
to answer the same questions; the performance of
each system is then examined to glean possible improvements.  
We investigated the potential of question-answering systems \cite{qa_memo2} and 
studied their construction by participating in the 
QAC \cite{qac1} at NTCIR 3 \cite{Murata_ntcir3_qa}. 

In this paper, we propose a new method of using 
multiple documents as evidence with decreased adding
to improve the performance of question-answering systems.
Often, the answer to a question may be found in multiple documents. 
In such cases, using multiple documents to predict the answer 
may produce higher accuracy than using only one document \cite{Charles,Dumis,magnini,takaki_ntcir3_qa}. 
In our method, information from multiple documents 
is used by adding the scores of candidate answers 
extracted from various documents \cite{Dumis,takaki_ntcir3_qa}. 
Because simply adding the scores degrades the performance of a question-answering system, 
our method adds the scores with progressively decreasing weights to overcome the problems associated with simple adding, i.e. 
the score of the $i$-th candidate answer is multiplied by a factor of $k^{(i-1)}$ 
before the score is added to the running total. 
The final answer is then determined based on the total score. 
For example, suppose that ``Tokyo'' is extracted as a candidate answer from three documents and 
has scores of ``26'', ``21'', and ``20'', and assume that $k$ is 0.3. 
In this case, the total score for ``Tokyo'' is ``34.1'' (= 26 + 21 $\times$ 0.3 + 20 $\times$ $0.3^2$). 
The score is calculated in the same way for each candidate 
and the answer with the highest score is taken to be the correct one.

To evaluate our method, we carried out experiments using the Question-Answering Challenge (QAC) test collection \cite{qac1}. 
The experiments were based on a 
question-answering system that we constructed. Five variations 
of the system were tested. 
The results showed that 
simply adding the scores from multiple documents, without employing decreasing weights, 
degraded the performance of the question-answering system in some cases. 
Our method is simple and easy to use, and 
in our experiments it improved the performance of the question-answering system, thus
demonstrating its feasibility and utility. 

In Section \ref{sec:method} of this paper, we explain our proposed method 
of using multiple documents as evidence with decreased adding. 
In the next section, we describe
the question-answering system used in our experiments. 
In Sections \ref{sec:experiments} and \ref{sec:discussions}, we describe
the experiments 
and discuss the results. 
Concluding remarks are presented in Section \ref{sec:conclusions}.  

\section{Use of Multiple Documents as Evidence with Decreased Adding}
\label{sec:method}

\begin{table}[t]
  \begin{center}
    \caption{Candidate answers with original scores, where ``Tokyo'' is the correct answer}
    \begin{tabular}{|c|l|r|l|} \hline
Rank & Candidate answer & Score & \multicolumn{1}{c|}{Document ID}\\\hline
1 & Kyoto & 3.3 & 926324\\
2 & Tokyo & 3.2 & 259312\\
3 & Tokyo & 2.8 & 451245\\
4 & Tokyo & 2.5 & 371922\\
5 & Tokyo & 2.4 & 221328\\
6 & Beijing & 2.3 &113127\\
... & ... & ...& ...\\\hline
\end{tabular}
\label{tab:plural_candidates}
\end{center}
\end{table}

Suppose that the question, ``What is the capital of Japan?'', 
is input to a question-answering system, with the goal of 
obtaining the correct answer, ``Tokyo''.
A typical question-answering system would output 
the candidate answers and scores listed in Table \ref{tab:plural_candidates}.
These systems also typically output a document ID 
corresponding to the document from which each candidate answer was extracted.

In the example shown in 
Table \ref{tab:plural_candidates}, 
the system outputs an incorrect answer, ``Kyoto'', as the highest ranked candidate. 

\begin{table}[t]
  \begin{center}
    \caption{Candidate answers with simply added scores where ``Tokyo'' is the correct answer}
    \begin{tabular}{|c|l|r|l|} \hline
Rank & Candidate answer & Score & \multicolumn{1}{c|}{Document ID}\\\hline
1 & Tokyo & 10.9 & 259312, 451245, 371922, 221328\\
2 & Kyoto & 3.3 & 926324\\
3 & Beijing & 2.3 &113127\\
... & ... & ...& ...\\\hline
\end{tabular}
\label{tab:plural_candidates2}
\end{center}
\end{table}

A method based on simply adding the scores of candidate answers 
was proposed previously \cite{Dumis,takaki_ntcir3_qa}. 
In the current example, this would produce
the results shown in Table \ref{tab:plural_candidates2}. 
In this case, the system outputs the correct answer, ``Tokyo'', as the highest ranked candidate. 
This method thus produces the correct answer using multiple documents as evidence. 

Unfortunately, this simple adding approach 
is more likely to select candidate answers with 
a higher frequency of occurrence, which 
is a serious problem from a performance standpoint. 
In the case of a system with good inherent performance, 
the original scores that it outputs are often more 
reliable than the simply added scores, so
the use of simple adding methods can degrade
system performance. 

To overcome this problem, we developed a new method 
of assessing potential answers from multiple documents. 
Instead of simply adding the scores of the candidate answers, 
we add the scores with progressively decreasing weights. 
This approach reduces the likelihood 
that a question-answering system will select candidate answers with high frequencies, 
while still improving the accuracy by adding the scores. 

\begin{table}[t]
  \begin{center}
    \caption{Candidate answers with original scores, where ``Kyoto'' is the correct answer}
    \begin{tabular}{|c|l|r|l|} \hline
Rank & Candidate answer & Score & \multicolumn{1}{c|}{Document ID}\\\hline
1 & Kyoto & 5.4 & 926324\\
2 & Tokyo & 2.1 & 259312\\
3 & Tokyo & 1.8 & 451245\\
4 & Tokyo & 1.5 & 371922\\
5 & Tokyo & 1.4 & 221328\\
6 & Beijing & 1.3 &113127\\
... & ... & ...& ...\\\hline
\end{tabular}
\label{tab:plural_candidates3}
\end{center}
\end{table}

The effect of our proposed method is demonstrated in the following example. 
Suppose that a question-answering system outputs 
Table \ref{tab:plural_candidates3} in response to the question, 
``What was the capital of Japan in AD 1000?''. 
The correct answer is ``Kyoto'', 
and the system outputs the correct answer as the highest ranked candidate. 


\begin{table}[t]
  \begin{center}
    \caption{Candidate answers with simply added scores where ``Kyoto'' is the correct answer}
    \begin{tabular}{|c|l|r|l|} \hline
Rank & Candidate answer & Score & \multicolumn{1}{c|}{Document ID}\\\hline
1 & Tokyo & 6.8 & 259312, 451245, 371922, 221328\\
2 & Kyoto & 5.4 & 926324\\
3 & Beijing & 1.3 &113127\\
... & ... & ...& ...\\\hline
\end{tabular}
\label{tab:plural_candidates4}
\end{center}
\end{table}

If we apply the method of simply adding scores 
to this system, however, 
we obtain the results shown in Table \ref{tab:plural_candidates4}. 
In this case, an incorrect answer, ``Tokyo'', is given the highest score. 

\begin{table}[t]
  \begin{center}
    \caption{Candidate answers obtained by decreased adding, where ``Kyoto'' is the correct answer}
    \begin{tabular}{|c|l|r|l|} \hline
Rank & Candidate answer & Score & \multicolumn{1}{c|}{Document ID}\\\hline
1 & Kyoto & 5.4 & 926324\\
2 & Tokyo & 2.8 & 259312, 451245, 371922, 221328\\
3 & Beijing & 1.3 &113127\\
... & ... & ...& ...\\\hline
\end{tabular}
\label{tab:plural_candidates5}
\end{center}
\end{table}

To attempt to overcome this problem, 
we applied our proposed method 
of adding candidate scores with decreasing weights. 
Suppose that 
we implement our method by multiplying the score of the $i$-th candidate by a weighting factor of $0.3^{(i-1)}$
before adding the score of this candidate to the running total. 
In this case, the score for ``Tokyo'' becomes 2.8 (= 2.1 $+$ $1.8 \times 0.3$ $+$ $1.5 \times 0.3^2$ $+$ $1.4 \times 0.3^3$) 
and we obtain the results shown in Table \ref{tab:plural_candidates5}. 
The correct answer, 
``Kyoto'', is given the highest score, 
while the score for ``Tokyo'' is significantly lower. 

\begin{table}[t]
  \begin{center}
    \caption{Candidate answers obtained by decreased adding, where ``Tokyo'' is the correct answer}
    \begin{tabular}{|c|l|r|l|} \hline
Rank & Candidate answer & Score & \multicolumn{1}{c|}{Document ID}\\\hline
1 & Tokyo & 4.3 & 259312, 451245, 371922, 221328\\
2 & Kyoto & 3.3 & 926324\\
3 & Beijing & 2.3 &113127\\
... & ... & ...& ...\\\hline
\end{tabular}
\label{tab:plural_candidates6}
\end{center}
\end{table}

We can also apply our method to
the first example question, ``What is the capital of Japan?''. 
In this case, 
the score for ``Tokyo'' becomes 4.3 (= 3.2 $+$ $2.8 \times 0.3$ $+$ $2.5 \times 0.3^2$ $+$ $2.4 \times 0.3^3$), 
and we obtain the results shown in Table \ref{tab:plural_candidates6}. 
As expected, ``Tokyo'' is given the highest score. 

As these simple examples show, 
our method of adding scores with decreasing weights 
produced the correct answer to both questions. 
This demonstrates the ability of our method
to reduce the likelihood
that a question-answering system will select answers that occur more frequently in texts, 
while at the same time improving the system's accuracy. 
To further examine the effectiveness of our method 
we carried out a series of experiments, as described in Section \ref{sec:experiments}. 

\section{Question-answering Systems Used in This Study}
\label{sec:qas}

In this section, we describe the question-answering system 
that we used
to experimentally assess the effectiveness of the proposed method. 

The system utilizes three basic components, which are described briefly below and 
in more detail in the following subsections:
\begin{enumerate}
\item \label{enum:expect_answer_type}
\underline{Answer type prediction} 

The system predicts the answer to be a particular type of question based on 
whether it includes an interrogative pronoun, 
an adjective, or an adverb. 
For example, in the input question, ``Who is the prime minister of Japan?'', 
the interrogative pronoun, "Who", suggests that the answer will be a person's name. 

\item \label{enum:document_retrieval}
\underline{Document retrieval} 

The system extracts terms from the input question and uses these terms to retrieve documents
that are likely to contain the 
correct answer.  
From the input question, ``Who is the prime minister of Japan?'', the system extracts 
``prime'', ``minister'', and ``Japan'' as terms and retrieves documents accordingly. 

\item 
\underline{Answer detection} 

The system extracts linguistic expressions that match the predicted answer type, 
as described above, from the retrieved documents. 
It then outputs the extracted expressions as candidate answers.  
For example, 
to answer the question, ``Who is the prime minister of Japan?'', the system extracts 
people's names as candidate answers 
from documents containing the terms ``prime'', ``minister'', and ``Japan''. 

\end{enumerate}

\begin{table}[t]
  \begin{center}
    \caption{Answer types used in the system}
    \begin{tabular}{|l|l|l|} \hline
Answer type  & Definition & Example \\\hline
TIME         & date and time   & {\it ichigatu} (January) \\
PERSON       & person's name       & {\it yamada san} (Mr. Yamada)\\ 
LOCATION     & name of a location  & {\it toukyou} (Tokyo)\\
ORGANIZATION & name of an organization & {\it shaapu} (Sharp)\\
ARTIFACT     & a product name, prize name, etc. & {\it Mac} (Mac)\\
COUNTRY      & name of a country      & {\it nihon} (Japan)\\
NUMERICAL    & an expression using a numerical expression & {\it 130 en} (130 yen)\\\hline
\end{tabular}
\label{tab:answer_types}
\end{center}
\end{table}

\subsection{Answer type prediction}
\label{sec:expectation}

The system predicts answer types, 
unit expressions, and NOUN-FOCUS. 
Since unit expressions are useful for answer detection, and 
NOUN-FOCUS is useful for answer-type prediction, 
we used them in addition to answer types. 
The system uses seven different answer types. 
These are shown in Table \ref{tab:answer_types}.
A unit expression is an expression used with 
a numerical expression. For example, 
a unit expression for the question 
``{\it ichi doru wa nan en desuka?}'' (How many yen to one dollar?) 
is ``{\it en}'' (yen) and 
an example of the answer is ``{\it 120 en}'' (120 yen). 
The unit expression is used as part of the answer expression. 
The NOUN-FOCUS means the noun that is the focus of the question sentence. 
For example, 
the NOUN-FOCUS for the question 
``{\it Mac wo tsukutta kaisha wa doko desuka?}'' (Which company 
developed the Mac?) 
is ``{\it kaisha}'' (the company) and 
the answer is ``{\it appuru}'' (Apple), which is a company name. 
If the NOUN-FOCUS is ``{\it kaisha}'' (the company), 
we find that the answer is the name of an organization (which has the answer type, ORGANIZATION, in Table \ref{tab:answer_types}). 
The NOUN-FOCUS is an important clue expression for question-answering systems 
and is used for both answer-type prediction 
and answer detection in our system. 

The system applies 39 manually 
defined heuristic rules to predict 
answer types, unit expressions, and NOUN-FOCUS. 
Some of them are listed here.
\begin{enumerate}
\item 
When ``{\it dare}'' (who) occurs in a question, PERSON is given as the answer type. 

\item 
When ``{\it itsu}'' (when) occurs in a question, TIME is given as the answer type. 

\item 
When ``{\it donokurai}'' (how many) occurs in a question, NUMERICAL is given as the answer type. 

\item 
When ``{\it nani} Suffix X'' (how many X) occurs in a question, ``Suffix X'' is extracted as a unit expression 
and NUMERICAL is given as the answer type. 

\item 
When ``X {\it wa doko/nani}'' (where/what is X) occurs in a question, expressions 
corresponding to X are extracted as the NOUN-FOCUS. 

\item 
When ``{\it doko}'' (where) occurs in a question and the NOUN-FOCUS is not ``{\it kaisha}'' (company), ``{\it soshiki}'' (organization), 
or a similar term, LOCATION is given as the answer type. 

\item 
When ``{\it doko}'' (where) occurs in a question and the NOUN-FOCUS is not ``{\it chiiki}'' (area), ``{\it basho}'' (location), 
or a similar term, ORGANIZATION is given as the answer type. 

\item 
When ``{\it doko no kuni}'' (what country) occurs in a question, COUNTRY is given as the answer type. 

\item 
When ``{\it nani}'' (what) occurs in a question and the NOUN-FOCUS is not ``{\it kaisha}'' (company), ``{\it chiki}'' (area), or so on, ARTIFACT is given as the answer type. 

\item 
When ``{\it nani}'' (what) occurs in a question and the NOUN-FOCUS is ``{\it chiiki}'' (area), or a similar term, LOCATION is given as the answer type. 

\end{enumerate}

In this system, 
more than one answer type can be given. 
For example, 
when both the sixth and seventh rules apply to a question, 
both LOCATION and ORGANIZATION are given as the answer type. 

\subsection{Document retrieval}
\label{sec:document_retrieval}

The system extracts terms from questions using a morphological 
analyzer, ChaSen\cite{chasen}. The analyzer first eliminates terms whose parts of speech are 
prepositions or similar. The system then performs retrieval by using the remaining extracted terms. 

Our system is capable of retrieving entire documents. 
In general, question-answering systems 
divide each document into small passages before executing document 
retrieval. For example, in our 
previous study, we divided documents into sets of three sentences \cite{qa_memo2,murata2000_1_nl_eng}. 
The answer to a question in a document often occurs 
near terms extracted from the question. 
Question-answering systems must therefore extract answers from documents in which 
the extracted terms occur near each other. 
Although this condition 
is very important, existing information retrieval methods have 
not taken it into consideration. As a result, divided passages have generally 
been used for document retrieval. 
Using these small divided passages, 
the systems check whether the extracted terms occur near each other within a small area. 
The document retrieval method used in our question-answering system, however, 
verifies that terms occur near each other by applying probability theory. 
This enables the system to utilize the entire document in the retrieval process. 

The document retrieval method operates as follows.
\begin{enumerate}
\item 
We first retrieve the top $k_{dr1}$ documents with the highest scores calculated from the following
equation

{
\begin{eqnarray}
  \label{eqn:robertson}
  & \hspace*{-1cm}
 Score(d) = \displaystyle \sum_{\begin{minipage}[h]{0.8cm}
    term $t$ 
  \end{minipage}} & \hspace*{-0.3cm} \left(  \displaystyle 
\frac{tf(d,t)}{\displaystyle 
tf(d,t) + k_{t} \frac{length(d)+k_{+}}{\Delta+k_{+}}} \times \ log\frac{N}{df(t)} \right),
\end{eqnarray}}

\noindent 
where $d$ is a document, $t$ is a term extracted from a question, $tf(d,t)$ is 
the frequency of $t$ in document $d$, $df(t)$ is the number of documents 
in which $t$ appears, $N$ is the total number of documents, $length(d)$ is the length of 
$d$ in characters, and $\Delta$ is the average length of all documents. 
$k_{t}$ and $k_{+}$ are constants defined according to experimental results. 

The values used in this study were $k_{t} = 0.00001$ and $k_{+} = 20$. 

We based this equation on Robertson's equation \cite{2poisson,robertson_trec3}. This 
approach has been shown to be effective, and we have used it extensively for information retrieval 
\cite{iral2000,Murata_ntcir2,Murata_ntcir3}. 

\item 
Next, we re-rank the extracted documents according to the following equation and extract 
the top $k_{dr2}$ documents, which are used in the ensuing answer 
extraction phase. 

{
\begin{eqnarray}
  \label{eqn:ir_near}
Score(d) & = &  \displaystyle - {min}_{t1 \in T} log \prod_{t2 \in T3} 
(2dist(t1,t2)\frac{df(t2)}{N})^{w_{dr2}(t2)} \\
& = &  \displaystyle {max}_{t1 \in T} \sum_{t2 \in T3} w_{dr2}(t2) 
log 
\frac{N}{2dist(t1,t2)*df(t2)},
\end{eqnarray}}

  {
    \begin{eqnarray}
      \label{eqn:new_test4}
      T3 = \{t| t \in T, 2dist(t1,t)\frac{df(t)}{N} \leq 1\},
    \end{eqnarray}}
  
\noindent 
where $d$ is a document, $T$ is the set of terms that appear in the question and the document, and 
$dist(t1,t2)$ is the distance between 
the terms $t1$ and $t2$ (defined as the number of characters between them), with $dist(t1,t2)$ = 
0.5 when $t1 = t2$. $df(t)$ is the number of times $t$ occurs in all documents, $N$ is 
the total number of characters in all documents, and $w_{dr2}(t2)$ is a function of 
$t2$ that is adjusted according to experimental results.  

Equation (\ref{eqn:ir_near}) can be deduced through probability theory 
in the case of $w_{dr2}(t2)=1$. 
We start by considering the probability that there is a pattern of occurrence for the extracted terms. 
First, we select $t1$. The occurrence probability of $t1$ is $\frac{df(t1)}{N}$. 
Next, we consider another term, $t2$. The probability of it occurring 
within $dist(t1,t2)$ of $t1$ is 
$2dist(t1,t2)\frac{df(t2)}{N}$\footnote{Strictly speaking, 
the probability is $1 - (1 - \frac{df(t2)}{N})^{2dist(t1,t2)}$. 
When we apply Taylor's formula ($(1 - x)^n = 1 - nx \ (x \ll 1)$), 
the probability becomes $2dist(t1,t2)\frac{df(t2)}{N}$.}. 
Therefore, the probability of an occurrence pattern is 

{
\begin{eqnarray}
\label{eqn:ir_near3}
\displaystyle \frac{df(t1)}{N} \prod_{t2 \in T3, t2 \neq t1} 
(2dist(t1,t2)\frac{df(t2)}{N}).
\end{eqnarray}}

The parameter value used in this study was $w_{dr2}(t) = 0.3$ 
when the term $t$ was a verb; otherwise, the value was 1. 

If we assume that $dist(t1,t2)$ = 0.5 when $t1 = t2$, the above equation can be written as follows:
{
\begin{eqnarray}
\label{eqn:ir_near3_add}
\displaystyle \prod_{t2 \in T3} 
(2dist(t1,t2)\frac{df(t2)}{N}),
\end{eqnarray}}
and it is similar to Eq. (\ref{eqn:ir_near}). When the probability of an occurrence pattern is 
low, 
the pattern is more valuable, so we preferentially extract documents with low scores from Eq. 
(\ref{eqn:ir_near3}). Since Eq. (\ref{eqn:ir_near}) is multiplied by a factor of $-1$, we 
can extract 
documents with high scores from Eq. (\ref{eqn:ir_near}). 

\end{enumerate}

Because our question-answering system can determine whether 
terms occur near each other by re-ranking them according to Eq. \ref{eqn:ir_near}, 
it can use entire documents for retrieval. 

In this study, we extracted 20 documents for retrieval.  
The following answer-detection procedure was 
applied to these 20 documents extracted by the system. 

\subsection{Answer detection}
\label{sec:detection_of_the_answer}

To detect answers, the system first generates candidate expressions for the answer 
from the extracted documents. We initially used morpheme n-grams for candidate 
expressions, but this approach generated too many candidates so we 
now 
use candidates consisting only of nouns, unknown words, and symbols. We also 
use the ChaSen analyzer to determine morphemes and their parts of speech. 

Our approach to judging whether a candidate is the correct answer is to add the
score ($Score_{near}(c)$) for the candidate, based on how near it is to an extracted term, to 
the score ($Score_{sem}(c)$) based on our heuristic rules according to the answer type. The system then selects the 
candidate with the highest total score as the correct answer. 

We used the following equation to calculate the score for a candidate $c$ 
under the condition that it must be near the extracted terms. 

{
\begin{eqnarray}
  \label{eqn:ir_near_}
Score_{near}(c) & = & \displaystyle - log \prod_{t2 \in T3} 
(2dist(c,t2)\frac{df(t2)}{N})^{w_{dr2}(t2)} \\
& = & \displaystyle \sum_{t2 \in T3} w_{dr2}(t2) log 
\frac{N}{2dist(c,t2)*df(t2)},
\end{eqnarray}}

  {
    \begin{eqnarray}
      \label{eqn:new_test5}
      T3 = \{t| t \in T, 2dist(c,t)\frac{df(t)}{N} \leq 1\},
    \end{eqnarray}}
  
\noindent 
where $c$ is a candidate for the correct answer, $T$ is a set of terms that appear in the question and the document, $dist(c,t2)$ is 
the distance between $c$ and $t2$, $df(t)$ is the number of times $t$ occurs in all the 
documents, $N$ is the total number of characters in all the documents, and 
$w_{dr2}(t2)$ is a function of $t2$, that is adjusted according to the experimental 
results.  The parameter value used in this study was $w_{dr2}(t) = 0.3$ 
when term $t$ was a verb, and 1 otherwise. 

$Score_{near}(c)$ is thus a heuristic function based on the condition that the answer 
must be near certain terms. 

Next, we describe how the score ($Score_{sem}(c)$) is calculated based on heuristic rules for the 
predicted answer type. In this study, we used 45 heuristic rules to award points to candidates and 
utilized the total points as the score. Some of the rules are listed
below. 
\begin{enumerate}
\item 
Add 1000 to candidates when they match one of the predicted answer types (LOCATION, 
ORGANIZATION, PERSON, ARTIFACT, or TIME) or consist of a numerical 
expression + NUMERICAL or a predicted unit expression, 
when they have job-related meanings and the NOUN-FOCUS 
is ``{\it shigoto}'' (work), ``{\it shokugyou}'' (job), or a similar term, or when they include ``Noun X'' and the question 
contains ``{\it nani} Noun X'' (what Noun X) or a similar expression.

This is the most fundamental rule, and we have adjusted it 
to ensure that a good candidate
answer will almost always satisfy it. 

We use named-entity (NE) extraction techniques to judge whether a candidate matches 
one of the predicted answer types: LOCATION, ORGANIZATION, PERSON, ARTIFACT, or TIME. We applied a machine-learning 
approach to NE extraction, and used CRL (Communications Research 
Laboratory) NE data for training \cite{acl_NE}. Only 
LOCATION, ORGANIZATION, PERSON, ARTIFACT, TIME, and DATE were selected from 
among the categories in the CRL NE data. We thus developed six individual NE extraction systems, i.e. 
one for each category. 
Each system judges whether a candidate expression belongs in its 
category. For example, the NE system for LOCATION judges whether a candidate expression is a 
LOCATION. Note that in our question-answering system, a candidate can belong to more than one NE category. 
For machine learning, we utilized a support vector machine
\cite{SVM,kudoh_svm,yamada_ipsj_svm_ne_eng}. When a candidate is judged as a 
DATE, we classify it as TIME because the question-answering system does not employ a DATE category. 

\item 
When the NOUN-FOCUS is ``{\it shigoto}'' (work), ``{\it shokugyou}'' (job), or a similar term, add 1000 to 
candidates with job-related meanings. Whether a candidate has 
this kind of meaning is determined from the EDR thesaurus \cite{EDR93e}. 

\item 
When COUNTRY is one of the predicted answer types, add 1000 to candidates 
found in our dictionary of countries, which includes the names of almost every country
(636 expressions). 

\item 
When the question includes ``{\it doko no todouhuken}'' (what state/province), add 1000 
to candidates that are location expressions with the suffix ``{\it to/dou/hu/ken}'' (state/province). 

\item 
When the NOUN-FOCUS is ``{\it meisaku}'' (masterpiece), ``{\it sakuhin}'' (literary work), or a similar term, 
add 1000 to candidates in quotation marks. 

\item 
When NUMERICAL is one of the predicted answer types, add 1000 to candidates 
consisting only of numerical expressions and unit expressions. 

\item 
When the question contains the expression, ``{\it denwabangou wa nani}'' (what is the telephone number), 
add 1000 to a candidate consisting of numerical expressions and the symbol ``-''. 

\item 
When the question contains ``{\it nani} Noun X'' (what Noun), add 1000 to candidates 
containing ``Noun X''. 

Because this rule is very similar to the fourth rule in the list in Section \ref{sec:expectation}, 
we will explain these two rules in more detail. 
The fourth rule in Section \ref{sec:expectation} is used for a question like 
``{\it ichi doru wa nan en desuka?}'' (How many yen to the dollar?) 
and an example of the answer is ``{\it 120 en}'' 120 yen as described previously. 
The eighth rule in Section \ref{sec:detection_of_the_answer} is used for a question like 
``{\it Koizumi shosho no shussin was nani daigaku desu ka?}'' (What university did Prime Minister Koizumi graduate from?) 
and the answer is ``{\it Keio daigaku}'' (Keio University). 
The difference between these two rules is whether 
the expression X following ``{\it nani}'' (what) is a suffix or a noun. 
When X is a suffix, 
the system regards it as a unit expression 
and selects a numerical expression with the unit expression as the answer. 
When X is a noun, 
the system considers it as the important content word, 
and selects a compound noun with the content word as the answer. 
A morphological analyzer is used to estimate whether a word is a suffix or a noun. 
Using these two rules, this method 
detects numerous correct answers but
in some cases also 
detects incorrect ones 
because the suffixes and nouns may be very similar, making it difficult to distinguish them. 

\item 
When a question contains a phrase, ``{\it nan youbi}'' (what day of the week), add 
10,000 to candidates containing a day expression like ``{\it nichiyoubi}'' (Sunday).

\end{enumerate}

The first rule in the above list is the most important. 
Using this rule, the system 
examines whether 
a candidate satisfies a condition based on 
six major answer types (LOCATION, ORGANIZATION, PERSON, 
ARTIFACT, TIME, or NUMERICAL). 
The other rules are additional. 
The more that are satisfied, 
the more plausible the answer. 
For example, to answer the question 
``jinkou ga mottomo ookii no wa doko no kuni desu ka?''
(What country has the biggest population?), 
the system predicts two answer types LOCATION and COUNTRY. 
In this case, 
candidate answers that satisfy
two rules, the first and 
third rules, using LOCATION and COUNTRY 
are more likely to be correct 
than 
candidate answers that satisfy
only the first rule relating to LOCATION. 
In our system, 
this is implemented by adding scores given by rules. 

An additional function of the system is 
that it compiles similar answers after the candidate answers are selected based 
on their scores. 
Here, similar answers are those that are part of another answer
and have a score difference of less than 1000. 
The compilation process eliminates answers other than the longest one. 
This function is performed after executing the decreased adding method. 


\section{Experiments}
\label{sec:experiments}

In this section, we describe 
the experiments we used to assess 
the effectiveness of the proposed method. 

\subsection{Data Used in the Experiments}
\label{sec:data}

We used the QAC data collection 
from the QAC at NTCIR 3
for our experiments. 
This data collection consists of 
pairs of questions and answers.
It was developed in Japanese 
and is based on articles in the Japanese Mainichi newspaper from 1998 and 1999. 

The QAC data collection includes three tasks: Task-1, Task-2, and Task-3. 
We chose Task-1 and Task-2 for these experiments. 
They each consist of the same 200 questions, but 
different methods are used to evaluate them. 
In Task-1, the mean reciprocal rank (MRR) is used for evaluation, while 
in Task-2, the mean of the f-measure (MF) is used. 
For Task-1, five answers can be output.
By applying the MRR, we obtain a score of $1/r$ when the r-th output answer is correct. 
For Task-2, we can output any number of answers for each question. 
When a question has more than one 
answer, we have to produce all of the correct answers to earn full marks.
An example of this type of question is, ``Which countries are permanent members of the United Nations Security Council?''. 
If the system outputs all five permanent members (``U.S.A.'', ``U.K.'', ``Russia'', ``China'', and ``France''), 
it earns full marks for the question. 
The answers generated in Task-2 are evaluated according to the f-measure, i.e. the inverse of the mean sum of 
the inverses of the recall rate and precision rate. 
The recall rate is the ratio of the number of correct answers 
output to the number of correct answers. The precision rate is the ratio of the 
number of correct answers output to the number of answers output. 

With these definitions and task requirements in mind, 
we set up our experimental system as follows: 
For Task-1, in which the MRR is 
used for evaluation, the system outputs the top five answers.
For Task-2, in which the MF is used for
evaluation, the system outputs only the top answer 
to maintain good precision 
because it does not have a function for selecting more than one answer 
from multiple candidates. 

\subsection{Methods of Adding Scores}
\label{sec:adding_scores}

In the experiments, we used each of the following methods 
of adding scores. 

\begin{itemize}
\item 
  Original Method

  This method simply outputs the answers generated by the question-answering system as they are, 
  without adding their scores. 
  
\item   
  Simple Adding Method
  
  This method adds the scores of each candidate answer
  as extracted from multiple documents. 
  It then outputs the answers according to their total scores. 

  In our question-answering system, 
  the significance of a candidate answer is greatly changed 
  by a score of 1000, because 
  the heuristic rules used to calculate $Score_{sem}(c)$ often produce scores in the order of 1000 or multiples thereof. 
  Therefore, we need to avoid skewing scores by adding the thousands digit and higher order digits 
  from each score. 
  For scores with the same values for the thousands and higher order digits, 
  the method extracts only digits 
  representing values below 1000 (i.e., hundreds, tens, and units) from each score. 
  It then adds all the extracted values to give a subtotal, which is 
  combined with the values of the thousands and higher order digits shared by the scores. 
  This method does not add the scores of candidates 
  with different values for the digits representing values of 1000 or more;
  instead, it simply takes the higher score as the total. 

  This method 
  of distinguishing the values of digits
  can be more clearly explained 
  by using the numerical terms $Score_{near}(c)$ and $Score_{sem}(c)$, defined in Section \ref{sec:detection_of_the_answer}. 
  We add the scores for $Score_{near}(c)$ 
  when the scores for $Score_{sem}(c)$ have the same value 
  because the values of $Score_{near}(c)$ are much less than 1000 and 
  those of $Score_{sem}(c)$ are multiples of 1000.

  For example, suppose that a candidate answer, $X$, appears twice, 
  with scores of 1025 and 1016. In this case, 25 and 16 are extracted 
  as digits representing values below 1000. These scores are added to obtain 41
  and finally 41 is added to 1000, giving a total score of 1041. 

  As another example, suppose that the candidate answer, $X$, appears twice, 
  with scores of 2025 and 2016. As in the previous example, 25 and 16 are extracted 
  and added to obtain 41, which is then added to 2000 to obtain a total score of 2041. 

  Finally, suppose that the candidate answer, $X$, appears twice, 
  with scores of 2025 and 1016. Here, the scores have 
  different values for the thousands digits. In this case, the scores are not added, 
  and 2025 is taken as the total score. 

\item   
  Decreased Adding Method

  This method adds the scores for each candidate answer
  extracted from multiple documents
  in almost the same way as the simple adding method.  
  It also uses the same approach to handle
  digits representing values of 1000 or more.  

  However, the actual method of adding is different. 
  The decreased adding method multiplies the score of the $i$-th candidate answer by a factor of $k^{(i-1)}$ 
  before adding it to the running total score. 
  This is expressed by the following equation:
  \begin{eqnarray}
    \label{eqn:decreased_adding}  
    Score_{decreased} = \displaystyle \sum_{1 \leq i \leq n} k^{i-1} \ score_{original}(i).
  \end{eqnarray}
  Here, $Score_{decreased}$ is the value of 
  the final score obtained by the decreased adding method 
  for digits representing values below 1000, while
  $score_{original}(i)$ is the value of the original score obtained by the question-answering system 
  for these digits. 
  $n$ is the number of occurrences of the same candidate answer extracted 
  from multiple documents with the same values 
  for the thousands and higher order digits. 
  Finally, $k$ is a constant set based on experimental results. 

  For example, suppose that a candidate answer, $X$, appears twice, 
  with scores of 2025 and 2016, and that $k = 0.3$. 25 and 16 are first extracted 
  as the values for digits below 1000. 
  These values are added by using Eq. \ref{eqn:decreased_adding},
  with $25 + 16 \times 0.3$ giving a result of 29.8; 
  29.8 is then added to 2000, producing a total score of 2029.8. 

  In the experiments, we used 0.01, 0.02, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9 as values of $k$. 

\item   
  Combined Method

  This method is a combination of 
  the original method, the simple adding method, and the decreased adding method
  (with the same 12 possible values of $k$ given above).
  First, the combined method identifies 
  the method that gives the best performance (as measured by the MRR/MF)
  for a set of training data. 
  It then uses the best method to output the answers. 

  In this study, we did not use any data other than the QAC test collection, 
  so we performed ten-fold cross validation for training purposes.
  Ten-fold cross validation is used to evaluate the performance of a system 
  when there is no set of training data available. 
  A set of test data is divided into ten data sets. 
  Each of the ten is analyzed after using the other data sets for training, 
  and then all the data sets are evaluated.

  The combined method involves two important considerations.

  First, it combines multiple methods, which 
  means that the best method can be selected for each case, 
  thus improving the performance of the system. 

  Secondly, it provides fair evaluation. 
  For example, in our experiments, the decreased adding method had
  12 possible values of $k$. 
  Because of large variations in $k$, 
  even if the system produces a good evaluation score with a certain value for $k$, 
  the score may be a fluke found only in the test data. 
  In general, to avoid unfair evaluation and 
  make sure that appropriate evaluation scores are calculated, 
  ten-fold cross validation is applied. 
  In this approach, 
  the value for $k$ is determined without using 
  an input question currently being answered by the system, 
  which ensures that 
  the evaluation is fair. 
  We therefore, applied the combined method 
  with training based on ten-fold cross validation. 

\end{itemize}

In addition, we employed the following four methods for comparison in the experiments. 
All these methods have been proposed in previous studies and include a function for adding scores. 
We handled digits representing values of 1000 or more 
using the same approach described above, 
which enabled us to use the information from $Score_{sem}(c)$ in these methods as well.
\begin{itemize}
\item 
Clarke et al.'s method

Clarke et al. use information from multiple documents for question answering, 
and call this information {\it redundancy} \cite{Charles}. 
Their method uses the following equation to extract answers:
  \begin{eqnarray}
    \label{eqn:Clarke1}
    Score_{Clarke}(c) = \displaystyle c_t \ log(\frac{N}{Freq(c)}),
  \end{eqnarray}
where $c$ is a candidate answer, $c_t$ is the number of documents 
including $c$ of the documents extracted during document retrieval, 
$N$ is the total number of characters in all the documents, 
and $Freq(c)$ is the number of times $c$ occurs in all the documents. 

\item 
Modified method of Clarke et al.

The Clarke et al.'s method cannot use $Score_{near}(c)$, so
it cannot take into account the condition that the answer must be near the input terms. 
We modified it so that it could utilize this condition. 
The modified method uses the following equation: 
  \begin{eqnarray}
    \label{eqn:Clarke2}
    Score_{Clarke2} = \displaystyle c_t \ score_{original}(1),
  \end{eqnarray}
where $c_t$ is the number of documents 
including candidate answers from the documents extracted during the document retrieval process. 
$score_{original}(1)$ is the value for the first item of the candidate answer in the original method, 
so it is the maximum value of the candidate answer in the original method. 

\item 
Xu et al.'s method

Xu et al.'s method uses the following equation \cite{Xu2003}: 
  \begin{eqnarray}
    \label{eqn:Xu}
    Score_{Xu} = \displaystyle score_{original}(1) + 0.001 \sum_{2 \leq i \leq n} score_{original}(i).
  \end{eqnarray}

\item 
Mori's method

Mori's method uses the following equation \cite{Mori_ntcir4_qa}: 
  \begin{eqnarray}
    \label{eqn:Mori}
    Score_{Mori} = \displaystyle (log_{10}(c_t)+1) \ score_{original}(1).
  \end{eqnarray}

\end{itemize}


\subsection{Systems Tested}
\label{sec:exp_system}

As noted above, we used the question-answering system 
described in Section \ref{sec:qas} for the experiments, 
but with five variations, as listed here. 
Note that the designation, Sys-5, refers to the base system, which is exactly as described in Section \ref{sec:qas}. 

We used these five systems 
to examine whether the proposed method
of using multiple documents as evidence with decreased adding
was effective for various types of question-answering systems. 

\begin{itemize}
\item 
  Sys-1
  
  Unlike Sys-5, this system does not use $Score_{sem}(c)$, and
  thus does not include scores based on heuristic rules according to the answer type.

\item 
  Sys-2

  Unlike Sys-5, which uses Eq. \ref{eqn:ir_near_}, 
  this system uses the following equation (Eq. \ref{eqn:ir_near_2}):
  
{
\begin{eqnarray}
  \label{eqn:ir_near_2}
Score_{near2}(c) & =  & \displaystyle \sum_{t2 \in T} w_{dr2}(t2) log \frac{N}{df(t2)}, 
\end{eqnarray}}
where $T$ is a set of terms that appear in the question and the document.

Thus, Sys-2 does not use 
the distances between a candidate answer and the terms extracted from the input question. 

\item 
  Sys-3

  This system divides documents into paragraphs during document retrieval, 
  and it also uses Eq. \ref{eqn:ir_near_2} instead of Eq. \ref{eqn:ir_near_}. 
  Like Sys-2, Sys-3 does not directly use 
  the distances between a candidate answer and the extracted terms. 
  However, because it divides documents into paragraphs, 
  it does check whether a candidate answer appears in the same paragraph as the terms. 
  This enables 
  Sys-3 to utilize a little more information related to 
  the distances between the candidate answer and the extracted terms, as compared to Sys-2. 

\item 
  Sys-4

  This system divides documents into paragraphs during document retrieval. 
  Current state-of-the-art question-answering systems 
  also divide documents into paragraphs, 
  so Sys-4 corresponds to the most common question-answering systems. 

\item
  Sys-5
  
  Sys-5 is the base question-answering system as described in Section \ref{sec:qas}. 

\end{itemize}

In Sys-3 and Sys-4, 
even when documents are divided into paragraphs, 
the adding methods are based on the original documents, 
not on the paragraphs. 
The scores of each candidate answer
extracted from multiple original documents 
are added in the adding methods. 


\subsection{Experimental Results}
\label{sec:results}

We conducted experiments with the QAC test collection for Task-1 and Task-2
using the methods described in Sections \ref{sec:adding_scores} and \ref{sec:exp_system}. 
The results are shown in Tables \ref{tab:task-1} and \ref{tab:task-2}, and 
Figures \ref{fig:task-1_results} and \ref{fig:task-2_results}. 
Table \ref{tab:task-1} lists the MRR values of each system for Task-1, while
Table \ref{tab:task-2} lists the MF values of each system for Task-2. 
In the two tables, 
the leftmost column indicates the adding method, while
the top line indicates the question-answering system. 
We used the two-sided Wilcoxon signed ranks test to test for significant differences, 
with the original method as the baseline method. 
When a method performed better than the baseline method at the 0.05 or 0.01 significance level, 
it was tagged with ``+'' or ``++'', respectively. 
Similarly, when a method performed worse than the baseline method at the 0.05 or 0.01 significance level, 
it was tagged with ``-'' or ``--'', respectively. 

\begin{table}[p]
  \begin{center}
    \caption{Results for Task-1 (MRR)}
    \begin{tabular}{|l|l|l|l|l|l|} \hline
         &  \multicolumn{1}{c|}{Sys-1}        & \multicolumn{1}{c|}{Sys-2}        & \multicolumn{1}{c|}{Sys-3}        & \multicolumn{1}{c|}{Sys-4}        & \multicolumn{1}{c|}{Sys-5}        \\\hline
Original &  0.130        & 0.294        & 0.405        & 0.526        & 0.541       \\\hline
Simple   &  0.051$^{--}$ & 0.387$^{+}$  & 0.474$^{+}$ & 0.523        & 0.449$^{--}$\\\hline
Combined &  0.097        & 0.437$^{++}$ & 0.506$^{++}$ & 0.567$^{+}$  & 0.597$^{++}$\\\hline
Decreased&               &              &              &              &             \\
k=0.01   &  0.115        & 0.432$^{++}$ & 0.498$^{++}$ & 0.544        & 0.551       \\
k=0.02   &  0.115        & 0.433$^{++}$ & 0.502$^{++}$ & 0.551$^{+}$  & 0.561       \\
k=0.05   &  0.115        & 0.440$^{++}$ & 0.510$^{++}$ & 0.549        & 0.563       \\
k=0.1    &  0.109        & 0.449$^{++}$ & 0.516$^{++}$ & 0.556        & 0.570       \\
k=0.2    &  0.129        & 0.446$^{++}$ & 0.509$^{++}$ & 0.574$^{++}$ & 0.590$^{++}$ \\
k=0.3    &  0.130        & 0.450$^{++}$ & 0.504$^{++}$ & 0.571$^{+}$  & 0.597$^{++}$\\
k=0.4    &  0.111        & 0.434$^{++}$ & 0.504$^{++}$ & 0.567$^{+}$  & 0.580       \\
k=0.5    &  0.101        & 0.428$^{++}$ & 0.509$^{++}$ & 0.565        & 0.565       \\
k=0.6    &  0.100        & 0.414$^{++}$ & 0.505$^{++}$ & 0.544        & 0.544       \\
k=0.7    &  0.081        & 0.411$^{++}$ & 0.498$^{++}$ & 0.531        & 0.537       \\
k=0.8    &  0.072$^{-}$  & 0.399$^{++}$ & 0.489$^{++}$ & 0.528        & 0.492       \\
k=0.9    &  0.062$^{--}$ & 0.390$^{++}$ & 0.480$^{++}$ & 0.532        & 0.472$^{-}$ \\\hline
Clarke   &  0.063$^{-}$  & 0.384$^{+}$  & 0.423        & 0.437$^{--}$ & 0.386$^{--}$\\\hline
Mod. Clarke&0.035$^{--}$ & 0.385$^{+}$  & 0.477$^{+}$  & 0.513        & 0.442$^{--}$\\\hline
Xu       &  0.110        & 0.441$^{++}$ & 0.506$^{++}$ & 0.542        & 0.556\\\hline
Mori     &  0.044$^{--}$ & 0.417$^{++}$ & 0.504$^{++}$ & 0.553        & 0.540\\\hline
\end{tabular}
\label{tab:task-1}
\end{center}
\end{table}

\begin{table}[p]
  \begin{center}
    \caption{Results for Task-2 (MF)}
    \begin{tabular}{|l|l|l|l|l|l|} \hline
         &  \multicolumn{1}{c|}{Sys-1}        & \multicolumn{1}{c|}{Sys-2}        & \multicolumn{1}{c|}{Sys-3}        & \multicolumn{1}{c|}{Sys-4}        & \multicolumn{1}{c|}{Sys-5}        \\\hline
Original &  0.065        & 0.172        & 0.246        & 0.359        & 0.376        \\\hline      
Simple   &  0.038        & 0.270$^{+}$  & 0.324$^{+}$  & 0.374        & 0.317        \\\hline      
Combined &  0.045        & 0.311$^{++}$ & 0.362$^{++}$ & 0.412$^{+}$  & 0.451$^{++}$ \\\hline
Decreased&               &              &              &              &              \\
k=0.01   &  0.060        & 0.291$^{++}$ & 0.360$^{++}$ & 0.381        & 0.381        \\
k=0.02   &  0.064        & 0.291$^{++}$ & 0.360$^{++}$ & 0.388$^{+}$  & 0.393        \\      
k=0.05   &  0.059        & 0.296$^{++}$ & 0.365$^{++}$ & 0.395$^{+}$  & 0.399        \\      
k=0.1    &  0.051        & 0.311$^{++}$ & 0.365$^{++}$ & 0.402$^{+}$  & 0.416$^{+}$  \\      
k=0.2    &  0.066        & 0.321$^{++}$ & 0.361$^{++}$ & 0.435$^{++}$ & 0.446$^{++}$ \\      
k=0.3    &  0.066        & 0.318$^{++}$ & 0.357$^{++}$ & 0.432$^{++}$ & 0.451$^{++}$ \\      
k=0.4    &  0.044        & 0.305$^{++}$ & 0.362$^{++}$ & 0.427$^{+}$  & 0.421        \\      
k=0.5    &  0.039        & 0.300$^{++}$ & 0.372$^{++}$ & 0.431$^{+}$  & 0.421        \\      
k=0.6    &  0.049        & 0.285$^{++}$ & 0.372$^{++}$ & 0.405        & 0.401        \\      
k=0.7    &  0.033        & 0.285$^{++}$ & 0.362$^{++}$ & 0.388        & 0.399        \\      
k=0.8    &  0.035        & 0.277$^{+}$  & 0.350$^{++}$ & 0.383        & 0.351        \\      
k=0.9    &  0.042        & 0.270$^{+}$  & 0.339$^{++}$ & 0.385        & 0.336        \\\hline
Clarke   &  0.038        & 0.278$^{+}$  & 0.286        & 0.293$^{-}$  & 0.283$^{-}$\\\hline      
Mod. Clarke&0.022$^{-}$  & 0.261        & 0.331$^{+}$  & 0.364        & 0.317\\\hline      
Xu       &  0.053        & 0.309$^{++}$ & 0.371$^{++}$ & 0.384$^{+}$  & 0.386\\\hline      
Mori     &  0.024$^{-}$  & 0.288$^{++}$ & 0.365$^{++}$ & 0.415        & 0.403\\\hline      
\end{tabular}
\label{tab:task-2}
\end{center}
\end{table}

\begin{table}[t]
  \begin{center}
    \caption{Comparison of the combined method and other methods for Task-1 (MRR)}
    \begin{tabular}{|l|l|l|l|l|l|} \hline
         &  \multicolumn{1}{c|}{Sys-1}        & \multicolumn{1}{c|}{Sys-2}        & \multicolumn{1}{c|}{Sys-3}        & \multicolumn{1}{c|}{Sys-4}        & \multicolumn{1}{c|}{Sys-5}        \\\hline
Combined &  0.097        & 0.437        & 0.506        & 0.567        & 0.597      \\\hline
Simple   &  0.051        & 0.387$^{-}$  & 0.474$^{-}$  & 0.523$^{-}$  & 0.449$^{--}$\\
Clarke   &  0.063        & 0.384        & 0.423$^{--}$ & 0.437$^{--}$ & 0.386$^{--}$\\
Mod. Clarke&0.035$^{--}$ & 0.385$^{-}$  & 0.477        & 0.513$^{--}$ & 0.442$^{--}$\\
Xu       &  0.110        & 0.441        & 0.506        & 0.542        & 0.556$^{-}$\\
Mori     &  0.044$^{-}$  & 0.417        & 0.504        & 0.553        & 0.540$^{--}$\\\hline
\end{tabular}
\label{tab:task-1-comp}
\end{center}
\end{table}

\begin{table}[t]
  \begin{center}
    \caption{Comparison of the combined method and other methods for Task-2 (MF)}
    \begin{tabular}{|l|l|l|l|l|l|} \hline
         &  \multicolumn{1}{c|}{Sys-1}        & \multicolumn{1}{c|}{Sys-2}        & \multicolumn{1}{c|}{Sys-3}        & \multicolumn{1}{c|}{Sys-4}        & \multicolumn{1}{c|}{Sys-5}        \\\hline
Combined &  0.045        & 0.311        & 0.362        & 0.412        & 0.451        \\\hline
Simple   &  0.038        & 0.270$^{-}$  & 0.324$^{-}$  & 0.374        & 0.317$^{--}$ \\
Clarke   &  0.038        & 0.278        & 0.286$^{--}$ & 0.293$^{--}$ & 0.283$^{--}$\\
Mod. Clarke&0.022        & 0.261$^{-}$  & 0.331        & 0.364        & 0.317$^{--}$\\
Xu       &  0.053        & 0.309        & 0.371        & 0.384        & 0.386$^{--}$\\
Mori     &  0.024        & 0.288        & 0.365        & 0.415        & 0.403$^{-}$\\\hline      
\end{tabular}
\label{tab:task-2-comp}
\end{center}
\end{table}

\begin{figure}[p]
      \begin{center}
      \epsfile{file=data/bou3_.eps,height=8cm,width=12cm} 
      \end{center}
      \caption{Results as a function of k for Task-1 (MRR)}
    \label{fig:task-1_results}
\end{figure}

\begin{figure}[p]
      \begin{center}
      \epsfile{file=data/bou4_.eps,height=8cm,width=12cm} 
      \end{center}
      \caption{Results as a function of k for Task-2 (MF)}
      \label{fig:task-2_results}
\end{figure}


We applied the same statistical test to compare the combined method to the other methods. 
The results are shown in Tables \ref{tab:task-1-comp} and \ref{tab:task-2-comp}
using the same labels described above to indicate better or worse performance than the combined method at 
the 0.05 and 0.01 significance levels.
The results for the decreased adding method, shown in Tables \ref{tab:task-1} and \ref{tab:task-2}, are also expressed 
as line charts in Figures \ref{fig:task-1_results} and \ref{fig:task-2_results}. 

We also applied ten-fold cross validation to the combined method
and counted the frequency with which each method was used. 
The results are shown in Figures \ref{fig:task-1_freq} and \ref{fig:task-2_freq}. 

\section{Discussion}
\label{sec:discussions}


\subsection{Comparison among the Original, Simple Adding, and Combined Methods}

\subsubsection{Comparison between the Original and Simple Adding Methods}

As seen from Tables \ref{tab:task-1} and \ref{tab:task-2}, 
the simple adding method performed better than the original method 
with Sys-2 and Sys-3 for Task-1, and 
with Sys-2, Sys-3, and Sys-4 for Task-2, but 
it performed worse otherwise. 
The simple adding method improved the performance
of Sys-2 and Sys-3, which did not use the distances between 
a candidate answer and the terms extracted from the input question. 
In contrast, 
this method resulted in a worse performance 
with Sys-4 and Sys-5, which did use distance information. 
In particular, 
with Sys-1, Sys-4 and Sys-5 for Task-1 and 
Sys-1 and Sys-5 for Task-2, 
the simple adding method did not perform as well as
the original method. 
These results indicate that 
the performance of the simple adding method
depended on the system used.

\begin{figure}[p]
      \begin{center}
      \epsfile{file=data/bou1_.eps,height=19cm,width=14cm} 
      \end{center}
      \caption{Frequency of using each method in the combined method for Task-1}
      \label{fig:task-1_freq}
\end{figure}

\begin{figure}[p]
      \begin{center}
      \epsfile{file=data/bou2_.eps,height=19cm,width=14cm} 
      \end{center}
      \caption{Frequency of using each method in the combined method for Task-2}
      \label{fig:task-2_freq}
\end{figure}

\clearpage

\subsubsection{Comparison between the Original and Combined Methods}

As seen from Tables \ref{tab:task-1} and \ref{tab:task-2}, 
in cases other than Sys-1, 
the combined method performed better than 
the original method at the 0.05 significance level. 
We found that 
there were many cases in which 
the combined method produced better results. 
The degree of improvement was quite large, ranging from 0.05 to 0.14. 
The combined method was simple to implement and produced
a large degree of improvement, thus indicating its effectiveness. 

\subsubsection{Comparison between the Simple Adding and Combined Methods}

As shown by Tables \ref{tab:task-1-comp} and \ref{tab:task-2-comp}, 
the combined method performed better than 
the simple adding method in every case. 
There was a significant difference 
for either Task-1 or Task-2 in all cases except Sys-1. 
The combined method clearly outperformed 
the simple adding method.

\subsection{Examination of the Decreased Adding Method}

We compared the decreased adding method to the original method, as shown in 
Tables \ref{tab:task-1} and \ref{tab:task-2}. 
In many cases, 
the decreased adding method performed better
than the original method at the significance level of 0.05, although
in a few cases (Sys-1, and k = 0.7-0.9 with Sys-5), 
did not perform as well. 
For k = 0.2 and k = 0.3 in systems other than Sys-1, 
the decreased adding method performed better 
than the original method at the 0.05 significance level. 
The lines in Figures \ref{fig:task-1_results} and \ref{fig:task-2_results} 
roughly express curves with peaks around k = 0.2-0.3. 
In addition, as shown in Figures \ref{fig:task-1_freq} and \ref{fig:task-2_freq}, 
we found that the decreased adding method with k = 0.2 or 0.3 was often 
selected by the combined method, 
indicating that 0.2 and 0.3 were good choices for k.

\subsection{Examination of the four previous methods}

We compared our combined method 
to four existing methods: Clarke et al.'s method, 
a modified Clarke et al.'s method, 
Xu et al.'s method, and Mori's method. 
As seen from Tables \ref{tab:task-1-comp} and \ref{tab:task-2-comp}, 
the combined method produced almost the same or better precision
than these four methods.
The combined method 
performed better than each of the four methods 
at the 0.05 significant level at least once, and 
never performed worse 
than any of these methods at that level. 
These results indicate that the combined method 
was more effective than the four existing methods. 

Next, we analyzed the results obtained using each of the four methods. 

Clarke et al.'s method uses Equation (\ref{eqn:Clarke1}) 
and does not use the condition 
that the answer must be near the terms extracted from the input question (according to $Score_{near}(c)$).
As a result, 
it produced almost the same or better precision 
than the modified Clarke et al.'s method 
in Sys-1 and Sys-2, which did not use 
the distances between a candidate answer and the extracted terms. 
In Sys-3, Sys-4, and Sys-5,  
which did use
those distances, 
the modified Clarke et al.'s method 
achieved greater precision than Clarke et al.'s method. 

The modified Clarke et al.'s method is very similar 
to the simple adding method. 
The equations used to add scores in the two methods can be 
expressed as follows:
  \begin{eqnarray}
    \label{eqn:Clarke1_}
    Score_{Clarke} = \displaystyle \sum_{1 \leq i \leq n} score_{original}(1),
  \end{eqnarray}
  \begin{eqnarray}
    \label{eqn:simple_adding}
    Score_{simple \ adding} = \displaystyle \sum_{1 \leq i \leq n} score_{original}(i).
  \end{eqnarray}
The difference is that 
the modified Clarke et al.'s method adds $score_{original}(1)$,
while the simple adding method adds $score_{original}(i)$. 
Our results showed that these methods achieved similar precision
in all the cases shown in Tables \ref{tab:task-1} and \ref{tab:task-2}. 
Therefore, 
the modified Clarke et al.'s method exhibited
the same problem as the simple adding method
in that it was likely to select candidate answers occurring at
higher frequencies. 

Xu et al.'s method adds a very small number to the original score and 
is similar to our decreased adding method when $k$ is very small (e.g. $k=0.01$). 
Although this method offered some improvement, 
the improvement was smaller than with the combined method. 

Mori's method uses the numerical function $log$, 
so it can reduce the problem of selecting candidate answers with high frequencies. 
However, $log$ is a monotonically increasing function, so 
the method suffers to a comparatively large degree from this problem. 
This is because the value of $log$ goes to infinity, 
when the value of its argument goes to infinity. 
In contrast, the decreased adding method utilizes an exponential function, 
whose value converges to a finite value even when 
the value of its argument goes to infinity. 
Therefore, our method is less affected by this problem than Mori's method. 

\subsection{Examination of the Five Question-answering Systems}

In Section \ref{sec:comparison_five_systems}, 
we describe a fundamental comparison of five question-answering systems 
and in the following sections 
we examine the effectiveness of using our proposed method
with each of the five systems. 

\subsubsection{Comparison of five systems when using the original method}
\label{sec:comparison_five_systems}

In our experimental results, 
the precision using the original method increased stably 
in order from Sys-1 to Sys-5 in the five systems. 
These results indicted the following:
\begin{itemize}
\item 
  No use of semantic information ($Score_{sem}(c)$) resulted in the worst performance (Sys-1). 

\item 
  No use of location information 
  on the distances between a candidate answer 
  and the terms extracted from the input question (Sys-2 and Sys-3) 
  resulted in a worse performance than the use of location information (Sys-4 and Sys-5). 

\item 
  When location information was not used, 
  dividing documents into paragraphs during document retrieval (Sys-3)
  resulted in better performance than not dividing documents (Sys-2). 
  In contrast, 
  when location information was used, 
  dividing documents into paragraphs during document retrieval (Sys-4)
  resulted in worse performance than not dividing documents (Sys-5). 
\end{itemize}

\subsubsection{Sys-1}

We evaluated the effectiveness of using our proposed method
with each of the five question-answering systems. 

Sys-1 did not use any semantic information ($Score_{sem}(c)$) and 
did not perform well. 
As seen from Table \ref{tab:task-1}, 
the simple adding method produced a worse performance 
than the original method for Task-1 at the significance level of 0.01. 
The combined method also produced a worse performance 
than the original method, 
but the difference was not statistically significant. 

\subsubsection{Sys-2}

As described in Section \ref{sec:exp_system},
Sys-2 did not use the distances between a candidate answer 
and the terms extracted from the input question. 
It originally produced a poor performance. 
As seen from Table \ref{tab:task-1}, however, 
applying the simple adding method or 
the combined method greatly improved its performance, 
especially in the latter case. 

\subsubsection{Sys-3}

Like Sys-2, 
Sys-3 did not use the distances between a candidate answer 
and the extracted terms, though it did 
divide documents into paragraphs during document retrieval.
The original performance of Sys-3 was not especially poor, 
but applying the simple adding method or 
the combined method greatly improved its performance. 
In the QAC contest\footnote{In the QAC contest, 
15 and 13 groups participated in Task-1 and 
Task-2, respectively.}, the second best score for Task-1 was 0.524, and 
the best score for Task-2 was 0.364. 
The improved performance of Sys-3 thus matched some of the higher 
performance levels achieved in that contest. 
Note that although the simple adding method 
improved the performance of Sys-3, 
the combined method improved its performance to a greater extent. 

\subsubsection{Sys-4}

As described in Section \ref{sec:exp_system}, 
Sys-4 both used the distances between a candidate answer 
and the extracted terms and 
divided documents into paragraphs, 
as do most conventional question-answering systems. 
It originally performed well, but
applying the simple adding method or, 
in particular, the combined method, improved its performance. 
The improvements with the combined method 
were quite notable. 

\subsubsection{Sys-5}

As described in Section \ref{sec:exp_system}, 
Sys-5 used the distances between a candidate answer 
and the extracted terms but
did not divide documents into paragraphs. 
This system performed best of all. 
It originally performed well, but 
its performance was improved further 
with the combined method. 
The scores were 0.597 and 0.451 for Task-1 and Task-2, respectively. 
In the QAC contest, the best score for Task-1 was 0.608, and 
that for Task-2 was 0.364. 
The scores for Sys-5 with the combined method 
were thus very near the best score for Task-1 and 
much higher than the best score for Task-2. 
The simple adding method, on the other hand, reduced the performance 
of this system, 
because it 
too often selected candidate answers that occurred at high frequencies. 

\subsubsection{Characteristics of our proposed method}

Our proposed methods (the decreased adding and combined methods) 
did not perform any better than the original method 
in Sys-1, which did not use any semantic information, 
indicating that these methods are not effective 
when no semantic information is used. 

Because our proposed methods produced a significant improvement 
in Sys-2 and Sys-3, which did not use location information 
on the distance between a candidate answer 
and the terms extracted from the input question, 
we found that our methods were very effective 
when location information was not used.  

However, our proposed methods resulted in improvements 
in Sys-4 and Sys-5, which use both 
semantic and location information. 
This showed that 
our methods 
when both semantic and location information were used 
were less effective than when 
location information was not used, but were effective. 

Our proposed methods produced better results with Sys-2 and Sys-3 
than with Sys-4 and Sys-5 
because 
Sys-2 and Sys-3 do not use location information and 
use less information overall than Sys-4 and Sys-5.
Therefore, the effect of using information from plural documents 
with our proposed methods in Sys-2 and Sys-3 was larger than 
in Sys-4 and Sys-5, and consequently
the improvement in Sys-2 and Sys-3 
was larger than in Sys-4 and Sys-5. 

The differences between Sys-2 and Sys-3 or Sys-4 and Sys-5 
relate to whether or not documents are divided into paragraphs during document retrieval. 
The improvements gained by using our proposed method were 
almost the same between Sys-2 and Sys-3 or Sys-4 and Sys-5, 
indicating that 
the effectiveness of our proposed methods was not affected by 
whether or not documents were divided into paragraphs during document retrieval. 

Based on this evaluation of our proposed methods using the five systems (Sys-1 to Sys-5), 
below we summarize the characteristics of our proposed method. 
\begin{itemize}
\item 
  Our methods were not effective 
  when semantic information was not used. 

\item 
  Our methods were very effective 
  when location information was not used.  

\item 
  Our methods 
  when both semantic and location information were used 
  were less effective 
  than when not using location information, 
  but were effective. 

\item 
  The effectiveness of our proposed methods was not affected by 
  whether or not documents were divided into paragraphs during document retrieval. 
\end{itemize}

The above characteristics, 
indicate that
our proposed methods are effective except when 
semantic information is not used. 

\begin{table}[t]
  \begin{center}
    \caption{Results when paragraphs are treated as documents by the adding methods}
    \begin{tabular}{|l|l|l|l|l|} \hline
         &  \multicolumn{2}{c|}{Task-1 (MRR)}        & \multicolumn{2}{c|}{Task-2 (MF)}\\\hline
         &  \multicolumn{1}{c|}{Sys-3}        & \multicolumn{1}{c|}{Sys-4}        &  \multicolumn{1}{c|}{Sys-3}        & \multicolumn{1}{c|}{Sys-4}        \\\hline
Original &  0.405        & 0.526        & 0.246        & 0.359\\\hline
Simple   &  0.478$^{++}$ & 0.508        & 0.341$^{++}$ & 0.366\\\hline
Combined &  0.522$^{++}$ & 0.565$^{+}$  & 0.367$^{++}$ & 0.397\\\hline
Decreased&               &              &              &                           \\
k=0.01   &  0.499$^{++}$ & 0.541        & 0.356$^{++}$ & 0.378\\
k=0.02   &  0.503$^{++}$ & 0.545$^{+}$  & 0.356$^{++}$ & 0.383\\
k=0.05   &  0.514$^{++}$ & 0.551        & 0.366$^{++}$ & 0.392$^{+}$\\
k=0.1    &  0.522$^{++}$ & 0.558$^{+}$  & 0.371$^{++}$ & 0.402$^{+}$\\
k=0.2    &  0.515$^{++}$ & 0.565$^{+}$  & 0.372$^{++}$ & 0.421$^{++}$\\
k=0.3    &  0.511$^{++}$ & 0.560        & 0.368$^{++}$ & 0.422$^{++}$\\
k=0.4    &  0.505$^{++}$ & 0.549        & 0.363$^{++}$ & 0.415$^{+}$\\
k=0.5    &  0.494$^{++}$ & 0.547        & 0.358$^{++}$ & 0.417$^{+}$\\
k=0.6    &  0.494$^{++}$ & 0.535        & 0.358$^{++}$ & 0.399\\
k=0.7    &  0.489$^{++}$ & 0.524        & 0.353$^{++}$ & 0.389\\
k=0.8    &  0.487$^{++}$ & 0.521        & 0.353$^{++}$ & 0.386\\
k=0.9    &  0.487$^{++}$ & 0.511        & 0.356$^{++}$ & 0.371\\\hline
Clarke   &  0.437        & 0.444$^{--}$ & 0.284        & 0.289$^{-}$\\\hline
Mod. Clarke&0.475$^{++}$ & 0.501        & 0.340$^{++}$ & 0.358\\\hline
Xu       &  0.517$^{++}$ & 0.545$^{+}$  & 0.379$^{++}$ & 0.389\\\hline
Mori     &  0.506$^{++}$ & 0.547        & 0.365$^{++}$ & 0.411\\\hline
\end{tabular}
\label{tab:dividing_method_results}
\end{center}
\end{table}

\subsubsection{Handling plural documents in Sys-3 and Sys-4}

In terms of the experiments shown in Tables \ref{tab:task-1} and \ref{tab:task-2}, 
in Sys-3 and Sys-4, even when 
documents were divided into paragraphs, 
the original documents, 
not the paragraphs, were treated as documents by the adding methods. 
(In the experiments, the scores of each candidate answer
extracted from multiple original documents
were added in the adding methods.)
In this section, we described experiments 
performed using paragraphs from documents 
which were treated as documents by the adding methods. 
(In the experiments, the scores of each candidate answer
extracted from multiple paragraphs
were added in the adding methods.)
The results are shown in Table \ref{tab:dividing_method_results}. 
We used the two-sided Wilcoxon signed ranks test to determine significant differences 
as in Tables \ref{tab:task-1} and \ref{tab:task-2} for Table \ref{tab:dividing_method_results}. 

When we compared the results with those in Tables \ref{tab:task-1} and \ref{tab:task-2}, 
we found that 
when paragraphs were treated as documents by the adding methods, 
the precision achieved  using our proposed methods increased by about 0.01 in Sys-3
and decreased by about 0.01 in Sys-4. 
The reasons would be as follows. 
Since Sys-3 did not use location information 
and used less information than Sys-4, 
information using plural documents or 
frequently occurring candidates was more important for Sys-3 than for Sys-4. 
Therefore, a method using paragraphs as documents 
that contained more information on frequently occurring candidates 
was more effective for Sys-3 than Sys-4. 
In contrast, Sys-4 used location information, which 
was more important than 
information based on plural documents or frequently occurring candidates. 
Therefore, using paragraphs as documents 
containing more information on frequently occurring candidates 
was not effective for Sys-4. 

\begin{table}[t]
  \begin{center}
    \caption{Task-1 results for Sys-5 with various numbers of documents ($x$) including answers (MRR)}
    \begin{tabular}{|l@{ }|l@{ }|l@{ }|l@{ }|l@{ }|l@{ }|l@{ }|l@{ }|} \hline
         &  \multicolumn{1}{c|}{$x=0$}        &  \multicolumn{1}{c|}{$0<x\leq1$}        & \multicolumn{1}{c|}{$1<x\leq2$}        & \multicolumn{1}{c|}{$2<x\leq3$}        & \multicolumn{1}{c|}{$3<x\leq5$}        & \multicolumn{1}{c|}{$5<x\leq10$}        & \multicolumn{1}{c|}{$x>10$}        \\\hline
No. of  & \multicolumn{1}{c|}{5}            & \multicolumn{1}{c|}{30}           & \multicolumn{1}{c|}{33}           & \multicolumn{1}{c|}{36}           & \multicolumn{1}{c|}{28}           & \multicolumn{1}{c|}{43}           & \multicolumn{1}{c|}{25}\\[-0.1cm]
Question  &  & & & & & & \\\hline
Original         & 0.000        & 0.493        & 0.412        & 0.694        & 0.529        & 0.563        & 0.632\\\hline
Simple           & 0.000        & 0.267$^{--}$ & 0.170$^{--}$ & 0.428$^{--}$ & 0.514        & 0.651        & 0.728\\\hline
Combined         & 0.000        & 0.427        & 0.424        & 0.744        & 0.636$^{+}$  & 0.670$^{+}$  & 0.776\\\hline
Decreased        &              &              &              &              &              &              &      \\
k=0.01           & 0.000        & 0.493        & 0.412        & 0.700        & 0.529        & 0.558        & 0.712\\
k=0.02           & 0.000        & 0.493        & 0.412        & 0.717        & 0.564        & 0.567        & 0.712\\
k=0.05           & 0.000        & 0.480        & 0.412        & 0.706        & 0.557        & 0.586        & 0.736\\
k=0.1            & 0.000        & 0.453        & 0.436        & 0.700        & 0.564        & 0.609        & 0.752\\
k=0.2            & 0.000        & 0.447        & 0.418        & 0.733        & 0.629        & 0.651$^{+}$  & 0.752\\
k=0.3            & 0.000        & 0.427        & 0.424        & 0.744        & 0.636$^{+}$  & 0.670$^{+}$  & 0.776\\
k=0.4            & 0.000        & 0.380$^{-}$  & 0.370        & 0.728        & 0.614        & 0.679$^{+}$  & 0.792$^{+}$\\
k=0.5            & 0.000        & 0.360$^{-}$  & 0.333$^{-}$  & 0.656        & 0.621        & 0.712$^{+}$  & 0.792$^{+}$\\
k=0.6            & 0.000        & 0.353$^{-}$  & 0.291$^{--}$ & 0.572        & 0.614        & 0.721$^{+}$  & 0.792\\
k=0.7            & 0.000        & 0.293$^{--}$ & 0.285$^{--}$ & 0.567$^{-}$  & 0.614        & 0.740$^{++}$ & 0.792\\
k=0.8            & 0.000        & 0.287$^{--}$ & 0.248$^{--}$ & 0.489        & 0.536        & 0.679        & 0.792\\
k=0.9            & 0.000        & 0.280$^{--}$ & 0.212$^{--}$ & 0.461$^{--}$ & 0.521        & 0.670        & 0.752\\\hline
Clarke           & 0.000        & 0.007$^{--}$ & 0.109$^{--}$ & 0.372$^{--}$ & 0.457        & 0.660        & 0.704\\\hline
Mod. Clarke      & 0.000        & 0.213$^{--}$ & 0.182$^{--}$ & 0.422$^{--}$ & 0.507        & 0.651        & 0.752\\\hline
Xu               & 0.000        & 0.493        & 0.418        & 0.694        & 0.536        & 0.558        & 0.744\\\hline
Mori             & 0.000        & 0.380$^{-}$  & 0.267$^{--}$ & 0.561$^{-}$  & 0.607        & 0.744$^{++}$ & 0.752\\\hline
\end{tabular}
\label{tab:task-1-docnum}
\end{center}
\end{table}

\begin{table}[t]
  \begin{center}
    \caption{Task-2 results for Sys-5 with various numbers of documents ($x$) including answers (MF)}
    \begin{tabular}{|l@{ }|l@{ }|l@{ }|l@{ }|l@{ }|l@{ }|l@{ }|l@{ }|} \hline
         &  \multicolumn{1}{c|}{$x=0$}        &  \multicolumn{1}{c|}{$0<x\leq1$}        & \multicolumn{1}{c|}{$1<x\leq2$}        & \multicolumn{1}{c|}{$2<x\leq3$}        & \multicolumn{1}{c|}{$3<x\leq5$}        & \multicolumn{1}{c|}{$5<x\leq10$}        & \multicolumn{1}{c|}{$x>10$}        \\\hline
No. of & \multicolumn{1}{c|}{5}            & \multicolumn{1}{c|}{30}           & \multicolumn{1}{c|}{33}           & \multicolumn{1}{c|}{36}           & \multicolumn{1}{c|}{28}           & \multicolumn{1}{c|}{43}           & \multicolumn{1}{c|}{25}\\[-0.1cm]
Question  &  & & & & & & \\\hline
Original          & 0.000        & 0.413        & 0.285        & 0.522        & 0.293        & 0.353        & 0.456\\\hline
Simple            & 0.000        & 0.187$^{--}$ & 0.030$^{--}$ & 0.278$^{-}$  & 0.336        & 0.526$^{+}$  & 0.592\\\hline
Combined          & 0.000        & 0.353        & 0.333        & 0.583        & 0.329        & 0.512$^{++}$ & 0.656$^{+}$\\\hline
Decreased         &              &              &              &              &              &              &      \\
k=0.01            & 0.000        & 0.413        & 0.285        & 0.522        & 0.257        & 0.353        & 0.536\\
k=0.02            & 0.000        & 0.413        & 0.285        & 0.539        & 0.293        & 0.367        & 0.536\\
k=0.05            & 0.000        & 0.413        & 0.285        & 0.528        & 0.293        & 0.386        & 0.576\\
k=0.1             & 0.000        & 0.387        & 0.345        & 0.528        & 0.293        & 0.405        & 0.616$^{+}$\\
k=0.2             & 0.000        & 0.387        & 0.333        & 0.556        & 0.364        & 0.488$^{+}$  & 0.616$^{+}$\\
k=0.3             & 0.000        & 0.353        & 0.333        & 0.583        & 0.329        & 0.512$^{++}$ & 0.656$^{+}$\\
k=0.4             & 0.000        & 0.287$^{-}$  & 0.242        & 0.539        & 0.336        & 0.512$^{+}$  & 0.672$^{+}$\\
k=0.5             & 0.000        & 0.287$^{-}$  & 0.212        & 0.467        & 0.371        & 0.572$^{++}$ & 0.672$^{+}$\\
k=0.6             & 0.000        & 0.287$^{-}$  & 0.182        & 0.361        & 0.371        & 0.591$^{++}$ & 0.672$^{+}$\\
k=0.7             & 0.000        & 0.187$^{--}$ & 0.182        & 0.389        & 0.371        & 0.628$^{++}$ & 0.672$^{+}$\\
k=0.8             & 0.000        & 0.187$^{--}$ & 0.121$^{-}$  & 0.306$^{-}$  & 0.336        & 0.544$^{+}$  & 0.672$^{+}$\\
k=0.9             & 0.000        & 0.187$^{--}$ & 0.061$^{--}$ & 0.306$^{-}$  & 0.336        & 0.544$^{+}$  & 0.632$^{+}$\\\hline
Clarke            & 0.000        & 0.000$^{--}$ & 0.030$^{-}$  & 0.206$^{--}$ & 0.300        & 0.549        & 0.576\\\hline
Mod. Clarke       & 0.000        & 0.120$^{--}$ & 0.091$^{--}$ & 0.250$^{-}$  & 0.336        & 0.526$^{+}$  & 0.632$^{+}$\\\hline
Xu                & 0.000        & 0.413        & 0.285        & 0.522        & 0.257        & 0.353        & 0.576\\\hline
Mori              & 0.000        & 0.287$^{-}$  & 0.182        & 0.344$^{-}$  & 0.379        & 0.628$^{++}$ & 0.632$^{+}$\\\hline
\end{tabular}
\label{tab:task-2-docnum}
\end{center}
\end{table}

\subsection{Relationship between Performance and Number of Documents Including Answers}

A problem occurred with the proposed method of using multiple documents as evidence with decreased adding 
when there was only one document including the answers. 
In this situation, 
the method could not add the scores for answers from multiple documents, 
which adversely affected
the performance of the system. 
We examined this problem using Sys-5, the best performing system. 
We calculated the system performance of Sys-5 
using different numbers of documents ($x$) including answers from 
the top 20 documents obtained during document retrieval, as described in Section \ref{sec:document_retrieval}. 
The results are shown in Tables \ref{tab:task-1-docnum} and \ref{tab:task-2-docnum}, and 
in Figures \ref{fig:task-1_results-docnum} and \ref{fig:task-2_results-docnum}. 
Table \ref{tab:task-1-docnum} lists the MRR values obtained by Sys-5 for Task-1 with various numbers of documents, while
Table \ref{tab:task-2-docnum} lists the MF values obtained by Sys-5 for Task-2. 
In the tables, the adding method is indicated in the leftmost column, while
the top line indicates the number of documents including answers ($x$). 
For questions with multiple answers, 
$x$ is the average of the number of documents including each answer.
We again applied the two-sided Wilcoxon signed ranks test to determine significant differences, 
using the original method as the baseline, as in Tables \ref{tab:task-1} and \ref{tab:task-2}. 
The results for the decreased adding method, as shown in Tables \ref{tab:task-1-docnum} and \ref{tab:task-2-docnum}, are also expressed 
as line charts in Figures \ref{fig:task-1_results-docnum} and \ref{fig:task-2_results-docnum}. 

There were five questions for which
the top 20 documents did not include any answers. 
In these cases, every score was 0. 

As expected, in the case of 0$<$x$\leq$1, 
the combined method performed worse than 
the original method.
Otherwise, however, the combined method 
performed better. 

The simple adding method did not perform as well as 
the original method 
with 0$<$x$\leq$5 for Task-1 and 0$<$x$\leq$3 for Task-2. 
It performed more poorly than the original method 
in more cases than did the combined method, 
showing that 
the combined method was more effective than the simple adding method.

Next, we examined the decreased adding method 
using the data shown in Figures \ref{fig:task-1_results-docnum} and \ref{fig:task-2_results-docnum}. 
The three lines for the cases of 0$<$x$\leq$1, 1$<$x$\leq$2, and 2$<$x$\leq$3 
decreased at higher values of $k$. 
This indicates that when the number of documents including answers is smaller, 
a lower value of $k$ is needed to maintain a good performance. 
On the other hand, the two lines for the cases of 5$<$x$\leq$10 and x$>$10 increased at higher values of $k$. 
This indicates that when the number of documents including answers is larger, 
the value of $k$ must also be larger to maintain a good performance. 
The values of 0.2 and 0.3, which were often used for $k$ in the combined method, 
as shown in Figures \ref{fig:task-1_freq} and \ref{fig:task-2_freq}, 
produced good, stable performance in every case. 

Next, we examined the results for the four other proposed methods (Clarke et al.'s method, modified Clarke et al.'s method, 
Xu et al.'s method, and Mori's method), as shown in Tables \ref{tab:task-1-docnum} and \ref{tab:task-2-docnum}. 
Xu et al.'s method obtained almost the same precision as the original method for x$\leq$10 and 
higher precisions for x$>$10. 
The other three methods 
did not perform as well as the original method 
at the 0.05 significant level for x$\leq$3 in Task-1, 
indicating that these methods were especially weak 
when x was small. 

\begin{figure}[p]
      \begin{center}
      \epsfile{file=data/graph3_.eps,height=7.2cm,width=12cm} 
      \end{center}
      \caption{Task-1 results as a function of k for Sys-5 with various numbers of documents ($x$)\\ including answers (MRR)}
    \label{fig:task-1_results-docnum}
\end{figure}

\begin{figure}[p]
      \begin{center}
      \epsfile{file=data/graph4_.eps,height=7.2cm,width=12cm} 
      \end{center}
      \caption{Task-2 results as a function of k for Sys-5 with various numbers of documents ($x$)\\ including answers (MF)}
      \label{fig:task-2_results-docnum}
\end{figure}

\clearpage

As described before, in the case of 0$<$x$\leq$1, 
the combined method did not perform as well as the original method.
We consider that the following approaches have the potential to 
further improve the system.
\begin{itemize}
\item 
  Increasing the size of the document set
  
  As shown by the results, the system performed better
  when the number of documents including answers was larger. 
  Therefore, if we were to use 
  very large document sets, unlike those used in these experiments, 
  such as web texts, 
  the number of documents including answers would be increased,
  thus improving the performance of the system.

\item 
  Identifing whether 
  the number of documents including answers 
  for an input question is 
  one or more
  
  In the case of 0$<$x$\leq$1, 
  the combined method did not perform as well as the original method. 
  If we could identify whether 
  the number of documents including answers was only one or more than one, 
  we would be able to use the original method for the case of 0$<$x$\leq$1 
  and the combined method for all other cases, 
  thus producing higher performance. 
  
\end{itemize}

\subsection{Examples of Results}

In this section, we show some examples of the results 
obtained using Sys-5, which performed best in our experiments. 
The first example is a case in which 
the combined method produced the correct answer and 
the original method produced an incorrect one.
The input question was
``What was the world's first retortable food product?''. 
The correct answer was ``Bon Curry'' (a Japanese curry product).
The combined method gave the correct answer, while
the original method incorrectly answered ``Otsuka Food'' (a food company). 
In the document with the highest score, 
``Otsuka Food'' occurred near the terms extracted from the input question 
and obtained a score of 1035.5, while
``Bon Curry'' obtained a slightly lower score of 1035.0.
Therefore, the original method extracted 
``Otsuka Food'' as the answer.
``Otsuka Food'' occurred seven times and 
``Bon Curry'' eight times in the top 20 documents. 
The combined method added the scores with decreasing weights, and as a result, 
the scores of ``Otsuka Food'' and ``Bon Curry'' were 1047.8
and 1048.4, respectively. 
Thus, the combined method produced the correct answer, ``Bon Curry'', as the first answer. 

The second example is a case in which 
the original method produced the correct answer and
the combined method produced an incorrect one.
The input question was ``What is the name of the brand that Toyota, Matsushita,
Asahi, Kao, and Kinki Nippon Tourist are developing jointly?''. 
``WiLL'' was the correct answer. 
The combined method produced an incorrect answer of ``integrated brand''. 
In the document with the highest score, 
``WiLL'' occurred near the terms extracted from the input question 
and obtained a score of 1076. 
``integrated brand'' obtained a slightly lower score of 1068, 
so the original method correctly extracted ``WiLL'' as the answer. 
``WiLL'' occurred 12 times and 
``integrated brand'' occurred 13 times in the top 20 documents. 
Using the combined method, 
i.e., adding scores with decreasing weights, 
the scores for ``WiLL'' and ``integrated brand'' increased to 1093 
and 1096, respectively.
Thus, the combined method selected the incorrect answer, ``integrated brand'', as the first answer. 

The third example is a case in which 
both the original method and the combined method selected the correct answer. 
The input question was ``When does NTT Communications plan to take over NTT-WN?''. 
``October 1'' was the correct answer. 
Both the original and combined methods produced the correct answer. 
The second highest scoring candidate answer was ``July 7''. 
In the document with the highest score, 
``October 1'' and ``July 7'' both occurred near the terms extracted from the input question. 
``October 1'' and ``July 7'' received scores of 1066.7 and 1064.2, respectively. 
With its slightly higher score, 
``October 1'' was correctly extracted as the answer by the original method. 
Furthermore, ``October 1'' and 
``July 7'' both occurred twice in the top 20 documents. 
In the combined method, 
by adding the scores with decreasing weights, 
the scores for ``October 1'' and ``July 7'' increased to 1078.1 and 1064.2, respectively. 
Thus, the combined method also produced the correct answer. 

\subsection{Summary of Discussion}

\begin{itemize}
\item 
  The simple adding method did not perform as well as the original method in some cases. 
  
\item 
  The combined method, which included our proposed method of adding scores with progressively decreasing weights, 
  performed better than the original method with four of the five question-answering systems tested, i.e., all except Sys-1.

\item 
  The combined method always performed better than the simple adding method. 

\item 
  Our proposed method produced a significant improvement, with values of 0.05 to 0.14 for the evaluation scores (MRR/MF).

\item 
  In the decreased adding method, 0.2 and 0.3 were appropriate values for $k$.  

\item 
  The combined method was more effective than the four other methods that we examined (i.e., Clarke et al.'s method, 
  the modified version of Clarke et al.'s method, Xu et al.'s method, and Mori's method).

\item 
  With our best question-answering system (Sys-5), 
  the combined method produced scores of 0.597 and 0.451 for Task-1 and Task-2, respectively, 
  while the original method produced scores of 0.541 and 0.376. 

\item 
  The proposed methods (i.e., the combined method including the decreased adding method) have 
  the following characteristics:

  \begin{itemize}
  \item 
    The proposed methods (i.e., the combined method including the decreased adding method) 
    were not effective 
    when no semantic information was used. 

\item 
  The proposed methods were very effective 
  when location information was not used.  

\item 
  The proposed methods 
  when we used both the two kinds of information
  were less effective than when 
  not using location information, but were effective. 

\item 
  The effectiveness of our proposed methods was not affected by 
  whether or not documents were divided into paragraphs during document retrieval. 
\end{itemize}

\item 
  One problem with the proposed methods
  was that they did not perform as well as the original method 
  when the number of documents including answers was only one. 
  To solve this problem and further improve the methods, 
  we could greatly increase the size of the document set or 
  identify whether the number of documents including answers for an input question was
  only one or more than one.
  
\end{itemize}

\section{Conclusion}
\label{sec:conclusions}

Question-answering systems are likely to become increasingly important as 
convenient alternatives to systems designed for information retrieval, and as 
basic components of future artificial intelligence systems. 
We therefore developed a new method of using 
multiple documents as evidence with decreased adding
to improve the performance of question-answering systems. 
Because the answer to a question may be found in multiple documents, 
using multiple documents to predict an answer 
should result in a better performance than using only one document. 
In our proposed method, information from multiple documents 
is utilized by adding the scores for the same candidate answers
extracted from various documents. 
Because simply adding these scores can degrade the performance of 
a question-answering system, 
we instead add the scores with progressively decreasing weights. 
We carried out experiments using the QAC test collection and 
the results showed that our method produced a statistically significant improvement. 
We also found that 
simply adding scores without decreasing weights 
resulted in a poorer performance in some cases. 
Our decreased adding method multiplies 
the score of the $i$-th candidate by a factor of $k^{(i-1)}$ before adding the score 
to the running total. 
We found experimentally that 0.2 and 0.3 were appropriate values for $k$. 
We also conducted experiments comparing our method to 
several previously proposed methods using multiple documents and 
found that our methods was more effective. 
The proposed method is simple and easy to use, 
and it produced very large score improvements. 
These results indicate
the potential feasibility and utility of our method in question-answering systems.
When we also applied the decreased adding method to various types of question-answering systems, 
it improved the performance in every case except one. 
The experiments indicated that 
our methods were not effective 
when no semantic information was used; 
they were very effective 
when location information was not used; and they
were less effective when 
using both semantic and location information 
compared to not using location information. 

In future work, we will 
greatly increase the size of the document set used 
in document retrieval by employing web texts. 
We will also try to improve our approach by identifying 
whether the number of documents including the answers for an input question is 
only one or more than one.
If these studies prove successful, 
we expect the system to achieve further improvement. 
Eventually, 
we hope to construct 
a high-performing system that can also be used 
as a component of future artificial intelligence systems. 

\bibliographystyle{nlpbbl}
\begin{thebibliography}{}

\bibitem[\protect\BCAY{Clarke, Cormack, \BBA\ Lynam}{Clarke
  et~al.}{2001}]{Charles}
Clarke, C. L.~A., Cormack, G.~V., \BBA\ Lynam, T.~R. \BBOP 2001\BBCP.
\newblock \BBOQ Exploiting Redundancy in Question Answering\BBCQ\
\newblock In {\Bem Proceedings of the 24th Annual International ACM SIGIR
  Conference on Research and Development in Information Retrieval}.

\bibitem[\protect\BCAY{Cristianini \BBA\ Shawe-Taylor}{Cristianini \BBA\
  Shawe-Taylor}{2000}]{SVM}
Cristianini, N.\BBACOMMA\  \BBA\ Shawe-Taylor, J. \BBOP 2000\BBCP.
\newblock {\Bem An Introduction to Support Vector Machines and Other
  Kernel-based Learning Methods}.
\newblock Cambridge University Press.

\bibitem[\protect\BCAY{Dumis, Banko, Brill, Lin, \BBA\ Ng}{Dumis
  et~al.}{2002}]{Dumis}
Dumis, S., Banko, M., Brill, E., Lin, J., \BBA\ Ng, A. \BBOP 2002\BBCP.
\newblock \BBOQ Web Question Answering: Is More Always Better?\BBCQ\
\newblock In {\Bem Proceedings of the 25th Annual International ACM SIGIR
  Conference on Research and Development in Information Retrieval}.

\bibitem[\protect\BCAY{EDR}{EDR}{1993}]{EDR93e}
EDR \BBOP 1993\BBCP.
\newblock {\Bem EDR Electronic Dictionary Technical Guide}.
\newblock EDR (Japan Electronic Dictionary Research Institute, Ltd.).

\bibitem[\protect\BCAY{Ittycheriah, Franz, Zhu, \BBA\ Ratnaparkhi}{Ittycheriah
  et~al.}{2001}]{IBM_QA2001}
Ittycheriah, A., Franz, M., Zhu, W.-J., \BBA\ Ratnaparkhi, A. \BBOP 2001\BBCP.
\newblock \BBOQ {IBM's Statistical Question Answering System}\BBCQ\
\newblock In {\Bem TREC-9 Proceedings}.

\bibitem[\protect\BCAY{Kudoh}{Kudoh}{2000}]{kudoh_svm}
Kudoh, T. \BBOP 2000\BBCP.
\newblock \BBOQ {TinySVM: Support Vector Machines}\BBCQ\
\newblock http://cl.aist-nara.ac.jp/~taku-ku// software/TinySVM/ index.html.

\bibitem[\protect\BCAY{Kupiec}{Kupiec}{1993}]{MURAX}
Kupiec, J. \BBOP 1993\BBCP.
\newblock \BBOQ {MURAX: A} Robust Linguistic Approach For Question Answering
  Using An On-Line Encyclopedia\BBCQ\
\newblock In {\Bem Proceedings of the Sixteenth Annual International ACM SIGIR
  Conference on Research and Development in Information Retrieval}.

\bibitem[\protect\BCAY{Magnini, Negri, Prevete, \BBA\ Tanev}{Magnini
  et~al.}{2002}]{magnini}
Magnini, B., Negri, M., Prevete, R., \BBA\ Tanev, H. \BBOP 2002\BBCP.
\newblock \BBOQ Is It the Right Answer? {Exploiting} Web Redundancy for Answer
  Validation\BBCQ\
\newblock In {\Bem Proceedings of the 41st Annual Meeting of the Association
  for Computational Linguistics}.

\bibitem[\protect\BCAY{Matsumoto, Kitauchi, Yamashita, Hirano, Matsuda, \BBA\
  Asahara}{Matsumoto et~al.}{1999}]{chasen}
Matsumoto, Y., Kitauchi, A., Yamashita, T., Hirano, Y., Matsuda, H., \BBA\
  Asahara, M. \BBOP 1999\BBCP.
\newblock \BBOQ Japanese Morphological Analysis System {ChaSen} version 2.0
  Manual 2nd edition\BBCQ.

\bibitem[\protect\BCAY{Moldovan, Pasca, \BBA\ Sanda~Harabagiu}{Moldovan
  et~al.}{2003}]{Harabagiu_tois}
Moldovan, D., Pasca, M., \BBA\ Sanda~Harabagiu, M.~S. \BBOP 2003\BBCP.
\newblock \BBOQ Performance Issues and Error Analysis in an Open-Domain
  Question Answering System\BBCQ\
\newblock {\Bem ACM Transactions on Information Systems}, {\Bem 21\/}(2),
  133--154.

\bibitem[\protect\BCAY{Mori}{Mori}{2004}]{Mori_ntcir4_qa}
Mori, T. \BBOP 2004\BBCP.
\newblock \BBOQ Japanese Q/A System Using A* Search and Its Improvement\BBCQ\
\newblock {\Bem Proceedings of the Fourth {NTCIR} Workshop}.

\bibitem[\protect\BCAY{Murata, Ma, \BBA\ Isahara}{Murata
  et~al.}{2002}]{Murata_ntcir3}
Murata, M., Ma, Q., \BBA\ Isahara, H. \BBOP 2002\BBCP.
\newblock \BBOQ High Performance Information Retrieval Using Many
  Characteristics and Many Techniques\BBCQ\
\newblock {\Bem Proceedings of the Third {NTCIR} Workshop ({CLIR})}.

\bibitem[\protect\BCAY{Murata, Uchimoto, Ozaku, Ma, Utiyama, \BBA\
  Isahara}{Murata et~al.}{2000}]{iral2000}
Murata, M., Uchimoto, K., Ozaku, H., Ma, Q., Utiyama, M., \BBA\ Isahara, H.
  \BBOP 2000\BBCP.
\newblock \BBOQ Japanese Probabilistic Information Retrieval Using Location and
  Category Information\BBCQ\
\newblock {\Bem The Fifth International Workshop on Information Retrieval with
  Asian Languages}, 81--88.

\bibitem[\protect\BCAY{Murata, Utiyama, \BBA\ Isahara}{Murata
  et~al.}{1999}]{qa_memo2}
Murata, M., Utiyama, M., \BBA\ Isahara, H. \BBOP 1999\BBCP.
\newblock
\newblock \BBOQ Question Answering System Using Syntactic Information\BBCQ.
\newblock http://xxx.lanl.gov/abs/cs.CL/9911006.

\bibitem[\protect\BCAY{Murata, Utiyama, \BBA\ Isahara}{Murata
  et~al.}{2000}]{murata2000_1_nl_eng}
Murata, M., Utiyama, M., \BBA\ Isahara, H. \BBOP 2000\BBCP.
\newblock \BBOQ Question Answering System Using Similarity-Guided
  Reasoning\BBCQ\
\newblock {\Bem Information Processing Society of Japan, WGNL 2000-NL-135},
  181--188.

\bibitem[\protect\BCAY{Murata, Utiyama, \BBA\ Isahara}{Murata
  et~al.}{2002}]{Murata_ntcir3_qa}
Murata, M., Utiyama, M., \BBA\ Isahara, H. \BBOP 2002\BBCP.
\newblock \BBOQ A Question-Answering System Using Unit Estimation and
  Probabilistic Near-Terms {IR}\BBCQ\
\newblock {\Bem Proceedings of the Third {NTCIR} Workshop ({QAC})}.

\bibitem[\protect\BCAY{Murata, Utiyama, Ma, Ozaku, \BBA\ Isahara}{Murata
  et~al.}{2001}]{Murata_ntcir2}
Murata, M., Utiyama, M., Ma, Q., Ozaku, H., \BBA\ Isahara, H. \BBOP 2001\BBCP.
\newblock \BBOQ {CRL} at {NTCIR2}\BBCQ\
\newblock {\Bem Proceedings of the Second {NTCIR} Workshop Meeting on
  Evaluation of {Chinese} \& {Japanese} Text Retrieval and Text Summarization},
  5--21--5--31.

\bibitem[\protect\BCAY{{National Institute of Informatics}}{{National Institute
  of Informatics}}{2002}]{qac1}
{National Institute of Informatics} \BBOP 2002\BBCP.
\newblock {\Bem Proceedings of the Third {NTCIR} Workshop ({QAC})}.

\bibitem[\protect\BCAY{{NTCIR-3 QAC task committee}}{{NTCIR-3 QAC task
  committee}}{2001}]{qac_hp}
{NTCIR-3 QAC task committee} \BBOP 2001\BBCP.
\newblock
\newblock \BBOQ {Question Answering Challenge} (QAC) Home Page\BBCQ.
\newblock {http://www.nlp.cs.ritsumei.ac.jp/qac/}.

\bibitem[\protect\BCAY{Robertson \BBA\ Walker}{Robertson \BBA\
  Walker}{1994}]{2poisson}
Robertson, S.~E.\BBACOMMA\  \BBA\ Walker, S. \BBOP 1994\BBCP.
\newblock \BBOQ Some Simple Effective Approximations to the 2-Poisson Model for
  Probabilistic Weighted Retrieval\BBCQ\
\newblock In {\Bem Proceedings of the Seventeenth Annual International ACM
  SIGIR Conference on Research and Development in Information Retrieval}.

\bibitem[\protect\BCAY{Robertson, Walker, Jones, Hancock-Beaulieu, \BBA\
  Gatford}{Robertson et~al.}{1994}]{robertson_trec3}
Robertson, S.~E., Walker, S., Jones, S., Hancock-Beaulieu, M.~M., \BBA\
  Gatford, M. \BBOP 1994\BBCP.
\newblock \BBOQ Okapi at TREC-3\BBCQ\
\newblock In {\Bem TREC-3}.

\bibitem[\protect\BCAY{Sasaki}{Sasaki}{2003}]{Sasaki_yutaka_ieice}
Sasaki, Y. \BBOP 2003\BBCP.
\newblock \BBOQ Question Answering as Abduction: A Feasibility Study at NTCIR
  QAC1\BBCQ\
\newblock {\Bem IEICE Transactions on Information and Systems}, {\Bem
  E86--D\/}(9), 1669--1676.

\bibitem[\protect\BCAY{Takahashi, Nawata, Inui, \BBA\ Matsumoto}{Takahashi
  et~al.}{2003}]{takahashi_ieice_qa}
Takahashi, T., Nawata, K., Inui, K., \BBA\ Matsumoto, Y. \BBOP 2003\BBCP.
\newblock \BBOQ Effect of Structural Matching and Paraphrasing in Question
  Answering\BBCQ\
\newblock {\Bem IEICE Transactions on Information and Systems}, {\Bem
  E86--D\/}(9), 1677--1685.

\bibitem[\protect\BCAY{Takaki \BBA\ Eriguchi}{Takaki \BBA\
  Eriguchi}{2002}]{takaki_ntcir3_qa}
Takaki, T.\BBACOMMA\  \BBA\ Eriguchi, Y. \BBOP 2002\BBCP.
\newblock \BBOQ {NTT DATA} Question-Answering Experiment at the {NTCIR-3
  QAC}\BBCQ\
\newblock {\Bem Proceedings of the Third {NTCIR} Workshop ({QAC})}.

\bibitem[\protect\BCAY{{TREC-8 committee}}{{TREC-8 committee}}{1999}]{trec8qa}
{TREC-8 committee} \BBOP 1999\BBCP.
\newblock
\newblock \BBOQ TREC-8 Question Answering Track\BBCQ.
\newblock {http://www.re-search.att.com/\verb+~+singhal/qa-track.html}.

\bibitem[\protect\BCAY{Uchimoto, Murata, Ozaku, Ma, \BBA\ Isahara}{Uchimoto
  et~al.}{2000}]{acl_NE}
Uchimoto, K., Murata, M., Ozaku, H., Ma, Q., \BBA\ Isahara, H. \BBOP 2000\BBCP.
\newblock \BBOQ Named Entity Extraction Based on Maximum Entropy Model and
  Transformation Rules\BBCQ\
\newblock {\Bem Proceedings of the 38th Annual Meeting of the Association of
  Computational Linguistics}.

\bibitem[\protect\BCAY{Xu, Licuanan, \BBA\ Weischedel}{Xu
  et~al.}{2003}]{Xu2003}
Xu, J., Licuanan, A., \BBA\ Weischedel, R. \BBOP 2003\BBCP.
\newblock \BBOQ {TREC2003 QA at BBN:} {Answering} Definitional Questions\BBCQ\
\newblock In {\Bem TREC-2003 Proceedings}.

\bibitem[\protect\BCAY{Yamada, Kudo, \BBA\ Matsumoto}{Yamada
  et~al.}{2002}]{yamada_ipsj_svm_ne_eng}
Yamada, H., Kudo, T., \BBA\ Matsumoto, Y. \BBOP 2002\BBCP.
\newblock \BBOQ Japanese Named Entity Extraction Using Support Vector
  Machine\BBCQ\
\newblock {\Bem Transactions of Information Processing Society of Japan}, {\Bem
  43\/}(1), 44--53.
\newblock (in Japanese).

\end{thebibliography}


\begin{biography}

\biotitle{}

\bioauthor{Masaki Murata}
{
Masaki Murata received his Bachelor's,
  Master's, and Doctorate degrees in engineering from Kyoto University in 1993, 1995, 
  and 1997, respectively.  
He is a senior researcher at the National Institute of Information and Communications Technology, Japan, 
an independent administrative institution.  
He is a member of the Information Processing Society of Japan,
the Japanese Society for Artificial Intelligence, 
the Institute of Electronics, Information and Communication Engineers,
the Mathematical Linguistic Society of Japan, 
and the Association for Computational Linguistics.
His research interests include natural language processing, machine translation, 
information retrieval, and question answering.
}

\bioauthor{Masao Utiyama}
{
Masao Utiyama is a senior researcher at the National Institute of
Information and Communications Technology, Japan, an independent administrative institution. 
His main research field is natural language processing. He completed his
doctoral dissertation at the University of Tsukuba in 1997. His
current research interests are exploring models of natural
languages and their practical applications.
}

\bioauthor{Hitoshi Isahara}
{
Hitoshi Isahara received his Bachelor's,
Master's, and Doctoral degrees in engineering from Kyoto University in 1978, 1980,
and 1995, respectively.  He is currently a leader of the Computational Linguistic Group  
at the National Institute of Information and Communications Technology, Japan, 
an independent administrative institution.  
He is a member of the Information Processing Society of Japan, 
the Japanese Society for Artificial Intelligence, 
the Japanese Cognitive Science Society, 
and the Association for Computational Linguistics.
His research
interests include natural language processing, machine translation, and
lexical semantics.
}


\bioreceived{Received}
\biorevised{Revised}
\bioaccepted{Accepted}

\end{biography}

\end{document}

