    \documentclass[english]{jnlp_1.4_rep}

\usepackage{jnlpbbl_1.3}
\usepackage[dvipdfm]{graphicx}
\usepackage{amsmath}
\usepackage{hangcaption_jnlp}
\usepackage{udline}
\setulminsep{1.2ex}{0.2ex}
\usepackage{array}

\Volume{21}
\Number{4}
\Month{September}
\Year{2014}

\received{2005}{2}{7}
\revised{2005}{4}{19}
\accepted{2005}{5}{23}

\setcounter{page}{707}

\etitle{Improvements of Katz K Mixture Model\footnotetext{\llap{*~}This article has been partially revised for better understanding of overseas readers.}}
\eauthor{Yinghui Xu\affiref{ToyohashiUniv} \and Kyoji Umemura\affiref{ToyohashiUniv}}
\eabstract{
A simpler distribution that fits empirical word distribution about as well as a negative binomial is the Katz K mixture. In the K 
mixture model, the basic assumption is that the conditional probabilities of repeats for a given word are determined by a constant decay factor that 
is independent of the number of occurrences which have taken place. However, the probabilities of the repeat occurrences are generally lower than 
the constant decay factor for the content-bearing words with few occurrences that have taken place. To solve this deficiency of the K mixture model, 
in-depth exploration of the characteristics of the conditional probabilities of repetitions, decay factors and their influences on modeling term 
distributions was conducted. Based on the results of this study, it appears that both ends of the distribution can be used to fit models. That is, 
not only can document frequencies be used when the instances of a word are few, but also tail probabilities (the accumulation of document 
frequencies). Both document frequencies for few instances of a word and tail probabilities for large instances are often relatively easy to estimate 
empirically. Therefore, we propose an effective approach for improving the K mixture model, where the decay factor is the combination of two 
possible decay factors interpolated by a function depending on the number of instances of a word in a document. Results show that the proposed model 
can generate a statistically significant better estimation of frequencies, especially the frequency estimation for a word with two instances in a 
document. In addition, it is shown that the advantages of this approach will become more evident in two cases, modeling the term distribution for 
the frequently used content-bearing word and modeling the term distribution for a corpus with a wide range of document length.}
\ekeywords{Term Distribution Model, Katz K mixture, decay factor}

\headauthor{Xu and Umemura}
\headtitle{Improvements of Katz K Mixture Model}

\affilabel{ToyohashiUniv}{}{Toyohashi University of Technology}

\Reprint{Vol.~12, No.~5, pp.~131--155}

\begin{document}

\maketitle

\section{Introduction}

Term distribution models are a subject of great general interest and a matter of even greater practical importance. In addition to information
retrieval \cite{Robertson80}, a good understanding of distribution patterns is useful wherever we want to assess the likelihood of a certain number
of occurrences of a specific word in a unit of a text.

Linguistic count data often violates the simplest assumptions of 
\pagebreak
standard probability models like a binomial or Poisson distribution. In particular,
the inadequacy of the Poisson distribution for modeling word frequency is well known, and alternatives have been proposed 
    (Church and Gale 1995; Katz 1996; Manber and Myers 1990)\nocite{Manber90}. 
In the case of the Poisson, a mixture of Poisson distribution has been a
commonly-accepted model for the distribution of content words in text documents, with the two-Poisson model viewed as a meaningfully interpretable
approximation of a general model \cite{Robertson94}. But, the underlying mechanism of the Poisson process for generating content words in text is
incompatible with empirical data. The observed occurrences are not independent of each other, and multiple instances of content words are observed
close to each other far too frequently than it would have been if they, indeed, were the outcome of a Poisson process. Another commonly used robust
alternative is the negative binomial distribution \cite{Church95}, which has the ability to capture extra-Poisson variations in the data. The
negative binomial fits term distribution better than one or two Poisson, but can be hard to work with in practice because it involves the
computation of large binomial coefficients.

The Katz K mixture model \cite{Katz96} is known to be one of the best term distribution models to capture the linguistic properties of term
occurrences. There are two kinds of K mixture models proposed by Katz: the two-parameter model and three-parameter model. In the Katz K mixture,
terms with specific frequencies are counted according to his innovative definitions of the three-way classifications of documents which are: among
documents containing occurrences of any particular content word, documents with a single instance of that word are assumed to be non-topical
documents while documents including multiple instances of that word are assumed to be topical documents; All other documents containing no
occurrences of that word are considered unrelated documents. Both Katz models assume that the conditional probabilities of repeats are independent
of the number of previously-observed occurrences and approximate them by some constant. Whereas, in contrast to the two-parameter models which does
not treat the non-topical and topical documents differently, the three-parameter K mixture model handles the non-topical and topical documents
separately, and the probabilities of repeat occurrences of the content word are assumed to be the constant only among topical documents. However,
while reviewing the Katz K mixture models, although considerable improvements in the modeling of term distribution are made by the three-parameter
Katz K mixture compared with the two-parameter model, there are still comparatively critical misjudgments in the distribution of term frequencies
observed in the experiments, especially for a word with two or three instances. After examining the assumptions carefully, we noted that the
accidental repeats of non-topically-used content words are overlooked in the Katz three-parameter model since the conditional probabilities of
repeats in the topical documents are treated equally without discrimination. Nonetheless, cases of accidental repeats are possible, particularly for
frequently used content words or in a long document. In our view, the accidental repeats of a word in a document will probably occur when there are
few instances of the word. In this paper, an investigation on the practical characteristic of the conditional probabilities of repeats is conducted.
It was discovered that: for function words, the assumption above is acceptable while for content-bearing words, a word with more occurrences will
have a greater chance of being repeated and that chance tends to stabilize with an increase in the instances of the word. At the same time, the
decay factor, as the deterministic component of the K mixture model, was also explored. The findings in the experiments indicate it is possible to
use both ends of the distribution to fit models. That is, not only can document frequencies be used when instances of a word are few, but also tail
probabilities can be employed. Both document frequencies for a few instances of a word and tail probabilities for many instances are often fairly
easy to estimate empirically. Therefore, we propose an effective approach for improving the K mixture model, where the decay factor is the
combination of two possible decay factors interpolated by a function depending on the number of instances of a word in a document. Results show that
the proposed model will generate a statistical-significantly improved estimation of frequencies, especially the frequency estimation for a word with
exactly two instances in a document. In addition, it is shown that the advantages of our approach will become more obvious in two cases, modeling
the term distribution for the content-bearing word that is used frequently and modeling the term distribution for a corpus with a wide range of
document length. Our model deals with only the distribution of single term, and thus is simpler than other researches that treat relationship
between plural words, such as, Probabilistic Latent Semantic Indexing (PLSI) \cite{Hoffman}, Latent Dirichlet Allocation (LDA) \cite{Daivd} and
Dirichlet Mixtures (DM) \cite{Mikio}. Treating the relationship between plural words will make the model more interesting and is quite possible to
form the theoretical framework of our proposed model, which will be the research of our future work.

This paper is organized as follows: experiment data is firstly described in Section 2. In Section 3, important notations often used in the paper are
defined. The Katz K mixture model and experimental results are introduced in Section 4. In Section 5, investigations on several important issues
related to the K mixture model are conducted. Based on findings in Section 5, an improved model for the K mixture is proposed in Section 6.
Conclusions are presented in Section 7.


\section{Experiment Data}

The experimental part of this study involves a large collection, English Gigaword Corpora, of
English news text \cite{LDC}. There are four distinct international sources of English newswires:
\begin{itemize}
    \item AFE  Agence France Presse English Service
    \item NYT  The New York Times Newswire Service
    \item APW  Associated Press Worldstream English Service
    \item XIE  The Xinhua News Agency English Service
\end{itemize}
The results reported in the paper are based on the experiments using AFE collections, which contain six years news text. There are about $170,969$
keywords and $656,269$ documents in the entire corpus. To explore the term distribution model for this large corpus, the frequencies of certain
items (words, word pairs, substrings, etc.) were counted. Word-based suffix array \cite{Manber90} was used to accumulate the necessary frequency and
rates (counts of frequency) information. To avoid some unpredictable influences on the modeling term distributions, the experimental conditions were
limited to the following:
\begin{itemize}
    \item The range of the document length in the AFE collection is quite wide, from 1 to 6,950, and
    the document length of most of the documents is within the range from 200 to 800. To avoid influences on
    term distribution, statistical information of the terms was extracted from a subset of the total corpus
    in which document length were limited from 300 to 500. The number of documents in this subset is
    153,132.
    \item In the experiments, to find the characteristics of word repetitions, words with fat-tail properties were used.
    A Fat-tail properties means that there are two or more document having 10 or more appearances of the word in the test corpus.
    \item There were many types of words in the test corpus. They were classified into two groups based on their linguistic
    functionality. Firstly, words such as adjectives, pronouns, conjunctions, prepositions,
    and auxiliary verbs were chosen and called function words.  Their occurrences
    are independent from what the document is about. Second, in contrast to function words, some proper
    nouns and compound noun terms, intrinsically content entities, are extracted as
    content-bearing words. An examination of distribution was conducted, respectively, for these two types.
    \item The counting of terms was case-insensitive, and no stemming treatment was conducted on
    the corpus.
\end{itemize}


\section{Notations}

Before introducing the Katz K mixture model, some often used symbols which represent empirical statistical values in the collection were
defined as follows: \\
\textbf{Definitions 1}\\
    $w$:  \mbox{terms (words, word pairs, substrings etc.) that need to be counted.} \\
    $d$:  \mbox{a document containing a sequence of terms.} \\
    $D$:  \mbox{a set of documents, usually called the corpus.} \\
    $N$:  $N=|D|$, \mbox{the size of the corpus.} \\
\textbf{Definitions 2} \\
    $\mathit{tf}(w,d)$: the number of occurrences of $w$ in $d$ \\
    $\mathit{cf}(w) \equiv \sum\limits_{d \in D}\mathit{tf}(w,d)$ \\
    $\mathit{df}(w) \equiv \vert\{d \vert \mathit{tf}(w,d)\ge 1\}\vert$ \\
These three definitions are commonly used in information retrieval literature. They are usually called within-document term frequency, corpus
frequency, and document frequency respectively. \\
\textbf{Definitions 3} \\
  $\mathit{df}(k;w) \equiv \vert\{d \vert \mathit{tf}(w,d)\ge k, d \in D \}\vert $ \\
  The $\mathit{df}(k;w)$ value, as a kind of generalized document frequency, is the number of documents that
	contain $w$ for $k$ or more times. In particular, we have: $\mathit{df}(1;w)=\mathit{df}(w)$ and $N=\mathit{df}(0;w)$. \\
\textbf{Definitions 4} \\
    $\mathit{ddf}(k;w)\equiv \vert \{d \vert \mathit{tf}(w,d)=k,d \in D \} \vert $ \\
    The $\mathit{ddf}(k;w)$ value, as a type of differential document frequency, 
	is the number of documents containing the word $w$ exactly $k$ times.
    With the above definitions, it is easy to obtain:
\begin{gather*}
  \mathit{df}(k;w) \equiv \sum\limits_{i \ge k}\mathit{ddf}(i;w) \\
  \mathit{ddf}(k;w)=\mathit{df}(k;w)-\mathit{df}(k+1;w).
\end{gather*}
\textbf{Definitions 5} \\
  $\mathit{cdf}(k;w)\equiv \sum\limits_{i \ge k}\mathit{df}(i;w)$ \\
  For expository convenience, the definition of $\mathit{cdf}(k;w)$, as a kind of cumulative document 
\linebreak
frequency, is introduced in this paper. The
  generalized document frequency, $\mathit{df}(k;w)$, can be recovered from $\mathit{cdf}(k;w)$. The properties of the cumulative frequency are:
\begin{gather*}
  \mathit{cdf}(1;w)=\mathit{cf}(w)  \\
  \mathit{df}(k;w)= \mathit{cdf}(k;w)-\mathit{cdf}(k+1;w) 
\end{gather*}
\textbf{Definitions 6} \\
  $P_{w}(k+1|k) \equiv P(\frac{\mathit{tf}(w,X) \ge k+1}{\mathit{tf}(w,X) \ge k})$
  \mbox{ where: $X$ is the random variable for $d$} \\
  The $P_{w}(k+1|k)$ value, used by Katz \cite{Katz96}, represents the likelihood of a term having more than $k$ occurrences in a document given that
exactly $k$ occurrences have already taken place. Through the maximum likelihood estimation (MLE), the value of the conditional probabilities can be
estimated empirically as $\hat{P_{w}}(k+1|k)=\mathit{df}(k+1;w) / \mathit{df}(k;w)$. Particularly, in the case of $k=1$, the $\hat{P_{w}}(2|1)$ is
estimated by $\mathit{df}(2;w) / \mathit{df}(w)$ and this statistic had been used as the approximation of adaptation \cite{Church2000}. Adaptation
is a term borrowed from the literature on language modeling for speech recognition \cite{Jelinek99}.


\section{Katz K Mixture}

The Katz K mixture is a simpler distribution that fits empirical word distributions as well as a negative binomial model \cite{Manning99}. There are
two kinds of K mixture models proposed by Katz, the two-parameter model and a three-parameter model. First, we have introduced the definition of a
three-way classification of documents \cite{Bookstein74} as used by Katz in his model descriptions. In the proportion of documents that contain
occurrences of any particular content word, documents with only a single instance of that word are assumed to be non-topical documents, while
documents including multiple instances of that word are assumed to be topical documents; All other documents containing no occurrences of that word
are deemed as unrelated documents. Both Katz models assume that the conditional probabilities of repeats are independent from the number of
previously-observed occurrences, and approximate them by some constant. Descriptions of these two models are presented, respectively, in the
following sections. Here we need to point out that to simplify the model description, the statistics of the words,
$\mathit{df}$,$\mathit{cf}$,$\mathit{df}(k)$, $\mathit{ddf}(k)$, and $\mathit{cdf}(k)$ are used in what follows without specifying the word $w$.


\subsection{The Conventional Katz K Mixture---Two-Parameter Model}
In the two-parameter model, occurrences in non-topical documents and topical documents are not treated differently. Accordingly, the probabilities
of repeats as the probabilities of a word which has exactly $k$ occurrences in a document is modeled by:
\begin{align*}
 & P_{\mathit{Katz}2}(\mathit{tf}(w,X)=k) = (1 - \alpha)\delta_{k,0} + \frac{\alpha}{\beta + 1}\left(\frac{\beta}{\beta +1}\right)^k \\
 & \mbox{where: } \quad
  \delta _{k,0} = \left\{
            \begin{array}{l}
                1 \quad \mbox{iff} \quad k = 0 \\
                0 \quad \mbox{otherwise}\\
            \end{array} \right. 
\end{align*}

The parameter $\alpha$ and $\beta$ are estimated using the observed
mean $\lambda$ and the observed inverse document frequency
($\mathit{IDF}$) as follows:
\begin{align*}
 & \mbox{observed mean:  }
 	\hat{\lambda} = \frac{\mathit{cf}}{N}, \qquad
 	\hat{\mathit{IDF}}     = \log _2\frac{N}{\mathit{df}} \\
 & \hat{\alpha}  = \frac{\hat{\lambda}}{\hat{\beta}}= \frac{\mathit{cf}}{N} \times \frac{\mathit{df}}{\mathit{cf} - \mathit{df}} 
	= \frac{\mathit{df}}{\mathit{cdf}(2)} \\
 & \hat{\beta}   = \hat{\lambda} \times 2^{\hat{\mathit{IDF}}}-1 = \frac{\mathit{cf} - \mathit{df}}{\mathit{df}}
	=\frac{\mathit{cdf}(2)}{\mathit{df}}
\end{align*}

The parameter $\alpha$ captures the absolute frequency of the term. Two terms with the same $\beta$ have identical ratios of collection frequency to
document frequency, but different values for $\alpha$ if their collection frequencies are different. The parameter $\beta$ is the number of ``extra
terms'' per document in which the term occurs (compared to the case where a term has only one occurrence per document). The decay factor 
$\hat{\beta}/ (\hat{\beta}+1)= \mathit{cdf}(2) / \mathit{cf}$ (extra terms per term occurrence) determines the ratio 
$P_{\mathit{katz}}(k+1) /P_{\mathit{katz}}(k)$, which is the quantity used to measure conditional probability, $P_{w}(k+1|k)$.


\subsection{Katz's Three-parameter Model}

The conventional Katz K mixture model is called a two-parameter model in Katz's paper. He pointed out that the two-parameter model does not have
enough parameters to treat $P_{w}(2|1)$ differently from the rest of the conditional probabilities, $P_{w}(k+1|k),k\ge 2$. Accordingly, the
three-parameter model, where the difference between $P_{w}(2|1)$ and the rest of conditional probabilities, $P_{w}(k+1 \vert k)$ for all $k \geq 2$
are considered, is also proposed in his paper. The model descriptions are shown as follows:
{\allowdisplaybreaks
\begin{align*}
 & P_{\mathit{Katz3}}(\mathit{tf}(w,X)=k) \\
 & \quad = (1 - \alpha)\delta_{k,0} + \alpha \times(1-\gamma)\times \delta _{k,1} + \frac{\alpha \times
	\gamma }{\beta + 1}\left(\frac{\beta}{\beta + 1}\right)^{k -2}\times(1-\delta_{k,0}-\delta _{k,1}) \\
 & \mbox{where:} \qquad
	 \delta _{i,j}= \left\{
            \begin{array}{l}
             1 \quad \mbox{iff} \quad i = j \\
             0 \quad \mbox{otherwise} \\
            \end{array} \right.\\
 & \mbox{The parameter $\alpha$, $\gamma$ and $\beta$ are estimated by:} \\
 & \qquad \left\{\begin{array}{l}
		\hat{\alpha} = \frac{\mathit{df}(1)}{N} \\
		\hat{\gamma}  = \frac{\mathit{df}(2)}{\mathit{df}(1)} \\
		\hat{\beta}  = \frac{\mathit{cf}-\mathit{df}(1)-\mathit{df}(2)}{\mathit{df}(2)} = \frac{\mathit{cdf}(3)}{\mathit{df}(2)} \\
	  \end{array}\right.
\end{align*}
}

We can denote the two Katz models as $P_{\mathit{katz}}(k;\alpha,\beta)$ and $P_{\mathit{katz'}}(k;\alpha,\gamma,\beta)$, respectively. 
\linebreak
According to
the experimental reports in Katz's paper, the three-parameter model, called the G-model by Katz, works much better than the well-known conventional
Katz K mixture. In the G-model, the frequencies of zero, single and multiple occurrences are treated separately and consequently, the estimations
for the $\mathit{ddf}(0)$ and $\mathit{ddf}(1)$, computed by $\mathit{ddf}(0)=N \times P_{\mathit{katz'}}(0;\alpha,\gamma,\beta)$ and
$\mathit{ddf}(1)=N \times P_{\mathit{katz'}}(1;\alpha,\gamma,\beta)$, respectively, are always fitted exactly by such construction. The decay
factor, $\beta / (\beta+1)$, for representing the remaining conditional probabilities is equal to $\mathit{cdf}(3) / \mathit{cdf}(2)$. To illustrate
the problems of the K mixture for modeling term distribution, let us first look at some examples. The number of documents with $k$ (from 1 to 9)
occurrences in the test corpus for both six function words and six content-bearing words are presented in Table~1. In the cases of function words,
we found that the two K mixture models are both able to model a reasonable frequency distribution, and there is no great difference between their
estimations. However, for content-bearing words, the situation is less favorable. Two Katz models behave differently. The conventional model is far
worse than the three-parameter model. The estimations by the three-parameter model are quite reasonable for the frequencies of the tail in the
distribution, but comparatively serious misjudgments of frequencies occur in the beginning of the distribution except for the points 0 and 1. To our
understanding, the kernel part of the K mixture model lies in the method to decide the decay factor. The more precisely the conditional
probabilities, $P_{w}(k+1|k)$, $k\ge 1$, are reflected by the decay factor, the better the predictions of $\mathit{ddf}(k)$ will be. But what is the
characteristic of the conditional probabilities, $P_{w}(k+1|k)$, in the case of a different $k$? What are the differences among some possible decay
factors, and how do they influence the prediction results? Answers to these questions will help find improvements of the K mixture model and make it
more attractive for content-bearing words.

\begin{table}[p]
\hangcaption{Actual and estimated numbers of documents with $k$ occurrences for six function words and six content-bearing terms}
\input{04table01.txt}
\vspace{4pt}\small
K'1 represents the conventional Katz model and K'2 is the Katz's three parameter model. Samples show that two Katz's K mixture model are both work well for function words. While, for content-bearing words, the Katz's three parameter model performs much better than the conventional model.
\par
\end{table}


\section{Observations}

\subsection{Observation on the Conditional Probabilities}

The basic assumption of the K mixture model is that the conditional probability of a term is independent of the number $k$ and approximates them by
some constant. How is the constant determined empirically based on the assumption of K mixture? Although there are a number of equivalent ways to
explain this, we have chosen the following method to better explain why the process goes beyond the K mixture model.
\begin{align*}
 & \mbox{Given the assumption: } \\
 & P_{w}(k + 1\vert k) = C; \\
 & \mbox{Emprical Estimation of the conditional probability given $k$: }\\
 & \hspace{30pt} k = 1:\qquad \hat{P_{w}}(2 \vert 1)   = \frac{P(\mathit{tf}(w,X)\ge 2)}{P(\mathit{tf}(w,X)\ge 1)}= \frac{\mathit{df}(2) }{\mathit{df}(1) } \\
 & \hspace{30pt}k = 2:\qquad \hat{P_{w}}(3 \vert 2)   = \frac{P(\mathit{tf}(w,X)\ge 3)}{P(\mathit{tf}(w,X)\ge 2)}= \frac{\mathit{df}(3) }{\mathit{df}(2) } \\
 & \hspace{30pt}\cdots \\
 & \hspace{30pt}k= k+1:\qquad\hat{P_{w}}(k+1 \vert k) = \frac{P(\mathit{tf}(w,X)\ge k+1)}{P(\mathit{tf}(w,X)\ge k)} = \frac{\mathit{df}(k+1)}{\mathit{df}(k)}\\
 & \mbox{For the conventional K mixture model: }\\
 & \frac{\mathit{df}(2) }{\mathit{df}(1) } = \frac{\mathit{df}(3) }{\mathit{df}(2) }
	    = \cdots
	    = \frac{\mathit{df}(k+1) }{\mathit{df}(k) } = \frac{\beta }{\beta+ 1} = C  \\
 & \Rightarrow \quad 
    C = \frac{\mathit{df}(2) + \mathit{df}(3) + \cdots + \mathit{df}(k+1) }{\mathit{df}(1) + \mathit{df}(2) +\cdots + \mathit{df}(k) } 
	\equiv \frac{\mathit{cf} - \mathit{df}(1) }{\mathit{cf}} = \frac{\mathit{cdf}(2)}{\mathit{cdf}(1)}  \\
 & \mbox{For the three parameter model: } \\
 & \frac{\mathit{df}(3) }{\mathit{df}(2) }
    = \cdots
    = \frac{\mathit{df}(k+1) }{\mathit{df}(k) } = \frac{\beta }{\beta+ 1} = C' \\
 & \Rightarrow \quad
    C'= \frac{\mathit{df}(3) + \cdots + \mathit{df}(k+1) }{\mathit{df}(2) +\cdots + \mathit{df}(k)} 
	\equiv \frac{\mathit{cf} - \mathit{df}(1)-\mathit{df}(2) }{\mathit{cf}-\mathit{df}(1)}= \frac{\mathit{cdf}(3)}{\mathit{cdf}(2)}
\end{align*}
\begin{figure}[p]
\begin{center}
\includegraphics{21-4ia4f1.eps}
\end{center}
\hangcaption{Comparison of the conditional probability $Pr(k+1|k)$ for $1 \le k\le9$
  using sample words with fat-tail properties, including eight content-bearing words and eight function words.
  There are only slight differences in the conditional probabilities for function words,
  while, for content-bearing words, there are clear changes between $P(3|2)$ and $P(2|1)$.}
\end{figure}
It is apparent that the derived value $\mathit{cdf}(2) / \mathit{cdf}(1)$ in the conventional K mixture model does not differentiate the case of
$P_{w}(2|1)$ from the rest of the conditional probabilities $P_{w}(k+1|k),k\geq 2$ while the Katz's three-parameter model does. As pointed out by
Katz, in practice, the conditional probability $P_{w}(2|1)$ is an entity that is different from the other conditional probabilities, 
\linebreak
$P_{w}(k+1|k)$,
$k\geq 2$, since they are related to different properties. $P_{w}(2|1)$ is about how likely a word is to be used again once it first occurs, while
the latter are about the probabilities of consecutive repeats provided that the word had already been used repeatedly. Meanwhile, intuitively
speaking, a word having occurred, does not necessarily tend to repeat, whereas, an often-repeated word will probably be repeated again. The
comparison of conditional probabilities under different $k$ values for some content-bearing words and function words with notable characteristic are
plotted in Figure 1. The distinct phenomenona are: for the content-bearing words, $P_{w}(2 \vert 1)$ are generally less than $P_{w}(k+1 \vert k)$
and when $k \geq 2$, $P_{w}(k+1 \vert k)$ is tend to stabilize with the increment of $k$, while for the function words, there are no great
differences in $P_{w}(2 \vert 1)$ from the rest of the conditional probabilities. This could be explained by the linguistic background. The cause of
repetitions of function words is the requirement of forming an integrated language structure for a document. Since the structure of a language is
the common rule no matter how many times a word has already been repeated in all kinds of documents, the repetitive features of such types of words
will likely be captured by the frequency statistics from large-scale text collection, and where the variance degree of the conditional probabilities
is somewhat small. However, the cause of repetitions of content-bearing words is probably determined by the requirements of the author himself to
ensure that the meaning of the document is clear. Particularly, the conditional probability reflecting the repetition characteristics will become
larger and more stable with an increment in the number of instances of a content-bearing word in a document due to its tendency to be a
topic-related word. That is to say, the single occurrence of a word may not cover the topic of the document while a word with multiple occurrences
will probably be a topic-related word. Accordingly, it is reasonable for the Katz three-parameter model to treat the $P_{w}(2|1)$ and
$P_{w}(k+1|k)$, $k\geq 2$ differently. However, accidental repeats of non-topically used content words are possible, particularly for the
frequently-used content words or in a long document. This is overlooked in the Katz three-parameter model since the conditional probabilities,
$P_{w}(k+1|k)$, $k\geq 2$, are treated equally without discrimination. To further investigate the conditional probabilities, experiments were conducted
on 289 content-bearing words. The mean of $P_{w}(k+1 \vert k)$, $k\ge 1 $ and other advanced descriptive statistics including $\mathit{P25}$,
$\mathit{P75}$\footnote{$\mathit{P25}$ and $\mathit{P75}$ means 25th percentiles and 75th percentiles, respectively} and Median are all plotted in
Figure 2. In addition to the phenomenon of $P_{w}(2|1)<P_{w}(k+1|k),k\geq 2$, we note that $P_{w}(3|2)$ is also comparatively lower than the other
conditional probabilities. This characteristic can be explained by the accidental repeats of content-bearing words. Considering the comparison
between $P_{w}(3|2)$ and $P_{w}(4|3)$, they are empirically estimated by $\mathit{df}(3) / \mathit{df}(2)$ and $\mathit{df}(4) / \mathit{df}(3)$,
respectively. Without loss of generality, we assume that the accidental repeats are all accounted for in $\mathit{df}(2)$, $\mathit{df}(3)$ and
$\mathit{df}(4)$. From the linguistic intuitions, it is reasonable to say that the accidental components in $\mathit{df}(2)$ are the largest among
them, and the accidental components of $\mathit{df}(3)$ are larger than $\mathit{df}(4)$. Also, the difference in accidental components between
$\mathit{df}(2)$ and $\mathit{df}(3)$ is bigger than that between $\mathit{df}(3)$ and $\mathit{df}(4)$. Accordingly, we see that $P_{w}(3|2)$ is
less than $P_{w}(4|3)$. Moreover, since the occurrence possibility of the accidental repeats phenomenon will become smaller with the increment of
instances of a content-bearing word in a document, the accidental components in the count for $\mathit{df}(k)$ tends to be smaller, and
topic-repetitions will play a dominant role in determining the value of $P_{w}(k+1|k)$. Therefore, the plots in the figure shows that the
conditional probabilities rise initially and then tend to become stable gradually.

\begin{figure}[t]
\begin{center}
\includegraphics{21-4ia4f2.eps}
\end{center}
\hangcaption{The comparison of conditional probability $Pr(k+1|k)$ for $1 \le k\le9$. There are
  $289$ content-bearing words with fat-tail properties used in the experiments.}
\end{figure}


\subsection{Observations on Decay Factors}

After learning about the characteristics of the conditional probabilities, let us now study the
relationship of the decay factors with the conditional probabilities, and how the decay factors are
used in the Katz models to reflect conditional probabilities. Now, going back to the derivations in
Section 5.1, the equations for calculating the constant $C$ can be expressed as follows:
{\allowdisplaybreaks
\begin{align*}
 & C = \frac{\beta }{\beta + 1} = \frac{\mathit{cdf}(2)}{\mathit{cdf}(1)}
   = \frac{\mathit{df}(2) + \sum_{k \ge 3} {\mathit{df}(k)}}{\mathit{df}(1) +\sum_{k \ge 2} {\mathit{df}(k) } } \\
 & \mbox{Then, we decompose the conventional ratio into two seperated ratio factors: } \\
 & {C}'  = \frac{\mathit{df}(2) }{\mathit{df}(1) } \qquad \mbox{and} \qquad
	{C}'' = \frac{\sum_{k \ge 3} {\mathit{df}(k) } }{\sum_{k \ge 2}{\mathit{df}(k)}} \equiv \frac{\mathit{cdf}(3)}{\mathit{cdf}(2)}
\end{align*}
}

$C'$ is the conditional probability corresponding to $P_{w}(2 \vert 1)$. As for $C''$, as used in the Katz three-parameter model, it is the average
value of the conditional probabilities $P_{w}(k+1 \vert k)$ for all $k \geq 2$. The initial definition of this value given by Katz is that this
average represents a ratio of the expected number of repeats to the expected number of instances that could have been followed by the repeats. We
are expecting that there are different properties among the three possible measures $C$, $C'$ and $C''$ for the conditional probabilities of a given
word. Therefore, the question arises: What is the practical comparison among $\mathit{cdf}(2) / \mathit{cdf}(1)$, $\mathit{df}(2) / \mathit{df}(1)$
and $\mathit{cdf}(3) / \mathit{cdf}(2)$?

Our investigations were also conducted on the 289 content-bearing words used in Section 5.1. The mean and other advanced descriptive statistics of
$\mathit{cdf}(2) / \mathit{cdf}(1)$ and $\mathit{cdf}(3) / \mathit{cdf}(2)$ for these words are appended in Figure 2. The comparisons among the
three possible decay factors are: the value of $\mathit{cdf}(3) / \mathit{cdf}(2)$ is larger than $\mathit{df}(2) / \mathit{df}(1)$, and the values
of $\mathit{cdf}(2) / \mathit{cdf}(1)$ and $\mathit{df}(3) / \mathit{df}(2)$ are located in between. Meanwhile, plots of the decay factors and the
conditional probabilities indicate that: the $P_{w}(2|1)$ is almost always overestimated, and the $P_{w}(k+1|k,k\ge 2)$ are on average
underestimated by the decay factor $C$ used in the conventional model. In the three parameter Katz model, although $P_{w}(2|1)$ is treated
individually, $P_{w}(3|2)$ is overestimated and the remaining conditional probabilities are underestimated by the decay factor $C''$. If the decay
factor $\mathit{df}(2) / \mathit{df}(1)$ is selected, $P_{w}(2|1)$ can be exactly estimated, and all the remaining conditional probabilities will be
greatly underestimated. These relationships help to understand the characteristics of the estimation results reported in Section 4. Further
comparisons of the decay factors word-by-word for 100 words are also presented in Figure 3. A consistent phenomenon is observed. Among these
observations, we are particularly interested in the the conditional probability $P_{w}(3|2)$ and its relationship with those decay factors. The Katz
three-parameter model was proposed based on the three-way classification (unrelated, non-topical, topical) of documents and the states of the
content-bearing words with occurrences information are only non-topical or topical and are treated by $P_{w}(2|1)$ and $P_{w}(k+1|k)$, $k\ge 2$
separately. However, there appears to be a transitive state between the non-topical and topical states. Our intuition is that for a word with a
transitive state, its repetitions will be influenced by both non-topical repeats (accidental repeats) and topical repeats. Accordingly, the
accidental repeats of such types of words should be considered. Corresponding to practical instances, the conditional probability, $P_{w}(3|2)$, is
something like a transitive state (where the word is), while $P_{w}(2|1)$ is the non-topical state and $P_{w}(k+1|k)$, $k\ge 2$ is the topical state.

\begin{figure}[t]
\begin{center}
\includegraphics{21-4ia4f3.eps}
\end{center}
\caption{Comparison among $\frac{\mathit{cdf}(3)}{\mathit{cdf}(2)}$, $\frac{\mathit{cdf}(2)}{\mathit{cdf}(1)}$, $\frac{\mathit{df}(2)}{\mathit{df}(1)}$ and $\frac{\mathit{df}(3)}{\mathit{df}(2)}$ of the $100$ content-bearing words.}
\label{the comparison of parameters}
\end{figure}


\subsection{Influences of the Decay Factors on Estimation Results}

In this section, some practical experimental results are reported using the decay factor $C'$ and
$C''$, respectively. Instead of treating the non-topical and topical documents differently for a
given word, with a slight difference from the Katz three-parameter model, the common decay factors
for both non-topical and topical documents are used rather than treating them differently. The
method is as follows:
{\allowdisplaybreaks
\begin{align*}
 & P_{\mathit{var}}(\mathit{tf}(w,X)=k) = (1 - \alpha)\delta_{k,0} + \frac{\alpha \times\gamma }{\beta + 1}
	\left(\frac{\beta}{\beta + 1}\right)^{k -2}\times(1-\delta_{k,0}) \\
 & \mbox{where:} \qquad
	 \delta _{k,0}= \left\{\begin{array}{l}
			1 \quad \mbox{iff} \quad k = 0 \\
			0 \quad \mbox{otherwise} \\
		\end{array}\right. \\
 &\qquad \hat{\alpha}  = \frac{\mathit{df}(1)}{N} \qquad \hat{\gamma}  = \frac{\mathit{df}(2)}{\mathit{df}(1)} \\
 & \mbox{Variation 1 (Adap):  decay factor $\frac{\beta}{\beta+1}$ are estimated by:} \\
 & \qquad \frac{\hat{\beta}}{\hat{\beta}+1}  = \frac{\mathit{df}(2)}{\mathit{df}(1)} \qquad \hat{\beta} = \frac{\mathit{df}(2)}{\mathit{ddf}(1)} \\
 & \mbox{Variation 2 (B2): decay factor corresponding to the G-model: } \\
 & \qquad \frac{\hat{\beta}}{\hat{\beta}+1}  = \frac{\mathit{cf}-\mathit{df}(\ge 1)-\mathit{df}(\ge 2)}{\mathit{cf}-\mathit{df}(\ge 1)}
	=\frac{\mathit{cdf}(3)}{\mathit{cdf}(2)} 
	\qquad \hat{\beta} = \frac{\mathit{cdf}(3)}{\mathit{df}(2)}
\end{align*}
}

\begin{figure}[b]
\begin{center}
\includegraphics{21-4ia4f4.eps}
\end{center}
\hangcaption{Comparison experiments using different decay factor for the words ``bush'', ``congress'', ``investment'' and
  ``japanese''. since the statistic for the ``$k=1$'' is fitted exactly in the experiments using
  $\mathit{df}(2) / \mathit{df}(1)$, the range of the comparison is from 2 to 9. $\mathit{df}(2)/\mathit{df}(1)$
  systematically overestimates when $k$ is small, and underestimates when $\mathit{df}(k)$ is large. In contrast, $\mathit{cdf}(2)/\mathit{cdf}(1)$
  has the reverse problem. A hybrid of the two should perform better than either by itself.}
\end{figure}

\begin{table}[p]
\hangcaption{The comparison experiments between the Katz K mixture variations and the actual value for 5 content-bearing words}
\input{04table02.txt}
\end{table}

Since the experimental data is news data selected, to help us find a somewhat consistent phenomenon from the results, the comparison of K mixture
variations with the actual values for five content-bearing words is performed on the subset corpus with different time interval, respectively, and
results are shown in the Table 2. Meanwhile, the term distribution plots ($k$ from 2 to 9) of four content-bearing words, ``congress'', ``bush'',
``investment'' and ``japanese'' are illustrated as well. The phenomenons what we could find are:
\pagebreak
\begin{enumerate}
\item In the case of $k=1$, the variation using the decay factor $\mathit{df}(2) / \mathit{df}(1)$ could exactly estimate the value $\mathit{ddf}(1)$.
\item In the case of $k=2$, a biased estimation occurred in both the variation ``Adap'' and the variation ``B2''.
In most of the cases, the value of $\mathit{ddf}(2)$ will be overestimated by the variation ``Adap'' while underestimated by the variation ``B2''.
however, the estimation by the variation ``B2'' is comparatively close to the actual value.
\item From $k=4$, the estimation by the variation ``B2'' approaches the actual value closer than that by variation ``Adap''.
\item The decay degree of the variation ``Adap'' is greater than the variation ``B2''.
\end{enumerate}
From this practical comparison, we can conclude that the influences of the decay factor $\mathit{df}(2) / \mathit{df}(1)$ on estimating
$\mathit{ddf}(k)$ are negligible for a large $k$ while worthwhile considering for small $k$. It is reasonable for us to expect that the deficiency
of the variation ``B2'' on estimating the frequencies of a small $k$ can be made up by cooperating with the variation ``Adap''.



\section{Our Approach}

\subsection{Model Description}

The above variations of the Katz K mixture model could be denoted as $P_{\mathit{Adap}}(k;\alpha,\gamma,C')$ and $P_{B2}(k;\alpha,\gamma,C'')$,
respectively. Reviewing the comparison of the decay factors and considering on the estimation results, it appears that the decay factor, as one
deterministic component, is not just a constant, but is in fact a combination of two possible decay factors ($C'$ and $C''$) which is interpolated
by a function depending on the number of instances of a word in a document. With this approach, it is possible to achieve somewhat more precise
frequency estimation of the transitive state. Accordingly, our proposed model is:
{\allowdisplaybreaks
\begin{align*}
 & P_{\mathit{our}}(\mathit{tf}(w,X)=k) = (1 - \alpha)\delta_{k,0} + \frac{\alpha \times \gamma }{\beta_{k} + 1}
	\left(\frac{\beta_{k}}{\beta_{k} + 1}\right)^{k -2}\times(1-\delta_{k,0}) \\
 & \mbox{where:} \qquad
	\delta _{k,0}= \left\{\begin{array}{l}
		1 \quad \mbox{iff}\ k = 0 \\
		0 \quad \mbox{otherwise} \\
	    \end{array} \right. \\
 & \qquad \hat{\alpha}  = \frac{\mathit{df}(1)}{N} \qquad \hat{\gamma}  = \frac{\mathit{df}(2)}{\mathit{df}(1)} \\
 & \mbox{Decay factor $\frac{\beta_{k}}{\beta_{k}+1}$ are estimated by:} \\
 & \qquad \frac{\hat{\beta_{k}}}{\hat{\beta_{k}}+1}
                    = f(k) \times C' + (1-f(k))\times C'' \\
 & \quad \quad = f(k) \times \frac{\mathit{df}(2)}{\mathit{df}(1)} + (1-f(k))\times \frac{\mathit{cdf}(3)}{\mathit{cdf}(2)} \\
 & \mbox{Since factor $C''$ will play a dominant role with the increment of $k$, we choose:} \\
 & \qquad f(k) = a \times k^{-b}
\end{align*}
}

In our proposed model, in addition to using the three-way classification of documents for counting rate of words, the accidental repeats of
non-topically-used content words are also taken into consideration. Experimental results shown above indicate that there is a transitive state
between the non-topical and topical state and word repetitions will be influenced by both non-topical repeats and topical repeats and there is
possibility to get better estimations using interpolation of two possible decay factors. Therefore the function $f(k)$ is introduced to make the
model more linguistic favoritism. It might be possible to form the theoretical models for interpolation factor $f(k)$, and it will be our future
work.

In the function $f(k)$, two parameters, $a$ and $b$ need to be ascertained. According to the experimental results of K mixture variations, we know
that in the case of $k=1$, the decay factor will be decided by $\mathit{df}(2)/ \mathit{df}(1)$ itself. To make an exact estimation for
$\mathit{ddf}(1)$, parameter $a$ should be fixed at 1. Accordingly, only parameter $b$ is required to be processed.

On the basis of admitting the rationality of the K mixture on modeling term distribution, the introduced parameter $b$ is assumed to be chosen so
that the number of terms whose frequency distributions fit for the K mixture model should become maximum. For any individual terms, the $\chi^{2}$
statistics are chosen for goodness of fit. A $\chi^{2}$ statistic, $s$, for a given word has to be computed for each tested frequency distribution:
\begin{align*}
 & s=\sum\nolimits_{i = 1}^k {\frac{(\mathit{ddf}(i) - \hat{\mathit{ddf}(i)})^2}{\hat{\mathit{ddf}(i)}}}  \\
 &  \mbox{where:} \\
 & \qquad 
  \left\{\begin{array}{l}
	\mbox{$\mathit{ddf}(i)$: actual frequencies} \\
	\mbox{$\hat{\mathit{ddf}(i)}$: estimated frequencies; $\hat{\mathit{ddf}(i)}=N\times P_{our}(i;\alpha,\gamma,\beta_{i}(b))$ } \\
	\end{array}\right.
\end{align*}
For test reliability, the cells for which the estimated frequencies $\hat{\mathit{ddf}(i)}$ are smaller than five have to be combined with adjacent
cells so that each combined cell's estimate is no less than five. In our experiments, because the K mixture is a monotonous function diminishing
successively, we can safely combine the cell having the estimated frequency less than five with the remaining cells. The estimation is computed by
the identities $\sum_{r\ge k}C^{r}=C^{k}/(1-C)$. Meanwhile, the degrees of freedom ($d.f.$) are adjusted correspondingly. Since the Katz
three-parameter model and our proposed K mixture both fit exactly for $\mathit{ddf}(0)$ and $\mathit{ddf}(1)$, the truncated frequency distribution
(frequency of the cells at the interval from 2 to 8) is tested. The critical values of the $\chi^{2}$-test are selected according to a suitable
$d.f.$ at the significant level 0.005. If the value of $s$ for a given term is less than the critical value, it will be counted as a term whose
frequency distribution could be generated by the hypothesized distribution.

\begin{table}[b]
\hangcaption{Experiments for 245 content-bearing words using different parameter $b$ based on the three subset corpus (94--95,96--97,01--02) with the range of document length from 300 to 700}
\input{04table03.txt}
\vspace{4pt}\small
The number of documents of each subset corpus for 94--95, 96--97 and 01--02 are 78,673, 96,494 and 45,899, 
respectively. The value in the table is the percentage of terms which pass the $\chi^{2}$-test.
\par
\end{table}

\begin{figure}[p]
\begin{center}
\includegraphics{21-4ia4f5.eps}
\end{center}
\hangcaption{AWK code fragment for parameter tuning; Maximum value of parameter $b$, denoted as $powv$ in the program, is set to 5, the initial value and increasing step are all 0.1.}
\label{code fragment}
\end{figure}

This paper proposes two ways for constructing the models. One is to set up the parameter $b$ as a constant value which lies in the range from 2.0 to
3.0. Since the experimental results in Table 3 show that our approach using a different parameter $b$ all performed better than that of the Katz
three-parameter model, an acceptable solution would be that we select a possible value for $b$ in the range from 2.0 to 3.0. From this point of
view, the proposed method can be regarded as a kind of three-parameter model. Secondly, to gain much greater improvements, parameter $b$ can be
adjusted by a sampling of words. In this paper, the 100 content-bearing words are used as training sets for tuning the parameter $b$. The parameter
tuning process is shown in a fragment of the AWK code. After the system running, the most suitable range of parameter $b$ is from 2.5 to 2.8. In the
model comparison experiments, the parameter $b$ is fixed at 2.6.


\subsection{Model Comparison}

In this section, the comparison experiments among the conventional Katz K mixture, denoted as K'1,
Katz three-parameter model denoted as K'2, and our proposed K mixture are conducted.

\begin{table}[b]
\hangcaption{Experiments for 127 content-bearing words based on the subset corpus with different range of document length}
\input{04table04.txt}
\small
The value in the table is the percentage of terms which pass the $\chi^{2}$-test. 
\par
\end{table}

As we mentioned in Section 5.1, the accidental repeats phenomenon will be dramatic in two cases, one where the content word is frequently used and
the other where the word occurs in a long document. To investigate the effectiveness of our proposed approach in these two cases, two kinds of
experiments were performed. The first experiments for 127 content-bearing words based on the whole and a subset of the corpus with different ranges
of document length are conducted, and the results presented in Table 4. The value presented in the column $\chi^{2}$-test is the number of terms
passing the statistical test. We note that the range of document length in the corpus does have distinct influences on model effectiveness, and the
longer the document is, the worse the estimation results will be. Compared with the Katz three-parameter model, our approach will show dramatic
improvements in estimation $\mathit{ddf}(2)$ and slight improvements in the estimation of the tail frequencies; The degree of improvements is clear
on a test corpus with a wider range of document length. In the second experiment, 583 content-bearing words, including nouns, biographical names and
geographical names, with fat-tail properties are involved, and the corpus with a range of documents from 300 to 500 is used. They are grouped by the
$\mathit{df}$ value and comparisons are performed for each group, respectively. Results are presented in Table 5. In the case of the bin with a
small $\mathit{df}$ value, our proposed approach and the Katz three-parameter model have the same accuracy but with the increment of the
$\mathit{df}$ value of the bin, our proposed approach shows advantages, especially for the estimations of $\mathit{ddf}(2)$. From the above
experimental results, it is evident that our proposal can help the K mixture model provide more precise estimations, especially in the case of small
k. Evidence of the effectiveness of our approach can also be found in Figure 6, where, in addition to the statistical mean, comparisons using the
advanced statistics including the median and the 25th and 75th percentiles for 127 content-bearing words are all presented.

\begin{table}[t]
\hangcaption{583 content-bearing words are grouped according to their $df$ value and comparison experiments are performed for each group}
\input{04table05.txt}
\vspace{4pt}\small
The value in the table is the percentage of terms which pass the $\chi^{2}$-test.
\par
\end{table}

\begin{figure}[t]
\begin{center}
\includegraphics{21-4ia4f6.eps}
\end{center}
\hangcaption{Model comparison using descriptive statistics ``Mean'', ``P25'', ``P75'' and ``Median'' of 127 sample words.}
\label{model comparison using descriptive statistics}
\end{figure}


\section{Conclusion}

In this paper, an in-depth exploration on the well-known Katz K mixture model is deeply conducted. The interesting phenomena found in the
experiments about the characteristics of conditional probabilities of repetition and the exploration of the decay factors are reported in detail.
Based on the findings in the experiments, it is possible to use both ends of the distribution to fit models. That is, not only can document
frequencies be used when instances of a word are few, but tail probabilities can also be used. Both document frequencies for small instances of a
word and the tail probabilities for large instances are rather easy to estimate empirically. Therefore, we propose an effective approach for
improving the K mixture model, where the decay factor is the combination of two possible decay factors interpolated by a function depending on the
number of instances of the word in the document. Results show that the proposed model can generate a statistically-significant better estimation of
frequencies, especially the frequency estimation for a word with two mentions in a document. In addition, it is shown that the advantages of our
approach will become more evident in two cases: modeling the term distribution for the content-bearing word used frequently and modeling the term
distribution for the corpus with a wide range of document length.

\vspace{-0.5\Cvs}
\acknowledgment

\vspace{-0.5\Cvs}
The authors would like to acknowledge Kenneth W. Church for comments and suggestions and to thank Okabe Masayuki and Nakase Kenta for their help on
preparing the draft. This work was supported by the 21st Century COE program ``Intelligent Human Sensing'', from the Ministry of Education, Culture,
Sports, Science and Technology.

\bibliographystyle{jnlpbbl_1.5}
\begin{thebibliography}{}

\bibitem[\protect\BCAY{Blei, Ng, \BBA\ Jordan}{Blei et~al.}{2003}]{Daivd}
Blei, D.~M., Ng, A.~Y., \BBA\ Jordan, M.~I. \BBOP 2003\BBCP.
\newblock \BBOQ Latent Dirichlet Allocation.\BBCQ\
\newblock {\Bem Journal of Machine Learning Research}, {\Bbf 3}, \mbox{\BPGS\
  993--1022}.

\bibitem[\protect\BCAY{Bookstein \BBA\ Swanson}{Bookstein \BBA\
  Swanson}{1974}]{Bookstein74}
Bookstein, A.\BBACOMMA\ \BBA\ Swanson, D.~R. \BBOP 1974\BBCP.
\newblock \BBOQ Probabilistic Models for Automatic Indexing.\BBCQ\
\newblock {\Bem Journal of the American Society for Information Science}, {\Bbf
  25}  (5), \mbox{\BPGS\ 312--318}.

\bibitem[\protect\BCAY{Church}{Church}{2000}]{Church2000}
Church, K.~W. \BBOP 2000\BBCP.
\newblock \BBOQ Empirical Estimates of Adaptation: The Chance of Two Noriegas
  is Closer to $p/2$ than $p^{2}$.\BBCQ\
\newblock In {\Bem Proceedings of the 18th Conference on Computational
  Linguistics (COLING '00)}, \mbox{\BPGS\ 180--186}.

\bibitem[\protect\BCAY{Church \BBA\ Gale}{Church \BBA\ Gale}{1995}]{Church95}
Church, K.~W.\BBACOMMA\ \BBA\ Gale, W. \BBOP 1995\BBCP.
\newblock \BBOQ Poisson Mixtures.\BBCQ\
\newblock {\Bem Natural Language Engineering}, {\Bbf 1}  (2), \mbox{\BPGS\
  163--190}.

\bibitem[\protect\BCAY{Hoffman}{Hoffman}{1999}]{Hoffman}
Hoffman, T. \BBOP 1999\BBCP.
\newblock \BBOQ Probabilistic Latent Semantic Indexing.\BBCQ\
\newblock In {\Bem Proceedings of the 22th Annual International SIGIR
  Conference on Research and Development in Information Retrieval},
  \mbox{\BPGS\ 50--57}.

\bibitem[\protect\BCAY{Jelinek}{Jelinek}{1999}]{Jelinek99}
Jelinek, F. \BBOP 1999\BBCP.
\newblock {\Bem Statistical Methods for Speech Recognition}.
\newblock MIT Press, Cambridge \mbox{Massachusetts}.

\bibitem[\protect\BCAY{Katz}{Katz}{1996}]{Katz96}
Katz, S.~M. \BBOP 1996\BBCP.
\newblock \BBOQ Distribution of Content Words and Phrases in Text and Language
  Modeling.\BBCQ\
\newblock {\Bem Natural Language Engineering}, {\Bbf 2}  (1), \mbox{\BPGS\
  15--59}.

\bibitem[\protect\BCAY{LDC}{LDC}{}]{LDC}
LDC\ (2004). \texttt{http://www.ldc.upenn.edu/}.

\bibitem[\protect\BCAY{Manber \BBA\ Myers}{Manber \BBA\ Myers}{1990}]{Manber90}
Manber, U.\BBACOMMA\ \BBA\ Myers, G. \BBOP 1990\BBCP.
\newblock \BBOQ Suffix Arrays: A New Method for On-line String Searches.\BBCQ\
\newblock In {\Bem Proceedings of the 1st Annual ACM-SIAM Symposium on Discrete
  Algorithms}, \mbox{\BPGS\ 319--327}.

\bibitem[\protect\BCAY{Manning \BBA\ Sch{\"u}tze}{Manning \BBA\
  Sch{\"u}tze}{1999}]{Manning99}
Manning, C.~D.\BBACOMMA\ \BBA\ Sch{\"u}tze, H. \BBOP 1999\BBCP.
\newblock {\Bem Foundation of Statistical Natural Language Processing}.
\newblock The MIT Press, Cambridge Massachusetts, London, England.

\bibitem[\protect\BCAY{Robertson, van Rijsbergen, \BBA\ Porter}{Robertson
  et~al.}{1980}]{Robertson80}
Robertson, S.~E., van Rijsbergen, C.~J., \BBA\ Porter, M.~F. \BBOP 1980\BBCP.
\newblock \BBOQ Probabilistic Models of Indexing and Searching.\BBCQ\
\newblock In {\Bem Proceedings of the 3rd Annual ACM Conference on Research and
  Development in Information Retrieval (SIGIR '80)}, \mbox{\BPGS\ 35--56}.

\bibitem[\protect\BCAY{Robertson \BBA\ Walker}{Robertson \BBA\
  Walker}{1994}]{Robertson94}
Robertson, S.~E.\BBACOMMA\ \BBA\ Walker, S. \BBOP 1994\BBCP.
\newblock \BBOQ Some Simple Effective Approximations to the 2-poisson Model for
  Probabilitic Weighted Retrieval.\BBCQ\
\newblock In {\Bem Proceedings of the 17th Annual International ACM SIGIR
  Conference on Research and Development in Information Retrieval},
  \mbox{\BPGS\ 232--241}.

\bibitem[\protect\BCAY{Yamamoto, Sadamitsu, \BBA\ Mishina}{Yamamoto
  et~al.}{2003}]{Mikio}
Yamamoto, M., Sadamitsu, K., \BBA\ Mishina, T. \BBOP 2003\BBCP.
\newblock \BBOQ Context Modeling using Dirichlet Mixtures and its Applications
  to Language Models.\BBCQ\
\newblock {\Bem IPSJ SIG Technical Report}, {\Bbf 48}  (5), \mbox{\BPGS\
  29--34}.

\end{thebibliography}

\begin{biography}

\bioauthor[:]{Yinghui Xu}{Received the B.E. degree from Xi'an Architecture and Technology University, Xi'an, China in 1996 and the M.E. degree from
University of Science And Technology Beijing, Beijing, China in 2000. He is now a postgraduate student for doctor degree in the department of
Information and Computer Sciences, Toyohashi University of Technology. His research interests include Information Retrieval, Web Mining, Natural
Language Processing. }

\bioauthor[:]{Kyoji Umemura}{Received B.E., M.E. and D.E. degrees from the University of Tokyo, Tokyo, Japan in 1981, 1983 and 1991, respectively.
Currently he is a Professor in the department of Information and Computer Sciences, Toyohashi University of Technology. His research interests
include Information Retrieval, Natural Language Processing, Data Mining, Lisp, Symbolic Computation, Compilers and Operating System. }

\end{biography}

\biodate


\end{document}
