\documentstyle[epsf,theapa]{jnlp_e_b5_old}

\input{prepictex}
\input{pictex}
\input{postpictex}

\setcounter{page}{75}
\setcounter{巻数}{2}
\setcounter{号数}{2}
\受付{September}{30}{1994}
\再受付{December}{8}{1994}
\採録{January}{10}{1995}

\setcounter{secnumdepth}{2}

\title{}
\author{}
\jkeywords{}

\etitle{A New Direction for Sublanguage N.L.P.}

\eauthor{Satoshi Sekine\affiref{NewYorkUniv}}

\headauthor{Satoshi Sekine}
\headtitle{A New Direction for Sublanguage N.L.P.}

\affilabel{NewYorkUniv}
          {New York University, Computer Science Departent}
          {New York University, Computer Science Departent}

\eabstract{
 There have been a number of theoretical studies devoted to the notion of
sublanguage.
Furthermore, there are some successful natural language
processing systems which have explicitly or implicitly utilized
sublanguage restrictions. 
However, two big problems are still unsolved to utilize the sublanguage
notion:
1) automatic definition and dynamic identification of a text to sublanguage, and
2) automatic linguistic knowledge acquisition for sublanguage.
There are now new opportunities to address these problems
owing to the appearance of large machine-readable corpora.
Although there have been several experiments to try to solve
the second problem listed above,
the first problem has not received so much attention.
In the previous sublanguage N.L.P. systems, 
the domain the system is dealing with was defined by a human.
This is actually one method to define the sublanguage of a text,
and, in a sense, it seems to work well.
However, it is not always possible and sometimes it may be wrong.
In order to maximize the benefit of the sublanguage notion,
we need automatic definition and dynamic sublanguage identification.
We will report preliminary experiments on sublanguage definition and
identification based on lexical appearance.
The results of the experiments show that
the methods proposed can be useful in processing a new text.
In particular, the fact that the first two sentences can reliably identify
a text's sublanguage encourages us in further investigation of this line of research.
In conclusion, it appears that the inductive definition of sublanguage
and sublanguage identification would be beneficial for natural language processing.
}

\ekeywords{Sublanguage N.L.P., Corpus-based N.L.P.}

\begin{document}

\maketitle

\section{Introduction}

 There have been a number of theoretical studies devoted to the notion of
sublanguage.
Some of them claim that the notion is important in processing
natural language text, owning to lexical, syntactic or semantic restrictions, etc.
A number of these studies have observed actual texts to verify the claim
\cite{Kittredge:82} 
\cite{Grishman:86}
\cite{Slocum:86}
\cite{Biber:93}.

  Furthermore, there are some successful natural language
processing systems which have explicitly or implicitly addressed
the sublanguage restrictions. 
For example, TAUM-METEO \cite{Isabelle:84} is a machine translation system
in which only sentences in the weather forecast domain are dealt with, 
and it works remarkably well.

However, two major problems are still unsolved to utilize the sublanguage notion.

\begin{enumerate}

\item Automatic definition and dynamic identification of a text to sublanguage
\item Automatic linguistic knowledge acquisition for sublanguage.

\end{enumerate}

Owing to the appearance of large machine-readable corpora,
there are now new opportunities to address these issues.
The explosion of large corpora has lead to a flowering of research on
linguistic knowledge acquisition from corpora 
\cite{Special_Issue:93a} \cite{Special_Issue:93b}.
Among them, several studies have mentioned the
importance of the sublanguage notion 
\cite{Grishman:92} \cite{Sekine:92}.
Although these are still small experiments in terms of accuracy and
coverage,
they addressed the second problem listed above, 
and could lead to a breakthrough
for future N.L.P. systems by reducing the costly
and errorsome task of linguistic knowledge encoding by human linguists.

The first problem has not received so much attention.
In the previous sublanguage N.L.P. systems, 
the domain the system is dealing with was defined manually.
For example, the sublanguages of ``weather forecasts'',
``medical reports'' or ``computer manuals''
are artificially or intuitively  defined by a human.
This is actually one method to define the sublanguage of the text,
and, in a sense, it seems to work well.
However, it is not always possible and sometimes it may be wrong.
For example, in a large vocabulary speech recognition system,
which may has to handle a variety of domains,
it is impossible to tune to a sublanguage in advance.
Also, if we take the ``computer manual domain'' as a sublanguage,
the domain contains a mixture of linguistic phenomena.
Because ``computer manual domain'' ranges from a ``word processor manual for novices''
to a ``UNIX reference manual'',
and their linguistic usages are different.
In short, a new step is long-awaited to expand the sublanguage research.


\section{Two Definitions of Sublanguage}

The notion of sublanguage has been well discussed in the last decades.
We can find two kinds of definitions, although the difference
has not received serious attention up to now.
One definition can be inferable from Zellig Harris \cite{Harris:68}:

\begin{quotation}
{\it Certain proper subsets of the sentences of a language may be closed
under some or all of the operations defined in the language,
and thus constitute a sublanguage of it.}
\end{quotation}

From this definition, we can infer that a sublanguage can be defined empirically
by observing language behavior.
The other type of definition is exemplified by the following sentence in
Bross, Shapio and Anderson \cite{Bross:72}:

\begin{quotation}
{\it Informally, we can define a sublanguage as the language used by a particular
community of speakers, say, those concerned with a particular
subject matter or those engaged in a specialized occupation.}
\end{quotation}

In this paper, the former definition will be called 
an {\it `inductive definition'}, since a sublanguage can be defined by
observation of data.
The latter definition, on the other hand, will be called
a {\it `deductive definition'}, since it is based on the principle that
a particular subject matter would define a sublanguage.

Many practical projects have been using the deductive definition
intentionally or unintentionally.
For example, the TAUM project took two subject domains for their MT applications;
one is weather forecasts and the other is aircraft maintenance manuals.
This is a deductive definition of sublanguage, 
because they took a particular subject as their target.
However, the deductive definition is not always possible and
sometimes it may be wrong, as we discussed before.


\section{Benefit from Large Corpora}

Recently, a number of large corpora have become available in machine-readable form.
The appearance of large corpora would be of beneficial to sublanguage study.
In particular, it can be useful for the inductive definition of sublanguage.
Although Harris's notion was mainly concerned with the operations on sentences,
there are several other factors which are related to the sublanguage notion.
For example lexical, syntactic and semantic restrictions,
a high frequency of certain constructions and use of special symbols
(these factors are borrowed from Lehrberger \cite{Lehrberger:82}).
These factors can be useful for the inductive definition of sublanguage, as well as 
themselves being characteristics of sublanguage.
In other words, we may be able to find sublanguages
by observing texts in terms of similarities of these characteristics.
The other factors in Leherberger's list of sublanguage characteristics -
limited subject matter,
deviant rules of grammar and text structure - are harder to utilize
for the inductive definition of sublanguage.

The strategy for inductive definition of a sublanguage using corpora is
to analyze the corpora in term of lexical, syntactic or semantic characteristics, 
and then make clusters in which the values of the factors are similar.
However, in practice, it is still difficult to analyze a sentence correctly 
in term of syntax and semantics,
so the preliminary experiment in this paper employs only the lexical
factor to build up a sublanguage as an initial attempt.


\section{Sublanguage Definition}

Our experiments can be divided into two parts: the definition of sublanguage
and the identification of a new text to the sublanguage. 
We will describe the experiment of the sublanguage definition, first, in this section, 
and in the next section, we will discuss the experiments of
the sublanguage identification.

We use a newspaper (7.5MB of San Jose Mercury, 2147 articles) in our experiments.
Each article in the newspaper is regarded as a unit of data, and
a sublanguage will be formed by gathering similar units in term of word appearance.
This is almost identical to the text clustering technique,
which has been well studied in the field of information retrieval 
\cite{Willett:88}.
However the aim of their study is slightly different from ours.
They try to make clusters which are useful for human information retrieval purposes,
so the linguistic features of the clusters are not so important.
In contrast, our purpose is to find a sublanguage which
is useful for N.L.P. systems, so we are very interested 
in the linguistic features of clusters.

One of the problems in making clusters is how to define the number and size of clusters.
The number of cluster can range from 1, with all the articles in the cluster, 
to the number of articles, each cluster contains just one article.
The problem here is that we have no objective means to decide the number and the size.
This problem is very crucial for automatic sublanguage definition,
because otherwise we need manual intervention or artificial thresholds.
In order to explore this problem, we introduce the following statistics. \\

\begin{flushleft}
\underline{Perplexity and Expected Perplexity} \\
\end{flushleft}

Perplexity, more precisely `uni-gram perplexity' used in this paper, is 
a notion from Information Theory.
The formal definition of it is defined by using Uni-gram Entropy $H$:

\begin{equation}
	H = - \sum_{w} \; p(w) log_{2}(p(w))
\end{equation}
\begin{equation}
	PP = 2^{H}
\end{equation}

Here $p(w)$ is a probability of a token $w$ in the source.
This probability may be estimated from a text 
by dividing the number of the token by the number of token in the text.
From this estimated probability, we can get estimated entropy and perplexity.
Roughly speaking, perplexity indicates the amount of information in the source.
On condition that the sizes of two texts are the same in terms of number of tokens,
then we can roughly say that the text with larger perplexity has 
the greater variety of the tokens.

We can also calculate perplexity for a set of texts,
treating the set as a single text.
If two texts have large number of overlapping tokens, 
which means the two texts are similar in terms of word apparency,
the perplexity for the combined text will be smaller than
that of a text which has tokens chosen at random from the entire corpus.
In short, if the perplexity for a text set is small 
in comparison to perplexity for a random text,
we may say that the set has sublanguage tendency.

In order to observe perplexities of texts,
we made clusters based on similarities of articles.
Clusters are grown from one initial article by adding similar articles
in the order of their similarity to the initial article.
The similarity measurement is borrowed from information retrieval research
\cite{Frankes:92}.
The idea of the formula is based on Inversed document frequency (IDF)
as be shown in the following:
\begin{equation}
  S(A,B) = \sum_{w \; in \; A \;and \;B} \; \frac{1}{log(F_{w}) * N_{A} * N_{B}}
\end{equation}
Here, $S(A,B)$ is the similarity between article A and B,
$F_{w}$ is frequency of word $w$ throughout the corpus, and
$N_{X}$ is the number of tokens in the article $X$.
We set the cut-of-number 200
(tokens which occur more than 200 times in the corpora are not taken into account),
the minimum occurrence 2 (only the tokens which occur 2 or more times
in the article is considered) and the minimum co-occurrence 2
(relationship would be established is the two article have 2 or more
tokens overlapped).
Those parameters and numbers are experimentally decided.
We grow a cluster in this way around each article in the corpus.

In order to make objective measurement,
we want to compare these estimated perplexity value with the
perplexity of random text.
Because the estimated perplexity depends on the size of the sample text, 
we need to compare the cluster to a sample text of the same size text
obtained by selecting tokens at random from the entire corpus.
The expected perplexities used in the experiment is the average of
three of these trials.

Figure \ref{Fig_perplexity} shows some examples of the relationship
between number of tokens in clusters and ratio of its perplexity to
that of random estimation.
The points in the graph starting  from the left side indicate data for the first article,
the first article and the closest article combined, and so on.

\begin{figure}
\beginpicture
\setcoordinatesystem units <0.006pt,200pt>
\setplotarea x from 0 to 50000, y from 0 to 1.0

\axis bottom label {Number of Tokens} 
      ticks short quantity 6 numbered at 0 20000 40000 /  / 
\axis left                            
      ticks short quantity 6 numbered from 0 to 1.0 by 0.2 / 

\multiput {\small $\circ$}   at 
676 0.60319
1061 0.515636
1516 0.470162
2049 0.44925
2092 0.461901
2522 0.497011
2655 0.493394
3174 0.465605
3291 0.465766
3377 0.465469
4193 0.461492
4343 0.460855
4625 0.468466
4757 0.46783
4959 0.483791
5617 0.454087
5901 0.459108
6201 0.446392
6565 0.448444
6696 0.450666
7158 0.448892
7584 0.444614
8072 0.441307
8174 0.443427
8877 0.432026
9291 0.439095
9813 0.441111
10040 0.444172
10308 0.447449
11264 0.449687
12180 0.452687
12702 0.462136
12819 0.463911
13486 0.465379
13847 0.467312
14193 0.467086
14914 0.467823
15088 0.469692
15524 0.471523
15857 0.47049
16394 0.481133
17095 0.482216
17770 0.489453
18111 0.498178
18376 0.499515
18805 0.503994
19037 0.504701
19433 0.508655
19842 0.515144
20204 0.515899
20463 0.515717
20632 0.520197
21143 0.516339
21395 0.517854
22081 0.516602
22302 0.520047
23032 0.525991
23288 0.526213
23609 0.528285
23779 0.529632
24395 0.533466
25255 0.53617
25623 0.535692
26340 0.533546
26917 0.535312
27100 0.537906
27424 0.538506
27783 0.542803
28236 0.55145
28482 0.553901
28907 0.556832
29154 0.55588
29901 0.57719
30894 0.580425
31521 0.578655
31765 0.582595
31997 0.582115
32350 0.585011
32744 0.584508
33151 0.58393
33589 0.584359
34345 0.597545
34591 0.601962
35388 0.604511
35579 0.60691
35959 0.609248
36755 0.606828
37162 0.609287
37498 0.611413
38305 0.617037
39091 0.622267
39626 0.623503
40165 0.623314
40399 0.625269
41148 0.629563
41704 0.629254
42332 0.634521
42681 0.637629
43295 0.635482
43585 0.63564
44098 0.637821
44956 0.638932
45451 0.638477
46243 0.634617
46612 0.638891
47036 0.640724
47370 0.641575
47753 0.641925
48406 0.643159
48789 0.64424
49411 0.649144 /

\setlinear \plot
676 0.60319
1061 0.515636
1516 0.470162
2049 0.44925
2092 0.461901
2522 0.497011
2655 0.493394
3174 0.465605
3291 0.465766
3377 0.465469
4193 0.461492
4343 0.460855
4625 0.468466
4757 0.46783
4959 0.483791
5617 0.454087
5901 0.459108
6201 0.446392
6565 0.448444
6696 0.450666
7158 0.448892
7584 0.444614
8072 0.441307
8174 0.443427
8877 0.432026
9291 0.439095
9813 0.441111
10040 0.444172
10308 0.447449
11264 0.449687
12180 0.452687
12702 0.462136
12819 0.463911
13486 0.465379
13847 0.467312
14193 0.467086
14914 0.467823
15088 0.469692
15524 0.471523
15857 0.47049
16394 0.481133
17095 0.482216
17770 0.489453
18111 0.498178
18376 0.499515
18805 0.503994
19037 0.504701
19433 0.508655
19842 0.515144
20204 0.515899
20463 0.515717
20632 0.520197
21143 0.516339
21395 0.517854
22081 0.516602
22302 0.520047
23032 0.525991
23288 0.526213
23609 0.528285
23779 0.529632
24395 0.533466
25255 0.53617
25623 0.535692
26340 0.533546
26917 0.535312
27100 0.537906
27424 0.538506
27783 0.542803
28236 0.55145
28482 0.553901
28907 0.556832
29154 0.55588
29901 0.57719
30894 0.580425
31521 0.578655
31765 0.582595
31997 0.582115
32350 0.585011
32744 0.584508
33151 0.58393
33589 0.584359
34345 0.597545
34591 0.601962
35388 0.604511
35579 0.60691
35959 0.609248
36755 0.606828
37162 0.609287
37498 0.611413
38305 0.617037
39091 0.622267
39626 0.623503
40165 0.623314
40399 0.625269
41148 0.629563
41704 0.629254
42332 0.634521
42681 0.637629
43295 0.635482
43585 0.63564
44098 0.637821
44956 0.638932
45451 0.638477
46243 0.634617
46612 0.638891
47036 0.640724
47370 0.641575
47753 0.641925
48406 0.643159
48789 0.64424
49411 0.649144 /

\multiput {\small $\bullet$}    at
814 0.401626
1061 0.327864
1583 0.284104
2249 0.254501
2880 0.229073
3064 0.228226
3265 0.224909
4049 0.205439
4442 0.20036
4646 0.214434
4763 0.226049
4873 0.237223
5242 0.267621
5575 0.287914
5791 0.300548
6132 0.310053
6389 0.314091
6472 0.317365
6764 0.333691
7113 0.355723
7490 0.366559
7778 0.356889
9077 0.413586
9710 0.429317
10105 0.441562
10525 0.459953
11348 0.497038
11713 0.512154
12889 0.543191
13526 0.541246
14480 0.549713
15229 0.548922
15649 0.555226
15994 0.569025
16298 0.572319
16615 0.579442
16925 0.587654
17388 0.593498
19015 0.62207
19572 0.622507
19796 0.624785
20552 0.619723
20849 0.629257
21086 0.633379
21331 0.646451
22192 0.636136
22456 0.642249
22687 0.649234
23029 0.653762
23776 0.659931
24122 0.664673
25012 0.684186
25281 0.689194
26363 0.718318
27357 0.70656
27985 0.722315
28247 0.727934
28943 0.733497
29418 0.740427
29707 0.744448
29983 0.749898
30656 0.757916
30991 0.760979
31318 0.761463
31608 0.763962
31957 0.767426
32471 0.772811
33066 0.779805
33601 0.794685
33874 0.799352
34243 0.803049
34894 0.807272
35531 0.812564
35830 0.817107
36199 0.8204
36697 0.826561
37176 0.832233
37700 0.828978
39086 0.835704
39525 0.838903
40549 0.839919
40867 0.841844
41176 0.843373
41563 0.852689
42250 0.855715
43104 0.858196
43560 0.858652
43957 0.860455
44427 0.860806
44775 0.864185
45408 0.865651
45831 0.867661
46222 0.871616
46572 0.876397
46982 0.877272
47470 0.877959
48137 0.881709
48670 0.88382
49396 0.887565 /

\setlinear \plot
814 0.401626
1061 0.327864
1583 0.284104
2249 0.254501
2880 0.229073
3064 0.228226
3265 0.224909
4049 0.205439
4442 0.20036
4646 0.214434
4763 0.226049
4873 0.237223
5242 0.267621
5575 0.287914
5791 0.300548
6132 0.310053
6389 0.314091
6472 0.317365
6764 0.333691
7113 0.355723
7490 0.366559
7778 0.356889
9077 0.413586
9710 0.429317
10105 0.441562
10525 0.459953
11348 0.497038
11713 0.512154
12889 0.543191
13526 0.541246
14480 0.549713
15229 0.548922
15649 0.555226
15994 0.569025
16298 0.572319
16615 0.579442
16925 0.587654
17388 0.593498
19015 0.62207
19572 0.622507
19796 0.624785
20552 0.619723
20849 0.629257
21086 0.633379
21331 0.646451
22192 0.636136
22456 0.642249
22687 0.649234
23029 0.653762
23776 0.659931
24122 0.664673
25012 0.684186
25281 0.689194
26363 0.718318
27357 0.70656
27985 0.722315
28247 0.727934
28943 0.733497
29418 0.740427
29707 0.744448
29983 0.749898
30656 0.757916
30991 0.760979
31318 0.761463
31608 0.763962
31957 0.767426
32471 0.772811
33066 0.779805
33601 0.794685
33874 0.799352
34243 0.803049
34894 0.807272
35531 0.812564
35830 0.817107
36199 0.8204
36697 0.826561
37176 0.832233
37700 0.828978
39086 0.835704
39525 0.838903
40549 0.839919
40867 0.841844
41176 0.843373
41563 0.852689
42250 0.855715
43104 0.858196
43560 0.858652
43957 0.860455
44427 0.860806
44775 0.864185
45408 0.865651
45831 0.867661
46222 0.871616
46572 0.876397
46982 0.877272
47470 0.877959
48137 0.881709
48670 0.88382
49396 0.887565 /

\multiput {\small $\ast$}    at
301 0.825856
371 0.759172
732 0.752913
1031 0.678214
1577 0.64972
1904 0.61681
2225 0.614006
2747 0.60534
3149 0.552511
3395 0.557177
3871 0.579517
4251 0.567893
4431 0.566565
4824 0.59209
4955 0.606643
5356 0.606787
5526 0.611764
5802 0.6184
6551 0.626781
8559 0.612896
8941 0.616118
9329 0.634535
9961 0.627795
10196 0.62746
10691 0.62545
11530 0.626365
11991 0.63033
12797 0.625126
13475 0.624992
13778 0.639606
14114 0.643631
14512 0.637239
15136 0.624849
15437 0.631617
15783 0.634868
16187 0.637009
16642 0.632266
17115 0.631324
17842 0.631567
18597 0.631141
19014 0.641654
19636 0.636636
20182 0.632694
20532 0.642491
21089 0.634698
22000 0.626511
22749 0.637693
23392 0.637404
23931 0.64322
24420 0.653705
25236 0.668296
25775 0.676346
26338 0.684651
27109 0.681733
27781 0.681222
28573 0.687707
29799 0.695534
30651 0.710261
31388 0.710149
32345 0.702944
33318 0.711894
34533 0.719775
35582 0.731294
36992 0.75203
41175 0.787166 /

\setlinear \plot
301 0.825856
371 0.759172
732 0.752913
1031 0.678214
1577 0.64972
1904 0.61681
2225 0.614006
2747 0.60534
3149 0.552511
3395 0.557177
3871 0.579517
4251 0.567893
4431 0.566565
4824 0.59209
4955 0.606643
5356 0.606787
5526 0.611764
5802 0.6184
6551 0.626781
8559 0.612896
8941 0.616118
9329 0.634535
9961 0.627795
10196 0.62746
10691 0.62545
11530 0.626365
11991 0.63033
12797 0.625126
13475 0.624992
13778 0.639606
14114 0.643631
14512 0.637239
15136 0.624849
15437 0.631617
15783 0.634868
16187 0.637009
16642 0.632266
17115 0.631324
17842 0.631567
18597 0.631141
19014 0.641654
19636 0.636636
20182 0.632694
20532 0.642491
21089 0.634698
22000 0.626511
22749 0.637693
23392 0.637404
23931 0.64322
24420 0.653705
25236 0.668296
25775 0.676346
26338 0.684651
27109 0.681733
27781 0.681222
28573 0.687707
29799 0.695534
30651 0.710261
31388 0.710149
32345 0.702944
33318 0.711894
34533 0.719775
35582 0.731294
36992 0.75203
41175 0.787166 /

\multiput {\small $\diamond$}    at
163 0.819706
233 0.844079
550 0.666578
672 0.632548
937 0.620808
1760 0.590522
2542 0.527222
2843 0.576554
3219 0.610384
4171 0.66515
4656 0.712431
5183 0.750316
6359 0.737976
7441 0.824726
8084 0.83585
8724 0.8451
9501 0.8451
10782 0.908312
11692 0.913342
13293 0.928898
14435 0.927513 /

\setlinear \plot
163 0.819706
233 0.844079
550 0.666578
672 0.632548
937 0.620808
1760 0.590522
2542 0.527222
2843 0.576554
3219 0.610384
4171 0.66515
4656 0.712431
5183 0.750316
6359 0.737976
7441 0.824726
8084 0.83585
8724 0.8451
9501 0.8451
10782 0.908312
11692 0.913342
13293 0.928898
14435 0.927513 /

\put {Ratio} at 0 1.1

\put {{\small $\circ$}    Example 1}  at 40000 0.4
\put {{\small $\bullet$}  Example 2}  at 40000 0.3
\put {{\small $\ast$}     Example 3}  at 40000 0.2
\put {{\small $\diamond$} Example 4} at 40000 0.1

\endpicture
\caption{\label{Fig_perplexity}Ratio of perplexities}
\end{figure}

\begin{flushleft}
\underline{Observations on the Graph} \\
\end{flushleft}

It is natural that the ratio moves toward 1.0 as the number of tokens becomes bigger, 
because the cluster is approaching the set of all articles combined.
However, we can see that in all of the four example,
there is a minimum point at a small number of tokens.
This could happen if the combining texts have larger number of 
overlapping words than expected.
Now, we propose that this minimum point suggests one of the definitions of
sublanguage in terms of word appearance.
This phenomena is observed not only at the four examples shown, 
but also for many of the other clusters.
We have examined the articles involved and we find a promising pattern.
In Example 2, all 9 articles from the beginning until the minimum point
are all obituaries, while the next three are real estate articles.
In Example 1, the first article and 54\% of articles up to the minimum point are
college basketball articles.
The rest are either high-school and professional basketball (24\% and 8\%, respectively),
hockey (8\%) or football (4\%) articles.
On the other hand, only 16\% of the next 25 articles and
6\% of the rest of the articles up to the 185th are college basketball articles.
These example intuitively support the claim that
the minimum point of each line at the graph would be
an objective indicator of the size of a text cluster or a sublanguage.

\begin{flushleft}
\underline{Sublanguage Clusters} \\
\end{flushleft}

We take the articles from the beginning until the minimum point as a sublanguage. 
These clusters are used in the next experiment, identification of a new text
to sublanguage.
As there are 2147 articles in the corpus, the same number of clusters exist
(not necessary distinct).
The number of article in a cluster ranges from one to 31, 
and the average number is 4.26 articles.


\section{Sublanguage Identification for a New Text}

We take 244 test articles from new San Jose Mercury articles
which are not used in the experiment of sublanguage definition described above.
In this section, the method to identify the closest cluster
for each test article will be described.

For each test article, similarity measures to all the clusters are calculated 
to find the closest one.
Basically, the similarity measure is the same as the one in the previous experiment.
The similarity between a test article and a cluster is set to equal to
the maximum similarity between the test article and any of the articles in the cluster.
If there is a tie, the cluster which contains the smallest number of articles wins.
This calculation is not expensive, because we have a limited number of
the words which have to be taken into account in the calculation.
The number is normally much less than the number of tokens in the test articles.
Also, the number of clusters to be examined for each word
is less than the cut off number (50 in the experiment).
So this calculation is almost linear in the number of words in the test article,
if there is enough space to store the word index.

In this experiment, we set the parameters for similarity calculation
slightly different from the ones in the definition experiment.
The cut off number is 50, minimum occurrence is 1 and minimum co-occurrence
is set to 3, based on empirical adjustments.

\begin{flushleft}
\underline{Result of Identification} \\
\end{flushleft}

The result of the identification experiment will be described.
The evaluation measure is based on how many tokens in the test article
also exist in the closest cluster,
i.e. number of tokens which overlap between the test article and the closest cluster.
A straightforward measurement is its coverage, which is
how many tokens are overlapping against the number of tokens in the article.
This figure could be useful in deciding how useful this method is,
but it might still be difficult to make an objective decision.
Then, the number of overlapping tokens is compared with 
an expected number of overlapping tokens.
It is computed by taking two sets randomly which have the same
number of tokens to the test article and the selected cluster,
then calculate average number of overlapping tokens between them.
This expected number ($E$) is calculated by the following formula.

\begin{equation}
 E =  n_{t} \; \cdot \; \frac{\sum_{w} \; \frac{N_{t} * f_{w}}{N} 
              \; ( 1 - ( 1 - \frac{N_{c}}{N} )^{f_{w}})}
	{\sum_{w} \; \frac{N_{t} * f_{w}}{N}}
\end{equation}

Here, the denominator is the expected number of tokens in the article,
the numerator is expected number of overlapping tokens based on expected number of token,
and $E$ is adjusted number of overlapping tokens.
$N_{t}$, $n_{t}$ are the numbers of tokens in the test article,
$N_{c}$ is the number of tokens in the cluster,
$N$ is the number of tokens in the entire corpus,
and $f_{w}$ is the frequency of word $w$.
The expected number of overlapping tokens can be classified into certain frequency ranges
applying the variables $f_{w}$ and $n_{t}$ to the range.

It is well known that high frequency words like ``the'' or ``of'',
are relatively common regardless of the topic or sublanguage.
On the other hand, low frequency words, which are often regarded as
indicating the topic, are expected to co-occur together in same articles.
So, we can anticipate in this evaluation that
low frequency words overlap between test articles and
the closest clusters more than expected values.
(Note that, because words with $f_{w}$ $<$ 50 are used
 in the similarity calculation,
 those words must be specially treated in the evaluation.)
To observe such details, we classified the result based on document frequency of words
in the entire corpus.
Table \ref{T2} shows an example of the results.
In this example, the number of tokens in the article is 129 and
the number of tokens in the cluster is 1265 and the cluster consists of 5 articles.

\begin{table}[hbt]
\begin{center}
\begin{tabular}{|c||c|c|c|c|}
\hline
{\tiny Frequency} & {\tiny Number} & {\tiny Overlap} & {\tiny Coverage} & {\tiny Ratio} \\
                  & {\tiny of words} &  {\tiny words} & {\tiny (\%)} &         \\
\hline

0       &    0      &          &          & \\
        &   (4.6)   &          &          & \\
1-49    &    20     &     6    &   30.0   &  6.01  \\
        &   (34.9)  &  (1.0)   &          & \\
50-99   &    11     &     5    &   45.5   &  2.40  \\
        &   (11.6)  &  (2.1)   &          & \\
100-199 &    25     &    15    &   60.0   &  1.82  \\
        &   (12.0)  &  (8.2)   &          & \\
200-299 &    10     &     9    &   90.0   &  1.57  \\
        &   (6.9)   &  (5.7)   &          & \\
300-499 &    10     &     7    &   70.0   &  0.94  \\
        &   (7.1)   &  (7.5)   &          & \\
500-999 &    12     &    12    &  100.0   &  1.03  \\
        &   (16.9)  & (11.7)   &          & \\
1000-   &    41     &    41    &  100.0   &  1.00  \\
        &   (35.1)  & (41.0)   &          & \\
\hline
Total   &   129     &   95    &    73.6   &  1.23  \\
        &  (129.0)  &  (67.6)  &          & \\
\hline
\end{tabular}
\caption{\label{T2} An example of the result}
\end{center}
\end{table}

The first column shows the number of tokens and its expected number
in the brackets below each number.
For example, the expected number of tokens whose frequency ranges from 100 to 199 is
12.0, but is actually 25 in the article.
These two figures show the balance of the word frequency distribution in the article,
and we can see that most of them are balanced.

The second column shows information about overlapping tokens.
For example, 95 tokens out of 129 in the entire
article are overlapping to the closest cluster.

The third column shows the coverage of overlapping tokens in the test article.
We can see the entire overall coverage was 73.6\% in this sample.

The figure in the fourth column indicates that tokens in the article
are overlapping 1.23 times of the expected value.
Also 6.01 and 2.40 times of expected number of overlapping tokens are found in the article
in frequency ranges from 1 to 49 and from 50 to 99, respectively. 
As mentioned before, the result for words whose frequency ranges between
1 to 49 can not be evaluated directly by the figure.
As the method to find the closest cluster is like the single linkage technique,
so the words in the closest article is used in the cluster search.
The number of tokens overlapping between the test article and the closest article
in frequency 1 to 50, i.e. number of tokens used in the similarity
calculation, is 3 for this sample (not shown in the table).
This means that out of 5 overlapping tokens between the test article and the
closest cluster, 3 tokens are found in the closest article and the other 2 tokens are
found in the rest of the articles in the cluster.
So these figures show the benefit of the clustering in increasing
the overlapping tokens.

The average of the coverage and the ratio throughout the 236 test articles are given
in Table \ref{T3}. Note that, 8 articles can't find their closest
clusters because of the thresholds.

\begin{table}[hbt]
\begin{center}
\begin{tabular}{|c||c|c|}
\hline
Frequency &   Coverage &  Ratio   \\
          &   (\%)     &          \\
\hline
1 - 49    &     16.7   &  7.98  \\
50 - 99   &     26.4   &  1.94  \\
100 - 199 &     34.5   &  1.90  \\
200 - 299 &     45.9   &  1.03  \\
300 - 499 &     57.7   &  1.19  \\
500 - 999 &     76.6   &  1.00  \\
1000 -    &     95.8   &  0.98  \\
\hline
Total     &     51.1   &  1.18  \\
\hline
\end{tabular}
\caption{\label{T3} Average coverage and ratio}
\end{center}
\end{table}

The second column of the table shows the percentage of overlapping tokens in
the test articles, and the third column shows the
average ratio of overlapping tokens to expected overlapping tokens.
From the figures in Table \ref{T3}, we can observe the success of the method.
For example, tokens whose frequency range from 50 to 99 are found with 1.94 times of
the expected value in the closest cluster.
Also the ratio in range from 100 to 199 is 1.90, which is well more than 1.0.
These can be explained by saying that lower frequency words co-occur together more than expected.
This is the very fact that we had expected and prove the existence of sublanguage.
On the other hand, ratio at higher frequency is about 100\%.
It means the number of these words are almost same as that of expected.
This can be understandable, because many high frequency words are closed class words
and these words can occur in any of the articles.
So the expected number of overlapping is about the same as the number of tokens,
as can be seen in Table \ref{T2}.
Actually, almost all of the words whose frequency is more than 1000
are closed class words, like ``the'', ``of'' or ``it'' and
occur in many of the articles (there are only 68 words
which have frequencies over 1000. The total number of words is 39217).


\section{Experiment with First Two Sentences}

For some N.L.P. applications, it may be impossible to see all the sentences
of a text in advance of the processing.
For example, spontaneous speech recognition systems have to process each
utterance at a time.
The algorithm described above can't be applied to these applications directly,
because the system can't pre-scan the material before the processing.
Therefore we conducted an experiment using the first two sentences in the test articles,
instead of all the sentences in the article, to find the closest cluster.
This process can be called dynamic sublanguage identification,
because it utilizes the knowledge which could be acquired during the process.
If it can find a good cluster (i.e. sublanguage) which is useful for the rest of the article,
we can say the dynamic sublanguage identification is well performed.

Table \ref{T4} shows the result of this experiment.


\begin{table}[hbt]
\begin{center}
\begin{tabular}{|c||c|c|}
\hline
Frequency &   Coverage &  Ratio   \\
          &   (\%)     &          \\
\hline
1 - 49    &     17.2   &  6.40  \\
50 - 99   &     26.7   &  1.67  \\
100 - 199 &     34.9   &  1.90  \\
200 - 299 &     47.7   &  0.95  \\
300 - 499 &     56.8   &  1.16  \\
500 - 999 &     75.4   &  0.96  \\
1000 -    &     95.6   &  0.98  \\
\hline
Total     &     51.2   &  1.18  \\
\hline
\end{tabular}
\caption{\label{T4} Average coverage and ratio}
\end{center}
\end{table}

Surprisingly, the result is almost the same as the previous one.
Actually, 72.5\% of the test articles (177 out of 244 articles) select the same cluster
which was selected in the previous experiment.
This result is intuitively understandable, 
because the first sentences in newspaper articles normally indicate its topic and
the sentences could be helpful to identify its sublanguage.
This result indicates that this method can be applicable to
speech recognition systems or others in which it is impossible to pre-scan all the
sentences before processing.

\section{Discussion}

We propose ``Inductive definition of sublanguage'' and
``Dynamic sublanguage identification''.
We don't deny the metric of the conventional approach to sublanguage definition.
Obviously, the domain of a text is one of the very important keys
of sublanguage definition.
However, the inductive definition could be useful in situations
where the domain name is not available or the domain definition is too vague.
Our results suggest that the perplexity measurement can play a significant role
in automating the definition process.
This is one of the fruitful results in this paper.

The result of sublanguage identification using the first
two sentences suggests the possibility of practical applications of this method.
For example, we can construct an adaptable machine translation system,
which can take inputs from any kinds of subject domain or sublanguage.
The system would find which sublanguage the input belongs to in its initial process,
and adapt knowledge of the sublanguage to analyze the following sentences.
Intuitively, this procedure looks very similar to what a human is doing.

As it is easily inferable from the sublanguage literature,
the definition of sublanguage may differ based on its objective or usage.
In this experiment, the sublanguage notion is aimed to utilize
language restrictions (word usage),
so the inductive definition based on lexical appearance is suitable.
In the future, we could probably extend this technique by using
other sublanguage characteristics, i.e. syntactic and semantic restrictions.
One of the recent corpus observation by Gale et al. \cite{Gale:92}
supports the usefulness of semantic restrictions.
Furthermore, we could seek a hybrid concept of the two, deductive and inductive
sublanguage definitions, in order to maximize the profit of the sublanguage notion.

We can identify several directions for future study.
One is, of course, to refine the algorithm and to make larger experiments.
With regard to applications, as has been repeatedly mentioned, speech recognition
is one of the most interesting applications.
The result of that sort of experiments may tell us much more about the value of this approach.
The way in which it is applied may heavily depend on the features of the application system.
However, we believe that sublanguage concept is one of the common problems
for many kind of N.L.P. systems.

\section{Conclusion}

In conclusion, it appears that the inductive definition of sublanguage
could become realistic by using large scale corpus.
This would be very beneficial for natural language processing
since the evaluation results show that
automatically identified sublanguage knowledge is useful in processing
of a new text.
The second experiment proves that the method is applicable 
not only for static applications which can pre-scan the material
before its actual processing,
but also for applications which process the sentence sequentially,
such as a speech recognition system.


\section{Acknowledgments}

The work reported here was supported by the Advanced Research Projects Agency
under contract DABT63-93-C-0058 from the Department of the Army.
We would like to thank our colleagues at NYU, in particular
Prof.Grishman, whose comments have been very useful.



\bibliographystyle{theapa}
\bibliography{sekine}

\begin{biography}

\biotitle{}

\bioauthor{Satoshi Sekine}
{
Satoshi Sekine received a B.S. degree in applied physics from
Tokyo Institute of Technology in 1987.
From 1987 to 1993, he worked for Matsushita Electric Industry Co. Ltd., Tokyo, Japan.
During that time, he was a member of EDR from 1989 to 1990. He was
also a visiting researcher at the University of Manchester Institute of
Science and Technology, England from 1990 to 1992.
Since 1994, he has been an assistant research scientist
at New York University, New York, USA.
He is a member of 
the Information Processing Society of Japan,
the Japanese Society for Artificial Intelligence
and the Association for Computational Linguistics.
His current research interests are natural language processing, 
in particular sublanguage and corpus linguistics.
}

\bioreceived{Received}
\biorevised{Revised}
\bioaccepted{Accepted}

\end{biography}

\end{document}



