    \documentclass[english]{jnlp_1.4}

\usepackage{jnlpbbl_1.2}
\usepackage[dvips]{graphicx}
\usepackage{udline}
\setulminsep{1.2ex}{0.2ex}
\let\underline
\usepackage{biodateX}
\usepackage{amsmath}


\Volume{16}
\Number{4}
\Month{October}
\Year{2009}

\received{2008}{8}{30}
\revised{2008}{10}{31}
    \rerevisedX{December 20, 2008; February 28, 2009; April 9, 2009}
\accepted{2009}{7}{17}

\setcounter{page}{3}

\etitle{A Corpus-based E-learning System \\for Japanese Vocabulary}
\eauthor{Gregory Hazelbeck\affiref{Author_1} \and Hiroaki Saito\affiref{Author_1}} 
\eabstract{
This study presents an initial version of an e-learning system that assists learners of Japanese with their study of vocabulary. The system uses sentences from a corpus to generate context-based exercises. The sentences used in the context-based exercises are selected using a readability formula developed for this system. We used the system with two different types of corpora, a web corpus that we constructed for this system and a sample of the recently released Balanced Corpus of Contemporary Written Japanese (BCCWJ). We compared the two corpora and while the BCCWJ has better word coverage, our web corpus still covered a majority (96.1\%) of the target vocabulary words even though it's relatively small. Evaluation of this system showed that the readability formula performs well, especially when sentences contain the system's target set of vocabulary words. A group of learners of Japanese were also asked to use the system and then fill out a survey. Results of the survey indicate that the learners thought the system was easy to use. Most of the learners also expressed a desire to use this type of system when studying vocabulary.
}
\ekeywords{e-learning, web corpus, BCCWJ}

\headauthor{Hazelbeck and Saito}
\headtitle{A Corpus-based E-learning System for Japanese Vocabulary}

\affilabel{Author_1}{}{Graduate School of Science and Technology, Keio University}


\begin{document}
\setcounter{secnumdepth}{3}

\maketitle

\section{Introduction}

The advent of the Internet has brought many new opportunities for the use of technology in learning \cite{Book_Bransford}. In recent years there has been a large focus on using the Internet for collaborative learning \cite{Article_Shih}. However, the access to the mass amount of information that the Internet provides has been ignored by many \cite{Article_Fletcher}. The task of finding quality information is probably the largest factor that prevents many from using the Internet as a linguistic resource \cite{Article_Fletcher}. As \cite{Article_Fletcher} states, ``quantity of information online greatly surpasses its overall quality'' and ``online documents generally seem to consist more of accumulations of fragments, stock phrases and bulleted lists than of original extended text.'' Even though there are issues that must be resolved about the quality of information, because the Internet is constantly being updated it contains current usage of the target language \cite{Article_Fletcher} making it very appealing for use during language instruction.

In order to support the utilization of the large amount of information on the Internet as well as content from other media for language instruction, we have begun developing an e-learning system that can use both online and offline corpora to construct exercises. In this paper, we present the initial version of this e-learning system that assists English-speaking learners of Japanese with their studies of vocabulary and kanji for the Japanese Language Proficiency Test (JLPT). The JLPT was selected because it is the most widely recognized Japanese language test in the world \cite{Web_JLPT}. 

In preparation for the JLPT, learners of Japanese must study many vocabulary words as well as the characters used to write them. There are three types of characters in written Japanese: hiragana, katakana, and kanji. Hiragana and katakana are syllabaries where each character has a unique reading (or pronunciation). Learners of Japanese learn all of the hiragana and katakana characters at the very beginning of their studies. On the other hand, kanji is logographic characters that represent words. A single kanji character can be used for many different words and can have multiple readings. There are thousands of kanji characters so it takes learners of Japanese a very long time to learn them. Japanese words are also often written in a romanized form in books and software targeted at beginning learners. There has been much debate as to whether roman characters should be used when teaching Japanese at the entry level. However, a study conducted by \cite{Article_Okuyama} with English-speaking learners of entry-level Japanese, found that the presence of roman characters in a computer-assisted language learning program did not facilitate their vocabulary intake. Based on the results of that study, we have decided to not use any roman characters when displaying vocabulary words in the present e-learning system.

When studying a vocabulary word for the JLPT, a learner must study the meaning, reading, and written form of the word. The exercises used in this e-learning system have been developed to facilitate the learning process of these three elements of each vocabulary word. The main type of exercise used in this e-learning system is called a context-based exercise. In this type of exercise, learners are able to observe the vocabulary words and the kanji that they are studying in sentences. This type of exercise gives the learner more opportunities to read and see the word that they are studying in context, which is very important when learning vocabulary \cite{Article_Chall}. During these exercises, learners also have access to glosses of each word in the sentence. The glosses were included because it has been found that they have a positive effect when students are learning new vocabulary \cite{Article_Hulstijn}. 

The sentences used in the context-based exercises are taken from either an online or offline corpus. These corpora often contain sentences that are not conducive to learners' vocabulary studies because they are excessively long and use many proper nouns or technical terminology. In order to prevent the system from using these types of sentences, we propose the use of a readability formula to determine which sentence should be selected for a particular exercise. Use of the readability formula allows the system to select shorter sentences with easy vocabulary to use in exercises where the learner is first starting to study a word. Longer sentences with more difficult vocabulary are used only after the learner has correctly recalled the target word in exercises with shorter sentences.

\section{Related work}

This section describes other research being done in subjects relating to e-learning systems for Japanese, readability formula, and linguistic resources.

\subsection{E-learning systems for Japanese}

Many other e-learning systems have been developed for the Japanese language. However, these systems often use pre-created content which limits the amount of exercises that the learner can perform. While we were unable to find another e-learning system for Japanese that creates exercises from sentences in either online or offline corpora, we found a tool that helped learners select example sentences from corpora called the NAIST Language Tutor \cite{Inproc_Mizuno}. This tool takes a sentence or a list of vocabulary words as input and returns a list of example sentences that use the inputted words sorted by difficulty. The user can also specify ranges for how long they would like the example sentence to be, as well as the word, kanji, and grammar level that they want. Like the e-learning system presented in this paper, this tool also uses the JLPT levels to categorize vocabulary words and kanji by difficulty. To determine the difficulty of each example sentence they developed a formula that takes into account sentence length, word level, kanji level, and grammar level. The example sentences used in this tool were taken from newspapers and the Internet. Currently, no evaluation results of this tool have been published. The differences of this tool's method for ranking sentences and our e-learning system's method are discussed further in section 4.1.1.

\subsection{Readability formula}

In this e-learning system, a simple readability formula is used to prevent the system from using sentences that are not conducive to learners' vocabulary studies. Several different methods have been developed to measure the readability of Japanese texts. \cite{Inproc_Sato} presented a method that uses a language modeling approach. A textbook corpus is used along with a character unigram model that considers the three types of characters in Japanese (hiragana, katakana, and kanji). The textbook corpus spans across 12 grades (elementary school to high school) with an additional grade being added for texts that are above high school level. Using their language model they are able to categorize a given text into one of the 13 grades. Evaluation indicates that their method performs very well with long texts achieving a high correlation coefficient (above 0.9). It also performs well with shorts texts, achieving a correlation coefficient of 0.83. This method of measuring readability was developed to measure the readability of websites in terms of a school grade level in order to help improve accessibility. This method of measuring readability was not used in this e-learning system because it was developed for a different purpose which does not meet the objectives of this e-learning system.

Another method for measuring the readability of Japanese texts was presented by \cite{Inproc_Tateisi,Article_Tateisi}. They presented a readability formula derived by analyzing data from 77 documents using principal component analysis. The first equation they derived uses ten factors to evaluate readability. They further simplify this equation so that only six factors are used. The simplified version of the formula is as follows:
\begin{equation}\label{rf1}
RS = -0.121s - 1.37la + 7.41h - 23.18lc - 5.4lk - 4.67cp + 115.79
\end{equation}
where $RS$ is the readability score, $ls$ is the average number of characters per sentence, $la$, $lh$, $lc$, and $lk$ are the average number of roman letters/symbols, hiragana characters, kanji characters, and katakana characters per run, and $cp$ is the ratio of commas to periods. In this case, a run is defined as a string that consists of only one type of character. Evaluation of this readability formula included an experiment where the examinees were asked to fill in one blanked out word in a text. Examinees took less time to fill-in the blank on texts with a higher $RS$ score than texts with a lower $RS$ score. This readability formula was not used in this e-learning system because it does not take into account JLPT vocabulary or kanji. Vocabulary and kanji lists from the JLPT are already being used in other parts of the e-learning system so we want the readability formula to take advantage of these lists.

\subsection{Linguistic resources}

Large-scale corpora, like those used in this e-learning system, are a very important linguistic resource that provides a wealth of information to both researchers and learners. In this e-learning system, learners are able to read sentences from these corpora while performing exercises to study vocabulary. While reading these sentences, learners can discover certain lexical relationships between words, like the adjective 冷たい(cold) is often used with the noun 水(water). Lexical relationships are often observed from these commonly occurring sequences of words called collocations. However, \cite{Inproc_Joyce} argued that in addition to collocations found in large corpora, word association norms also contain some unique lexical relationships. They compared lexical relationships found in Japanese collocation data and a word association database. Their comparisons indicated that some lexical relationships were found in only one of the resources. This led the authors to conclude that both types of resources are important in order to cover the wide variety of lexical relationships.


\section{System overview}

This section describes the overall design and operation of the e-learning system.

\subsection{System architecture}

The architecture of the e-learning system is broken down into two parts, the client and server, in order to support both online and offline corpora. The client program is installed on the learner's computer and is used by the learner to practice vocabulary and kanji. The server program is installed on a single computer and continually collects new textual data from the Internet to be added to the online corpus. When the system is using an online corpus there are databases on both the client and server that contain the corpus. The database on the server is continually updated with new texts that have been mined from the Internet while the database on the client will only be updated when the learner initiates an update process. During this update process, texts that are relevant to the learner's studies are downloaded from the server and put into the client's database. This gives the learner access to a large amount of texts but also minimizes the amount that they have to store on their local computer. In the case in which the system is only using an offline corpus, the server is not used because the entire corpus is included with the client program. The databases on both the client and server programs also contain a Japanese-English dictionary called EDICT\footnote{http://www.csse.monash.edu.au/{\textasciitilde}jwb/edict.html}, the system's main dictionary, along with a list of JLPT vocabulary words and kanji.

\subsection{Client program}

The client program provides the main interface that learners use to study vocabulary and kanji. This program is separated into three main sections called: exercises, vocabulary, and profile. The exercises section is where the learner selects the type of exercise that they wish to perform, as well as the vocabulary list that contains the words that they wish to study. The vocabulary section is where the learners can create, edit, and remove vocabulary lists and the profile section is where learners can view the results of past exercises. The general use of the client program's various sections by the learner is outlined in the steps below:

\begin{enumerate}
\item The learner first logs into the program by selecting or creating a profile. When creating a new profile the learner must input their name and JLPT level.

\item After logging in, the learner must first create vocabulary lists of words that they want to practice. If the learner has previously created one or more vocabulary lists then they can jump to step 4. Creation of a vocabulary list is performed in the vocabulary section of the program. Once in that section, the learner creates a new empty vocabulary list, and then inputs a word that they want to practice into the system. The system then shows a list of words from its dictionary that match the inputted word. The learner then selects the word that they want to practice from the list and adds it to the new vocabulary list. The learner can repeat this process and add as many words as they want to the vocabulary list. The only constraint is that the system does not allow words that do not exist in the dictionary to be added to a vocabulary list. 

\item When the system is being used with an online corpus, the learner has the option of updating the program's database after they have created a new vocabulary list. If they initiate this process, new sentences from the server will be downloaded to the learner's computer.

\item Next the learner moves to the exercise section of the program. In that section they will select the vocabulary list that contains the words that they wish to practice and the type of exercise that they wish to perform. 

\item Once the learner has selected a type of exercise and vocabulary list, the system will begin displaying exercises for each vocabulary word.

\item When the learner is finished with all of the exercises the results will be displayed. The learner can then return to step 4 and perform another exercise or view their overall results displayed in the profile section.
\end{enumerate}

Each section of the client program was designed to be as user-friendly as possible to ensure that learners would be able to quickly understand how to use the program. Furthermore, the program was developed using Python and the cross-platform GUI library wxWidgets, enabling it to be used on any of the major computing platforms.


\subsubsection{Types of exercises}

The client program provides two types of exercises: context-based and flashcards. In this paper, we focus mainly on the context-based exercise. A context-based exercise is where the learner is given an example sentence that contains one of the words that they are studying. The learner is asked to read the sentence and input the reading of the word being studied, in hiragana, into an input box that is displayed above the word. If they input the correct reading, the input box will turn green and they will be able to advance to the next problem. Otherwise, the correct reading will be displayed to them and then they will be allowed to advance to the next problem. A context-based exercise is shown in Fig.~\ref{fig1}.

\begin{figure}[b]
\vspace{0.5\baselineskip}
\begin{center}
  \includegraphics{16-5ia1f1.eps}
\end{center}
  \caption{A context-based exercise}
  \label{fig1}
\end{figure}

In the context-based exercises, the system also provides a gloss for each word that can be accessed by clicking on the word. This feature allows learners to quickly look-up words that they have not learned or are unable to recall the meaning of. The use of the gloss feature is shown in Fig.~\ref{fig2}.

\begin{figure}[t]
\begin{center}
  \includegraphics{16-5ia1f2.eps}
\end{center}
  \caption{The use of the gloss feature}
  \label{fig2}
\vspace{-1\baselineskip}
\end{figure}

Some words in the context-based exercises also have their reading displayed in hiragana above them. When a word in a sentence is at or above the learner's JLPT level and does not exist in one of the learner's vocabulary lists, a reading in hiragana is displayed. This is to help the learner read harder vocabulary that they might not know. It also forces the learner to recall the reading of vocabulary words that they have already studied. When the reading is not displayed, it is put into the gloss of that word so that if the learner is unable to recall it they can easily look it up.


\subsubsection{Repetition of exercise questions}

When practicing a set of vocabulary words the learner can select a repetition/order mode. The repetition/order modes dictate in what order the questions in each respective exercise type will be given as well as how many times they will be displayed. If the random mode is selected then each question will be asked once in random order. If the adaptive mode is selected then a method developed by \cite{Article_Mondria} is used. In this system's implementation of the method, the questions are repeatedly asked to the learner with the order being shuffled for each iteration. When the learner has answered a question correctly five times in a row the question will no longer be asked.

\subsubsection{Exercise scoring}

Scoring of the exercises depends on what repetition/order mode was selected. If random is selected it simply records which exercises are answered correctly and which exercises are answered incorrectly. If adaptive is selected, it marks exercises as correct only once they have been repeated five or six times (i.e. the learner always answered the exercise correctly or only made a mistake on the first time), otherwise it marks them as incorrect.


\subsection{Server program}

The server is composed of several programs written in both Python and PHP which manipulate textual data stored in an SQLite database. The main component of the server is a web crawler, written in Python, that crawls the Internet and fetches new textual data. The other component is a set of PHP scripts that transfer data to the client program and manage the server-side data. These PHP scripts run on an Apache web server and are accessed by the client program when the learner updates their client program's database.



\section{System Components}

\subsection{Exercise generation}

This section explains how the system generates a context-based exercise. An outline of the generation process is illustrated in Fig.~\ref{fig3}.

\begin{figure}[b]
\begin{center}
  \includegraphics{16-5ia1f3.eps}
\end{center}
  \caption{An outline of the exercise generation process}
  \label{fig3}
\end{figure}


\subsubsection{Selecting a sentence for the exercise}

When an exercise is started by the learner, the system will receive the vocabulary list that he/she selected. For each word in the list, the system selects all sentences containing that word from the database. Next, the system calculates the readability scores for all of the retrieved sentences using the following readability formula that we developed:
\begin{equation}\label{rf2}
S = \frac{w_1vg + w_2kg}{sl}
\end{equation}

Where $S$ is the readability score, $vg$ is the sentence's vocabulary JLPT level, $kg$ is the sentence's kanji JLPT level, $sl$ is the sentence length, and $w_1$ and $w_2$ are adjustable weights. Using this formula, easy sentences will receive high scores.

In this formula, a sentence's vocabulary level ranges between 0-4 with 1-4 being the respective JLPT levels and 0 being the level used for words that do not exist in the JLPT. A sentence's vocabulary level is calculated by first creating a list of words that are used in each sentence retrieved from the database. These words are then looked up in the system's dictionary where the JLPT level is also stored. The number of words in each level is counted and the level with the highest amount is assigned to $vg$ in the readability formula. The sentence's kanji level is calculated in the same way as the sentence's vocabulary level. However, the kanji level ranges from 0-5, with 1-4 being the respective JLPT levels and 0 assigned to kanji that do not exist in the JLPT, and 5 being the level used for when the sentence contains no kanji. The weights ($w_1$ and $w_2$) in the formula can be adjusted by the learner from within the client program. This allows the learner to adjust how the system considers vocabulary word level and kanji level when selecting sentences for the exercises.

This formula uses some of the same factors as the formula that \cite{Inproc_Mizuno} presented. However, we divide the weighted sum of the vocabulary word level and kanji level by the sentence length, whereas \cite{Inproc_Mizuno} adopted the weighted sum of all the factors. We found that taking a different approach to sentence length performed well during our tests. The formula that \cite{Inproc_Mizuno} presented also considers grammar as one of its factors. While grammar is a very important factor in understanding a sentence, we did not include it in the formula for the first version of this e-learning system because it was not required to meet our objective. The main purpose for using this formula is to prevent the use of excessively long sentences with difficult vocabulary in exercises when a learner is beginning to study a new word. The current version of the formula is able to assign low scores to these types of sentences, preventing them from being used before shorter sentences with easier vocabulary (that receive high scores). Therefore, this formula is able to satisfy our main objective without considering grammar.

After readability scores are calculated for all of the sentences retrieved from the database, a ranked list is created by sorting the sentences by their score in descending order. Next, the system selects which sentence to use in the exercise. This selection process is illustrated in Fig.~\ref{fig4}. 

To make this selection, the system uses information stored in the learner's profile, which included a history of completed exercises. In this history, sentences that were used in exercises answered correctly by the learner for each previously studied word are recorded. The system starts at the top of the ranked list and checks each one to see if it is already in the learner's history. The first sentence that is not in the learner's history is selected for use in the exercise. If all of the sentences are already in the learner's history (i.e. the learner has answered all of the possible exercises correctly), a random sentence is selected from the ranked list. 

This selection process enables the system to display short sentences that use simple vocabulary when the learner is first starting to study a word. This is important because the learner could quickly become frustrated if the system displays a long sentence with difficult vocabulary when they are first studying a new word. The selection process also enables the system to introduce the learner to a wide variety of usages of the word that they are studying. As long as the learner keeps recalling the word and answering the exercises correctly, a different sentence will be displayed every time allowing the learner to observe the word in more contexts.

\begin{figure}[t]
\begin{center}
  \includegraphics{16-5ia1f4.eps}
\end{center}
  \caption{The sentence selection process}
  \label{fig4}
\end{figure}


\subsubsection{Customizing the exercise for the learner}

After a sentence is selected, the system determines how the selected sentence will be displayed in the exercise. First, using the Japanese morphological analyzer tool called Mecab\footnote{http://mecab.sourceforge.net/}, the sentence is broken down into word segments. Next, the word that the learner is practicing is found in the sentence and marked so that later the program can display an input box above it. If the word appears in its hiragana form, it is replaced by its kanji form. After the target word is found, the system attaches readings to words that have JLPT levels at or below the learner's JLPT level. Finally, a gloss is attached to each word in the sentence. This gloss is retrieved from the system's dictionary. When multiple definitions of a word exist, all definitions are included in the gloss.

Examples of exercises that the system generated using both online and offline corpora are shown in Table 5 and 6 in the appendixes. These exercises were generated when the word 行く(meaning: to go) was input into the system. A JLPT level 2 learner profile was used in order to demonstrate how the system displays readings according to learner level. The readability score is also displayed beside each exercise to indicate how the system actually ranked each sentence. 


\subsubsection{Performance}

The time taken to generate an exercise largely depends on how many words are in the vocabulary list. During our tests on a computer with an Intel Core Duo 1.83~GHz CPU and 1~GB of RAM, it took the system about 2 seconds per word in the selected vocabulary list to generate the exercises. This is the time taken for the first time that exercises are generated for a vocabulary list. Subsequent times are almost instantaneous because the database is cached. While this is acceptable performance for the first version of this e-learning system, in the future we plan to look into ways of optimizing this process so that exercises can be generated faster.

\subsection{Web crawler}

This section describes the part of the server called the web crawler. The operations of the web crawler are illustrated in Fig.~\ref{fig5}.

\begin{figure}[b]
\begin{center}
  \includegraphics{16-5ia1f5.eps}
\end{center}
  \caption{Operations of the web crawler}
  \label{fig5}
\end{figure}

The web crawler is first given a set of seed URLs from where it begins looking for texts. Websites outside of the domain names of these seed URLs are not crawled. Once the web crawler is started it uses a series of templates and rules to extract sentences from websites. It first takes the HTML text of a website and splits it up according to sentence ending punctuation marks. Sentence ending punctuation marks are defined as one of the following characters:（。），（！），or（？）．The commonly used period punctuation mark in Japanese differs from the dot used for a decimal point so it can be used to reliably determine an ending of a sentence. These pieces of text are then put through several filters to determine if they are a suitable sentence or not. First, any text that contain HTML tags or other code are immediately disregarded. Next, any text containing punctuation that denoted quotation is removed. Any text containing symbols or numbering is also removed. Finally, if a piece of text only contains nouns or verbs it is also removed.

Pieces of text that make it through all of the filters are finally judged to be a sentence and stored in the database. At this point, the server uses Mecab to perform morphological analysis on the sentence. The output of Mecab is used to create a list of words that appear in that sentence. The server removes any numbers, punctuation marks, and particles from the list and then stores it in the database.

The operations of the web crawler are fully automated meaning that the instructor or learner only needs to provide a URL of the website that they want to extract texts from. The web crawler will then periodically crawl that website and extract any new texts that are found. This allows the instructor or learner to easily and quickly add new sources of texts without having to worry about actually building and maintaining a corpus. The types of websites that the web crawler is currently targeted at are websites containing professional level writing. These websites have the most suitable material for learners of Japanese because they contain high quality texts that are least likely to have errors. The uniform usage of sentence punctuation types on these websites also ensures that the web crawler will be able to successfully extract sentences. This web crawler is not meant to be used on personal blogs or bulletin boards where the writing style may not be appropriate for learners and irregular usage of punctuation could prevent the web crawler from extracting useful sentences.


\subsection{Corpora}

This section describes the corpora that were used with the system for this paper. One online and offline corpus each were selected and used with the system. The online corpus is a collection of texts taken from websites on the Internet. This type of corpus is usually collected automatically and can always be updated because many websites are updated with new content every day. For the online corpus used in this paper, we collected texts from websites containing professional level writing in order to ensure that only high quality texts were mined. These websites included newspaper websites, technology-related news websites, and websites targeted at children. We refer to this online corpus simply as the Web corpus. As of writing this paper, the web corpus contains 150,000 sentences.

In contrast to an online corpus, an offline corpus is usually constructed from a collection of publications, such as books or newspapers, and is released in one package that cannot be updated like an online corpus. The offline corpus used in this paper was the recently released sample of the Balanced Corpus of Contemporary Written Japanese (BCCWJ) \cite{Book_BCCWJ}. It contains texts from white papers, recordings of the National Diet, and many different types of publications like books and newspapers. It also contains texts from a website called Yahoo! Chiebukuro, a community-driven question and answer website where users can both submit questions and answer questions asked by others. While the BCCWJ does contain some texts from the Internet, it is treated as an offline corpus in this paper because it cannot be updated. The same method and filters for the web corpus was used to insert sentences from the BCCWJ into the system's database. In total, around 668,000 sentences were inserted into the database.

These two corpora were selected in order to enable the system to cover the wide range of vocabulary and kanji that are in the JLPT and allow many different types of learners to use the system. However, the system can also use much more specialized corpora that would support a learner's specific needs. The use of these types of corpora is discussed in the next section.



\section{System use scenarios}

This section presents some usage scenarios of the e-learning system. The section is split into two parts, one focusing on beginning learners and the other focusing on advanced learners.


\subsection{Beginning learners}

Learners of Japanese studying at the beginning level are learning basic vocabulary, grammar patterns, and relatively few kanji. These types of learners are studying under levels 3 and 4 of the JLPT. As a minimum requirement for using the context-based exercises in the system, a learner must be able to read hiragana and input it into a computer.

In order to use the current system effectively, it is necessary for beginning learners to use the system in conjunction with a Japanese course. The course instructor can create a corpus of sentences they have either written or taken from materials used in the course. Instructors do not need to provide the definitions or kanji readings for the words used in the sentences because the system provides them automatically. Learners will then be able to easily review and practice their vocabulary that they are studying outside of the classroom.


\subsection{Advanced learners}

Learners of Japanese studying at an advanced level have already learned all of the basic vocabulary, kanji, and grammar patterns. Such learners study at level 1 or 2 of the JLPT. The current version of this e-learning system is most effective for learners studying in these advanced levels.

Learners in the advanced levels often take Japanese classes that are comprised of reading exercises. In such classes, the learners are typically given an article (often taken from a magazine or newspaper) along with a list of vocabulary and grammar in that article. The learners will then read and discuss the article together in class. The instructor will explain the grammar and some of the more difficult to understand vocabulary. However, it is often up to the learners to study and memorize the vocabulary and kanji by themselves, outside of the classroom. This is where this e-learning system can be most beneficial to the learner. When the learner is away from the classroom, they can input the vocabulary on the list into the system. They will then be able to study and memorize the word's reading as well as discover new ways to use the word that they could not have found just by reading the article from class.

The system can also be used by advanced learners that are not taking a class but studying by themselves. Learners using textbooks or books designed for self-study can input vocabulary introduced in each section into the system. Learners using websites on the Internet that teach Japanese will also have no problem using the system because those websites often take the vocabulary and kanji directly from the JLPT lists.

Advanced learners also have many options when choosing the kind of corpus to use with the system. However, using an online corpus is where the learner is able to customize the system to their own specific interests. For example, learners who are interested in technology can provide the system's web crawler the URLs of their favorite Japanese technology websites. Texts from these websites will then be collected by the web crawler and used in the context-based exercises when the learner studies vocabulary. This allows the learner to study the usage of their vocabulary within the context of a field that they are interested in as well as gain exposure to new vocabulary used in that field.


\section{Corpus analysis}

This section describes the analysis performed to compare the web corpus with the BCCWJ. The two aspects on which the two corpora were compared were word coverage and sentence word level distributions.

\subsection{Word coverage}

The first way in which the two corpora were compared was in terms of word coverage. The word coverage of the corpora is measured by counting the number of words in the JLPT lists that were used in sentences in the two corpora. Table 1 shows the number of words found in each corpus for each JLPT level.

\begin{table}[b]
  \caption{Corpus comparison by word coverage}
  \label{table1}
\input{01table01.txt}
\end{table}

As Table 1 shows, the larger BCCWJ does have better word coverage than the web corpus. Some examples of where the BCCWJ proved better than the web corpus are words that can be written in several different ways like 明るい（明るい vs. 明い）or 落とす（落とす vs. 落す）．Both versions of these words were found in the BCCWJ but only the more common version was found in our web corpus. While these types of omissions in the corpus will not entirely prevent the learner from studying one of these words, it will prevent the system from being able to show the learner sentences with both versions.

Another factor that was measured was the number of JLPT words used in useful sentences in the corpora. A useful sentence is defined to be a sentence with a word level that is the same or lower than the level of the target word. In the system, the easiest sentence containing the target word is always selected from the database. In the best case, the system will select a sentence with a level 4 word level for a level 4 vocabulary word. For a level 2 vocabulary word, the system will select a sentence with a word level of 4, 3, or 2. If the learner was studying a level 4 vocabulary word and the system presented them with a sentence with word level of 2, there is a good chance that the leaner would not understand any of the vocabulary in that sentence. So it is important that the corpus contains useful sentences for the JLPT words that can be used in the system. Table 2 shows the number of the words in each JLPT level that have useful sentences for each corpus.

\begin{table}[t]
  \caption{Number of words with useful sentences}
  \label{table2}
\input{01table02.txt}
\end{table}

While Table 2 indicates that the BCCWJ provides more words with useful sentences, the web corpus still provides a majority of the words with useful sentences even though it is less than one-quarter size of the BCCWJ.

The analysis of both the web corpus and the BCCWJ shows that while the BCCWJ has overall better word coverage, the web corpus, despite its smaller size, can still cover the majority of the JLPT words. As a result, we believe that it is acceptable to use this web corpus for the current version of our e-learning system.


\subsection{Sentence word level distribution}

The amount of sentences at each word level in a corpus is also an important factor because there must be a balance between easy and difficult sentences. Fig.~\ref{fig6} presents the percentage of sentences at each word level for each corpus.

\begin{figure}[t]
\begin{center}
  \includegraphics{16-5ia1f6.eps}
\end{center}
  \caption{Percentage of sentences in each word level for each corpus}
  \label{fig6}
\end{figure}

In the graph in Fig.~\ref{fig6}, the 0 level is the level used for a sentence that mostly contains words that are not in the JLPT lists. This graph shows that, especially for the web corpus, there is a large amount of sentences which contain these unlisted words. We believe that one reason why such sentences are so numerous in the web corpus is that we have included a lot of texts from technology-related websites. These texts will have more technical terms in them that might not be in the JLPT lists. The graph also shows that there are significantly less sentences at level 3 than at level 2. The reason for this is that words in level 3 often appear together with many level 4 words in simple sentences. Since the readability formula currently assigns a sentence's word level to the level with highest word count, many sentences with level 3 words will be assigned a word level 4 because they contain more level 4 words.

The graph in Fig.~\ref{fig6} also shows that both of the corpora have the largest amount of sentences at level 2 and 4. It is good to see that neither corpus is overly biased towards either easy or hard words. However, the web corpus does have less level 4 sentences than the BCCWJ. In the future we would like to try to balance the web corpus so that it has an equal amount of sentences with easy and hard words as well as look into methods of dealing with all of the unlisted words.


\section{Experiments}

This section describes the three experiments performed to evaluate the system. The first two experiments were performed to empirically evaluate the readability formula that was developed for this e-learning system. The final experiment was conducted to evaluate the usability of the e-learning system. During all experimentation default weights ($w_1 = 0.5$ and $w_2 = 0.3$) were used for formula \ref{rf2} when calculating the readability score. Results and an explanation of each experiment are first given followed by a discussion of all the results at the end of the section.


\subsection{Experiment 1}

This experiment evaluated the ability of the readability formula to correctly rank sentences that contained vocabulary and kanji from the JLPT. 50 sentences from levels 4, 3, 2, and 1 respectively were selected from past JLPTs. These sentences were then split up into 50 groups, each group containing one sentence from the level 4, 3, 2, and 1 tests. Then, the proposed readability formula was used to rank each group of sentences. A sentence taken from the level 4 test should be ranked as the easiest with the sentences from levels 3, 2, and 1 following respectively. The correlation coefficient ($r$) was calculated between the correct ranking and the ranking obtained by the proposed readability formula. To prevent ordering of the sentences from affecting the results, correlation coefficients were taken for every possible ordering of the sentences. The final result was obtained by selecting the median of these correlation coefficients. The proposed readability formula obtained a correlation coefficient of $r = 0.95$.


\subsection{Experiment 2}

This experiment evaluated the ability of the proposed readability formula to correctly rank sentences contained in both our web corpus and the BCCWJ was evaluated. A group of native speakers of Japanese either selected several vocabulary words or words were chosen at random for them. Then sentences that contained these words were retrieved from both the web corpus and the BCCWJ. The sentences for each word were ranked by difficulty using the proposed readability formula. The top (easiest), bottom (hardest) and middle sentences were selected from the ranked list of each word and displayed to the student. The native Japanese speaker participant was first asked to rank the sentences from the web corpus for each word. Then they were asked to rank the sentences from the BCCWJ for each word.

The participant's responses were taken to be the true ranking and were compared with the ranking obtained using the proposed readability formula. In total, 40 vocabulary words were used to rank 120 sentences. Once all of the rankings were collected, the correlation coefficient $(r)$ was calculated between the correct ranking and the ranking obtained by the proposed readability formula. The correlation coefficients for both are shown in Table 3.

\begin{table}[b]
  \caption{The correlation coefficients obtained in the second experiment}
  \label{table3}
\input{01table03.txt}
\end{table}


\subsection{Experiment 3}

This experiment evaluated the usability of the system. Nine learners of Japanese of all levels were asked to use the system with a focus being put on the context-based exercises. They were then asked to complete a short survey where they were asked several questions and then were able to write down their impressions of the e-learning system. This experiment was performed using the version of the system that uses the web corpus and not the BCCWJ. The results of the survey are shown in Table 4.

\begin{table}[t]
  \caption{Results obtained by the survey given in the third experiment}
  \label{table4}
\input{01table04.txt}
\end{table}

While the current version of the system can be regarded as being more effective for advanced learners of Japanese, we decided to include learners of all levels in this experiment because the final objective of developing this e-learning system is to support all levels of learners. Therefore, by including learners from all levels in this experiment we are able to not only evaluate the usability of the current version of the system, but also receive valuable feedback that will help shape the future development of the system.


\subsection{Discussion of results}

In the first experiment, the proposed readability formula performed very well. These results indicate that the proposed readability formula performs very well for sentences that contain the system's target vocabulary and kanji. In the second experiment the proposed readability formula yielded a fair correlation coefficient when used with both the web corpus and the BCCWJ. The results of that experiment showed a decrease in performance compared to the first experiment. We believe the main reason for this is that sentences from both corpora often contain words that do not exist in the JLPT.

The readability formula performs slightly better when used with the web corpus rather than the BCCWJ in the second experiment. One problem that was pointed out to us by a student during the second experiment was that one of the sentences used an obsolete hiragana character （ゐ） which caused them to rank the sentence as being difficult. This type of character is not in the JLPT so our readability formula is not able to correctly rank a sentence that contains them.

In the third experiment, the learners' opinions of the system were positive overall. All of the learners thought that the program was at least fairly easy to use. During the entire experiment there was only one word that a learner could not study with the system. This word was not parsed correctly by the morphological analyzer tool used in the system so no sentences with that word could be found.

Half of the learners felt that the sentences used in the e-learning system were at least fairly hard to understand. Three out of the four of these learners had only been studying Japanese for a year or less. This highlights the need for the system to be able to provide exercises that even learners with a low proficiency level can complete. However, even though some learners felt that sentences being used were above their level, almost all of them said that they would like to use this kind of system to study vocabulary words. This is a very encouraging result because it suggests these learners enjoyed using this type of system for studying vocabulary.

Comments received from the learners were mostly about specific parts of the interface that they would like changed or parts of the interface that did not work how they were suppose to. All of these issues will be addressed during the development of the next version of this system. One learner commented that the system was good for vocabulary, but that there might be some grammar that they did not understand in the sentences. This highlights the need for the system to provide support for grammar as well as vocabulary. Methods of supporting grammar are being investigated for the next version of the system.


\section{Future work}

Research and development for the next version of the system will largely focus on improving the way that the system selects sentences. Much time will be dedicated to investigating other factors that facilitate the selection of more appropriate sentences, and also filter out sentences that do not support learning. Furthermore, new kinds of exercises useful to both beginning and advanced learners of Japanese will also be developed in order to allow learners of all levels to successfully study Japanese vocabulary using this system.


\section{Conclusion}

In this paper the first version of an e-learning system that uses a corpus to generate context-based exercises was presented. The objective of developing this e-learning system is to assist English-speaking learners of Japanese with their studies of vocabulary and kanji for the JLPT. The different components of the e-learning system were explained and a readability formula was proposed as a way to select appropriate sentences for use in the context-based exercises. Use scenarios of the e-learning system were also described for both beginning and advanced learners with the system being most effective for advanced learners.

A web corpus was compared with the BCCWJ and while the larger BCCWJ provided better word coverage, the web corpus covers a majority of the target words even with its relatively small size. We argue that the level of coverage realized with the web corpus is acceptable for use in the current version of the e-learning system.

Three experiments were performed for the evaluation of the e-learning system. The first experiment using data from the JLPT yielded a good performance with a high correlation coefficient $(r = 0.95)$ while the second experiment using data from both the web corpus and the BCCWJ resulted in fair performance with correlation coefficients of $r = 0.775$ and $r = 0.700$ respectively. In the last experiment, results of a survey of nine learners of Japanese were positive overall with most of the learners saying that they would use the system to study vocabulary.

Development of the next version of this e-learning system will concentrate on improving the system's ability to select appropriate sentences for use in the context-based exercises as well as developing new exercises that can accommodate learners with a low proficiency level.


\acknowledgment

We would like to thank all of the students who took the time to participate in the evaluation of this e-learning system. Their comments were invaluable in helping shape future development of the system. We would also like to thank the reviewers for providing valuable feedback during the writing of this paper. Finally, we would like to thank the National Institute for Japanese Language for letting us use the BCCWJ.

\bibliographystyle{jnlpbbl_1.4}
\begin{thebibliography}{}

\bibitem[\protect\BCAY{Bransford et~al.}{Bransford et~al.
  }{2003}]{Book_Bransford}
Bransford, J.~D. et~al.\BEDS\ \BBOP 2003\BBCP.
\newblock {\Bem How People Learn: brain, mind, experience, and school}.
\newblock National Academy Press.

\bibitem[\protect\BCAY{Chall}{Chall}{1987}]{Article_Chall}
Chall, J.~S. \BBOP 1987\BBCP.
\newblock \BBOQ Two vocabularies for reading: recognition and meaning.\BBCQ\
\newblock In {\Bem The Nature of vocabulary acquisition}. Lawrence Erlbaum
  Associates.

\bibitem[\protect\BCAY{Fletcher}{Fletcher}{2004}]{Article_Fletcher}
Fletcher, W.~H. \BBOP 2004\BBCP.
\newblock \BBOQ Facilitating the compilation and dissemination of ad-hoc web
  corpora.\BBCQ\
\newblock In {\Bem Corpora and Language Learners}. John Benjamins Publishing
  Company.

\bibitem[\protect\BCAY{Hulstijn et~al.}{Hulstijn et~al.
  }{1996}]{Article_Hulstijn}
Hulstijn, J.~H. et~al. \BBOP 1996\BBCP.
\newblock \BBOQ Incidental Vocabulary Learning by Advanced Foreign Language
  Students: The Influence of Marginal Glosses, Dictionary Use, and Reoccurrence
  of Unknown Words.\BBCQ\
\newblock {\Bem The Modern Language Journal}, {\Bbf 80}  (3), \mbox{\BPGS\
  327--339}.

\bibitem[\protect\BCAY{{Japan Educational Exchanges and Services}}{{Japan
  Educational Exchanges and Services}}{2008}]{Web_JLPT}
{Japan Educational Exchanges and Services}.
\newblock \BBOQ JEES Japanese Language Proficiency Test Home.\BBCQ\
\newblock \Turl{http://www.jees.or.jp/jlpt/index.htm}.

\bibitem[\protect\BCAY{Joyce \BBA\ Srdanovi\'{c}}{Joyce \BBA\
  Srdanovi\'{c}}{2008}]{Inproc_Joyce}
Joyce, T.\BBACOMMA\ \BBA\ Srdanovi\'{c}, I. \BBOP 2008\BBCP.
\newblock \BBOQ Comparing Lexical Relationships Observed within Japanese
  Collocation Data and Japanese Word Association Norms.\BBCQ\
\newblock In {\Bem Proceedings of the workshop on Cognitive Aspects of the
  Lexicon (COGALEX 2008)}.

\bibitem[\protect\BCAY{国立国語研究所}{国立国語研究所}{2008}]{Book_BCCWJ}
国立国語研究所 \BBOP 2008\BBCP.
\newblock \Jem{現代日本語書き言葉均衡コーパス (BCCWJ)
  モニター公開データ（2008年度版）}.

\bibitem[\protect\BCAY{Mizuno et~al.}{Mizuno et~al. }{2008}]{Inproc_Mizuno}
Mizuno, J. et~al. \BBOP 2008\BBCP.
\newblock \BBOQ Construction of an Example Extraction System to Assist Japanese
  Reading Comprehension.\BBCQ\
\newblock In {\Bem Proceedings of the Workshop on Natural Language Processing
  that Supports Education and Learning}.

\bibitem[\protect\BCAY{Mondria \BBA\ {Mondria-De Vries}}{Mondria \BBA\
  {Mondria-De Vries}}{1994}]{Article_Mondria}
Mondria, J.-A.\BBACOMMA\ \BBA\ {Mondria-De Vries}, S. \BBOP 1994\BBCP.
\newblock \BBOQ Efficiently Memorizing Words with the Help of Word Cards and
  ``Hand Computer'': Theory and Applications.\BBCQ\
\newblock {\Bem System}, {\Bbf 22}  (1), \mbox{\BPGS\ 47--57}.

\bibitem[\protect\BCAY{Okuyama}{Okuyama}{2007}]{Article_Okuyama}
Okuyama, Y. \BBOP 2007\BBCP.
\newblock \BBOQ CALL Vocabulary Learning in Japanese: Does Romaji Help
  Beginners Learn More Words?\BBCQ\
\newblock {\Bem CALICO Journal}, {\Bbf 24}  (2), \mbox{\BPGS\ 355--380}.

\bibitem[\protect\BCAY{Sato et~al.}{Sato et~al. }{2008}]{Inproc_Sato}
Sato, S. et~al. \BBOP 2008\BBCP.
\newblock \BBOQ Automatic Assessment of Japanese Text Readability Based on a
  Textbook Corpus.\BBCQ\
\newblock In {\Bem Proceedings of the Sixth International Language Resources
  and Evaluation}.

\bibitem[\protect\BCAY{Shih et~al.}{Shih et~al. }{2008}]{Article_Shih}
Shih, M. et~al. \BBOP 2008\BBCP.
\newblock \BBOQ Research and trends in the field of e-learning from 2001 to
  2005: A content analysis of cognitive studies in selected journals.\BBCQ\
\newblock {\Bem Computers \& Education}, {\Bbf 51}  (2), \mbox{\BPGS\
  955--967}.

\bibitem[\protect\BCAY{Tateisi et~al.}{Tateisi et~al. }{1988a}]{Inproc_Tateisi}
Tateisi, Y. et~al. \BBOP 1988a\BBCP.
\newblock \BBOQ A computer readability formula of Japanese texts for machine
  scoring.\BBCQ\
\newblock In {\Bem Proceedings of the 12th Conference on Computational
  Linguistics}.

\bibitem[\protect\BCAY{Tateisi et~al.}{Tateisi et~al.
  }{1988b}]{Article_Tateisi}
Tateisi, Y. et~al. \BBOP 1988b\BBCP.
\newblock \BBOQ Derivation of readability formula of Japanese texts.\BBCQ\
\newblock {\Bem Information Processing Society of Japan SIG Note DPHI}, {\Bbf
  18}  (4).

\end{thebibliography}

\clearpage

\appendix
\section{Sample exercises using the web corpus}

\makeatletter
\newcommand{\tblcaption}[1]{}
\setlength\belowcaptionskip{8\p@}
\makeatother
  \tblcaption{Sample exercises when studying the word 行く with the web corpus}
  \label{table5}
\input{01table05.txt}

\section{Sample exercises using the BCCWJ}

  \tblcaption{Sample exercises when studying the word 行く with the BCCWJ}
  \label{table6}
\input{01table06.txt}

\clearpage

\begin{biography}

\bioauthor[:]{Gregory Hazelbeck}{
Gregory Hazelbeck received his B.S. in computer science from the University of Kansas in 2006, and M.S. degree from Keio University in 2008. Since 2008 he has been a Ph.D. student in the Graduate School of Science and Technology at Keio University. His research interests include natural language processing, computer-assisted language learning, and e-learning.
}

\bioauthor[:]{Hiroaki Saito}{
Hiroaki Saito received his Ph.D. degree in 1991. He is currently an associate professor at Keio University. His research interests include natural language processing and music comprehension.
}
\end{biography}

\biodate








































\end{document}
