\documentclass[english]{jnlp_3.5}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%\usepackage{multirow}
\usepackage{booktabs}
%%%%\usepackage{comment}
%%%%\usepackage[export]{adjustbox}
\newcommand{\linestack}[1]{
    \def\arraystretch{0.9}\begin{tabular}[c]{@{}c@{}} #1 \end{tabular}}
\DeclareMathOperator*{\argmax}{arg\,max}
%%%%\usepackage{framed}
\usepackage{algpseudocode}
%%%%\newcommand{\emoimage}[3][width=10pt]{
%%%%\includegraphics[scale=0.1,trim=0.1cm 0.1cm 0.1cm 0.1cm]{image/emojis/#2_#3.pdf}
%%%%}
%%%%\newcommand{\emoji}[2]{\begin{minipage}{11pt}\emoimage[width=12pt]{#1}{#2}\end{minipage}}
    \newcommand{\emoji}[1]{\raisebox{-1pt}[0pt][0pt]{\includegraphics{#1}}}
    \newcommand{\emojifortable}[1]{\raisebox{-3pt}{\includegraphics{#1}}}
%%%%\usepackage{amsfonts}
\usepackage{arydshln}
%%%%\usepackage{framed}
%%%%\renewenvironment{framed}[1][\hsize]
%%%%   {\MakeFramed{\hsize#1\advance\hsize-\width \FrameRestore}}%
%%%%   {\endMakeFramed}
\newcommand{\quotes}[1]{``#1''}
\usepackage{algorithm}
%%%%\newcommand{\newcite}[1]{\citeauthor{#1} \shortcite{#1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



\Volume{29}
\Number{2}
\Month{June}
\Year{2022}
%%%%\Issuetitle{Special Issue: XXXXXXXX}

\received{2021}{10}{31}
\revised{2022}{2}{7}
%%%%\rerevised{2006}{5}{6}
\accepted{2022}{2}{28}

\setcounter{page}{467}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{General Paper}
\etitle{Joint Modeling of Emoji Position and Its Label for Better Understanding in Social Media}
\eauthor{Jingun Kwon\affiref{TokyoTech} \and Kobayashi Naoki\affiref{TokyoTech} \and Hidetaka Kamigaito\affiref{TokyoTech} \and \\
	Hiroya Takamura\affiref{TokyoTech}\affiref{AIST} \and Manabu Okumura\affiref{TokyoTech}} 
\eabstract{%
In social media, the frequent use of small images, called emojis, in posts has played a key role in recent communications. However, less attention has been paid to their positions in the given texts although users are known to carefully choose and place emojis that match their post. Exploring the position of emojis in texts is expected to enhance our understanding of the relationship between emojis and texts. In this paper, we propose a novel task of inserting an emoji at a position in a given tweet. We extend an emoji label prediction method considering the information of emoji positions, by jointly learning the emoji position in a tweet to predict the emoji label. Additional information on emoji position can improve the performance of emoji prediction. Human evaluations validate the existence of a suitable emoji position in a tweet. The proposed task makes tweets fancier and more natural. In addition, the emoji position can further improve the performance of irony detection compared to emoji label prediction. We also report the experimental results for the modified dataset, due to the problem of the original dataset for the first shared task to predict an emoji label in SemEval 2018.
}
\ekeywords{Social Media, Emojis, Irony Detection}

\headauthor{Kwon et al.}
\headtitle{Toward Better Understanding of Social Media with Emoji}

\affilabel{TokyoTech}{}{Department of Information and Communications Engineering, School of Engineering, Tokyo Institute of Technology}
\affilabel{AIST}{}{National Institute of Advanced Industrial Science and Technology (AIST)}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

The advent of emojis has dramatically changed the style of human communication. Using emojis in texts is interesting and attractive because the visual appearance of emojis can convey the meaning of the texts more effectively. Currently, emojis are widely used on social media platforms such as Twitter, Facebook, and Instagram. The prevalent use of emojis in the last few years, has made them the target of recent research. 

As reported by \citeauthor{Novak2015-SentimentOE} \citeyear{Novak2015-SentimentOE}, exploring the interplay between emojis and texts can provide clues for better natural language understanding (NLU) in social media datasets which are replete with emojis. In response to this report, \citeauthor{task-proposed} \citeyear{task-proposed} proposed a task to predict an emoji label given the text, which can contribute to various natural language processing (NLP) tasks such as information retrieval, social media content generation, and sentiment/emotion analysis. Subsequently, they proposed a method to solve this task using an LSTM-based tagger, achieving even higher performance than human prediction. 

In spite of the success of previous researches, the existing emoji prediction tasks predict only a single emoji given text without considering emoji position. \citeauthor{Novak2015-SentimentOE} \citeyear{Novak2015-SentimentOE} reported that social media users place emojis differently based on the type of emoji in a tweet. They analyzed 751 emoji labels by dividing them into positive, negative, and neutral types to show that the emoji type strongly correlates with the position in a text. This indicates that information on emoji position can be a good clue in facilitating emoji label prediction. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 1
\begin{table}[b]
\input{08table01.tex}
\hangcaption{Example tweets from the training dataset of SemEval 2018.
Our goal is to insert a suitable emoji into a suitable position in a given tweet.
In Examples 1 and 2, emojis are used instead of words and characters.}
\label{tab:task_discription}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Hence, it is necessary to consider the emoji label along with its position. Social media users do not simply place emojis at the end of a sentence. Automated systems for predicting an emoji along with its position consider the surrounding linguistic information to further enhance the aforementioned tasks in the fields of NLU and NLP. Table \ref{tab:task_discription} describes the difference between the previous task of emoji prediction and our proposed task of emoji insertion into texts. In the first and second examples, the position of the emojis, replacing a word or a character, is important and indispensable. In the third example, the position of the emoji necessitates examining the relationship between the emoji and multiple sentences. In considering the position of emoji labels, we first demonstrate that emoji position information can improve the performance of emoji label prediction by using the gold emoji position. 

The task of emoji prediction is then extended by proposing a novel task which inserts a suitable emoji into a suitable position in a given tweet. To solve this new task, bi-affine- and bi-LSTM-based models are proposed to jointly predict emoji labels and their positions without relying on any linguistic features. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[b]
\begin{center}
\includegraphics{29-2ia8f1.pdf}
\end{center}
\hangcaption{20 most frequent emojis used in our experiments. The black and grey bars indicate emotional and non-emotional types of emojis, respectively.}
\label{fig:emoji statistics}
\vspace{-1\Cvs}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Class-imbalanced data are a major concern in the field of emoji prediction \cite{task-proposed} because emojis with positive emotions are more frequently used on social media and dominate minor classes of negative and non-emotional emojis \cite{Novak2015-SentimentOE}. It is difficult to improve the results for imbalanced emoji classes. Figure \ref{fig:emoji statistics} displays the imbalanced frequencies of emojis obeying a power law distribution. To deal with the imbalanced frequencies of emojis, we also present a novel method of \textit{contextualized dynamic-smoothing} (CDS), which can be incoporated into our proposed models. 

  The problems of the original dataset for the shared task of multilingual emoji prediction in SemEval 2018 \cite{barbieri-etal2018semeval}, which sometimes mistakenly contains ``traces'' of emojis in tweets, led us to prepare a modified dataset for our experiments.\footnote{The detailed meaning of \quotes{traces} is described in Section 4.1.1.}

Experimental results on the modified dataset show that incorporating the gold emoji position can improve emoji label prediction performance. In addition, the proposed models which jointly predict an emoji label and its position outperform the baseline model for emoji label prediction, which does not consider emoji position in terms of F-1 scores. Specifically, the results of our models with CDS in solving the problem of imbalanced emoji frequency, display additional improvement in infrequent and non-emotional emojis. Human evaluations demonstrated that there is a suitable emoji position in a tweet. In addition, the proposed task of inserting an emoji in a tweet can actually make the tweet fancier and more natural compared to the current emoji prediction task, and emoji positions can further improve the performance of irony detection compared to current emoji prediction.

In our preliminary study \cite{kwon-etal-2021-making}, jointly predicting the emoji position and its label improved the performance of the modified datasets without relying on external information. However, it is still unclear whether such hidden ``traces'' of emojis in the original datasets influence the performance of emoji label prediction. Thus, further  experiments were conducted using the proposed models on original datasets to judge how hidden \quotes{traces} affect the performance of emoji label prediction. As a result of additional investigation, it was concluded that such hidden \quotes{traces} influence the performance of emoji label prediction.

The remainder of this paper is organized as follows: in Section 2, works related to recent studies on emojis are described; in Section 3, the proposed models of jointly predicting emoji labels and  positions are explained; in Section 4 the experimental settings, including the baseline model as well as the original and modified datasets, are described; in Section 5, automatic and human evaluations are used to compare the baseline and proposed models; in Section 6, the effect of hidden \quotes{traces} is analyzed using original datasets; Section 7 concludes the paper.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}

Social media platforms contain various emotion expression methods, such as emojis and kaomojis \cite{kwon}. Recently, emojis have engaged a large proportion of the social media with the wealth of information they provide. Emoji information can enhance the quality of social media datasets for researches such as sentiment, irony, emotion, and sarcasm analysis \cite{felboetal2017-using,naacl-description}. Although emojis can be interpreted differently based on the social media platform \cite{emoji-wrong1}, their usage is similar in multiple countries \cite{Barbieri2016-howcosmopolitan}, meaning that they can be predicted in different languages, such as English, Spanish \cite{barbieri-etal2018semeval}, and Italian \cite{DBLP:conf/evalita/RonzanoBPPC18}. Emoji prediction can be enhanced by incorporating not only texts but also photos into the Instagram dataset \cite{multi-modal} and by using seasonal information such as spring, summer, autumn, and winter \cite{season-emoji}. Emoji prediction can directly improve the task of sentiment analysis \cite{Epred-Senti} and  can be used in dialogue systems to recommend a suitable emoji \cite{Xie2016-NeuralER}. Using emojis in chatbot systems is effective in attracting users, specifically in conversations for mental well-being  \cite{Fadhil2018-chatbot}. Recently, \citeauthor{ma-etal-2020-multi} \citeyear{ma-etal-2020-multi} released an emoji label prediction dataset with passage-level multiclass/multi-label and aspect-level multiclass annotations.

In this work, the prediction of both the emoji label and its position is in accordance with the sequential labeling tasks. In dependency parsing, a bi-affine dependency label classifier was proposed to study the relationship between two words, achieving state-of-the-art performance without external resources \cite{biaffine-layer}. Currently, bi-affine layers, owing to their effectiveness, have been widely adopted in various tasks, such as relation extraction \cite{biaffine-used2}, mention detection, and clustering \cite{biaffine-used1}. Because the performance of bi-affine layers stands out in the aforementioned tasks, here they are employed to learn emoji positions to predict emoji labels.  


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Models}

In this section, the proposed models are introduced for joint prediction of an emoji and its position in each tweet. Figure \ref{fig:structure} presents an overview of the proposed model. The joint prediction procedure is as follows:

\begin{enumerate}
{\item \textbf{Bi-LSTM}} layer receives a sequence of word embeddings and converts them to the hidden states $h_t$, at each time step. 
{\item \textbf{Selective Gate}} layer converts $h_{t}$ to $h^{'}_{t}$ by injecting sentence-level information into $h_{t}$ using a sigmoid gate.
{\item \textbf{Position Module}} calculates each position score between words $w_{t}$ and $w_{t-1}$ to predict a suitable position by applying a linear transformation to the concatenation of $h_{t}$ and $h_{t-1}$ or bi-affine transformation to the two vectors. 
{\item \textbf{Label Module}} calculates the probability of each emoji at the predicted position using linear projection. To address the problem of imbalanced emoji label frequencies, the proposed contextualized dynamic smoothing (CDS) is applied to the projection.
\end{enumerate}
The details of each module are explained in the following subsections.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 2
\begin{figure}[t]
\begin{center}
\includegraphics{29-2ia8f2.pdf}
\end{center}
\hangcaption{Model structure. The bi-affine layer is used to predict the position of an emoji. Then, the predicted emoji position is used to predict the label of the emoji with CDS.}
\label{fig:structure}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Bi-LSTM}

The proposed models employ Bi-LSTM networks \cite{bilstm-network} to capture the forward and backward context of the input text such that given words of the input text $S=\{w_0 ,w_1, w_2, ..., w_N, w_{N+1}\}$, which are encoded as word embeddings $\{e_0 ,e_1, e_2, ..., e_N, e_{N+1}\}$, $w_0$ and $w_{N+1}$ indicate ``\verb|<s>|'' and ``\verb|</s>|'' tokens for the beginning and end of a sentence, respectively. These word embeddings are then fed into the forward and backward LSTMs and converted into forward and backward hidden states $\overrightarrow{h}_t$ and $\overleftarrow{h}_t$, respectively. The concatenated hidden state $h_t = [\overrightarrow{h}_t; \overleftarrow{h}_t]$ constitutes the output of Bi-LSTM at each time step.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Selective Gate}

To enhance the information of each hidden state for the eligible emoji label and its position,  a selective module \cite{zhouetal2017-selective} was incorporated on top of the Bi-LSTM networks.
Specifically, each $h_t$, which is the output of Bi-LSTMs, is passed through the selective gate, $sGate_t$. The $sGate_t$ decides the importance for each hidden state by considering the entire sentence information that consists of $s=[\overleftarrow{h}_0;\overrightarrow{h}_{N+1}]$, where $\overleftarrow{h}_0$ is the first backward hidden state and $\overrightarrow{h}_{N+1}$ is the last forward hidden state. The formula for $sGate_t$ is defined as follows:
\begin{align}
    sGate_t &= \sigma (W_h h_t + W_s s + b), \\ 
    h_{t}^{'} &=  h_t \odot sGate_t,
\end{align}
where $W_h$ and $W_s$ are weight matrices; $b$ is the bias term; $\odot$ is a symbol for element-wise multiplication; and $\sigma$ is a sigmoid activation function for normalizing the ranges of gate outputs. The sequence of hidden states, that is, Bi-LSTM outputs, are passed to the selective gate. Then, a new sequence of embeddings, $\{h_{0}^{'}, h_{1}^{'}, h_{2}^{'}, ..., h_{N}^{'}, h_{N+1}^{'}\}$, which consider the sentence information for each time step, are generated.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Position Module}

This module predicts emoji position $\hat {pos}$. The emoji position is predicted using the following equation:
\begin{equation}
   \hat {pos} = \argmax_{pos} {\rm score_p}(pos),
   \label{eq:predicted_position}
\end{equation}
where $pos$ denotes possible emoji positions. 
To calculate the position score ${\rm score_p}(pos)$, the following two networks are proposed: 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Simple Concatenation}\label{sec:concate}

The concatenation of two neighboring hidden states, $[h'_i, h'_{i+1}]$, are used to calculate each position score, as follows:
\begin{align}
        {\rm score_p} (pos) &= v_p t_i, \label{eq:position_score2}\\
        t_i &= ReLU(W [h'_i;h'_{i+1}]+b),
    \label{eq:position_score1}
\end{align}
where $W$ is the parameter matrix, and $b$ is the bias term; $v_p$ is a learnable vector for emoji position prediction; the ReLU is the activation function of the rectified linear unit  \cite{relu-layer}. Function score$_{\rm p}$ returns a scalar value for each possible emoji position. The candidate position of $pos$ for inserting an emoji is always $i-1$ because the emoji is inserted between words.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Bi-Affine}\label{sec:biaffine}

The simple concatenation approach can explain the interaction between the two hidden states $h'_{i}$ and $h'_{i+1}$. However, clues from a single state without any interaction with another state may also be informative for emoji position prediction. To facilitate the extraction of pure clues, an additional method is proposed using a bi-affine-based transformation of the hidden states $h'_{i}$ and $h'_{i+1}$, which contain only the terms with $h'_{i}$ or $h'_{i+1}$. Specifically, instead of using Equations (\ref{eq:position_score2}) and (\ref{eq:position_score1}), the bi-affine function is used to calculate each position score, as follows:
\begin{align} 
    {\rm score_p}(pos) &= {\rm biaffine}(h'_i, h'_{i+1}), \\
    {\rm biaffine}(h_l, h_r) &= 
        v_{l}^{T} h_l + h_l^{T} W_b h_r + v_{r}^{T} h_r,
\end{align}
where $v_{l}^{T}$ and $v_{r}^{T}$ are the learnable vectors; $W_b$ denotes the weight matrix.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Label Module}

As a downstream procedure, this module predicts an emoji label, $\hat{label}$. The emoji label is predicted using the following equation:
\begin{equation} 
    \hat{label} = \argmax_{label} {\rm score_l}(label),
\end{equation}
where $label$ refers to emoji class labels.  
The following two types of networks are used for modeling the score function, ${\rm score_l}(label)$, to predict an emoji label.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Linear Projection}

 A commonly used linear projection layer is adopted to map the simple concatenated hidden states in Equation (\ref{eq:position_score1}) to the emoji label space. To consider the predicted emoji position when predicting an emoji label, $\hat{pos}$ is applied to calculate the score function:
\begin{equation}
    \label{eq:label_prob}
    {\rm score_l} (label) = W_l t_{\hat{pos}} + b_l,
\end{equation}
where $W_l$ and $b_l$ are the weight matrix and bias term for emoji label prediction, respectively.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Contextualized Dynamic-Smoothing (CDS)}

As shown in Figure \ref{fig:emoji statistics}, the frequencies of emojis are imbalanced, which can result in difficulties in obtaining a better performance in emoji label prediction \cite{task-proposed}. To address this issue, a novel smoothing method based on left and right context vectors for emoji label prediction was proposed. In Equation (\ref{eq:label_prob}), the bias term $b_l$ for predicting an emoji label is shared in all emoji positions, regardless of the predicted emoji position $\hat{pos}$. It would be beneficial for the bias term to consider the eligibility value at each emoji position in predicting an emoji label. Hence, the contextualized bias term was calculated based on the information of each emoji position using bi-affine layers as follows:
\begin{equation} 
     b_{\rm dynamic} = h_{l}^{'T} W_d h'_r,
\end{equation} 
where $W_d$ denotes the weight matrix; $h'_l$ and $h'_r$ are used to obtain various bias values to reflect the effect of smoothing on the distribution of emoji classes as social media users place an emoji differently in a tweet based on the emoji type  \cite{Novak2015-SentimentOE}; $b_{\rm dynamic}$ was added to calculate the emoji label score as follows:
\begin{equation}
\label{eq:jl4}
    {\rm score_l}(label) = W_l t_{\hat{pos}} + b_{\rm dynamic}.
\end{equation}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Objective Function}

The objective function consisted of two cross-entropy losses, for the emoji label and its position. 
C and P indicate the number of labels and number of possible positions, respectively. To learn the parameters by backpropagating the two losses, the losses are separated using the hyperparameter $\lambda$:
\begin{align}\label{eq:weight}
     loss &= loss_{label} + \lambda \cdot loss_{pos}, \\
     loss_{label} &= -P(label), \\
     loss_{pos} &=  -P(pos), \\
     P(label) &= \frac{{\rm exp(score}(label))}{\sum_{\hat{label} \in C} {\rm exp(score}(\hat{label}))}, \\
     P(pos) &= \frac{{\rm exp(score}(pos))}{\sum_{\hat{pos} \in P} {\rm exp(score}(\hat{pos}))}.
\end{align}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experimental Settings}

In this section, the problems of the original datasets from the SemEval 2018 task are reported. Then, the modified datasets used to predict emoji labels and their positions are described.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Datasets}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Original SemEval 2018 Datasets}

The SemEval 2018 task was the first shared task in predicting emoji labels with multilingual datasets: English and Spanish tweets \cite{barbieri-etal2018semeval}. Although good results were attained in predicting emoji labels using the systems discussed in the SemEval task, the released SemEval 2018 datasets had some problems. 

The trial and test datasets were composed of 50k tweets and divided into two files: target emoji labels and their corresponding input tweets for release. However, some English and Spanish input tweets contained hidden \quotes{traces} of the corresponding target emoji labels. Specifically, English input tweets in the test dataset with target emoji labels of \emoji{29-2ia8_411-1.pdf} and \emoji{29-2ia8_411-2.pdf} had such hidden \quotes{traces} of over 85\% and 98\%, respectively. The traces essentially consisted of two Unicode fragments of emojis: ``\verb|\ufe0f|'' and ``\verb|\ufe0e|'' indicating that the target emojis use the exact positions in the input tweets. Moreover, the released code, which splits the raw training dataset into two files of input tweets and target emoji labels in the preprocessing step, also had a problem. This resulted in hidden \quotes{traces} in the training dataset. The emoji \emoji{29-2ia8_411-3.pdf}, including the \quotes{traces}, is the most frequent in all training, trial, and test datasets. This may influence the performance of emoji label prediction. The hidden traces can be visualized by converting the input tweets into the json-string format. Table \ref{fig:Errorfig} shows an example of the hidden \quotes{traces} in the SemEval 2018 dataset. Finally, duplicate tweets were found in the trial and test datasets; 19,291 of 50k tweets in the trial dataset were also in the training dataset; and 1,557 out of 50k tweets in the test dataset were also in the training dataset. In addition, these datasets have been widely used to study emoji prediction \cite{semevalused1-emnlp,semevalused3,semevalused2}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 2
\begin{table}[t]
\input{08table02.tex}
\hangcaption{Example tweets in English SemEval 2018 training and test datasets. The unicode trace is hidden at its exact position.}
\label{fig:Errorfig}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Modified Dataset}\label{sec:ModifedData}

To remedy the above problems, the shared English dataset from the SemEval 2018 task was modified.
The lack of information on emoji positions and the problems of hidden \quotes{traces} discouraged us from using the original trial and test datasets. Neither did we use the released code, which divided the raw training dataset into two files: input tweets and target emoji labels.  

Thus, only 450,021 tweets were retrieved from the training dataset. To use the exact positions of emojis, a few tweets with emojis appearing more than once (e.g., \emoji{29-2ia8_412-1.pdf} \textit{My baby bear} \emoji{29-2ia8_412-2.pdf}) in the extracted training dataset, were discarded. As a result, the final dataset comprised 357,305 tweets with 20 emoji class labels. Figure \ref{fig:emoji statistics} presents the statistics on emojis for the final dataset. Subsets were also prepared for the ten (250,747 tweets) and five most frequent emojis (180,767 tweets) using the same procedure as in the previous work for emoji label prediction \cite{task-proposed,multi-modal}. The subsets were randomly shuffled and split into three parts: training (80\%), validation (10\%), and test (10\%) datasets.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Baseline Model}\label{sec:base} 

\citeauthor{task-proposed} \citeyear{task-proposed} proposed the task of emoji label prediction with a simple emoji classifier. Their Bi-LSTM classifier receives an input tweet without an emoji label and predicts a suitable emoji label. In this study, this network was used as the baseline model to examine whether emoji position information can improve emoji label prediction performance. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Parameter Settings}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Glove Embeddings}

Glove embeddings are well-known pretrained vector representations of words \cite{glove-emb}. Glove embeddings were concatenated with initialized word embeddings to obtain each word embedding; 27 billion pre-trained word embeddings of 200 dimensions trained on the Twitter dataset were used. During the training and updating of parameters in the training process, the glove embeddings were fixed.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Initialization and Parameters}

The experimental setup was common to all models: baseline, simple concatenation, bi-affine, and CDS. One-layer stacked Bi-LSTMs with 256 hidden states were used. The Adam optimizer was employed with a learning rate of 0.001, betas of 0.9 and 0.999, and epsilon of $1.0 \times 10^{-8}$. The initialized learning rate of each parameter decayed with an exponential learning rate of $1.0 \times 10^{-4}$. Gradient clipping was performed with a maximum normalization of 5.0. Words that appeared fewer than 5 times were replaced by the ``\verb|<UNK>|'' token. The value of $\lambda$ in Equation (\ref{eq:weight}) was 0.2, which was estimated based on the performance of the average macro F-1 score in emoji label prediction on the validation dataset. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments and Evaluation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Emoji Insertion}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Emoji Label Prediction}

For evaluation, the metrics officially used in the SemEval 2018 task were employed, namely, precision, recall, and macro F-1 scores.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 3
\begin{table}[b]
\input{08table03.tex}
\hangcaption{Average F-1 scores for emoji label prediction and accuracy for emoji position prediction. The best scores in all the models except Upper Bounds are shown in bold. $\dagger$ indicates that the improvement from the Baseline is statistically significant by using the paired bootstrap resampling method \protect\cite{koehn-sig} ($p < 0.001$).}
\label{tab:avg statistics}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In Table \ref{tab:avg statistics}, the results of emoji label prediction for the baseline and proposed models are compared for the gold and jointly learned emoji positions, respectively. The baseline was described in Section \ref{sec:base}. Upper Bounds 1 and 2 used the gold emoji positions to predict emoji labels with and without applying CDS, respectively. The results for Upper Bounds 1 and 2 demonstrate that the information of emoji positions can improve the performance of emoji label prediction. Specifically, using gold emoji positions in the model of Upper Bound 1 improved average F-1 scores on the modified datasets by +3, +3.27, and +2.36 for the top 5, 10, and 20 emojis, respectively, compared with the baseline that does not consider emoji positions. Incorporating the proposed CDS to deal with the class-imbalance problem yielded further improvements of +1.22, +1.14, and +1.57 on each dataset for Upper Bounds 1 and 2.  

Joint learning models 1 and 2 predicted the position of emojis using simple concatenation and bi-affine layers, respectively. Joint learning models 3 and 4 incorporated CDS to predict emoji labels. Joint learning models without external resources outperformed the baseline model. Joint learning 4, which uses bi-affine layers to predict the position of emojis and applies CDS to predict emoji labels had the highest F-1 score, improving the average F-1 scores on the modified datasets by +2.77, +2.5, and +1.93 for the top 5, 10, and 20 emojis, respectively, compared to the baseline. 

Table \ref{tab:full statistics} displays the F-1 scores for the top 20 individual emojis for the baseline and Upper Bound 2. Applying CDS with the position information can improve the performance of predicting non-emotional emojis. The performance, as measured by the F-1 scores for the non-emotional emojis, \emoji{29-2ia8t4-7.pdf}, \emoji{29-2ia8t4-9.pdf}, \emoji{29-2ia8t4-13.pdf}, \emoji{29-2ia8t4-14.pdf}, \emoji{29-2ia8t4-16.pdf}, \emoji{29-2ia8t4-17.pdf}, and \emoji{29-2ia8t4-19.pdf}, increased by +7.7, +2.6, +3.5, +2.5, +5.3, +4, and +5.3, respectively, compared with the baseline model. In addition, incorporating CDS is effective for the information on emoji positions for infrequent emojis, \emoji{29-2ia8_511-1.pdf} (+2.5), \emoji{29-2ia8_511-2.pdf} (+5.3), \emoji{29-2ia8_511-3.pdf} (+10.2). CDS also helps the model predict \emoji{29-2ia8_511-4.pdf} and \emoji{29-2ia8_511-5.pdf} with F-1 scores of 5 and 3, respectively.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 4
\begin{table}[t]
\input{08table04.tex}
\hangcaption{F1 scores for emoji label prediction with top 20 emojis. Base indicates the baseline model, and UB2 describes the upper bound, and 2\% denotes the proportion of each emoji in the test dataset.}
\label{tab:full statistics}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Emoji Position Prediction}


As an evaluation metric, the accuracy of emoji position prediction was employed. There was no need to measure precision, recall, and F-1 scores because each tweet contained only one emoji, and the accuracy, precision, recall, and F-1 scores were always equal.

The baseline model predicts emoji labels without considering emoji positions, and the upper-bound models use gold emoji positions. Therefore,  the performance of predicting emoji positions was compared among the joint learning models. The average accuracies for emoji position prediction are presented in Table \ref{tab:avg statistics}. Following the same tendency as emoji label prediction, the highest accuracy was obtained for Joint Learning 4 on the top 5, 10, and 20 modified datasets. 

Table \ref{tab:quali analysis} shows example outputs of the baseline and our best model, Joint Learning 4. As analyzed by \citeauthor{task-proposed} \citeyear{task-proposed}, it is difficult for the baseline model to predict \emoji{29-2ia8t5-1.pdf} as it prefers to select more frequent emojis of similar semantics, \emoji{29-2ia8_512-1.pdf}. Our best model can resolve this issue. In the first example, the baseline model prefers to choose the more frequent emojis of similar semantics, \emoji{29-2ia8_512-2.pdf}. Instead of choosing the more frequent emojis, our best model successfully inserted the predicted emoji, \emoji{29-2ia8t5-1.pdf}, into the exact position. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 5
\begin{table}[t]
\input{08table05.tex}
\hangcaption{Examples of predicting emoji labels and their positions with our best model, Joint Learning 4, compared with the baseline model.}
\label{tab:quali analysis}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In the second example, using emoji positions was more effective when emojis were used instead of words. Specifically, when \emoji{29-2ia8_512-3.pdf} was used instead of the word \quotes{love} between the two words \quotes{the} and \quotes{is}, the baseline model predicted \emoji{29-2ia8t5-1.pdf} instead of \emoji{29-2ia8_512-4.pdf}. However, the position information obtained enabled the proposed model to consider the relationship between the two words. Our best model recognized the differences between \emoji{29-2ia8_512-5.pdf} and \emoji{29-2ia8t5-1.pdf}. The model successfully predicted \emoji{29-2ia8_512-6.pdf} and inserted it between two words: \quotes{the} and \quotes{is}. 

The third example shows that our proposed task can handle multiple sentences to predict emoji labels. Our best model successfully predicted a suitable position, which was the end of the tweet, to predict the emoji label \emoji{29-2ia8_512-7.pdf}. Thus, our proposed task, which considers a suitable emoji position in a tweet, can bridge the gap between emojis and texts in determining which emoji suits the text in the tweet.

Figure \ref{fig:ranks} shows average estimated probabilities of emoji labels and their positions. 
The label ``label'' and ``pos'' bars are the average estimated probabilities of emoji labels and their positions, as determined by the Joint Learning 3 model. The ``CDS label'' bar denotes the average estimated probabilities of emoji labels from the model Joint Learning 4. The emoji labels and their positions were sorted in the order of probabilities obtained from the softmax layers. The probabilities were then averaged for each rank of labels and positions. Note that only  the top 20 ranks for emoji positions that have the same size as the emoji labels are visualized.
The rank probability from Joint Learning 3 shows that the probability distribution of emoji positions is in accordance with emoji labels, indicating that the probability distribution is not uniform. Thus, emoji position is predictable because the emoji label is predictable \cite{task-proposed}. Moreover, the probability distributions of emoji labels from Joint Learning models 3 and 4 demonstrated that the proposed CDS successfully smoothed the estimated probabilities of emoji labels, whereby the probabilities of low-ranked emoji labels increased.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 3
\begin{figure}[t]
\begin{center}
\includegraphics{29-2ia8f3.pdf}
\end{center}
\caption{Rank-probabilities of emoji labels and their positions from the top 20 test dataset.}
\label{fig:ranks}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% algo 1
\begin{algorithm}[t]
\caption{~Construction of the human evaluation dataset}\label{algo}
%%%%\begin{algorithmic}[0]
    \begin{algorithmic}
\Require~~\\
Tweets {\rm{T}} = $\{T_1, T_2, ..., T_n\}$ \\
tweetNLP = part-of-speech tagger \\
Initialize the filtered tweets, {\rm{F}} = [ ]
\For{$i=1$ to $n$}
\State{$tagged \leftarrow {\rm{tweetNLP(T_i)}}$}
\State{$v \leftarrow {\rm{tagged.count('v')}}$, $o \leftarrow {\rm{tagged.count('o')}}$}
    \If{$v > $2 \bf{and} $o > 2$}
        \State{${\rm{F.append(T_i)}}$}
    \EndIf
\EndFor
\\
\Return {\rm{F}}
\end{algorithmic}
\end{algorithm}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Human Evaluation}

Social media users tend to choose emoji positions differently based on the sentiment of tweets, which can express positive, neutral, and negative emotions \cite{Novak2015-SentimentOE}. 
 An experiment was conducted to demonstrate that there is a suitable emoji position for individual emojis in the tweets. Another experiment was then conducted to compare the current and proposed tasks. 

To encourage and guide human evaluation, a filtering step was applied to harvest suitable tweets from the SemEval 2018 training dataset. Algorithm \ref{algo} describes the method used to filter the tweets. First tweetNLP, which is a part-of-speech tagger was applied for tweets that consider emoji information \cite{tweetnlp}; tweetNLP can annotate part-of-speech tags for emojis. Specifically, emojis are tagged with \quotes{v} (Verb) when they are used instead of a word (e.g., \emoji{29-2ia8_52-1.pdf} is frequently used instead of \quotes{love}). Thus, rules were inserted for cases with number of \quotes{o} (Pronoun) and \quotes{v} (Verb) tags  greater than two to extract tweets including an emoji used instead of a word and tweets with multiple sentences. 

To show that there exists a suitable emoji position, a dataset, \textbf{Gold (Inside)} was constructed by randomly selecting 1,000 tweets in which emojis are not located at the end of the filtered dataset. An \textbf{End} dataset was prepared by moving emojis to the end of the tweet from the \textbf{Gold (Inside)} dataset. Because the gold emoji position can be the end of the tweet, a \textbf{Gold (End)} dataset was also constructed by randomly selecting 500 tweets in which emojis are located at the end of the filtered dataset. A \textbf{Random (Inside)} dataset was also prepared by randomly relocating emojis in each tweet from the \textbf{Gold (End)} dataset. Subsequently, \textbf{Current (End)} and \textbf{Proposal} datasets were constructed by applying the baseline model (see. \ref{sec:base}) and our best model, Joint Learning 4, to the modified test dataset. Because the baseline model cannot insert an emoji into a tweet, a predicted emoji was attached to the end of a tweet. \textbf{Current (End)} and \textbf{Proposal} datasets consist of randomly selected 500 tweets after the filtering step. 

Humans were asked to select better tweets to demonstrate that a suitable emoji position exists, and our proposed task can contribute to making tweets fancier and more natural compared to the current emoji prediction task. After presenting two tweets from the \textbf{Gold (Inside)} and \textbf{End}, \textbf{Gold (End)} and \textbf{Random (Inside)}, and \textbf{Current (End)} and \textbf{Proposal} datasets, the human annotators were asked two questions: \quotes{Which tweet is more fancy?} and \quotes{ Which tweets are more natural?}. The crowd-sourcing platform \quotes{Mechanical Turk} was used to design the experiment. The same tweets were displayed to five human annotators.\footnote{The annotators had their own Twitter accounts and were familiar with emoji usage on social media. Each annotator evaluated a maximum of 100 tweets.} The final decision was based on majority agreement. 

Table \ref{tab:human} shows the results of human evaluations for the gold emoji and relocated emoji positions. When the gold emoji position is inside of tweets, it can make tweets fancier and more natural 90\% and 87\% of the time, respectively, compared to moving emojis to the end of tweets. In addition, when the gold emoji position is at the end of tweets, it can make tweets fancier and more natural 73\% and 78\% of the time, respectively, compared to randomly relocating emojis in tweets. Thus, individual emojis are not always located at the end of tweets and there exists a suitable emoji position in tweets. To assess the reliability of the agreement between the human annotators and their decision, a statistical measure called Fleiss' Kappa \cite{kappa} was used. Kappa scores of 0.52 for Fancy and 0.68 for Natural were obtained for \textbf{Gold (Inside)} and \textbf{End} datasets, which indicate moderate and substantial agreement, respectively. In addition, Kappa scores of 0.41 for Fancy and 0.26 for Natural were obtained for \textbf{Random (Inside)} and \textbf{Gold (End)} datasets, which indicate moderate and fair agreement, respectively. Thus, considering emoji position is necessary to understand social media texts. As can be seen in Table \ref{tab:comparetask}, the proposed task can actually make tweets fancier and more natural compared to the current task.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 6
\begin{table}[t]
\input{08table06.tex}
\caption{Human evaluation of gold emoji and relocated emoji positions.}
\label{tab:human}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 7
\begin{table}[t]
\input{08table07.tex}
\caption{Human evaluation between current task and proposed task.}\label{tab:comparetask}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Irony Detection}

To validate the effectiveness of considering emoji position in understanding social media texts, a downstream task was conducted for irony detection. The 2-class (ironic or non-ironic) and 3-class (ironic by clash, other, or not ironic) raw irony detection datasets released by SemEval 2018 Task 3A and Task 3B were used, respectively,  \cite{van-hee-etal-2018-semeval}. The macro-average F1 score was used to evaluate the performance in accordance with previous work. Because the raw datasets were provided with training and test sets only, the training dataset was randomly shuffled and split into two parts: training (80\%) and validation (20\%). For each raw dataset, Emoji w/o position and Emoji w/ position datasets were constructed by inserting an emoji using four baseline and Joint Learning 4 models. Because baseline cannot insert an emoji, a predicted emoji was simply attached to the end of the sentence. Thus, three datasets of the released \textbf{raw}, constructed \textbf{Emoji w/o position}, and \textbf{Emoji w/ position} were used for irony detection. 

Table \ref{tab:irony} shows the results. The first block shows the reported scores of the top 2 previously submitted systems on the SemEval 2018 Task 3A and 3B. The second best model, NTUA-SLP based on Bi-LSTM with the self-attention network was used. The second block shows the re-run performances using the NTUA-SLP model\footnote{\url{https://github.com/cbaziotis/ntua-slp-semeval2018/}} with \textbf{raw}, \textbf{Emoji w/o position}, and \textbf{Emoji w/ position} datasets. Using the \textbf{Emoji w/o position} dataset improved the performance of irony detection compared to using the \textbf{raw} dataset for both task 3A and 3B. In addition, using \textbf{Emoji w/ position} dataset on the NTUA-SLP model outperformed \textbf{Emoji w/o position} dataset and achieved better performance compared to the best models of THU-NGN and UCDCC, in both tasks 3A and 3B. This improvement can be explained by the fact that social media users place an emoji carefully to convey that the meaning and position of emoji is effective in understanding social media texts.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 8
\begin{table}[t]
\input{08table08.tex}
\hangcaption{The results of irony detection. The left and right tables show the performance on SemEval 2018 task 3A (2-class) and 3B (3-class), respectively.}\label{tab:irony}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Original SemEval 2018 Datasets}

Table \ref{tab:semeval2018} shows the reported scores of the top 6 previously submitted systems that predict only emoji labels, using the original English dataset in SemEval 2018. These experimental results are attached to observe how hidden \quotes{traces} in the original datasets affect the performance of emoji label prediction. 
%%%%The experiments performed included the modified dataset. Because the first state-of-the-art system used support vector machines \cite{semeval1-sota1}, 
    The experiments performed included the modified dataset. Because the first state-of-the-art system used support vector machines (\mbox{\c{C}}{\"{o}}ltekin and Rama 2018)\nocite{semeval1-sota1}, 
it was incorporated into our model, Joint Learning 4 and the second state-of-the-art system, NTUA-SLP, which uses Bi-LSTM networks with an attention mechanism \cite{baziotis-etal2018ntuaslp}. In the calculation,
$r = \sum_{i=1}^{N} a_ih_i$, where $a_i =\frac{\exp(e_i)}{\sum_{t=1}^{N}\exp{(e_t)}}$, and $e_i = tanh(Wh_i+b)$.
where $W$ denotes the weight matrix; $h_i$ and $b$ are the hidden states and the bias term, respectively; $a_i$ indicates the attention weight ${\sum_{i=1}^{N} a_i = 1}$ and $a_i > 0$. To predict an emoji label, the final representation $r$ and weight matrix $W_a$ are applied to Equation (\ref{eq:jl4}) as follows:
\begin{equation}
    {\rm score_l}(label) = W_l t_{\hat{pos}} + W_a r+ b_{\rm dynamic},
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% table 9
\begin{table}[t]
\input{08table09.tex}
\hangcaption{Ranking of the submitted state-of-the-art systems in SemEval 2018 and our experimental results based on original and modified datasets. F-1, Prec., Recall, and Acc. indicate the average macro F-1, precision, recall, and accuracy for the English emoji label prediction, respectively.}
\label{tab:semeval2018}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

For the model \textbf{Bi-LSTM}, the Bi-LSTM classifier in Section \ref{sec:base} was used. \textbf{JL 4} is our best model, Joint Learning 4, that uses bi-affine layers to predict emoji positions and applies CDS to predict emoji labels. \textbf{NTUA-SLP+JL 4} is the model into which \textbf{JL 4} was incorporated \textbf{NTUA-SLP}.\footnote{\url{github.com/cbaziotis/ntua-slp-semeval2018}} Please note that \textbf{NTUA-SLP+JL 4} can jointly predict an emoji label and its position. In \textbf{NTUA-SLP+JL 4}, we followed the same preprocessing step as for model \textbf{NTUA-SLP}, that used ekphrasis\footnote{\url{github.com/cbaziotis/ekphrasis}} \cite{baziotis-etal2017datastoriessemeval} as a tweet preprocessor, and calculated word statistics and word embeddings from a large collection of unlabelled tweets \cite{baziotis-etal2018ntuaslp}.

In Table \ref{tab:semeval2018}, \quotes{-o} and \quotes{-m} indicate the use of the original training dataset and entire modified dataset for training the models, respectively. Although the original training dataset was composed of 500k tweets, only 450k tweets were used because the remaining tweets could not be retrieved. Thus, \quotes{357k} and \quotes{450k} indicate the size of the training dataset. To clearly show the effect of hidden \quotes{traces} in the original datasets, a dataset was prepared, the 357k-o, which is equal to the 357k-m dataset except for the presence of hidden \quotes{traces}. As can be seen from Table \ref{tab:semeval2018}, the average macro F-1 scores for Bi-LSTM-357k-o, JL 4-357k-o, NTUA-SLP-357k-o, and NTUA-SLP+JL 4-357k-o were 6.41, 6.12, 4.98, and 4.9 points higher than the ones for Bi-LSTM-357k-m, JL 4-357k-m, NTUA-SLP-357k-m, and NTUA-SLP+JL 4-357k-m, respectively. The hidden \quotes{traces} enhance the performance of emoji label prediction. Specifically, the F-1 score for an individual emoji \emoji{29-2ia8_6-1.pdf}, containing hidden \quotes{traces} most frequently was compared for the training, trial, and test datasets. JL 4-357k-m achieved an F-1 score of 47.32 for emoji \emoji{29-2ia8_6-2.pdf}, and JL 4-357k-o achieved an F-1 score of 81.06.

%%%%The higher F-1 score reported for NTUA-SLP (35.36) compared to that of NTUA-SLP-357k-o (34.41) was attributed to the difference in the training data size \cite{semeval1-sota1}. 
    The higher F-1 score reported for NTUA-SLP (35.36) compared to that of NTUA-SLP-357k-o (34.41) was attributed to the difference in the training data size (\mbox{\c{C}}{\"{o}}ltekin and Rama 2018). 
Please note that the JL 4 model with the 450k dataset could not be trained because it contains tweets with emojis appearing more than once, as described in Section \ref{sec:ModifedData}. Incorporating the best model, JL 4, into the second state-of-the-art model, NTUA-SLP, enhanced the performance of emoji label prediction in both the original and modified datasets. Specifically, jointly predicting emoji labels and their positions in the model of NTUA-SLP+JL4 improved the average F-1 scores by +0.56 and +0.64, on the original and modified datasets, respectively, compared with NTUA-SLP, which does not consider emoji positions. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

In this study, it was demonstrated that using emoji positions can boost emoji label prediction. Hence, a novel task was proposed to jointly predict an emoji label and its position by extending the recent task of emoji label prediction. 

The proposed model jointly predicted emoji positions and emoji labels using bi-affine. In addition, contextualized dynamic-smoothing (CDS) with bi-affine layers was proposed to solve the major problem of imbalanced label frequencies. The experimental results on modified datasets showed that emoji position information is important and can improve the performance of emoji label prediction, especially for infrequent and non-emotional emojis in the dataset without external resources. Human evaluation validated the existence of a suitable emoji position in a tweet, and our proposed task could make tweets fancier and more natural compared to current emoji prediction. Moreover, the emoji position is effective for irony detection. Because the original public datasets for emoji label prediction had problems, the influence of hidden \quotes{traces} was also evaluated. 

In the future, we will extend our proposed model to treat a dataset comprising multiple emojis in a tweet to improve the prediction performance. In addition, the proposed task will be applied to Japanese datasets, leveraging different character types of Japanese letters, such as Kanji, Hiragana, and Katakana for our method because they provide good clues for predicting and inserting emojis. For example, object emojis such as airplanes and cameras can be used instead of the corresponding kanji and katakana, respectively.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{08refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Jingun Kwon}{%
Jingun Kwon received his B.S from University of California, Los Angeles in 2018 and M.E from Tokyo Institute of Technology in 2020. He is currently a Ph.D. student of Department of information and Communications Engineering at Tokyo Institute of Technology. His current research interests are extractive and abstractive document summarization.
}
\bioauthor[:]{Naoki Kobayashi}{%
Naoki Kobayashi received his B.S from Ehime University in 2018 and M.E from Tokyo Institute of Technology in 2020. He is currently a Ph.D. student of Department of information and Communications Engineering at Tokyo Institute of Technology. His current research interests are discourse structure parsing of documents.
}
\bioauthor[:]{Hidetaka Kamigaito}{Hidetaka Kamigaito received his Ph.D. from Tokyo Institute of Technology, and is currently an assistant professor at Institute of Innovative Research, Tokyo Institute of Technology. His current research interests are in natural language processing with a specific focus on document-level text processing.}
\bioauthor{Hiroya Takamura}{%
Hiroya Takamura received his Ph.D. from the Nara Institute of Science and
Technology. He worked as a professor at the Tokyo Institute of Technology and is currently a research team leader at AI Research Center of National Institute of Advanced Industrial Science and Technology. His current research interests include natural language processing. 
}
\bioauthor[:]{Manabu Okumura}{%
Manabu Okumura was born in 1962. He received B.E., M.E. and Dr. Eng. from Tokyo Institute of Technology in 1984, 1986 and 1989 respectively. He was an assistant at the Department of Computer Science, Tokyo Institute of Technology from 1989 to 1992, and an associate professor at the School of Information Science, Japan Advance Institute of Science and Technology from 1992 to 2000. He is currently a professor at Institute of Innovative Research, Tokyo Institute of Technology. His current research interests include natural language processing, especially text summarization, computer assisted language learning, sentiment analysis, and text data mining.
}
\end{biography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\biodate



\end{document}
