\documentclass[english,other]{jnlp_3.3}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
%%%%\usepackage{hangcaption_jnlp}
%%%%\usepackage{array}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%\usepackage{comment}
%%%%\def\UrlBreaks{\do\A\do\B\do\C\do\D\do\E\do\F\do\G\do\H\do\I\do\J\do\K\do\L\do\M\do\N\do\O\do\P\do\Q\do\R\do\S\do\T\do\U\do\V\do\W\do\X\do\Y\do\Z\do\[\do\\\do\]\do\^\do\_\do\`\do\a\do\b\do\c\do\d\do\e\do\f\do\g\do\h\do\i\do\j\do\k\do\l\do\m\do\n\do\o\do\p\do\q\do\r\do\s\do\t\do\u\do\v\do\w\do\x\do\y\do\z\do\0\do\1\do\2\do\3\do\4\do\5\do\6\do\7\do\8\do\9\do\.\do\@\do\\\do\/\do\!\do\_\do\|\do\;\do\>\do\]\do\)\do\,\do\?\do\'\do+\do\=\do\#}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{28}
\Number{3}
\Month{September}
\Year{2021}
%%%%\Issuetitle{「　　　　」特集号}


\setcounter{page}{881}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{Society Column}
\etitle{When Creative AI Meets Conversational AI}
\eauthor{Xianchao Wu\affiref{Author_1}} 

\headauthor{Wu}
\headtitle{When Creative AI Meets Conversational AI}

\affilabel{Author_1}{}{NVIDIA, xianchaow@nvidia.com}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Following the rapid developing of artificial intelligence (AI) boosted by deep neural networks, we are  wondering how far is it going for AI to be really \emph{creative}. We know that human beings are intelligent in terms of creating novel things from scratch. Art can be selected as one type of \emph{Turing-Test}: given two paintings, can we distinguish which one is painted by AI and which one is by a real-world person? This same question stands for music creating, poem writing, singing and so on. We further ask ourselves, can we be involved in the whole AI creation process? Say, we communicate with AI to exchange ideas of what's the next painting topic or genre, or what's the next movement of the music. Through these interactive AI creation processes, we hope to inspire AI and be inspired by AI, just alike we learn from Alpha-Go\footnote{\url{https://deepmind.com/research/case-studies/alphago-the-story-so-far}} after it learnt from us first and then by itself through self-supervised learning. For Go gaming, we know the rules of win or lose. However, for AI creation, evaluation metrics are subjectively defined in creative ways.

In this year's (27th) Natural Language Processing (NLP 2021) conference, we proposed and organized the first workshop named ``when creative AI meets conversational AI'',\footnote{\url{https://sites.google.com/view/cai-workshop}} or, briefly ``CAI+CAI=CAI''.$^2$ In this one-day workshop, there are two technical papers and eight invited talks. These two technical papers respectively and interestingly cover one direction of text-to-image leveraging DM-GAN \cite{DBLP:conf/cvpr/ZhuP0019} and ManiGAN \cite{LQLT2020} for bird image generation \cite{cai2021-gamar} and its reversed direction of image-to-text by visual-text integrated Transformer \cite{cai2021-mori} for story generation. The eight invited talks cover most major directions of creative AI and conversational AI, such as voice conversion \cite{DBLP:journals/corr/abs-2005-07884}, image generation \cite{DBLP:journals/corr/abs-2102-07074}, contents retrieving \cite{DBLP:conf/wsdm/YuJJJY019-haitaoyu}, fine-art paintings \cite{DBLP:conf/eccv/HuckleGN20-art}, image-to-image translation \cite{jiaxian-guo-open-review-2021}, AI painting, music generating and singing \cite{wu2020transformerxl}, poem-writing \cite{nlp2017-haiku-wuxianchao}, and multi-modal learning for medical and healthcare applications \cite{MA2021107332,covid19-peiying-ruan}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Motivation of This Workshop}

Our workshop is inspired by the following facts. Creative AI, training \emph{generative} deep neural networks for NLP (such as poems, Haiku, stories), image (such as painting, animation), and speech (such as classic and popular music generation, singing), has achieved impressive milestones during recent years, thanks to deep neural networks such as attentive encoder-decoder architectures alike Transformers \cite{NIPS2017_3f5ee243}, generative-discriminative frameworks alike GANs \cite{NIPS2014_5ca3e9b1} and self-supervised encoders alike VAEs \cite{DBLP:journals/corr/abs-1906-00446}. In industry, conversational AI products such as Apple's Siri, Microsoft's Cortana, Google Home, Amazon's Alexa, XiaoICE \cite{zhou-etal-2020-design}, Rinna \cite{rinna-xianchaowu} that support text and speech based multi-modal communication between chatbots and human beings, have obtained millions of users in Japan and billions of users globally. 

In order to construct the strong \emph{persona} of conversational AI products, chatbots are enhanced to be able to interactively write poems, create songs, sing, and even tell stories, through multi-turn communications with end users. Furthermore, QA-style and IR-oriented chatbots of general domains and vertical domains such as finance, healthcare and even emotional pure-chatting are also requiring generative, creative and explainable AI models to support the multi-modal and multi-turn interactions with human beings. 

In this workshop, we were aiming at collecting, sharing, and discussing state-of-the-art research on creative AI and conversational AI, empowered by large-scale open datasets, open-source architectures, and distributed GPU platforms. Most importantly, with creative AI combined with conversational AI, we are aiming at bringing AI to help assisting under-represented groups' learning and communicating with the real world, such as interactive music therapy, children's painting guiding, emotional caring for social phobia and elderly cognition guarding. These goals derive our current and future CAI$^2$ workshops.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Roadmap of Technical Directions}

Figure \ref{figure:cai-single-combine-examples} illustrates three major directions of creative AI,\footnote{I also gave a same title speech in GTC 2021: S31384 \texttt{https://gtc21.event.nvidia.com/media/When\%20Creative \%20AI\%20Meets\%20Conversational\%20AI\%20\%5BS31384\%5D/1\_1nr6o531}} text, image and speech and further their combinations. First, GANs and generative large-scale pretrained language models such as GPT-3 have been employed for text generating. The up-left corner also shows a Haiku generated by Rinna \cite{nlp2017-haiku-wuxianchao}. Second, GANs and visual-transformers have been utilized for \emph{controllable} image generations. There are eight images for kerchiefs generated by inspiring customers' individual favours in the middle-left corner. Third, music and speech directions are of enormous market scale where AI has been investigated for piano generating that designed time-valued note tuples as inputs to transformer-xl \cite{dai-etal-2019-transformer} with joint pitch/velocity/melody learning \cite{wu2020transformerxl} and singing voice synthesis (e.g., popular music, Peking opera) using non-autoregressive frameworks such as Transformer's encoders following fastspeech \cite{lu2020xiaoicesing}, as shown in the bottom-left corner of Figure \ref{figure:cai-single-combine-examples}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[b]
\begin{center}
\includegraphics{28-3ia22f1.pdf}
\end{center}
\caption{Illustration of creative AI's single/combined directions.}
\label{figure:cai-single-combine-examples}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

These three directions can be combined together for multi-modal or cross-modal AI creation. First, text and image can be combined for directions such as text-guided painting. In the example given in the up-right corner of Figure \ref{figure:cai-single-combine-examples}, the input text is ``\emph{urbanization in China}'' and there are six paintings respectively drawn by historically famous artists such as Franz Marc, Rembrandt, Edvard Munch, Emile Bernard, van Gogh, and Gauguin. It is interesting to ask artists that lived centuries before to draw pictures of modern topics. In addition, the painting process is designed to be interactive: users are encouraged to keep guiding every time a painting is drawn by AI so that the images can be kept updating in multi-turns. This is helpful for users to better understand AI's painting process and for beginners to practice following hints from AI. Second, text and voice can be combined so that we can train an AI singer who can sing by given lyrics and melody and chord progress are generated on the fly \cite{10.1145/3219819.3220105}. The example given in right-middle corner is to ask AI to generate singing for a lyric sentence ``\emph{I and you}'' \cite{lu2020xiaoicesing}. Third, by combining images and voices following timeline, concerts have been hosted by AI singers or virtual singers, considering that there are a list of famous virtual singers.

Besides conversational AI products developed by a list of AI companies, there are open-source developer-oriented conversational AI platforms, such as NVIDIA's Jarvis\footnote{\url{https://developer.nvidia.com/nvidia-jarvis}} which includes 35 pretrained models of speech recognition, NLP (NER, QA, user intention classification, slot filling), and speech synthesis functions. Jarvis is a fully accelerated application framework for building multi-modal conversational AI services that use an end-to-end deep learning pipeline. Developers at enterprises can easily fine-tune state-of-the-art models on their data to achieve a deeper understanding of their specific context and optimize for inference to offer end-to-end real-time services that run in less than 300 milliseconds and delivers 7x higher throughput on GPUs compared with CPUs. Combining creative AI with these open-source conversational AI platforms can possibly save researchers and developers time for idea and product implementing. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

Human being constructed modern industrialized world since we are creative and converse with each other for cooperation. Now, we are empowering AI systems to be more and more creative so that we can cooperate with them as well through numerous multi-modal channels. Assisted by AI \underline{a}lgorithms, \underline{b}ig data and \underline{c}loud computing (ABC), CAI$^2$ is absorbing more focus and is supposed to contribute continually and impressively to a better society.   


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

The author would like to acknowledge the large contributions of Dr.~Gang Niu and Dr.~Lin Gu from RIKEN, Assistant Professor Haitao Yu from University of Tsukuba, Dr.~Yi Zhao (formerlly) from NII, Dr.~Yulan Yan from IBM and my colleagues Dr.~Peiying Ruan, Mana Murakami and Khanh Vo Duc from NVIDIA. We also express our thankfulness to the technical paper authors and the invited speakers.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{jnlpbbl_1.6}
\bibliography{22refs}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Xianchao Wu}{%
received his Ph.D. degree in Computer Science from The University of Tokyo in 2010, and a master's and bachelor's in Computer Science from Jilin University in 2004 and 2001, respectively. He also has an MBA degree of finance from Hitotsubashi University in 2020. He is currently a senior solution architect and data scientist at NVIDIA. He worked at Baidu until 2015 and then Microsoft until 2020. His research interests include large-scale pretrained language models, conversational AI, creative AI and financial NLP. He is an inventor and co-inventor of more than 90 patents.
}

\end{biography}

%%%% \biodate   %%%% Following settings will be edited by editors.


\end{document}





