\chapter{Imaging Novecento. A Mobile App for Automatic Recognition of Artworks and Transfer of Artistic Styles}\label{chap:imaging-900}
{

\begin{quote}
\textit{
Imaging Novecento} is a native mobile application that can be used to get insights on artworks in the \dquote{Museo Novecento} in Florence, IT. The App provides smart paradigms of interaction to ease the learning of the Italian art history of the 20$^{th}$ century. \emph{Imaging Novecento} exploits automatic approaches and gamification techniques with recreational and educational purposes. Its main goal is to reduce the cognitive effort of users \emph{versus} the complexity and the numerosity of artworks present in the museum. To achieve this the App provides automatic artwork recognition. It also uses gaming, in terms of a playful user interface which features state-of-the-art algorithms for artistic style transfer. Automated processes are exploited as a mean to attract visitors, approaching them to even lesser known aspects of the history of art.
\footnote{This Chapter has been published as \dquote{Imaging Novecento. A Mobile App for Automatic Recognition of Artworks and Transfer of Artistic Styles} in \textit{Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection: 6th International Conference, EuroMed 2016, Nicosia, Cyprus, October 31 -- November 5, 2016, Proceedings, Part I, pp.~781-791, 2016}~\cite{Becattini2016}.}

\end{quote}


\section{Introduction}

Modern museums can provide new paradigms for experiencing artworks. Thanks to the technological development, novel initiatives include pervasive uses of tech to create interactive experiences for visitors throughout a museum. However, making content relevant and appealing through these modern technologies is a difficult problem, requiring more and more interactivity as the audience is shifting towards a `multimedia point of view'. 
Moreover, while the massive amount of available artworks constitutes a huge resource for education and recreation purposes, it can also be a cognitive burden for visitors.

The cognitive process related to learning has been an active subject of study in recent decades. According to cognitive load theory, learners must cope with a certain level of cognitive effort to process new information~\cite{Park2015}. In this regard, multimedia education, defined as \dquote{presenting words and pictures that are intended to foster learning}~\cite{Moreno2003}, can be an effective remedy because it facilitates the activation of sensory and cognitive perceptions (e.g. visual and notional memory), avoiding visitors from information overloading. This can also be reinforced by gamification, that is the use of playful experience to help a user find personal motivations and engagement with serious content~\cite{Game2015}. This combination can enhance the visitor's involvement and further lower its cognitive effort. 
Using gamified applications, museum visitors have the opportunity to feel the emotion of a game, share results with friends on social networks or become part of a game community~\cite{Negrusa2015}.   
This aspect of learning through gaming is even more valuable in the context of the \dquote{Bring Your Own Device} (BYOD) approach \cite{ballagas-wude2004} that allows on demand access to digital content on personal devices. The BYOD approach and gamification have been identified in the NMC Horizon Report 2015 to be increasingly adopted by museums in one year's time or less for mobile and online engagement~\cite{Horizon2015}. 
In this Chapter we report our experience in embedding these concepts into \emph{Imaging Novecento}, a system built around a mobile application developed for the museum \dquote{Museo Novecento} in Florence, IT. We aimed at improving the learning process of the visitors by exploiting a simple gamification paradigm, and at reducing visitors cognitive load. To this end, we also developed a state-of-the-art computer vision system that is able to 1) recognize artworks from photos; 2) apply their style to user photos. 

\subsection{The Museo Novecento in Florence and Innovecento} \label{sec:museo}

The \dquote{Museo Novecento} in Florence, IT, is a museum opened on June 24, 2014. The museum is dedicated to the Italian art of the 20$^{th}$ century and offers a selection of about 300 artworks distributed in fifteen exhibition halls on two levels. The venue is located in the former hospital of the \dquote{Leopoldine} in Piazza Santa Maria Novella. The museum has been an example of innovation since its genesis, thanks to the prompt adoption of the latest multimedia technologies.

In March 2015, in order to improve the visitor experience, the Municipality of Florence has published an open call ~\dquote{INNOVecento - Novecento Museum Innovation Lab} inviting companies and professionals to propose ideas and solutions based on ICT. 

Five companies specialized in technologies applied to cultural heritage have already responded to the call which, at the time of writing, is still open. 

As NEMECH, centre of competence of the Tuscany region in Italy, we proposed~\emph{Imaging Novecento}. The App features automatic recognition of artworks through the visitor's smartphone and automatic transfer of artistic styles from artworks. These styles can be applied to user images. 


\subsection{Motivations and design} \label{sec:specifics}

The target of the App is rather wide. Although \emph{Imaging Novecento} can be used by anyone (e.g. tourists and residents), during the design process we identified a specific audience. We mainly target the App towards people in a relatively young age (between 14 and 30 years old), more accustomed to digital technologies, open to technological innovation and to gamification. 

One of the main ideas of the App is to exploit the pervasiveness of mobile cameras in modern smartphones to reduce the cognitive effort required to museum visitors. In fact, despite themed rooms and the ubiquitous explanatory cards, users can still be overwhelmed by the great number of artworks present in the museum. Labels in museums can be very concise or, on the contrary, can be filled with lots of explanation, often generic, not highlighting salient features of individual paintings.

By using \emph{Imaging Novecento}, the visitor can take a picture of the artwork he is interested in. The App will automatically recognize the painting and provide related information. 
Another reason for the adoption of this automatic process is the resistance of museums' curators to place or attach additional materials, such as QR codes or BLE iBeacon \cite{he-iccse2015}, next to artworks. 

Furthermore, tourists and school groups are usually `hit-and-run' visitors who tend to rapidly forget or do not have the time to process the overload of information. To solve this issue,~\emph{Imaging Novecento} leverages a playful feature that employs state-of-the-art algorithms for transferring artistic styles from recognized artworks to user images. This is done using a gamification paradigm at the interface level. Gamification techniques have been proved to be useful in engaging students in the learning process, improving their skills and maximizing their long-term memory~\cite{de2014systematic}.

\subsection{Previous work}

Several previous works have addressed the problem of providing an engaging experience to museum visitors. Rapid technological development has led to the implementation of a lot of applications.
There are several active trends for virtual museums: immersive reality~\cite{malomo2015virtualtour,fineschi20153d}, natural interaction installations~\cite{fanini2015engaging,baraldi2015gesture}, mixed reality, mobile applications~\cite{chen2014multimedia,yoon2014making}. While they all offer increasing engagement of visitors, only recently studies on the effects of audience have been carried out \cite{pagano2015virtual,Horizon2015}. In particular, a recent audience study has been conducted on the case of the ``Keys to Rome'' international exhibition, hosted at the~\dquote{Imperial Fora Museum} in Rome in 2015, to assess the impact of these technologies on cultural heritage. The exhibition was made up of 11 digital installations and applications, installed in the museum~\cite{pagano2015virtual}. The study highlights some fundamental aspects that must be taken into account when designing applications for virtual museums: 1) the majority of museum visitors are tourists and school groups; 2) visitors generally require applications with an high level of interactivity, particularly on their mobile devices; 3) it is essential for the UX design to use metaphors of informal learning capable to stimulate attention, memory and engagement (e.g. through gamification) in visitors. 

\paragraph{Automatic artwork recognition}
Automatic artwork recognition is a long standing problem in applications for cultural heritage.
Descriptors such as SIFT and SURF have been used for years in order to address this task \cite{ruf2008mobile,temmermans2011mobile} due to their accuracy in recognizing paintings. Crowley and Zisserman \cite{crowley2014state} retrieve artworks finding object correspondences between photos and paintings by using a deformable part based method. 
More recent approaches for artwork recognition adopt Convolutional Neural Networks (CNN) as in \cite{anwer2016combining}, where a holistic and a part based representation are combined. Peng and Chen \cite{peng2015cross} exploit CNNs to extract cross-layer features for artist and artistic style classification tasks. Artistic style recognition is also performed in \cite{karayev2013recognizing} on two novel large scale datasets. 
Similarly to these works, we explore the use of CNNs features but we aim to obtain a global representation that is semantically meaningful and also capable of retaining low level visual content information.

Artwork recognition has also been used with wearable devices, as in \cite{baraldi2015gesture} where the user's position is jointly estimated with what he is looking at. 

\paragraph{Artistic style transfer}
Regarding the application of artistic style to photos, a lot of research has been done in the past. The problem of rendering a given photo in the style of a particular artwork is known in literature as a branch of non photorealistic rendering \cite{kyprianidis-tvcg2013}.
This class of works use texture transfer \cite{efros-cgit2001,xie-dimea2007} to achieve style transfer. These techniques are non-parametric and directly alter image pixels of the content image into pre-defined styles. Another direction of work focuses on the idea of separating style and content in order to `remix' them together in different configurations. First works were evaluated on much simpler images such as characters in different handwritings \cite{tenenbaum-neural2000} or images representing human body configurations \cite{elgammal-cvpr2004}. Only recently, the breakthrough paper from Gatys \etal \cite{Gatys-arxiv2015} showed the possibility of disentangling the content from the style of natural images by using a convolutional neural network based representation. 

The advantage of this approach is the capability of performing style transfer from any painting to any kind of content images. The approach was recently extended with a more advanced perceptual loss \cite{johnson-arxiv2016} and also applied to movies \cite{anderson-arxiv2016} by considering the optical flow.

\section{The System} \label{sec:system}

The system is composed by two main components: a mobile App and a computer vision system responsible to address the two tasks of automatically recognize artworks and apply artwork styles to user photos. 
The mobile App is used by the visitor in the museum and is the \emph{fulcrum} of the user interaction. Once installed by the user in his mobile phone, it allows to take pictures, deliver artwork information and request the style transfer to new photos. 
Due to the limited amount of computational power available on most mobile devices, the computer vision system is deployed on a scalable web server system that processes requests from the mobile App. Since the two tasks use quite different technologies, we discuss them separately in the following sections.

\subsection{The Mobile App}

~\emph{Imaging Novecento} has been developed as an Android application using Ionic~\footnote{\url{http://ionicframework.com/}}. Ionic is a framework, based on Sass and AngularJS, for building highly interactive native web apps through mobile-optimized HTML, CSS and JS components and tools.
~\emph{Imaging Novecento} is a contextual App that can be used exclusively inside the Museo Novecento in Florence. An information flyer of the App is delivered to the visitor at the ticket office. In the flyer there are a QR code, through which the visitor can download the App from the Google Play store, and the list of the artworks on which the App can perform the automatic recognition and style transfer processes. The list comprises a selection of twenty artworks for which the museum's curators have provided multimedia materials. The App interface (Fig.~\ref{mobile-app}) is quite simple and is organized in two main views: 1) the Camera View and 2) the Artwork Details view.

\begin{figure}[!ht]
  \centering
    \includegraphics[width=1\textwidth]{figs/mobile-app.jpg}
     \caption{~\emph{Imaging Novecento} in action: 1) the user takes a picture of an artwork; 2) the artwork is recognized and insights are shown; 3) the user selects a photo from his own gallery in order to apply that artwork style and to share the results on social networks.}
     \label{mobile-app}
\end{figure}

The Camera View allows the visitor to frame one of the artworks on the list in order to have it immediately recognized by the automatic system. Proper feedback is given in case the recognition is not successful. Once the artwork is recognized, the Artwork Details view is activated. In this view, exhaustive but concise information about the author, the history of the artwork and its artistic style are given. 
An infographic is presented to the user. It works as a \dquote{call to action} for enabling the transfer of the recognized painting style to a photo from the user's device gallery. The infographic provides an animated preview that shows the result of the artistic style transfer on a predefined picture.
After the image has been successfully uploaded, the remote process for style transfer is performed. The result of the elaboration is then sent to the user in a few minutes by email. The image has a resolution of 900px wide preserving the original image aspect ratio and can be shared on the most popular social networks (e.g. Facebook).


\subsection{Automatic Artwork Recognition}

Artwork recognition is performed through a Python web server with a REST interface. The server processes the image and returns the ID of the recognized painting. The recognition step combines modern deep features with classical Support Vector Machines (SVM) in order to classify photos of paintings.
Image features are extracted using a deep convolutional neural network (CNN), and are then evaluated using a set of classifiers, one for each recognizable artwork.
The neural network we adopted is the Caffe reference model~\cite{jia2014caffe}, fine-tuned for style recognition using the FlickrStyle dataset~\cite{karayev2013recognizing}\footnote{the network is available online at~\url{http://caffe.berkeleyvision.org/gathered/examples/finetune_flickr_style.html}}.

In order to obtain a representation which is at the same time semantically meaningful and capable of retaining low level visual content information, we extract image features from an intermediate level of the network. In particular, we adopt the \textit{pool5} feature map, the latest one before the fully connected (FC) layers of the CNN. In fact, FC layers trade spatial information for a more semantic representation, which is highly coupled with the task and with the visual domain on which the network has been trained. This choice is therefore motivated by the fact that our visual domain, while being quite close, is different from the one of FlickrStyle.
\begin{figure*}[t]
\centering

\includegraphics[width=.8\columnwidth]{figs/grid-900}

\caption{Samples from the dataset collected at the museum. On the first row standard pictures are shown, depicting the painting in their entirety. On the second row instead, are reported more challenging photos, due to blur, occlusion or rotation.}
\label{fig:test_photos}
\end{figure*}
\begin{figure}[ht!]
\centering
\includegraphics[width=0.8\textwidth]{figs/confmat-images-crop.pdf}
\caption{Confusion matrix for the artwork recognition module. Each row indicates the percentages of correct and incorrect classifications for a given artwork.}
\label{confmatfig}
\end{figure}
Moreover, since a sufficiently large dataset was not available to perform a further fine-tuning step, SVM classifiers have been trained to adapt the framework to the App's domain and be able to classify artworks correctly.
For training the classifiers we used approximately 1,800 images, gathered at the museum using different smartphones and tablets, namely Galaxy S4, Galaxy Tab, iPhone 6, iPad Mini and OnePlus One. These images represent all of the twenty artworks plus a `negative' set of images containing other scenes and paintings inside of the museum. They are used to reduce the false positive rate when the user accidentally attempts to recognize other paintings.
All the classifiers are One vs All SVMs. During the evaluation phase, the ID of the highest scoring one is returned to the mobile App, if it scores above a cross validated threshold.

Details about the recognized artwork are then provided to the user, who can upload a personal photo to get the style of the painting transferred on to it.
Calls to the web server are handled asynchronously and each request takes approximately 300ms on a CPU.

In order to test the recognition accuracy we collected an additional set of photos which were not used for training. For each one of the twenty artworks in our system, we collected approximately 30 photos taken from different viewpoints, with different scales and degrees of occlusion. Fig. \ref{fig:test_photos} shows some of the photos from the test set. Some of them are \dquote{difficult} in a sense that might be blurred or taken from challenging viewpoints and artworks may be partially occluded by other visitors. Despite these difficulties our system achieves an overall good performance with a mean accuracy of 94.01\%. In detail, in Fig. \ref{confmatfig} we report the confusion matrix for the twenty artworks in the test set, showing how often each painting is correctly classified or confused with other artworks. As can be seen, the majority of artworks are perfectly recognized. Only four artworks have performance slightly inferior to 0.9, due to the difficult lightening conditions present in their specific locations at the museum. 

\subsection{Artistic style transfer}

From the Artwork Detail view of the mobile App, the user has the possibility to upload a personal image on which the style of the artwork will be applied. In this way, entertaining personal pictures that share similarities with the artworks can be obtained and shared on social networks. As a result, a visit at the museum can become a playful experience, combining gaming and learning aspects for young visitors. 

We base our approach on that of Gatys \etal \cite{Gatys-arxiv2015}, that is capable of freely mixing style and content of two different photos. 
The main advantage of this approach is its broad applicability to different styles, in contrast to fixed handcrafted styles \cite{efros-cgit2001,xie-dimea2007}. This allows a museum curator to easily add new artworks in the system without requiring the development of a new transfer style algorithm. 

Following \cite{Gatys-arxiv2015}, our approach uses a CNN to derive a neural representation of content and style. The feature responses of a pre-trained network on object recognition (VGG-19 \cite{simonyan-arxiv2014}) are used to capture the appearance of an artwork image and the content of a user photo under the form of texture information. 
We start from a blank novel image that is altered with back-propagation until its neural representation is similar in terms of euclidean distance to the style and content representations.

\begin{figure}[!ht]
  \centering
    \includegraphics[width=1\textwidth]{figs/neuralizer.jpg}
     \caption{Two examples of image stylization: 1) Baccio Maria Bacci, ``Il tram di Fiesole", applied to a picture of the Battistero in Florence, IT; 2) Alberto Moretti, ``Malcom X ed altri", applied to a picture of Piazza della Repubblica, also in Florence.}
     \label{neuralizerfig}
\end{figure}

Unfortunately, the generation of the image is quite computational intensive. For an image of 900 pixel large, it takes about $\sim$90 seconds on a K80 NVIDIA GPU. As a result, the requests have to be handled offline since it is not possibile to obtain the output image in few seconds.
Considering also that multiple requests can be made at the same time from multiple users, we implemented a scalable web server that is able to be easily deployed on several interconnected nodes. Web requests are handled in Python and enqueued to a distributed queue run by a Celery\footnote{\url{http://www.celeryproject.org}} server. By treating each request as a single unit task, it allows to process the images in a distributed batch fashion on several GPUs and several servers if available. After completing the computation, each output image is sent to the user via email, together with a description of the artwork. We also include links to share the image to several social media, with the aim of enabling viral publicity of the museum. 

\section{Conclusion} \label{sec:conclusion}
We presented the \textit{Imaging Novecento} App, recently developed for the \dquote{Museo Novecento} in Florence, IT.
Following previous studies on cultural heritage audience and applications, the App aims at enhancing the experience in the museum reducing cognitive load and exploiting gamification. The App automatically recognizes a selection of paintings and provides insights on artworks and their authors. The user can upload a personal picture with his smartphone to get it stylized with the recognized artwork style. He also has the possibility of sharing it on social networks. In the Chapter we show how computer vision technologies can be exploited to increase interactivity and reduce cognitive load. This can attract the targeted audience to the museum and further engage people with content. 

\subsubsection*{Acknowledgments.} We acknowledge the support of the \dquote{Museo Novecento} in Florence, IT, and the Municipality of Florence, IT. This research was supported by \dquote{THE SOCIAL MUSEUM AND SMART TOURISM}, MIUR project no. CTN01\_00034\_23154\_SMST.
