\chapter{Deep Artwork Detection and Retrieval for Context Aware Smart Audio Guides}\label{chap:deep-artwork}
{

\begin{quote}
\textit{
In this Chapter we address the problem of creating a smart audio guide that adapts to the actions and interests of museum visitors. As an autonomous agent, our guide perceives the context and is able to interact with users in an appropriate fashion. To do so, it understands what the visitor is looking at, if the visitor is moving inside the museum hall or if he is talking with a friend. The guide performs automatic recognition of artworks, and it provides configurable interface features to improve the user experience and the fruition of multimedia materials through semi-automatic interaction. Our smart audio guide is backed by a computer vision system capable to work in real-time on a mobile device, coupled with audio and motion sensors. We propose the use of a compact Convolutional Neural Network (CNN) that performs object classification and localization. Using the same CNN features computed for these tasks, we perform also robust artwork recognition. To improve the recognition accuracy we perform additional video processing using shape based filtering, artwork tracking and temporal filtering. The system has been deployed on a NVIDIA Jetson TK1 and a NVIDIA Shield Tablet K1, and tested in a real world environment (Bargello Museum of Florence). \footnote{This chapter has been published as \dquote{Deep Artwork Detection and Retrieval for Automatic Context-Aware Audio Guides} in \textit{ACM Transactions on Multimedia Computing Communications and Applications, 35, pp.~1-21, 2017}~\cite{seidenari2017} and as \dquote{Portable computer vision for new intelligent audio guides} in \textit{EVA 2017 Florence - Electroning Imaging \& the Visual Arts, 2017}~\cite{eva2017}}}
\end{quote}

\section{Introduction}\label{sec:intro}
Digital and mobile technologies are becoming a key factor to enhance visitors' experiences during a museum visit, e.g.~creating interactive and personalized visits. Personalization is viewed as a factor in enabling museums to change from ``talking to the visitor'' to ``talking with the visitors", turning a monologue to a dialogue. This applies especially to audio guides since, similarly to a real museum guide, they must adapt their content to the needs and interests of the visitors \cite{Bowen2004}. Whether personalization addresses on-line exhibitions \cite{Bowen2004}, on-site display of artworks \cite{Karaman2016}, or both on-line and on-site \cite{Wang2009}, there is a need to obtain information about the behavior of the visitor, e.g.~what he is looking at, for how long, and what other events happen during the visit.
In this paper we address the problem of creating a smart audio guide that adapts to the actions and interests of the visitor of a museum, understanding both the context of the visit and what the visitor is looking at.

The goal of this work is to implement a real-time computer vision system that can run on wearable devices to perform object classification and artwork recognition, to improve the experience of a museum visit through the automatic detection of the behavior of users.
Object classification, sensors and voice activity detection help to understand the context of the visit, e.g.~differentiating when a visitor is talking with people or his sight is occluded by other visitors, e.g.~understanding if he has friends that accompany him during the visit to the museum, or he is just wandering through the museum, or if he is looking at an exhibit that interests him.\footnote{\url{https://vimeo.com/187957085}} Artwork recognition allows to provide multimedia insights of the observed item automatically or to create a user profile based on what artworks a user is looking at and for how long.

\section{Related Work}\label{sec:previous-work}
\subsection*{Personalized museum experience.}
The personalization of a museum visit may address the on-line experience in a virtual museum, the on-site experience in the museum itself, or both cases.

In \cite{Bowen2004} web personalization in museums is motivated by the advantages that it provides in improving usability of museum web sites and the facilitation of the learning process implied in a visit. Personalization is considered a new communication strategy that improves relationships between visitors and the institution.
In \cite{Wang2009} it is presented the Cultural Heritage Information Personalization (CHIP) system, that bridges on-line and on-site tour guides creating a personalized visit tour through a web site and then downloading the guide on a mobile device with RFID sensors that track the visitor in the museum. Tour information and the rating of artworks, if provided by users, are then sent back to the web site to update the user profile.
Interactive digital guides have been used in \cite{Zancanaro2007} and \cite{Kuflik2012} to analyze and predict the behavioral patterns of museum visitors, according to four main patterns that were initially identified through ethnographic observations by \cite{Veron1991}. The works show that the four patterns can be identified using features such as average time spent on each artwork, percentage of observed artworks, etc.
In \cite{Keil2013} augmented reality (AR) on a mobile device is coupled with a personalized interactive storytelling experience, e.g.~adapting the guide based on the age of the visitor, providing a gamified experience to children.
In \cite{Karaman2016} a non-intrusive computer-vision system has been presented, based on person re-identification of museum visitors observed through surveillance cameras. The system identifies the artworks that are observed by museum visitors and measures how much time is spent looking at each artwork, to create a personalized user profile. At the end of the tour the user profile is used to create a personalized exploration of multimedia content on an interactive table, providing more information on the items that most attracted the visitor, and suggesting additional visits and tours. 

\subsection*{Object detection and recognition}
After the breakthrough of convolutional neural networks in image classification brought by 
Krizhevsky~\etal\cite{krizhevsky2012imagenet}, several  works have used similar or derived strategies to solve other image and video related tasks \cite{erhan2014scalable,girshick2015fast,girshick2014rich,ren2015faster}. A simple, yet dramatically effective strategy pioneered by Girshick~\etal is to extract CNN features from  regions of an image. Further improvements in localization and accuracy are obtained using a bounding box regressor and fine-tuning the CNN features on the detection task.
The task of computing a full forward pass for every sub-window is extremely time expensive even for moderately shallow networks. More up-to-date works~\cite{girshick2015fast,ren2015faster} avoid this burden by computing a single full resolution convolution on the whole frame and then performing classification and bounding box regression over a region of interest computed over the last convolutional layer.
Fast R-CNN avoided the computation of multiple full forward passes, nonetheless it required expensive resources to compute object proposals, often generated with low-level features such as edges \cite{zitnick-2014}. Ren~\etal\cite{ren2015faster} removed this further computation bottleneck by learning a lightweight object proposal sharing the same features of the network used for object detection. 

A more recent class of approaches tries to generate a set of class-labeled bounding boxes with a single pass of a convolutional network~\cite{redmon2015you,weiSSD2016}.
Redmon~\etal argue that You should Only Look Once (YOLO) at frames, using an architecture inspired by Inception \cite{szegedy2015going} focused on reducing the network size and the computation. The main idea is to produce, as an output, a tensor of size $N\times N\times|C|\times 5$, representing the coordinates and probabilities, for each of the $C$ categories, for $N^2$ evaluated locations.
Liu~\etal proposed an approach named Single-Shot Detection (SSD), which is very similar to YOLO, but differs in the fact that it removes all fully connected layers allowing to predict bounding box using small convolutional filters on the last convolutional activation map. One advantage of SSD is that it allows to evaluate more windows, at multiple scales, by computing convolutions on previous output layers.

\subsection*{Object detection and recognition}
After the breakthrough of convolutional neural networks in image classification brought by 
Krizhevsky~\etal\cite{krizhevsky2012imagenet}, several  works have used similar or derived strategies to solve other image and video related tasks \cite{erhan2014scalable,girshick2015fast,girshick2014rich,ren2015faster}. A simple, yet dramatically effective strategy pioneered by Girshick~\etal is to extract CNN features from  regions of an image. Further improvements in localization and accuracy are obtained using a bounding box regressor and fine-tuning the CNN features on the detection task.
The task of computing a full forward pass for every sub-window is extremely time expensive even for moderately shallow networks. More up-to-date works~\cite{girshick2015fast,ren2015faster} avoid this burden by computing a single full resolution convolution on the whole frame and then performing classification and bounding box regression over a region of interest computed over the last convolutional layer.
Fast R-CNN avoided the computation of multiple full forward passes, nonetheless it required expensive resources to compute object proposals, often generated with low-level features such as edges \cite{zitnick-2014}. Ren~\etal\cite{ren2015faster} removed this further computation bottleneck by learning a lightweight object proposal sharing the same features of the network used for object detection. 

A more recent class of approaches tries to generate a set of class-labeled bounding boxes with a single pass of a convolutional network~\cite{redmon2015you,weiSSD2016}.
Redmon~\etal argue that You should Only Look Once (YOLO) at frames, using an architecture inspired by Inception \cite{szegedy2015going} focused on reducing the network size and the computation. The main idea is to produce, as an output, a tensor of size $N\times N\times|C|\times 5$, representing the coordinates and probabilities, for each of the $C$ categories, for $N^2$ evaluated locations.
Liu~\etal proposed an approach named Single-Shot Detection (SSD), which is very similar to YOLO, but differs in the fact that it removes all fully connected layers allowing to predict bounding box using small convolutional filters on the last convolutional activation map. One advantage of SSD is that it allows to evaluate more windows, at multiple scales, by computing convolutions on previous output layers.


\subsection*{Object recognition on mobile devices}
The availability of multi-core CPUs and GPUs on mobile devices has recently allowed to implement multimedia and computer vision methods on smartphones, with particular attention to convolutional neural networks.

In \cite{Yanai2016} an analysis of the best CNN architectures for mobile devices has been performed, evaluating the impact of using NEON SIMD instructions available on ARM CPUs and BLAS routines. The authors propose to use a Network-In-Network (NIN) architecture, where neuron weights are compressed with Product Quantization, to reduce the memory occupation of the CNN network. This solution has been employed to implement a mobile system for food recognition, presented in \cite{Tanno2016}. The problem of food recognition using mobile devices has been addressed also in \cite{Meyers2015}, where different CNNs are used to segment food, estimate the 3D volume and classify food, so to provide an estimation of the calories; however only the CNN for food classification has been ported to a mobile device.
Speed improvement and memory requirements reduction of CNN execution, for mobile devices, has been obtained in \cite{Wu2016} through weights quantization of fully connected and convolutional layers, and applying an error correction technique to minimize the estimation error of each layer.
In \cite{Huynh2016} a framework to execute deep learning algorithms on mobile devices has been presented. The framework uses OpenCL to exploit the GPUs. The framework addresses the problem of thread divergence in GPUs through data padding.
In \cite{LatifiOskouei2016} has been presented a framework for GPU-accelerated CNNs on Android devices, that uses SIMD instruction on mobile GPUs, parallelizing some types of layers on GPUs and others, that are less computationally intensive, on CPUs. The framework has been released as open source.

\subsection*{Voice Activity Detection}
Voice activity detection (VAD) is the process of detecting when humans are speaking in a given audio stream. It is essential to improve further processing like automatic speech recognition or saving bandwidth in audio coding or conference systems. 

The first VAD system was first investigated in the fifties to be used on TASI systems \cite{bullington1959}. Early approaches to this problem were based on heuristics and simple energy modeling, by thresholding or observing zero-crossing rate rules \cite{woo2000}. These methods work well in settings where no background noise is present. More recent methods address this limitation by employing autoregressive models and line spectral frequencies \cite{mousazadeh2011} to observe signal statistics in current frame and compare it with the estimated noise statistics with some decision rules. 
However, most of these conventional algorithms assume that noise statistics are stationary over long periods of time, more than those of speech. Given the extreme diversity and rapid changes of noise in different environments, they can't detect occasional presence of speech. 
The most recent class of approaches for VAD are that of data-driven methods, that avoid to make assumption over the noise distribution. They usually use a classifier trained to predict speech vs non-speech given some acoustic features \cite{elizalde2013,misra2012}. Anyway, their performance degrades when the background noise resembles that of speech. The state-of-the-art methods exploit long-span context features learned through the use of recurrent neural networks \cite{eyben2013,drugman2016,vesperini2016} to adapt the classification on the basis of the previous frames.

\bigskip

The method presented in this paper addresses the problem of creating a personalized on-site museum experience using a non-intrusive computer vision algorithm that can be executed on board of an audio guide. Unlike works such as~\cite{LatifiOskouei2016} and \cite{Huynh2016}, no special framework has been used, and the problem of computational costs has been addressed using: \textit{i)} a CUDA implementation of a CNN running on NVIDIA portable GPUs, and \textit{ii)} designing the algorithm to exploit the same features used for object detection, classification and retrieval. The problem of the scarcity of training data, highlighted in \cite{Picard2015}, has been solved applying fine tuning to a pre-trained CNN. Moreover, we exploit on-board sensors and recent recurrent neural networks for voice detection to further understand the context of the wearer, like its movements and its interactions with other people.  

The remainder of the paper is organized as follows:in Sect.~\ref{sec:framework} we describe the overall system architecture and its sub-systems; in Sect.~\ref{sec:detection-classification} we describe our efficient method for detecting objects and how we obtain a reliable artwork identification using tracking and retrieval. Sect.~\ref{sec:context} outlines the context modeling module based on voice and sensor input processing. The full system, comprising also the Android App is described in Sect.~\ref{sec:system}. Finally in Sect.~\ref{sec:experiments} and Sect.~\ref{sec:user_evaluation} we present quantitative results on our system together with an user experience evaluation, then drawing conclusions in Sect.~\ref{sec:conclusions}.

\section{The System}\label{sec:framework}
\begin{figure*}[htb]
\includegraphics[width=\textwidth]{figs/system_architecture.pdf}
\caption{The overall system architecture.\label{fig:system_architecture}}
\end{figure*}

The system we propose comprises several components that work together to enable a smart experience. Fig. \ref{fig:system_architecture} shows an architectural diagram illustrating the main submodules of the system.
% The architecture of the system has been designed to cope with the reduced computational capabilities of mobile devices. In order to reduce the spatial and temporal complexity of the algorithm we devised a system pipeline which allows to optimize the computation taking into account the available resources.
From a higher level view of our system, two main sub-systems are identified, one responsible to recognize artworks, (providing \emph{Artwork id}) and one to model the \emph{User status}. They generate input signals for the \emph{Playback Control} module which is responsible to play descriptions at appropriate time.
 
Our system senses the environment through three main channels: a camera, a microphone and movement sensors.
The three sources are accessed through an Android App which is also responsible as a front-end of the whole system. 

The camera is used to understand what the user is looking at. A computer vision system is responsible to detect objects (\emph{Object Detector}) and recognize what artwork the user is looking at (\emph{Artwork Recognition}).  Two sub-modules are highlighted in the recognition step: the first retrieves the most similar artwork from a database of known artworks and the second performs tracking to smooth out wrong predictions.

The \emph{Context Modeling} module receives three behavioral signals: \emph{People Detections}, \emph{Voice Activity Detection}, and \emph{Walking Detection}. These signals concur in generating a \emph{User Status} signal. The microphone is used as a source for \emph{Voice Activity Detection},  and movement sensors are necessary for \emph{Walking Detection}.

\section{Efficient Object Detection and Recognition}\label{sec:detection-classification}

The smart audio guide we developed is based on an efficient computer vision pipeline that simultaneously performs artwork localization and recognition. The guide requires two main computer vision tasks to be solved:~\emph{i)} detection of relevant object categories: e.g. persons and artworks; and~\emph{ii)} for every detected artwork, reliable recognition of the specific artwork framed. Moreover, since we are dealing with a sequence of frames, in order to improve artwork recognition we take advantage from temporal coherence to make the output more stable.

Our system is based on YOLO \cite{redmon2015you}, that is demonstrated to obtain accurate results even for moderate size networks. The main advantage of YOLO can be read in its acronym, i.e. it requires to look at the image only once. The process to generate scored boxes for each category of interest can be summarized as in the following. The whole image is split in $7\times7$ blocks. For each of the 49 regions a tensor of $5\times2\times|C|$ is output. This tensor encodes two box predictions for each of the $|C|$ classes. Boxes are represented as a tuple $\langle x,y,h,w,s \rangle$. Non maximal suppression can be used to avoid multiple prediction for the same object. The confidence accounts for the accuracy of the bounding box and the probability of that class being present inside the given.

Differently from SSD \cite{Liu2015}, which is based on VGG-16, our YOLO-based classifier uses a much smaller network that allows the classifier to adhere with the memory requirements of an embedded system like the NVIDIA Tegra TK1 SoC.
The architecture is derived from \emph{Tiny Net}, a small CNN pre-trained on ImageNet, which allows the application to run at 10 FPS and fitting on the memory of a Shield Tablet.

\begin{figure}[htb]
\centering
\resizebox{0.8\columnwidth}{!}{\input{figs/proiezione_mappa_features.tex}}%\vspace{-4pt}
%\includegraphics[width=\columnwidth]{figs/proiezione_features.pdf}
%\vspace{-4pt}
\caption{Feature extraction procedure for an artwork detection on a single convolutional feature map.\label{fig:feat}}%\vspace{-14pt}
\end{figure}

The system network was fine-tuned to recognize artworks and people using our dataset. Recognizing people is relevant for two reasons: first we can exploit the presence of people in the field of view to create a better understanding of context, see Sect.~\ref{sec:context}; secondly, without learning a person model, it is hard to avoid false positives on people, since artwork training data contains statues, which may picture human figures. Learning jointly a person and an artwork model, the network features can be trained to discriminate between this two classes.

\subsection{Artwork recognition}\label{sec:retrieval}

\begin{figure*}[htb]
\includegraphics[width=\textwidth]{figs/features-pipeline.pdf}%\vspace{-8pt}
\caption{Our network architecture, with tensor size and layer numbering.\label{fig:arch}}
\end{figure*}%\vspace{-8pt}

The rich features computed by the convolutional layers are exploited and re-used to compute an object descriptor for artwork recognition.  

To ensure ease of deployment and update of the system, we base our artwork recognition system on a simple nearest neighbor step. We need to fulfill two important requirements: first our feature should be lightweight, i.e.~low dimensional, in order to be stored on the device and reduce the computation time for feature comparison; second we must compute a discriminative representation for a region of the frame that may differ in size and aspect ratio.

To obtain a low dimensional fixed size descriptor of a region, we apply a global max-pooling over convolutional feature activation maps, as shown in Fig.~\ref{fig:feat}. To increase the discriminative power, we concatenate such descriptor computed on two different feature maps. The region is remapped from the frame to the convolutional activation map with a simple similarity transformation.

Considering the activation map of the $n^{th}$ convolutional layer, we have a tensor dimension of $W_n \times H_n \times C_n$. After the reprojection of the bounding box onto the feature map we end up with a smaller tensor with a size of $W_{n,bb} \times H_{n,bb} \times C_n$; $W_{n,bb}$ and $H_{n,bb}$ depend both on the network layer and the bounding box geometry, while $C_n$ depends solely on the network layer and represents its number of channels. The max-pooling operation of the $C_n$ channels over the $W_{n,bb} \times H_{n,bb}$ values generate a feature vector that is independent from the dimension of the bounding box.

Considering the architecture in Fig.~\ref{fig:arch} one could wonder which features are best to recognize the specific framed artwork, since leftmost layers have higher resolution and  mostly represent the low-level structure of the image, while rightmost ones, are low resolution but encode higher level information, closer to the image semantics. 

After an experimental evaluation, which is detailed in Sect.\ref{sec:experiments}, we selected, as combination, the features from layers 3 and 4, yielding a feature of size 768. 
The final bounding box descriptor is obtained by concatenation of the two max-pooled regions values and is $L^2$-normalized.

Considering a pre-acquired dataset of artwork patches $p_i\in\mathcal{D}$ and their artwork labels $y$, for each detected artwork $d$ we predict a specific artwork label $y_{\hat{p}}$ finding the nearest neighbor patch
\begin{equation}
\hat{p} = \arg\max_i \langle p_i, d  \rangle\label{eq:nn}
\end{equation}%\vspace{-8pt}

\noindent The recognition system observes each frame independently and predicts artwork labels according to Eq.~\ref{eq:nn}, this approach, in case of motion blur or quick lighting changes may produce incorrect recognition results. In the following we detail how we exploit temporal coherence to produce a more stable recognition output.

\subsection{Artwork Tracking and Temporal Smoothing}\label{sec:tracking}

High recognition accuracy is a requirement for the audio-guide, since mistaking an artwork for another may result in a bad user experience, e.g.~this would result, at the interface level, in the audio guide presenting an artwork different from the one that is actually observed. 

This is an extremely critical aspect and must be addressed, 
in order to improve the stability of the recognition system.
We devise three strategies, based on the user location with respect to the artwork of interest and the continuous tracking of object bounding boxes. 

To reduce the error rate our idea is to avoid performing artwork recognition on objects that may be too far from the user. Farther objects are unlikely to be of interest for the user, moreover the feature computed on a smaller bounding box has little discriminative power and likely leads to erroneous recognition. 

Computing the actual metric distance from an artwork requires to perform real-time camera tracking and scene mapping. We believe that this accurate information is not required for our task and therefore we rely on a simple heuristic, comparing the areas of an artwork detection and the whole frame as in the following:

\begin{equation}
\frac{w_{bb} h_{bb}}{WH} > T\label{eq:reco-dist}
\end{equation}%\vspace{-8pt}

\noindent where $WH$ is the frame area and $w_{bb}$ and $h_{bb}$ are bounding box width and height respectively, and $T$ is a threshold (Fig.~\ref{fig:recognition-distance}) empirically fixed. We name this strategy \emph{Distance}. In our experiments we obtained the best results for $T = 0.1$, that as can be seen in Sec. \ref{sec:smoothing_evaluation}, allows to reduce false recognitions by 50\% w.r.t.~not using the heuristic, at the cost of introducing a small number of missed recognitions.
 
\begin{figure}[htb]
\centering
\includegraphics[width=.8\columnwidth]{figs/recognition_con_thresh_distanza_crop.jpg}%\vspace{-4pt}
\caption{Shape based filtering: artwork in yellow (left) is not considered for recognition, not satisfying Eq. \ref{eq:reco-dist}, while the other is recognized as ``marzocco" (the heraldic lion symbol of Florence).}\label{fig:recognition-distance}%\vspace{-8pt}
\end{figure}

Considering that there is continuity when the user walks around in the area, an artwork recognized frequently across a very short amount of time is probably the most correct.
To exploit this, we continuously predict artwork labels as described in Sec. \ref{sec:retrieval}, but we consider a prediction only after it persists for $M$ frames.
We name this strategy \emph{Consistency}. We implement it by tracking all artwork detection boxes with a greedy data association tracking-by-detection algorithm, requiring an IoU of consecutive bounding boxes of $50\%$. An example of this tracking is shown in Fig.~\ref{fig:tracking}.

\begin{figure*}[htb]
\centering
\includegraphics[width=.8\columnwidth]{figs/san-giovanni.jpg}

\caption{Example of artwork tracking, with $M=15$. Only after a stable recognition over $M$ frames the system labels the artwork.\label{fig:tracking}}
\end{figure*}

With the same principle, it is unlikely that the user moves quickly from an artwork to another in just few frames. So, after the system recognizes an artwork, it continuously output its label proportionally to the elapsed time since the recognition. We call this strategy \emph{Persistence}.
We increment a counter $p$ every time the recognition label for a box is unchanged, keeping track of the most frequent label  $\overline{y}$. Every time a label $y*$ is different from $\overline{y}$ we decrement $p$. We predict the artwork identity as $y*$ only if $p > P >M$. This technique greatly reduces the number of false recognitions. In our experiments best results were obtained for $M = 15$ and $P = 20$.

\section{Context Modeling}\label{sec:context}
To pursue the idea of an autonomous agent that is able to understand when it is the time to engage the user and when it should be inactive, it is essential to understand the context and the status of the wearer. In addition to the observation of the same scene the user is viewing through the wearable camera, we also try to understand if the user is busy following or participating in a conversation and if he is moving around the room, both independently from the visual data.

\subsection{Detecting conversations}
Our audio-guide should be able to understand when the user is engaging in a conversation, if his field of view is occluded by other visitors or he is paying attention to another person or an human guide. In that event, it is reasonable to stop the audio-guide or temporarily interrupt the reproduction of any content, in order to let the user carry out his conversation. This should be of high priority and it should be performed even if the user is standing in front of an artwork. 
To identify this scenario, we use the device microphone to detect the presence of a nearby voice. We chose to employ a Voice Activity Detection (VAD) system for this task. 

Typically, museums are mostly quiet environments where people tend to remain silent, to appreciate the artworks, and briefly talk between each other. Nonetheless, in some cases the environment can be noisy with the presence of music in background or some environmental noise. This requires the adoption of a VAD with automatic noise adaptation. The system will listen continuously to the environment, adapt to the local environment noise and detect when voice is present. Therefore, in order to run in real-time together with the computer vision module, it is essential to provide a lightweight system with low computational complexity.
We adopted the system from \cite{eyben2013}, that is a state-of-the-art method based on a Long Short Term Memory recurrent neural network. This approach is able to model long range dependencies between the inputs (and thus accurately model environmental noise) and is highly scalable. The computational complexity for evaluating the networks is linear with respect to the number of input frames. Only a constant number of operations needs to be performed for every audio frame. 
We use the open source implementation and model available in the OpenSMILE framework \footnote{http://audeering.com/research/opensmile/}. 

Considering that a positive voice identification stops the playing of any description, it is imperative that the classifier has a low false positive rate. Unexpectedly stopping the reproduction due to a classifier error may result in a poor user experience. 
To this end, we evaluate an entire second of audio before emitting the prediction. We choose to use a classifier with a granularity of 0.01, so that, by exploiting all the predictions in this time frame, we can increase the stability of the prediction. The final prediction is the mean over the single classifications. We threshold this value according to the expected false positive rate, measured empirically on our dataset.

\subsection{Sensors for walking detection}
An important hint for understanding the context of the user is given by its movements. Standing still, walking or sitting can signal if the user is paying attention to some artwork or if he is uninterested in what he is looking at. 

We make use of this information for mainly two purposes:
\begin{itemize}
	\item If the user is walking fast then he is not probably interested in the visible artworks. This means that even if the visual system detects and recognizes an artwork, the audio description should not be started. 
	\item If the user is standing still in front of an artwork and he is listening to the audio description, this should not be stopped, even if the visual system stops recognizing the artwork. This can happen mostly because of occlusions due to other people walking or standing between the visitor and the artwork. 
\end{itemize}

To perform walking detection we use accelerometer data. We estimate the mean and standard deviation of acceleration magnitude from the training set. We subtract the mean from the acceleration magnitude and then we filter out peaks below the standard deviation. We consider each peak as a step. We then take into account a sliding window of 1 second, and consider the subject walking if at least a step is detected in the given window.

To detect if a person changes the facing direction, we estimate the orientation variation using gyroscope data. We average the orientation vector over the same 1 second sliding window. The facing direction is considered changed if the current orientation vector differs from the average for at least $45^{\circ}$.

\section{System Implementation}\label{sec:system}
The proposed system has been initially developed using a NVIDIA Jetson TK1 board, to test the performance of the vision system, introduced in Sec.~\ref{sec:detection-classification}, using a device designed for embedded systems. The board has a NVIDIA Kepler GPU with 192 CUDA cores, and an NVIDIA 4-plus-1 Quad-core ARM Cortex A15 CPU.
Then the audio-guide application, named SeeForMe, has been deployed on an NVIDIA Shield Tablet K1 that has the same computational capabilities of the TK1 board, but it runs Android 5.0 instead of Linux, and it allows to develop a user friendly application that can support the visitor in his museum experience.

\begin{figure}[htb]
\centering
\includegraphics[width=.8\columnwidth]{figs/wearable-device-camera.jpg}%\vspace{-4pt}
\caption{A visitor with the device camera in the pocket}\label{fig:wearable-device}%\vspace{-8pt}
\end{figure}

We designed the application to handle three different user scenarios: \emph{(i)} the user makes use of the application in a fully-automated way (placing the device in a front pocket with the camera facing forward, or hanging it on the chest using a special support), as shown in Fig.~\ref{fig:wearable-device}. In this scenario the system does not need any interaction and continuously observes the surroundings using the camera, choosing when to start and stop the audio by analyzing the user's behavior; in this modality the user can still interact with the application by using voice commands that are elaborated by the operating system and translated in the form of actions such as start/stop the audio; \emph{(ii)} the user makes use of the application actively in a semi-automated way: after pointing the device towards the artworks the visitor is interested in, the system detects the artwork and provides the contextual audio guide, for which the audio can be started and stopped automatically or manually by the user; \emph{(iii)} the user has completed the tour and wants to revisit his experience: to this end, the application provides a visual history of the tour represented as a carousel of artworks in temporal order. Through the carousel the user can select the artworks he visited, have multimedia insights and replay the audio guide.

\begin{figure}[!htb]
    
        \includegraphics[width=0.95\linewidth]{figs/seeforme-app.jpg}
   
   
    \caption{\textit{(left)} the user is listening to the description of the artwork, \textit{(center)} the user is reviewing an item in the history, \textit{(right)} the user is speaking with someone not focusing on any artwork, }
    \label{fig:gui-listen-speak}
\end{figure}

\begin{figure}[htb]
\centering
\includegraphics[width=.8\columnwidth]{figs/seeforme-blur.jpg}%\vspace{-4pt}
\caption{The contextual menus to configure the app properties. Here it is shown the appearance of the interface in blur mode.}\label{fig:properties}%\vspace{-8pt}
\end{figure}

Several application properties and modalities can be configured in the mobile app guide through a contextual menus reachable from the top right corner of the navigation bar. In Fig.~\ref{fig:properties} it is shown the contextual menus where there are two main modes:~\emph{i)} Blur mode, ~\emph{ii) Auto mode}. The first one enables an app feature which blur the background of the artwork being framed by the visitor in order to focus his attention on the target. The Audio mode instead activates the automatic mode for the control of the guide audio stream.
Voice Commands and interaction can also be enabled and disabled. Finally, a range in seconds can be defined to set a custom temporal window between the instant that the system recognizes the artwork and the start of the audio guide reproduction (these delay is marked visually by the green line in the icon which animates until it closes the circle, as shown in Fig.~\ref{fig:properties}).

\bigskip

The mobile app has been developed using the Android SDK. The interface follows the guidelines of material design proposed by Google~\footnote{https://material.google.com/}. SQLite is used to persist the information on the device local storage. Communication between the app and the YOLO module is carried out using Java Native Interface (JNI) which enables the Java code running in the Java Virtual Machine (JVM) to call and be called by native applications. Data-interchange is performed through JSON messages. In particular, the YOLO module communicates with the mobile app passing data related to the current frame of the camera stream. This data comprises detected and recognized artworks and persons, with the coordinates of their bounding box, and booleans indicating if external speech and user movements have been detected.


\section{Experimental Results}\label{sec:experiments}

\subsection{Dataset}

We collected a dataset from footage captured in the Bargello Museum in Florence. Bargello Museum hosts a variety of artworks, featuring a large hall (Donatello Hall) with several masterpieces from Donatello. We use this hall as our testing ground. The collected data serves two distinct purposes: train and evaluate the object detector described in Sect.~\ref{sec:detection-classification}, and evaluate the full artwork recognition system. 

Artwork imagery has been collected in a diverse set of illumination and viewpoint conditions. In fact, the Donatello Hall is an extremely challenging environment featuring a high ceiling with large glass windows. Therefore depending on the time of the day and the weather condition, artwork appearance may change significantly, because of light diffusion and camera sensor saturation. We collect $1,237$ images from all the statues in the Donatello Hall, in different lighting and viewpoint conditions.

For the object detection task we extract a subset of annotated images, splitting the data in training and testing. Artworks appearing in the training set do not appear in the testing set to correctly evaluate the performance of the detector. We added person images from PASCAL VOC2007 in order to have a more balanced training set. Fine-tuning our small network does not require a huge amount of data; we simply collected a balanced dataset of $\sim300$ person and $\sim300$ artwork images. We used vertical image flipping in training as data augmentation.

To evaluate the recognition system we annotated a larger set of images with the artwork id. To easily collect our recognition database we developed a tool based on our detection pipeline. We use our artwork detector to generate bounding boxes and the tracker described in Sect.~\ref{sec:tracking}, to link all boxes in a sequence, after a sufficient amount of frames of an artwork has been collected the user may simply select an existing id or enter a new record in the database. Considering the non-parametric nature of the recognition system discussed in Sect.~\ref{sec:retrieval}, this process can be run multiple times to enrich the dataset. 

Finally, to test our full pipeline, we use sequences accounting for $8,820$ frames. We also pay attention to include shots where multiple artworks are visible. In each frame, we annotated the bounding box and the label of each visible artwork. At the end of the process we collected a total of $\sim250$ seconds of video with $7,956$ detections. 

\subsection{Artwork detection}

In the first experiment, we evaluate the performance of the artwork detection system. After performing the fine-tuning of the network on our dataset, we run the trained detector on the test set and measure the average precision. As described in Sec. \ref{sec:detection-classification}, we aim at detecting the artworks that are in front of the wearer and give less importance to the ones in the distance. As a result, we only consider detections of a minimum area $T$ that are indicative of a small distance from the user. We report in Fig. \ref{fig:detection} the average precision obtained by the detection system when varying the minimum area of the considered detections. The area is normalized with respect to the dimension of the video frame. 
It can be observed that the average precision increases with the minimum area of the box and reaches the maximum value of 0.9 at 40\% of the area. This means that the classifier is more effective at recognizing nearer artworks. We note that increasing the minimum box size area is not always a guarantee that the detector will be more precise. While far detections are very prone to errors due to the small object scale, some detection errors may also be present at a near distance due to blur. 

\begin{figure}[htb]	
\centering
\begin{minipage}[t]{0.44\textwidth}
\includegraphics[width=\textwidth]{figs/artwork-ap-large.pdf}%\vspace{-8pt}
\caption{Average precision of artwork detection when varying the minimum box area.\label{fig:detection}}
\end{minipage}\hspace{1em}
\begin{minipage}[t]{0.435\textwidth}
\includegraphics[width=\textwidth]{figs/artwork-pr-large.pdf}%\vspace{-8pt}
\caption{Precision-recall curve for artwork detection using a threshold $T = 0.1$.\label{fig:precision-recall}}
\end{minipage}
\end{figure}
Selecting a good value for the minimum box area is therefore a trade-off between a good precision and the proximity that a wearer has to be to an artwork. We chose the final value of $T = 0.1$, that provides a significant improvement of precision over the bare detector output and a maximum distance of $\sim5$ meters. In Fig.~\ref{fig:precision-recall} we show the precision-recall curve relative to the final $T$ value. Our system has a very good precision at high recall rates. Hence, the curve exhibit only a small amount of loss in terms of precision until ~0.8 recall. We note that higher recall can not be reached due to the T threshold selected according the results reported in Fig.~\ref{fig:detection}. For this reason, the curve is truncated at that point. 

\subsection{Artwork recognition: nearest neighbour evaluation}
In this experiment we evaluate the effect of the number of nearest neighbors on artwork recognition, in terms of precision. Descriptors are computed concatenating two layers of the network, according to the approach described in Sect.~\ref{sec:retrieval}. Results are plotted in Figure \ref{fig:layer34} with accuracy using 1 nearest neighbour, where features extracted from layers 3 or 4 are combined with the other layers. The figure shows again that the combination of the $3^{rd}$ and $4^{th}$ layers provides the best results. In Figure \ref{fig:neighbors} we report the accuracy when varying the number of nearest neighbours using the just selected best combination. We observe that 1 nearest neighbor provides the best performance in recognizing an artwork. Accuracy degrades when more nearest neighbours are used in voting the correct artwork id. This is due to the fact that the environment we are testing the system in, has high variability in lighting conditions. Moreover we acquired multiple poses for each artwork. It is clear that for each query only a few samples will be in the similar pose/lighting conditions while increasing the amount of neighbours will just add noisy data to the vote pool. 


\begin{figure}[htb]
\centering
\begin{minipage}[t]{0.45\textwidth}
	\includegraphics[width=\textwidth]{figs/layer3-4.pdf} %\vspace{-8pt}\\
	\caption{Recognition accuracy of combinations of layer 3 and 4 with layers $[3,\ldots,8]$. \label{fig:layer34}}
\end{minipage}%
\hspace{1em}
\begin{minipage}[t]{0.45\textwidth}
	\includegraphics[width=\textwidth]{figs/acc_nn.pdf}
	%\includegraphics[width=\textwidth]{figs/neighbors.pdf} %\vspace{-8pt}\\
	\caption{Recognition accuracy of the best layer combination (layers 3-4), varying the number of nearest neighbors.\label{fig:neighbors}}
	%\caption{Precision of the best layer combination (layers 3-4), varying the number of nearest neighbors.\label{fig:neighbors}}
\end{minipage}%
\end{figure}

\subsection{Temporal Processing Evaluation} \label{sec:smoothing_evaluation}
In order to measure the effectiveness of the three strategies for temporal processing described in Sec. \ref{sec:tracking}, we perform an experiment where several of their combinations are tested. The annotated video sequences are thus fed to a simulation of the system, where each combination of output bounding box and label is tracked and compared to the ground truth data. The thresholds are fixed at $T = 0.1$, $M = 15$ and $P = 20$. We measure the number of detections where the artworks are correctly and incorrectly labeled, and the number of times the system chose to output the \dquote{generic} artwork label. 

We report in Table \ref{tab:exp-tracking} the result of the evaluation. 
%We found out that the system was able to correctly detect the presence of all the artwork in each frame, thus the detection has a 100\% mAP. 
In the first test (T1), we measure the performance of the system without any additional criterion as baseline, i.e. the frame by frame output of the recognition system. We observe that the system outputs the correct artwork for the majority of the detections ($\sim$ 70\%), while about 30\% were labeled as an incorrect artwork. 
By adding the \emph{Distance} criterion, we see in test T2 that a slightly lower amount of detections were correctly labeled, but about half of the incorrect recognitions were considered generic, instead. This confirms that a large amount of errors are made on farthest artworks, since they are more difficult to recognize.
In test T3, we observe the outcome of the \emph{Consistency} strategy. In this case, almost all the incorrect artwork recognition are successfully exposed and classified as generic artwork output. This is due to the uncertainty of the vision system that rapidly shifts its prediction from frame to frame.
In test T4, as seen in T2, adding the \emph{Distance} criterion to \emph{Consistency}, reduces the Incorrect recognitions.
 While the \emph{Consistency} criterion by itself is able to almost nullify the incorrect recognitions, it is not robust to sparse errors. In fact, the system often swings from the correct recognitions to the generic label. 
This issue is resolved when combining this stringent strategy with the \emph{Persistence} one, in test T5. This is visible quantitatively in the gain of the number of correct recognitions and the relative decrease of generic outputs, at the expense of increasing the incorrect ones.
Combining all the criteria, as in T6, leads to a very low number of incorrect detections and a reasonable number of neutral artwork outputs, confirming our intuition about the efficacy of the three strategies. With only 22 wrong detections, the system predicts a wrong label approximately less than one cumulative second every $\sim5$ minutes of video.


\begin{table}[t]
\centering
\caption{\textbf{Performance by applying the three strategies for temporal smoothing:} C stands for Consistency, D for Distance and P for Persistence. We report the number of detections where, respectively, the artwork was correctly recognized, the artwork was misplaced for another one and where the system chose to output a generic \dquote{artwork} label.\label{tab:exp-tracking}}
{\begin{tabular}{c|c|c|c|c|c|c}
\toprule
\multicolumn{1}{c|}{Test} & \multicolumn{3}{|c|}{Strategy} & \multicolumn{1}{c|}{Correct} & \multicolumn{1}{c|}{Incorrect} & \multicolumn{1}{c}{Skipped}\\
& \multicolumn{1}{c|}{C} & \multicolumn{1}{c|}{D} & \multicolumn{1}{c|}{P} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c}{} \\ 
\hline 
T1 & \xmark & \xmark & \xmark & 5,598 ($\sim$70\%) & 2,358 ($\sim$30\%) &  0 (0\%)\\
T2 & \xmark & \cmark & \xmark & 5,334 ($\sim$67\%) & 1,267 ($\sim$16\%) & 1,355 ($\sim$17\%) \\
T3 & \cmark & \xmark & \xmark & 4,475 ($\sim$56\%)& 36 ($\sim$0\%) & 3,445 ($\sim$43\%) \\
T4 & \cmark & \cmark & \xmark & 4,363 ($\sim$55\%)& 11 ($\sim$0\%) & 3,582 ($\sim$45\%) \\
T5 & \cmark & \xmark & \cmark & 5,141 ($\sim$65\%) & 61 ($\sim$1\%) & 2,754 ($\sim$35\%) \\
T6 & \cmark & \cmark & \cmark & 4,966 ($\sim$62\%) & 22 ($\sim$0\%) & 2,968 ($\sim$37\%)\\
\bottomrule
\end{tabular}}
\end{table}

\subsection{Voice Detection Evaluation} \label{sec:audio_exp}
In this experiment we test the performance of the voice activity detection system on our dataset. We consider two simple strategies to emit a classification per second, namely Sample and Mean. The Sample strategy is just evaluating the classifier on a single audio frame per second, sampled at the beginning of a new second. This has the advantage to require only a single evaluation of the net. The Mean strategy, instead, consider all the predictions of net in a second and finally emits the mean of the values. This is more robust to the fluctuations of the classifier, at the expense of running the net continuously. 
With both strategy, in order to minimize the number of false positives, we measure the performance of the classifier varying the positive threshold. 

\begin{figure}[htb]
\centering
\includegraphics[width=0.5\textwidth]{figs/audio-rock.pdf} %\vspace{-8pt}\\
\caption{Receiver operating characteristic curve of the tested voice activity classifiers.\label{fig:audioroc}}
\end{figure}

We report the receiver operating characteristic (ROC) curve of the two strategies in Fig. \ref{fig:audioroc}. We observe that both strategies have a high area under the curve (AUC), meaning that they correctly predict the presence of the voice most of the time. The Mean strategy has a higher AUC and has always an higher true positive rate at the same false positive rate than Sample. This confirms that the Mean strategy is more robust than Sample. 

\section{User experience evaluation} \label{sec:user_evaluation}
Modern tourist guides have their origins dating up to $17^{th}$ and $18^{th}$ centuries \textit{Grand Tour}, and their role has become a key component in modern tourism experiences and applications. Guide functions can be highly specialized and require a lot of expertise and interpersonal skills to satisfy tourist needs. \cite{cohen1985tourist} describes guide roles as characterized by instrumental (guide), social (animator), interactional (leader) and communicative (intermediator) functions.
Instrumental functions represent services capable to convey essential tourism information such as path finder to artworks location and related infos. Interactional features offer the ability to create a relation between the user and the contextual environment (e.g.~informations about artworks). Improving this ability also means improving interaction. Sociality involves all the activities aiming at engaging the users with collaborative and not isolated experiences. Communicative functions facilitate access to artwork insights and targeted content, e.g.~pointing out objects of interest.
All these functionalities are fulfilled at their best by humans and modern audio guides have only partially replaced the complex role of the human guide. On the other hand the use of technology has improved aspects such as efficiency, sociality and autonomy in providing information communication under the so-called smart tourism paradigm. In Table \ref{tab:guide-comparison} we compare guide-role functions as provided by human and traditional audio-guides with those available in our system.

\begin{table}[t]
\centering
\caption{\textbf{Functions comparison of our guide with respect to human and traditional audio guides\label{tab:guide-comparison}}}
{\begin{tabular}{c|c|c|c|c}
\toprule
\multicolumn{1}{c|}{Type} & \multicolumn{1}{c|}{Instrumental} & \multicolumn{1}{c|}{Social} & \multicolumn{1}{c|}{Interactional} & \multicolumn{1}{c}{Communicative}\\
& \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} &  \multicolumn{1}{c|}{} & \multicolumn{1}{c}{} \\ 
\hline 
Human Guide & $\ast \ast \ast$  & $\ast \ast \ast$ & $\ast \ast \ast$ &  $\ast \ast \ast$ \\
Audio Guide & $\ast$ & -- & -- & $\ast \: \ast$ \\
SeeForMe & $\ast \: \ast$ & $\ast \: \ast$ & $\ast \: \ast$ & $\ast \: \ast$ \\

\bottomrule
\end{tabular}}
\end{table}

The main differences between traditional audio guides and our system can be found in the interactional and social aspects of the provided experience. In traditional audio-guides the fruition of content is for the most part passive and the user has a low control on the reproduction of content. As regard to this, SeeForMe offers a more user friendly experience giving the possibility to interrupt the audio playback manually and automatically, and to restart the reproduction from the last point. Playback control can be achieved also using voice commands. Furthermore, activation of contents in audio-guides is cumbersome: locations or room numbers have to be searched and inserted manually reducing usability whilst SeeForMe allows automatic artwork recognition; this fact results in further differentiation of the Instrumental function, even in case of automatically triggered guides (e.g.~those using RFID): an audioguide, being completely passive can not direct the visitor, while SeeForMe, highlighting the presence of other artworks as shown in Fig.~\ref{fig:properties}, can direct the visitor within the museum. As for sociality, if it is true that social networking mechanisms are commonly provided in tourism apps for mobile phones, these functionalities are intended for virtual or remote users and not real companions. Indeed, audio-guides hinder communication between visitors (especially group visitors) and make people feel isolated, causing them to stop using devices and applications in order to join others. SeeForMe in this sense is more social because it automatically understands the context detecting if the user loses attention or simply is speaking with someone else, adapting  the interaction with the system consequently.


In order to assess the whole experience offered by the system in a real environment, we conducted an  evaluation of its usability. According to ISO, usability is defined as ``the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use". However, there are several usability models and types of assessments, like ISO standards on quality models (ISO 9126), user-centered design (ISO 9241) or user-centered approaches. A review of techniques for mobile application usability evaluation is provided in \cite{nayebi2012}. 

The usability study was performed with the popular Standard Usability Scale (SUS)~\cite{brooke1996sus}, that follows a user-centered approach. Testing a user interface with SUS means, given a scenario of use and one or more tasks to solve, administer a 10 point questionnaire to a group of users. SUS is a Likert scale~\cite{trochim2006likert}, therefore questions address extreme cases, with opposite meaning and alternating positive with negative sentences. Answers to questions are numbers from 1 to 5, expressing all ranges from \dquote{Strongly Disagree} to \dquote{Strongly Agree}. This testing strategy has been proved effective in removing acquiescence bias. The alternation of positive and negative items makes sure that users read it carefully. Nielsen states that it is sufficient to collect 5 polls to find the 85\% of design errors of an interface or experience~\cite{Nielsen1990}; in these tests we recruited twelve persons, divided in two groups of six people each.

We tested two different scenarios, supervised and unsupervised, in which users were asked to perform two simple tasks: \emph{i}) \dquote{Activate the audio-guide for one or more artworks of your interest}, \emph{ii}) \dquote{After the visit, use the app to find again the information about one artwork you have seen}. In the former a group of people receives from the authors of the paper a spoken detailed description of our system, thoroughly explaining the Android app functionalities and also detailing insights on the recognition engine. In the latter scenario instead, users are given the same two simple tasks but without any explanation on the application functions.

After normalization SUS scores are expressed in the range $[0-100]$. They do not represent percentages but can be interpreted with an adjective rating~\cite{Bangor-2009}. A score over $68$ means that the user interface or experience design is above average \cite{Sauro-2012} and that tasks can be completed without too much fatigue. Scores above $80$ usually means that the interface is correctly designed and that the user experience is enjoyable. 

We obtained an average SUS of $74.0$ for the unsupervised scenario and $79.5$ for the supervised scenario. The small gap in scores measured in the two scenarios, and their closeness to $80$, means that the user interface is easy to use and that the training provided by expert users is not strictly required to perform tasks correctly. Nonetheless, considering that the user experience increased when users received a brief tutorial on the features and technical details, means that there is some room for further improving the design of interface and user experience of our app.

Users, when interviewed, mostly agreed that the automatic start/stop of the guide is the feature that makes the experience smooth. Regarding negative aspects of our system, most of the points made by users were about the need to access menus to change the language or other options.


\section{Conclusion}\label{sec:conclusions}
We have presented a system running on the NVIDIA Jetson TK1 and on NVIDIA Shield Tablet K1. Our approach jointly solves two problems: contextual analysis and object recognition. We apply our efficient video processing pipeline and multi-sensor analysis to improve museum experience. Our method allows to profile in real-time visitor interests and to provide instantaneous feedback on the artworks of interest. We exploit audio and sensor data to improve the user experience reducing the intrusiveness of the smart audioguide. 

Our Android app, allows users to switch between a fully automated experience to a more interactive mode. Moreover, after a visit is completed it is possible to for the user to look back and listen, or read, again about the artwork that gathered his interest.

Usability testing revealed few pitfalls of our experience design, but users where satisfied on average and provided some suggestions to improve the user interface further.

% Appendix

% Acknowledgments
% \begin{acks}
% The authors acknowledge and thank Giovanni Taverriti and Stefano Lombini for their work on the early version of the vision system, and Alessandro Sestini for his help in the development of the initial version of the Android front-end.
% \end{acks}






