\chapter{Item-Based Video Recommendation: an Hybrid Approach considering Human Factors}\label{chap:online-recomm}
{

\begin{quote}
\textit{In this chapter we propose a method for video recommendation in Social Networks based on crowdsourced and automatic video annotations of salient frames. We show how two human factors, users' self-expression in user profiles and perception of visual saliency in videos, can be exploited in order to stimulate annotations and to obtain an efficient representation of video content features. Results are assessed through experiments conducted on a prototype of social network for video sharing. Several baseline approaches are evaluated and we show how the proposed method improves over them.
\footnote{A preliminary version of the work presented in this chapter has been published as \dquote{A system for video recommendation using visual saliency, crowdsourced and automatic annotations} in \textit{Proceedings of the 23rd ACM international conference on Multimedia, 2015}~\cite{ferracani2015system} and then as~\dquote{Item-Based Video Recommendation: An Hybrid Approach considering Human Factors} in \textit{Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, 2016}~\cite{ferracani2016item}}.
}
\end{quote}

\section{Introduction and Related Work}\label{sec:intro}


Collaborative Filtering (CF) is a technique often used by Recommender Systems (RSs) which aims at predicting interesting items to a user based on the preferences, explicit and implicit, of other users.
A standard item-based video RS builds its prediction model considering user preferences for videos, expressed according to ratings, and suggests potential videos of interests comparing their distributions. Hybrid approaches in RSs have been proved to give best results~\cite{robillard2014}. These approaches combine CF with content-based techniques and reduce issues related to the large amount of data to be annotated and data sparsity. 
Recommending relevant videos can help users to find the most pertinent content according to their view habits or preferences. As shown in~\cite{zhou-2011}, recommendation is a powerful force in driving users to watch other videos, much more than direct search of new videos. Hybrid approaches presented in the literature typically exploit textual video metadata, sometimes complemented by multimedia content analysis \cite{Yang2007}, user profiling, social features  and User-Generated Content (UGC) \cite{Ma2013, bertini-2013, Cui-2014}. Crowdsourced data is usable information that can be leveraged to improve different online services.
 
In~\cite{Craggs-2014} crowdsourced annotations are used to create video previews that are more related to the queries of the users, to improve video retrieval. In~\cite{Sulser-2014} the performance of a video retrieval system based on crowdsourced annotations of sport videos shows that despite the heterogeneity and poor quality of the annotations, they are close to ground-truth.

Time accurate annotations of social videos, based on user comments and temporal, personalised topic modelling, has been proposed in~\cite{Wu-2014}. In~\cite{Xu-2014} a large crowdsourcing experiment has been carried out to analyse the differences between ``timed'' tags (i.e.~added to a specific timecode in a video) \emph{versus} ``timeless'' tags. The authors observed that most of the visually-related tags are relevant for short segments of the video, i.e.~people tend to tag when something is ``flashed'' in the video. We build on top of these studies and propose the adoption of an hybrid approach in which a brief and comprehensive representation of video content can improve the performance of a standard recommender based on CF (i.e.~using only ratings). 
The approach relies on content-based features gathered both through crowdsourced and CNN-based classifiers annotations. The dataset has been collected through a prototype of a Social Network (SN). Annotations collection is improved exploiting two human factors: \emph{i)} user profile interfaces and \emph{ii)} video frames visual saliency.
The main goals are: \emph{i)} to increase the number of crowdsourced annotations, that provide an enrichment of automatic video annotations; \emph{ii)} to improve the quality of video recommenders through video content analysis. 

The chapter is organised as follows: in Sect.~\ref{sec:system} the social network architecture and modules are described. Experimental results are presented in Sect.~\ref{sec:experiments} to show the influence of user profiles and visual saliency on the collection of user annotations of videos. Evidence is given that systems featuring a user profile interface stimulates user activity, increasing the number of annotations.  We also show that frames with an high visual saliency are more likely to be annotated; this can be used as a criterion \emph{i)} to suggest to users relevant frames; \emph{ii)} to filter relevant frames for automatic annotation. The recommender is evaluated in Sect.~\ref{sec:eval_rec}.

\section{The system}\label{sec:system}
The item-based RS has been implemented in a prototype of a SN\footnote{\texttt{http://fiona.micc.unifi.it/intime}}. The idea behind the SN is to exploit user profiling techniques to propose to the user targeted recommendations of videos, exploiting suggestions of topics of interest and similar users. This is achieved tracking user's activities on the SN, such as comments, number of video views, click-through data and video ratings. Users can comment videos at frame level tagging concepts derived from Wikipedia. All the concepts manually added are clustered in 54 categories using Fuzzy K-Means and classified using a semantic distance~\cite{Milne2008b} with a kNN approach. Categorised resources in videos are used to build a vector describing video content, then exploited in the RS. The SN also allows users to build a public personal profile of resources of interest from those extracted from comments or added by the SN users. The profile module is exposed relying on the hypothesis that self-expression and \emph{self-esteem} can be exploited to engage the user in the annotation process, in this way easing the collection of crowdsourced annotations. Salient frames of each video are extracted and related users activity on them is monitored in order to verify if visual saliency can affect user engagement with the system. The positive correlation, verified in Sec.~\ref{sec:eval_man}, is exploited at the interface level for easing the annotation process proposing a widget of most salient frames above each video. Automatic video annotations are then extracted using a CNN-classifier on the more salient frames. 

\paragraph*{User profile interface}\label{sec:profile} As noted in~\cite{zhao-2014}, profile curation is inherent to the use of SNs since management of personal content is integrated with its generation. The content that people choose to share online has to do with how they curate their self-image and present themselves to others. In a 2013 survey, participants ranked their relational identities as most important to them when sharing content on social media~\cite{morris2013}. SNs such as Facebook and LinkedIn, for example, are commonly regarded as a space for personal self-expression and self-promotion~\cite{van2013you}: users shape their identities in order to gain popularity and reach more and more recognition and connectedness. Our prototype system provides users with a public profile that can be curated in a semiautomatic way. The profile shows user's last comments and annotations as well as annotated video frames and tagged Wikipedia resources with thumbnails. A profiling algorithm categorizes annotations and automatically proposes inferred user interests. Each user can present himself with a set of categories that are visually shown on his profile. Resources annotated by SN users, automatically categorised, are suggested as items that users can drag and promote in their public profile for each detected user interest, as shown in Fig.~\ref{fig:profile_module}. 
\begin{figure}[!thb]
\centering
\includegraphics[width=1\columnwidth]{figs/profile_dragging_trimmed.png}
\caption{User profile interface: the user can publish resources of interest dragging suggestions from the below to the above carousel.}

\label{fig:profile_module}
\end{figure}

Many factors influence users' continued intention to use SNs such as social interactions, knowledge expansion and targeted recommendations~\cite{lin2011intention}. The users' desire of social interactions has been demonstrated to increase the number of \emph{likes} and comments in~\cite{huang2013}. The assessment of users engagement with content gives the opportunity to improve targeted services and recommendation. User propulsion at showing knowledge for self-promotion is used, for example, by platforms such as LinkedIn (Q\&A) and StackOverflow as a mean to increase the quality and number of crowdsourced annotations but, to the best of our knowledge, there is not a study in the literature which confirms that user profile interfaces affect and improve user activity in SNs.  In \ref{sec:profile_ex} we have conducted a controlled experiment to show how the user's effort to shape his public identity can be exploited to increase user's activity and production of crowdsourced annotations. 

\paragraph*{Visual saliency}\label{ssec:visualsal} 
We propose the use of visual saliency in SN systems and interfaces at two levels: \emph{i)} at the automatic annotation level to reduce the computational cost of processing all the frames; \emph{ii)} at the interface level to propose to the users possible frames of interest. The SN prototype also features a salient frames carousel above each video to ease the addition of crowdsourced comments. Videos are preprocessed to eliminate letterboxing (i.e.~black bars in videos). Then, visual saliency maps are extracted for all the video frames. Maps are defined by a visual attention model which uses a dynamic neural network on multiscale image features computed with the iLab Neuromorphic Toolkit~\cite{navalpakkam-2005}.
Salient frames in the video carousel are selected by identifying the peaks of saliency using the crest detection algorithm proposed in~\cite{Wang2013}. Automatic annotation is performed on frames selected computing the average saliency of the video and choosing those above the average, to have a dense sampling of video content.

\paragraph*{Crowdsourced annotations}\label{ssec:frontend} 
Users can comment videos at frame level and add semantic references to Wikipedia entities using an autosuggest widget, as shown in Fig.~\ref{fig:annotation_widget}. Wikipedia entities are also extracted automatically using entities detection.

\begin{figure}[!thb]
\centering
\includegraphics[width=1\columnwidth]{figs/annotation_trimmed.png}
\caption{Wikipedia annotation in video frame-level comments.}
\label{fig:annotation_widget}
\end{figure}

A carousel of the most salient frames is also shown above the video player as a video summary.
This facilitates fast and accurate annotations at exact timecodes, since users are more likely to interact with salient frames rather than with the less visually interesting ones.
A vector of categories $C$, with the same dimensionality of the SN categories taxonomy, is used to represent video content. 
Each category in $C$ is assigned with a weight defined by the average of the semantic distance of each annotation to the categories' taxonomy. This semantic relatedness between the terms is obtained using the Wikipedia Link-based Measure \cite{Milne2008b}. 

\paragraph*{Visual features}\label{ssec:visual-features}
Automatic annotation of all the frames of the videos in a SN is a time-consuming task which requires a lot of resources. In the proposed SN video frames are subsampled according to their visual saliency, allowing the system to scale while maintaining a reasonably dense sampling of video content. The convolutional network used was trained on the ImageNet ILSVRC 2014 dataset to detect 1000 synsets. 
A very deep CNN with 16 layers~\cite{chatfield-2014} was used to extract the final output layer for each frame, containing 1,000 object probabilities.
Video content is represented using a Bag-of-Words (BoW) approach. The features vector is computed using the frequency of occurrence of detected concepts with a probability above a threshold, then also complemented by crowdsourced annotations.


\paragraph*{The Recommender}\label{ssec:recommender}

Compared to user-based CF approaches, item-based recommenders minimise the sparse item ratings issue, are scalable and in general perform better than user-based recommenders~\cite{sarwar-2001}. 
The proposed hybrid RS adopts a solution that combines a semantic pre-filtering of content with an item-based algorithm. Videos are represented using a feature vector that concatenates the histogram of the categories of the crowdsourced comments and the BoW description obtained using the CNN classifier.
User's rating on a video is computed combining explicit and implicit activity. Users can explicitly vote a video on a 5 point scale with a visual widget. Number of visualizations, frame browsing and annotations are also taken into account. 
In order to reduce the dimensionality of the item-item matrix used by the algorithm, a pre-filtering on the set of possible videos to suggest is performed. Given a user $u$, we extract a set $F_{u}$ of videos for which $u$ generated a rating. For each video $v_{i}$ contained in $F_{u}$, the system selects the top-N similar videos creating a subset of similar videos $S_{i}$. The set of videos that will be used for the item-based recommender for user $u$ is then composed by the union of all the subsets  $S_{i}$, namely: 
\begin{equation}
R_u = \bigcup\limits_{i=1}^{|F_u|} S_i.
\end{equation}
The set of video $R_{u} \cup F_{u}$ is used to create the item-item matrix used for recommendation. This set is significantly smaller than the whole collection of videos contained in the system. 
The pre-filtering step uses several approaches in order to infer the top-N similar videos. These approaches, reported in Sect.~\ref{sec:eval_rec}, exploit automatic and crowdsourced annotations as well as visual saliency, and use distance measures to compute the overlap between histograms distributions.

\section{Experimental results}\label{sec:experiments}
Recommendation is a prediction problem: the system should be able to predict the user's level of interest in specific items (e.g. videos) and rank these according to their predicted values~\cite{Mobasher2007}. In order to evaluate the accuracy of the prediction, a percentage of the collected data, represented by users ratings on videos, is extracted and used as test data, not used to train the RS. The RS produces rating predictions for the missing test data, that are compared to the actual values in order to evaluate the accuracy. The performance is evaluated using Root Mean Square Error (RMSE). The more accurately the RS predicts user ratings, the lower the RMSE will result. The SN dataset is composed by 632 videos, of which 468 have been annotated with 1956 comments and 1802 annotations. 613 videos were rated by 950 of the 1108 users of the prototype SN. 

\subsection{User profile interface}\label{sec:profile_ex}
An A/B test experiment was conducted at the interface level to test the user profile influence on users' comments activity. The experiment was run on all the active users of the prototype SN for three months. Users were exposed to one of two variants of the SN, featuring (i.e.~the variant) or not (i.e.~the control) the profile curation interface. The variant was introduced in the third month, so that the number of users exposed to the variant is smaller than that of the control interface. Users who logged into the system and commented on videos since the third month were assigned to the variant (group B), whilst the others were assigned to the control group (group A). In this period of time there were 464 active users (321 in group A and 143 in group B) with a conversion rate of 3.75 and 5.81 average comments. User annotation average increased by a factor of 2.06. The result was statistically significant and validated by a \emph{t}-test that gave a \emph{t}-difference = -2.684. Minimum sample size for the evaluation criterion validity was calculated and resulted in 127 for both group A and group B with an optimum allocation ratio of 3.42. Results show a positive correlation between the use of the user profile interface and the increment in user annotations, and suggest that modules for profile curation can be effective in improving conversion rate in user online activity (e.g.~videos annotations).

\subsection{Visual saliency and manual annotations}\label{sec:eval_man}
The impact of the visual saliency of video frames on user comment activity has also been tested. In the experiment were considered: \emph{i)} the number of comments added without using the most salient frames carousel and \emph{ii)} all the comments, i.e.~adding also those coming from a click in the carousel. Results of case \emph{i} show that 53.5\% of user comments are on frames with a saliency above the average saliency of the videos, and that the percentage of frames above the average saliency is 46.5\%. Therefore, salient frames receive more attention by users, although not considerably. Results improve consistently considering also carousel driven annotations as in case \emph{ii}: in fact the percentage of comments increases to 65.24\%. Percentage of comments carousel driven is 24.01\% of the overall dataset, showing that one out of four comments are added using the carousel: it is an high percentage considering threaded comments, added by users in response to others. So, it can be said that salient frames suggestion can be useful if proposed in a web interface as to visually capture the user's attention and help in the annotation tasks.

\subsection{Recommendation}\label{sec:eval_rec}
The RS is evaluated, in terms of RMSE, comparing it to several baselines: \emph{i)} standard item-based RS, that considers users ratings of all the videos; \emph{ii)} RS working on a selection of videos, based on similarity computed using system categories only (no BoW content description); \emph{iii)} RS working on a selection of videos, based on content similarity (i.e. automatic annotations) computed on $n$ randomly selected frames;~\emph{iv)} RS working on a selection of videos, based on content similarity computed on $n$ frames with visual saliency above the average; \emph{v)} RS working on a selection of videos, based on content similarity computed on \emph{a)} frames with visual saliency score above the average and \emph{b)} crowdsourced annotations. 

Results are reported in Fig.~\ref{fig:recommender} and show how the proposed \emph{v)} approach results in a lower RMSE value than all the other approaches. In particular, it can be observed that video representation using salient frames improves over random selection, and that the addition of semantics extracted from manual annotations provides another improvement. In this experiment the threshold used to select the confidence scores of the classifiers is 0.85. In a second experiment we have evaluated the effect of the confidence of the classifier, using a threshold of 1. In this case the RMSE is further improved from 0.97 to 0.86.
\begin{figure}[!htb]
\centering
\includegraphics[width=0.99\columnwidth]{figs/RMSE_comparison}
\caption{Comparison of the proposed recommender (rightmost result) w.r.t.~baselines in terms of RMSE.}
\label{fig:recommender}
\end{figure}

\section{Conclusions}\label{sec:conclusions}
In this paper we presented a system to improve an item-based video RS. The RS uses a reduced item-item matrix, computed from content based description of videos obtained from crowdsourced and automatic annotations. User engagement through profile curation and visual saliency has been used \textit{i)} to increase the number of crowdsourced annotations, presenting the most relevant frames to users, and \textit{ii)} to address system scalability in terms of automatic annotation, reducing the number of frames to be processed. The effectiveness of exploiting human factors for user engagement (i.e.~self-esteem in user profile interfaces and visual saliency) is evaluated by user experiments on a SN prototype. Experiments show also that the proposed RS improves over the standard implementation of an item-based algorithm, and that the combination of manual and automatic annotations is more effective than the use of a single type of annotations. A positive correlation of the two human factors with the performance of the RS is not yet fully demonstrated, but it can be hypothesized and it is worth of further investigation.

\section{Acknowledgments}
This work is partially supported by the ``Social Museum and Smart Tourism'' MIUR project \linebreak (CTN01\_00034\_231545).


