A Netflix employee created a neural network for searching for kisses in movies.

Computer vision is almost perfect.recognizes objects in static images, and at the moment problems mainly arise only with videos. Nevertheless, developers continue to create neural networks for recognizing various actions on video, and one of the last and most interesting projects is the work of Netflix employee Amir Ziyah. Using a base of 100 Hollywood films, he trained the neural network to recognize scenes with kisses, while avoiding overly explicit scenes.

The neural network recognizes kisses, focusingon the movements of the lips of the actors and the sounds of touch. To show the neural network what she should pay attention to, Amir Ziay personally marked scenes with kisses in a sample of hundreds of Hollywood films. Among the films he processed were Anna Karenina (1935), Ghost (1990) and Casino Royal (2006). The developer did not expand the base of educational films, because the neural network stopped “getting wiser” after the eightieth film. In the end, Amir singled out 263 scenes with kisses and 363 scenes without kisses.

Neural network has learned to recognize kisses.

The created system of recognition of kisses consistsin two parts. The first analyzes the one-second video fragments and is called the binary classifier - it includes the convolutional ResNet neural network for analyzing visual data, and the VGG neural network for studying audio. The second part of the system, the segmenter, selects the fragments with kisses and gives them out so that they do not repeat.

Having received a video clip, the system studies it onthe presence of scenes with kisses and gives them in the form of passages. Testing has shown that the neural network does its job with 95% accuracy. However, it has been observed that the system often makes a mistake in some scenes where kisses occur against a background of vast landscapes, or where kisses occur too quickly.

Why do we need such a neural network?

Although Amir Ziay is a Netflix employee,he created a neural network has no relation to the service. And in vain - the function of recognition of kisses can be very useful in streaming services, as well as on Facebook, Instagram and TikTok. At a minimum, by slightly changing the algorithm, you can create a system for sorting unnecessarily explicit scenes. Also, such a system can automatically sort videos by thematic folders.

On the subject of neural networks, we recommend reading about itopportunities to revive static portraits. Thanks to the work of specialists from the Samsung Center for Artificial Intelligence, the portraits of Dostoevsky, Mona Lisa and Albert Einstein have literally come to life - you can see them in our material.

To discuss this, and other topics of science and technology, it is possible in our Telegram-chat. In addition to interesting interlocutors, there you will find announcements of fresh materials from our site!