Facebook's neural network allowed to control the movements of people shot on video

Neural networks already know how to create facesnon-existent people and even improve the quality of the video. The technology continues to evolve, and very rapidly - it seemed impossible, but from now on, people shot on video can be controlled using a computer keyboard. This was made possible by the work of the Facebook AI Research development team, which trained the neural network to recognize a moving person in a video, change the background image and control its movements. This is a must see - good, the developers shared a detailed video.

Based on technology invented by developersfrom Facebook, there are two neural network algorithms - Pose2Pose and Pose2Frame for transferring objects from one video to another. They also used the DensePose algorithm, which turns a two-dimensional image of a person filmed on a video into a three-dimensional model.

So to turn a regular video intocomputer game, the DensePose algorithm creates a 3D model of a specific person. Next, the Pose2Pose algorithm turns the recording with the animation of a moving person into a video with a black background. Already at this stage, the neural network is able to perceive keystrokes on the keyboard and move the “game character” in the right direction. Finally, the Pose2Frame algorithm creates the final image with a moving person and the corresponding animation.

</ p>

Other examples of work are also visible on the video.neural network. Of course, you can see a lot of image artifacts on them, but the fact that you can control a person from a video is very impressive. It is likely that this technology will be used in the gaming industry - how useful neural networks are in the field of games and cinema, read our special material.

Did the new neural network from Facebook impress you? Feel free to share your opinion in the comments, or in our Telegram-chat.