What benefits can neural networks have for cinema, video games and virtual reality?

With the development of neural networks and machinelearning, expanding and scope of their application. If earlier neural networks were used exclusively for carrying out complex mathematical, medical, physical, biological calculations and forecasting, now these technologies are gaining wide popularity in a more “mundane” environment - in the field of entertainment. Taking only the first steps in this direction, they are already able to demonstrate amazing and sometimes even outstanding results. Today we analyze several illustrative examples.

The video remastering process is so complex andtime consuming that many of the masterpieces of world classics, we probably would not have seen with a new, modern, clear and juicy picture. However, the world is full of intelligent fans and enthusiasts who are well versed in new technologies, and in particular neural networks and machine learning technologies, with which you can achieve amazing results even at home. For example, YouTube user Stefan Rumen with the pseudonym CaptRobau decided to demonstrate some of the capabilities of neural networks in video processing of an old fantastic series.

His earlier work is Remako Mod -“HD remake” of the classic and very popular Japanese RPG Final Fantasy VII called. For this, he used the AI ​​Gigapixel AI algorithm, with which he was able to scale the image of the original image 4 times, transferring it to HD resolution without any significant changes in the original art design. Thus, while you are waiting for the next decade until the moment when the Japanese developer and publisher of computer games Square Enix officially releases the remaster, perhaps one of the best parts of this game series, you can try the Stefan Rumen mod yourself by downloading it from this site.

</ p>

By the way, recently the technology of neural networksfor remastering old games and bringing them to a more current and modern look without changing the overall original concept has become a real trend among various modders. For example, not so long ago we talked about the ESRGAN technology (Enhanced Super Resolution Generative Adversarial Networks), with the help of which image scaling technologies with a 2-8-fold increase in quality are implemented. The algorithm is “fed” with the original low-resolution image, after which it not only increases the original resolution of the latter, but also improves the image quality by painting realistic details and making the textures “more natural.”

Texture quality comparison: on the left the original texture from the game Morrowind, on the right - processed by the neural network

Character from Doom (on the left - it was, on the right - it became)

Resident Evil 3 background processing

Read in more detail and see examples by clicking on this link.

Anyway, in betweenRemastered The Seventh Finals Stefan Rumen decided to start another project - using the same machine learning technology, but already for processing frames of the classic science fiction series of the 90s. Rumen chose Star Trek: Deep Space Nine as an object for experimentation.

Scale TV series “live image”the complexity is very different from scaling a pre-rendered image of Final Fantasy VII, the author notes, so the final result looks much better than the original materials in low resolution, but this picture is still far from the ideal you could have dreamed of since the market for the first blu-ray players. Occasionally small “artifacts” appear on the screen. But, again, in general, everything looks more than worthy. And, in general, see for yourself.

</ p>

Rumen also used for this projectAI Gigapixel algorithm, which was trained in editing images based on real photos. The author notes that the new picture was obtained in 1080p and 4k format, but since Rumen does not have a TV or monitor with a native 4K resolution, he cannot adequately evaluate the 4K version.

</ p>

Unfortunately, see the entire series in FullHD quality is impossible. The processing of the entire source material would take a very long time, so Rumen used only separate frames from different series for the demonstration. According to him, he took up this project for one reason only - to show that this is indeed possible. In his opinion, a whole team of professionals working in a large television company and having at their disposal more suitable and powerful computer equipment for such work will be able to cope with this task much better.

Using neural networks to simplify work onThe processing of old images from video games and movies are not the only areas where such technologies are able to show their talents. In the modern world, where panoramic panoramic cameras, capable of producing at 360 degrees, as well as virtual reality headsets, are becoming increasingly popular, developers have begun to actively explore the potential of panoramic shooting.

One of the latest developments in this directionis a neural network capable of articulating panoramic static images. Its authors are machine learning specialists from the University of Massachusetts, Columbia Universities and George Mason University.

The created algorithm determines the type of environment andobjects in the photograph, and then selects and places sounds from the database in use in accordance with the spatial calculation of the distance to their sources in this image. Thanks to this panoramic image acquires realistic and surround sound, allowing you to completely evaluate the presented image in a completely new way.

</ p>

According to the developers of this neural network,The technology may find interest among developers of VR content (movies and games). The latter in this case will not have to impose all sounds on the panoramic image manually, the neural network will be able to do all this on its own.

Subscribe to our Yandex. Dzen. There you can find materials that are not published on the site.