A schoolboy from Russia won a Google contest for developing a sign language interpreter.

Can you help deaf people to express theirthoughts, automatically translating gestures with the hands they show into words? This question was asked Russian tenth grader Daniil Kazantsev from Yekaterinburg Lyceum. The decision to understand this problem led Daniel to victory in one of the categories of the young science contest Google Science Fair 2019. He developed a system that can interpret sign language and replace a sign language interpreter.

As indicated on the Google Contest website, Kazantsevwas selected among thousands of other participants of the competition and became one of 24 finalists from 14 countries of the world. As a result, he received the Lego Education Builder Award for developing his sign language interpreter.

All finalists of the competition. Daniel is on the left.

What is electromyography?

The basis of the development of the young Russian genius iselectromyography technology, which determines the electromechanical activity of muscles taking into account the changing position of the hand and fingers. In this case, each movement of the limb creates tiny electrical signals that can be read and then translated into computer-readable commands. In particular, this technology is used to control artificial prostheses.

See also: How can dumb people regain their speechlessness?

But the Russian schoolchild has developed a system based on electromyography technology that can translate electrical muscle signals into words.

Blue and white crosses indicate the location of the electrodes, which read the electrical impulses of the muscles

How a sign language translator works

The device consists of a cuff that is put on.on the forearm. The cuff has 24 electrodes, each of which reads certain electrical signals from the muscles of the forearm. In addition, it uses a special sensor that detects the position of the forearm in space. On the basis of all these signals, a unique “imprint” of the position of each separate part of the brush in space is formed.

All collected signals are amplified and filtered.from excess electromagnetic noise, and then transmitted by wire to an external microcomputer. This microcomputer digitizes signals from sensors and electrodes, after which it transfers them to a powerful computing server with a specially trained neural network. It is with the help of the neural network that interpretation (translation into words) of data on electrical signals responsible for each language gesture is carried out.

How was the gesture translator checked

Daniel spent checking the system on himself and two other volunteers. He used 5 gestures and the popular sign language ASL (American Sign Language) as the basis for translation.

Hello Yes (yes); No; Please; I love you (I love you)

For testing the effectiveness of the system, 10 attempts were used for each gesture recognition. According to the author of the development, the system was able to show 92.6 percent accuracy in recognition.

The table indicates the participants of the experiment, the names of the gestures that are required to be recognized by the system, as well as the number of correct recognitions

Video verification technology can be viewed below:

</ p>

</ p>

Daniel notes, will continue to work on autonomoussignal interpreter. There are several points that do not yet allow using it as a turnkey solution. For example, now the system can only translate individual gestures. But she is not yet able to form sentences from the words she received. Thus, now the system cannot be used as a complete replacement for the sign language interpreter. The solution of this issue will be one of the future tasks of the young scientist.

See also: This neural network knows how you will look in 50 years. How does she work?

In addition, the electronic sign language requiresconstant access to a powerful computing server with a neural network that recognizes the signals of language gestures and translates them into words. However, the author notes that the modern speed of data transmission over the mobile Internet allows providing constant access to the server, thereby partially compensating for this disadvantage. The author also considers the possibility of making the device more compact by reducing the area of ​​the location of the electrodes to 7 centimeters.

In general, it is noted that when finalizing suchtechnology, it can be used with any sign dictionary. In this case, the user will not need any special training for the possibility of using such a system.

For more information on the development of Daniel Kazantsev can be found on his project page.

If you are interested in the news of science and technology, subscribe to our channel in Yandex. Dzen. There you will find exclusive materials that have not been published on the site!