On April 23, Zhejiang University’s Brain-Machine Interface Team unveiled their latest research findings at the Second Affiliated Hospital of Zhejiang University School of Medicine. It’s reported that their most recent study marks the first successful implementation of a brain-machine interface for writing Chinese characters, achieving the futuristic concept of “mind-controlled writing” seen previously only in science fiction.
Since 2006, Zhejiang University’s Brain-Machine Interface Team has been at the forefront of interdisciplinary research on brain-machine interfaces. In 2012, they accomplished the first domestic case of decoding precise hand gestures like hooking, grabbing, pinching, and holding by implanting electrodes into the motor cortex of a monkey’s brain. Subsequently, in 2014, they achieved another milestone by successfully decoding actions such as rock-paper-scissors in the first clinical case of electrode implantation in a patient’s skull. In 2020, they accomplished the first domestic and international instance of a volunteer using an invasive brain-controlled mechanical hand for activities like drinking, eating, and shaking hands.
Currently, the team’s latest research primarily focuses on brain-controlled Chinese character writing. With this technology, patients only need to imagine the normal writing process, and the neural activity in the motor area reflecting this process can be captured. By analyzing the neural signals from the motor area, the intended writing trajectory can be obtained, enabling control of the robotic arm for writing.
During the presentation, the research team smoothly wrote “Zhejiang University” and “Brain-Machine Interface” using the brain-controlled robotic arm based on the previously extracted electroencephalogram (EEG) signals from patient Zhang Dabo.
“This breakthrough allows some stroke and ALS patients, who have lost their ability to write and speak, to communicate with the outside world by controlling external mechanical devices to write the words they want through extracted EEG signals,” said Zhang Jianmin, Director of the Neurological Diseases Division of the Zhejiang University Brain-Machine Regulation Clinical Translational Research Center and Director of Neurosurgery at the Second Affiliated Hospital of Zhejiang University School of Medicine.
In practice, brain-controlled Chinese character writing encounters numerous challenges. Firstly, the control principle of traditional brain-machine interfaces for robotic arm movements involves analyzing gross motor movements, whereas the process of writing Chinese characters requires fine motor skills. Additionally, compared to English, Chinese characters are logograms with complex structures, more strokes, intricate classifications, and a larger character set, making decoding more challenging.
“We know that writing Chinese characters involves many aspects such as radicals, components, and stroke order. Sometimes a slight difference in stroke length can completely change the meaning. Moreover, there are no teams abroad studying Chinese characters, so the successful decoding of Chinese characters in this study is a crucial breakthrough in brain-machine interface research,” Zhang Jianmin added.
According to Latest report, Zhejiang University’s Brain-Machine Interface Team has overcome the special encoding mechanism of Chinese character writing, developing new techniques for decoding Chinese character trajectories. In offline mode, the classification accuracy of 100 commonly used Chinese characters has reached 91.3%, which increases to 96.2% with language model assistance.
Now, the implementation of brain-controlled writing of Chinese characters using invasive brain-machine interfaces has been achieved, marking a new breakthrough in the development of brain-machine interfaces. However, further research is needed to optimize how this technology can better serve the daily lives of patients.
“The future of brain-machine interfaces is promising. When the hardware becomes smaller, less invasive, and wireless, we can implant them in partially paralyzed or ALS patients to enable them to perform functions they couldn’t before, such as controlling exoskeletons for walking or assisting with daily tasks,” said Zhu Junming, Deputy Director of Neurosurgery and Head of the Functional Neurosurgery Group at the Second Affiliated Hospital of Zhejiang University School of Medicine.
