Training of real-time robotic grasp decoding from neuronal population activity in macaque motor cortex (M1)

The primate cortex can be viewed as a closed-loop dynamical system with very high dimensionality. Extracting a sub-sample of this activity and using it for the decoding of any encoded intention is a challenging task by virtue of the size of the sample relative to the full population. The problem is amplified by the brain’s plasticity, which renders any effort towards decoding ephemeral. Here we employed a modified Kalman filter to decode the grip of a robotic hand from population spiking activity recorded from two 64 channel Utah arrays implanted in the hand area of primary motor cortex (area M1) of a macaque monkey. After initial training in a reach and grasp task and array implantation, the head-fixated monkey was trained in the closing and opening of a prosthetic hand, visible on a screen, to grasp an object that he was previously trained to grasp with his own hand. The task paradigm specified a start cue and fixed maximal times for closing, holding, and releasing of the object. The monkey was initially observing the robot hand performing the required movement under the control of a control computer for 20 trials. After this, the decoder was trained on the neural population activity recorded from the electrode arrays during action observation. Then the monkey and the control computer shared control of the hand, such that its commanded position was a weighted average of the two estimates: brain control and computer control. With the progression of time, control was gradually transferred from the computer to the animal, eventually giving full control to the monkey. Our results confirm that a modified Kalman Filter (see: Agudelo-Toro et al., bioRxiv 2023) is suitable to successfully translate cortical activity to motor commands of a robot hand in real time. We also demonstrate the gradual learning by observation and internalization of the task by the monkey using two metrics - the reduction in the transitory nature of the decoder and the speed of the transfer of control of the prosthesis to the monkey. In early experimental sessions, the monkey was slow to take over control and the decoder required retraining within the experiment session, whereas in later sessions decoder training was required only at the beginning of the session and transfer of control was rapid. Furthermore, decoder training involved not only the training of the decoder on the neural activity, but it also adapted brain activity on the decoder, and an equilibrium had to be maintained between the learning processes. This equilibrium was achieved faster and maintained more robustly as the monkey became more proficient in the task, hence demonstrating the viability of this learning-by-observation paradigm.