LISA

Research project MEG · Speech · Interpretability

Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval

LISA is a compact neural decoder that retrieves heard speech from brain activity and keeps a path back to the cortical sources, temporal dynamics, and stimulus features behind each decision.

Ilia Semenkov Daria Kleeva Ivan Dakhtin Zarina Maksudova Alex Ossadtchi

AXXX · HSE University · ITMO University

Publication Cite the paper View code
Model overview3 s MEG → audio embedding
LISA architecture: MEG passes through spatial attention, subject-specific spatial filters, temporal filters, and a compact convolutional decoder to produce an audio-aligned embedding.
A 3-second, 208-channel MEG segment passes through a geometry-aware front end, 25 interpretable spatiotemporal branches, and a two-block convolutional decoder.
39.75 ± 0.34%

Top-1 accuracy among 1,005 candidate speech segments

Chance: 0.10%; mean across six trained solutions
70.4%

Top-10 accuracy among the same 1,005 candidates

Chance: 1.0%
486,619

trainable parameters in the complete MEG decoder

The decoder portion uses about 1/20 as many parameters as Défossez et al.
15 of 19

tested stimulus features contribute to retrieval

Paired MEG occlusion; FWER-corrected

01 / Overview

Accurate retrieval does not explain what was decoded.

The problem

Conventional decoders stop at performance.

Deep networks can match short MEG segments to the speech a person is hearing, but their learned representations do not correspond to cortical locations, neural rhythms, or time courses. A correct retrieval therefore says little about the neural evidence that produced it.

LISA’s answer

Make the evidence readable by construction.

LISA uses a physically grounded, factorized front end whose spatial and temporal weights can be converted into source patterns and dynamics. Paired MEG interventions then test which properties of the speech stream actually support retrieval.

02 / Method

Physics first. Then physiology. Then retrieval.

Each constraint preserves a direct interpretation without giving up the expressive power needed for natural speech.

01

Geometry-aware spatial attention

Spherical harmonics parameterize attention directly on the three-dimensional MEG helmet instead of flattening the sensor layout.

02

Interpretable branches

Twenty-five subject-specific branches pair spatial filters with 150 ms temporal kernels, matching each branch to a source in space and time.

03

Compact contrastive decoder

Two residual convolutional blocks align three-second MEG segments with wav2vec 2.0 audio embeddings using a CLIP-style objective.

208 MEG sensors
270 virtual channels
25 interpretable branches
2 residual blocks
768 × 149 aligned embedding

03 / Findings

From a correct match back to its neural evidence.

Finding 01

Learned weights recover a speech-perception network.

Spatial patterns projected to source space identify generators in bilateral auditory cortex, medial temporal regions, and frontal cortex. Left-localized branches carry higher-frequency rhythmic components not evident on the right.

  • Spatial filters are converted to activation patterns.
  • Temporal kernels reveal branch-specific dynamics.
  • Source estimates land on a shared cortical template.
Axial, coronal, and sagittal views showing fitted source dipoles concentrated around bilateral auditory, temporal, and frontal regions.
Dominant dipoles from the leading cross-subject spatial patterns, shown in axial, coronal, and sagittal views.

Inside the branches

Space, time, rhythm, and source remain connected.

Each column follows one learned branch cluster from its sensor topography through its temporal response and spectrum to a cortical source estimate.

Six representative learned branch clusters, each shown from sensor topography through temporal pattern and spectrum to cortical source estimates.
Six representative clusters from the main 25-branch model. Every column preserves the complete path from sensor space to source space.

Finding 02

The decoder relies most on acoustic state and transitions.

Paired MEG occlusion compares matched replacements drawn from feature-present and feature-absent intervals. Fifteen of nineteen tested features contribute reliably; silence, sound intensity, vowels, and acoustic onsets produce the largest effects.

A revealing reversal: replacing random-word-list activity with narrative MEG improves retrieval, suggesting that coherent speech evokes more recoverable information.

Participant-level rank differences for nineteen stimulus-feature interventions, with null-distribution violins and significance markers.
Gray points show participants, diamonds show observed effects, and green violins show the paired null distribution. Positive values mean that feature-absent replacement worsened retrieval more than the matched control. Significance markers reflect one-sided tests for positive effects.

Finding 03

The recoverable target space is surprisingly small.

A projection learned with the retrieval objective compresses the 768-dimensional wav2vec 2.0 target to about twelve dimensions with essentially no loss. Unsupervised PCA degrades much sooner, showing that MEG-relevant directions are not simply the directions of greatest audio variance.

64×feature compression
without accuracy loss
Top-1 and Top-10 accuracy across embedding dimensions. Learned linear reduction remains near full-model accuracy down to roughly twelve dimensions, while PCA degrades earlier.
Task-driven feature reduction preserves retrieval in a compact subspace.

Additional result

Accuracy rises steadily with segment duration.

For the two-block model, Top-1 retrieval climbs from 14.37% at 1.5 seconds to 62.20% at 5 seconds. The monotonic trend holds with zero, two, and five downstream convolutional blocks.

Top-1 and Top-10 retrieval accuracy from 1.5 to 5 second MEG segments for decoders with zero, two, and five convolutional blocks.
All durations use the same 991-candidate test set. The monotonic trend holds for decoders with zero, two, and five convolutional blocks.

04 / Resources

Read, reproduce, and adapt.

PDF PublicationSemenkov et al. (2026) · arXiv:2608.01481 GIT Source codeTraining, evaluation, and analysis DOC Reproduce the paperComplete experiment, table, and figure workflow DOC Use LISA on your dataAdapt the encoder to another task or dataset DATA MEG-MASCNaturalistic speech MEG dataset

05 / Citation

If this work helps your research, please cite it.

Publication

Semenkov, I., Kleeva, D., Dakhtin, I., Maksudova, Z., & Ossadtchi, A. (2026). Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval. arXiv:2608.01481 [cs.LG].

@misc{semenkov2026interpretablemegdecodingperceived,
      title={Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval},
      author={Ilia Semenkov and Daria Kleeva and Ivan Dakhtin and Zarina Maksudova and Alex Ossadtchi},
      year={2026},
      eprint={2608.01481},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.01481},
}