A study, recently published in Science Advances, takes a new approach. Researchers involved in the study have developed what they refer to as a “mind-captioning” technique that uses an iterative optimization process, where a masked language model (MLM) generates text descriptions by aligning text features with brain-decoded features.

The technique also incorporates linear models trained to decode semantic features from a deep language model using brain activity from functional magnetic resonance imaging (fMRI). The result is a detailed text description of what a participant is seeing in their brain.