Rxivist logo

End-to-end deep image reconstruction from human brain activity

By Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, Yukiyasu Kamitani

Posted 27 Feb 2018
bioRxiv DOI: 10.1101/272518 (published DOI: 10.3389/fncom.2019.00021)

Deep neural networks (DNNs) have recently been applied successfully to brain decoding and image reconstruction from functional magnetic resonance imaging (fMRI) activity. However, direct training of a DNN with fMRI data is often avoided because the size of available data is thought to be insufficient to train a complex network with numerous parameters. Instead, a pre-trained DNN has served as a proxy for hierarchical visual representations, and fMRI data were used to decode individual DNN features of a stimulus image using a simple linear model, which were then passed to a reconstruction module. Here, we present our attempt to directly train a DNN model with fMRI data and the corresponding stimulus images to build an end-to-end reconstruction model. We trained a generative adversarial network with an additional loss term defined in a high-level feature space (feature loss) using up to 6,000 training data points (natural images and the fMRI responses). The trained deep generator network was tested on an independent dataset, directly producing a reconstructed image given an fMRI pattern as the input. The reconstructions obtained from the proposed method showed resemblance with both natural and artificial test stimuli. The accuracy increased as a function of the training data size, though not outperforming the decoded feature-based method with the available data size. Ablation analyses indicated that the feature loss played a critical role to achieve accurate reconstruction. Our results suggest a potential for the end-to-end framework to learn a direct mapping between brain activity and perception given even larger datasets.

Download data

  • Downloaded 5,750 times
  • Download rankings, all-time:
    • Site-wide: 533 out of 84,050
    • In neuroscience: 54 out of 14,962
  • Year to date:
    • Site-wide: 22,405 out of 84,050
  • Since beginning of last month:
    • Site-wide: 22,887 out of 84,050

Altmetric data


Downloads over time

Distribution of downloads per paper, site-wide


PanLingua

Sign up for the Rxivist weekly newsletter! (Click here for more details.)


News