数据集获取

git clone https://github.com/gegen666/EEGTo3D.git

curl -L -o repo.zip https://github.com/gegen666/EEGTo3D/archive/refs/heads/main.zip
unzip repo.zip

源站 README 摘录（使用方式）

3D-Telepathy: EEG-to-3D using 2D Diffusion

1.Abstract

Reconstructing 3D visual stimulus scenes from electroencephalogram (EEG) data holds significant potential applications in brain-computer interface and aiding expression in populations with communication impairments. As of now, no work on EEG to 3D conversion has been proposed. The principal challenge is EEG itself has numerous sources of noise, rendering it challenging to extract visual 3D information from it. To address the groundbreaking task of EEG-to-3D reconstruction, we use dual masked contrastive learning and multimodal joint self-supervised learning to obtain the EEG encoder. By leveraging 2D diffusion as a prior distribution and training neural radiance fields (NeRF) through variational score distillation (VSD), we obtain the 3D scenes with both visually pleasing effects and well-defined structures from EEG. Additionally, during the 3D generation, we could obtain a set of images depicting the same object from varied perspectives. This implies that we can derive multi-view visual and 3D-space information from EEG data that originally contained only single-view visual information. Overall, our work inaugurates a novel investigative trajectory by integrating the fields of neural signal decoding and 3D generation. It also offers novel insights into the neural representations of human perception of three-dimensional space.

2.Method Overview

3.Preparatory work

The datasets folder and pretrains folder are not included in this repository. Please download the EEG data from eeg. We also provide a copy of the Imagenet subset imagenet.
Throughout the entire process, we utilized the standard SD1.5.You can sownload the parameter from stable-diffusion-v1-5.

4.Training Procedure

4.1 MERL Stage1

python stageA1_mbm_pretrain_contrast.py \
output_path . \
lr 0.001 \ 
—-batch_size 10 \
—-do_self_contrast True \
—-do_cross_contrast True \
self_contrast_loss_weight 1 \ 
cross_contrast_loss_weight 0.5 \
—mask_ratio 0.75 \
—num_epoch 150 

4.2 MERL Stage2

python stageA2_mbm_finetune_cross.py \
pretrain_mbm_path your_pretrained_ckpt_from_phase1 \
batch_size 4 \
num_epoch 6


## 数据加载示例（图像类）

```python
from PIL import Image
import glob, os

files = (glob.glob(os.path.join(path, "**", "*.png"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.jpg"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.tif"), recursive=True))
print("图像文件数:", len(files))
img = Image.open(files[0]); print("尺寸/模式:", img.size, img.mode)

# torchvision Dataset 方式:
# from torchvision import datasets
# ds = datasets.ImageFolder(path)  # 要求 子目录=类别

目录组织与标注格式以源站说明和下载后实际文件为准。

完整仓库：github.com/gegen666/EEGTo3D