GitHub - Sys-Nexus/DreamConnect: [VINT '25] Connecting Dreams with Visual Brainstorming Instruction (original) (raw)

Connecting Dreams with Visual Brainstorming Instruction

Yasheng Sun, Bohan Li, Mingchen Zhuge, Deng-Ping Fan, Salman Khan, Fahad Shahbaz Khan,Hideki Koike

Paper

We aim to develop a straightforward framework that uses other modalities, such as natural language, to translate the original “dreamland”. We present DreamConnect, employing a dual-stream diffusion framework to manipulate visually stimulated brain signals. By integrating an asynchronous diffusion strategy, our framework establishes an effective interface with human “dreams”, progressively refining their final imagery synthesis.

Table of Content

News

Step-by-step Installation Instructions

**a. Create a conda virtual environment and activate it.**It requires python >= 3.7 as base environment.

conda create -n sssp python=3.7 -y conda activate sssp

b. Install PyTorch and torchvision following the official instructions.

conda install pytorch==1.10.0 torchvision==0.8.2 -c pytorch -c conda-forge

**c. Install other dependencies.**We simply freeze our environments. Other environments might also works. Here we provide requirements.txt file for reference.

pip install -r requirements.txt

Note that the transformers==1.19.2 is strictly required.

Prepare Data

Pretrained Model

Instructions for Testing the Model

Step 1: Update the Checkpoint Path

Open the configuration file located at configs/test/test_res_value_inject_idback_css15.yaml and update the checkpoint path to match the paths of the downloaded models.

Step 2: Pre-Aligned Features for Convenience

For ease of use, we provide the following pre-aligned features:

If you are interested in training a alignment model by yourself, please follow the overall procedure fMRI-reconstruction-NSD. We provide the our trained alignment model for img_clip and text_clip. Download them and place to the directory of train_logs/latent_diffusion_image_fp32_resume/ and train_logs/latent_diffusion_text_fp32_resume2/ accordingly. Then, you can run below commands to obtain the above provided img_clip and text_clip files.

bash experiments/diffusion_test.sh image
bash experiments/diffusion_test.sh text

Step 3: Testing the Model

Once the paths are updated, you can test the model by running the following command:

bash experiments/test_language_control.sh

Acknowledgements

Many thanks to these excellent open source projects:

Citation

If you find our paper and code useful for your research, please consider citing:

@misc{sun2024connectingdreamsvisualbrainstorming, title={Connecting Dreams with Visual Brainstorming Instruction}, author={Yasheng Sun and Bohan Li and Mingchen Zhuge and Deng-Ping Fan and Salman Khan and Fahad Shahbaz Khan and Hideki Koike}, year={2025}, journal={Viusal Intelligence} }