salesforce /BLIP

PyTorch code for BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

★ 5,700⑂ 764👁 31⑃ 24● 211▤ 0
Jupyter NotebookBSD-3-ClauseCreated 2022-01-25
More from this author
+0
Daily
5,700
stars
764
forks
31
watchers
24
prs
211
issues
0
releases

README

This repo is DEPRECATED and no longer supported - We recommend that you do not use the content in this repo for any production or sensitive purpose

 BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

 Announcement: BLIP is now officially integrated into LAVIS - a one-stop library for language-and-vision research and applications!



This is the PyTorch code of the BLIP paper [blog]. The code has been tested on PyTorch 1.10.
To install the dependencies, run pip install -r requirements.txt 

Catalog:
- [x] Inference demo
- [x] Pre-trained and finetuned checkpoints
- [x] Finetuning code for Image-Text Retrieval, Image Captioning, VQA, and NLVR2
- [x] Pre-training code
- [x] Zero-shot video-text retrieval
- [x] Download of bootstrapped pre-training datasets 


 Inference demo:
Run our interactive demo using Colab notebook (no GPU needed).
The demo includes code for: 
1. Image captioning
2. Open-ended visual question answering
3. Multimodal / unimodal feature extraction
4. Image-text matching

Try out the Web demo, integrated into Huggingface Spaces 🤗 using Gradio. 

Replicate web demo and Docker image is also available at 

 Pre-trained checkpoints:
Num. pre-train images | BLIP w/ ViT-B | BLIP w/ ViT-B and CapFilt-L | BLIP w/ ViT-L 
--- | :---: | :---: | :---: 
14M | Download| - | -
129M | Download| Download | Download

 Finetuned checkpoints:
Task | BLIP w/ ViT-B | BLIP w/ ViT-B and CapFilt-L | BLIP w/ ViT-L 
--- | :---: | :---: | :---:
Image-Text Retrieval (COCO) | Download| - | Download
Image-Text Retrieval (Flickr30k) | Download|  - | Download
Image Captioning (COCO) | - | Download| Download | 
VQA | Download| Download | - 
NLVR2 | Download| - | - 


 Image-Text Retrieval:
1. Download COCO and Flickr30k datasets from the original websites, and set 'image_root' in configs/retrieval_{dataset}.yaml accordingly.
2. To evaluate the finetuned BLIP model on COCO, run:
python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \
--config ./configs/retrieval_coco.yaml \
--output_dir output/retrieval_coco \
--evaluate 
3. To finetune the pre-trained checkpoint using 8 A100 GPUs, first set 'pretrained' in configs/retrieval_coco.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth". Then run:
python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \
--config ./configs/retrieval_coco.yaml \
--output_dir output/retrieval_coco  

 Image-Text Captioning:
1. Download COCO and NoCaps datasets from the original websites, and set 'image_root' in configs/caption_coco.yaml and configs/nocaps.yaml accordingly.
2. To evaluate the finetuned BLIP model on COCO, run:
python -m torch.distributed.run --nproc_per_node=8 train_caption.py --evaluate 
3. To evaluate the finetuned BLIP model on NoCaps, generate results with: (evaluation needs to be performed on official server)
python -m torch.distributed.run --nproc_per_node=8 eval_nocaps.py  
4. To finetun

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON