salesforce /BLIP
PyTorch code for BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
★ 5,700⑂ 764👁 31⑃ 24● 211▤ 0
vision-languagevision-and-language-pre-trainingimage-text-retrievalimage-captioningvisual-question-answeringvisual-reasoningvision-language-transformer
More from this author
+0
Daily
5,700
stars
764
forks
31
watchers
24
prs
211
issues
0
releases
README
This repo is DEPRECATED and no longer supported - We recommend that you do not use the content in this repo for any production or sensitive purpose
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Announcement: BLIP is now officially integrated into LAVIS - a one-stop library for language-and-vision research and applications!
This is the PyTorch code of the BLIP paper [blog]. The code has been tested on PyTorch 1.10.
To install the dependencies, run pip install -r requirements.txt
Catalog:
- [x] Inference demo
- [x] Pre-trained and finetuned checkpoints
- [x] Finetuning code for Image-Text Retrieval, Image Captioning, VQA, and NLVR2
- [x] Pre-training code
- [x] Zero-shot video-text retrieval
- [x] Download of bootstrapped pre-training datasets
Inference demo:
Run our interactive demo using Colab notebook (no GPU needed).
The demo includes code for:
1. Image captioning
2. Open-ended visual question answering
3. Multimodal / unimodal feature extraction
4. Image-text matching
Try out the Web demo, integrated into Huggingface Spaces 🤗 using Gradio.
Replicate web demo and Docker image is also available at
Pre-trained checkpoints:
Num. pre-train images | BLIP w/ ViT-B | BLIP w/ ViT-B and CapFilt-L | BLIP w/ ViT-L
--- | :---: | :---: | :---:
14M | Download| - | -
129M | Download| Download | Download
Finetuned checkpoints:
Task | BLIP w/ ViT-B | BLIP w/ ViT-B and CapFilt-L | BLIP w/ ViT-L
--- | :---: | :---: | :---:
Image-Text Retrieval (COCO) | Download| - | Download
Image-Text Retrieval (Flickr30k) | Download| - | Download
Image Captioning (COCO) | - | Download| Download |
VQA | Download| Download | -
NLVR2 | Download| - | -
Image-Text Retrieval:
1. Download COCO and Flickr30k datasets from the original websites, and set 'image_root' in configs/retrieval_{dataset}.yaml accordingly.
2. To evaluate the finetuned BLIP model on COCO, run:
python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \
--config ./configs/retrieval_coco.yaml \
--output_dir output/retrieval_coco \
--evaluate
3. To finetune the pre-trained checkpoint using 8 A100 GPUs, first set 'pretrained' in configs/retrieval_coco.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth". Then run:
python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \
--config ./configs/retrieval_coco.yaml \
--output_dir output/retrieval_coco
Image-Text Captioning:
1. Download COCO and NoCaps datasets from the original websites, and set 'image_root' in configs/caption_coco.yaml and configs/nocaps.yaml accordingly.
2. To evaluate the finetuned BLIP model on COCO, run:
python -m torch.distributed.run --nproc_per_node=8 train_caption.py --evaluate
3. To evaluate the finetuned BLIP model on NoCaps, generate results with: (evaluation needs to be performed on official server)
python -m torch.distributed.run --nproc_per_node=8 eval_nocaps.py
4. To finetunTop issues
#165The size of tensor a (3) must match the size of tensor b (9) at non-singleton dimension 021 comments · 2023-07-07#37Fine tune BLIP Image Captioning to custom dataset21 comments · 2022-03-22#4minimizing difference across image and text features13 comments · 2022-02-02#77Continuously increasing RAM with Pre-training12 comments · 2022-07-05#43Training with one GPU12 comments · 2022-03-31#155BLIP Image Captioning GradCAM?8 comments · 2023-05-22#18Inference code for video-text retrieval on MSRVTT!8 comments · 2022-02-17#175Can BLIP generate more words image caption?7 comments · 2023-08-09