facebookresearch /ImageBind
ImageBind One Embedding Space to Bind Them All
★ 9,082⑂ 841👁 95⑃ 41● 103▤ 0
+0
Daily
9,082
stars
841
forks
95
watchers
41
prs
103
issues
0
releases
README
ImageBind: One Embedding Space To Bind Them All
FAIR, Meta AI
Rohit Girdhar,
Alaaeldin El-Nouby,
Zhuang Liu,
Mannat Singh,
Kalyan Vasudev Alwala,
Armand Joulin,
Ishan Misra
To appear at CVPR 2023 (Highlighted paper)
[Paper] [Blog] [Demo] [Supplementary Video] [BibTex]
PyTorch implementation and pretrained models for ImageBind. For details, see the paper: ImageBind: One Embedding Space To Bind Them All.
ImageBind learns a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. It enables novel emergent applications ‘out-of-the-box’ including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation.
!ImageBind
ImageBind model
Emergent zero-shot classification performance.
Model
IN1k
K400
NYU-D
ESC
LLVIP
Ego4D
download
imagebind_huge
77.7
50.0
54.0
66.9
63.4
25.0
checkpoint
Usage
Install pytorch 2.0+ and other 3rd party dependencies.
shell
conda create --name imagebind python=3.10 -y
conda activate imagebind
pip install .
For windows users, you might need to install soundfile for reading/writing audio files. (Thanks @congyue1977)
pip install soundfile
Extract and compare features across modalities (e.g. Image, Text and Audio).
python
from imagebind import data
import torch
from imagebind.models import imagebind_model
from imagebind.models.imagebind_model import ModalityType
text_list=["A dog.", "A car", "A bird"]
image_paths=[".assets/dog_image.jpg", ".assets/car_image.jpg", ".assets/bird_image.jpg"]
audio_paths=[".assets/dog_audio.wav", ".assets/car_audio.wav", ".assets/bird_audio.wav"]
device = "cuda:0" if torch.cuda.is_available() else "cpu"
Instantiate model
model = imagebind_model.imagebind_huge(pretrained=True)
model.eval()
model.to(device)
Load data
inputs = {
ModalityType.TEXT: data.load_and_transform_text(text_list, device),
ModalityType.VISION: data.load_and_transform_vision_data(image_paths, device),
ModalityType.AUDIO: data.load_and_transform_audio_data(audio_paths, device),
}
with torch.no_grad():
embeddings = model(inputs)
print(
"Vision x Text: ",
torch.softmax(embeddings[ModalityType.VISION] @ embeddings[ModalityType.TEXT].T, dim=-1),
)
print(
"Audio x Text: ",
torch.softmax(embeddings[ModalityType.AUDIO] @ embeddings[ModalityType.TEXT].T, dim=-1),
)
print(
"Vision x Audio: ",
torch.softmax(embeddings[ModalityType.VISION] @ embeddings[ModalityType.AUDIO].T, dim=-1),
)
Expected output:
Vision x Text:
tensor([[9.9761e-01, 2.3694e-03, 1.8612e-05],
[3.3836e-05, 9.9994e-01, 2.4118e-05],
[4.7997e-05, 1.3496e-02, 9.8646e-01]])
Audio x Text:
tensor([[1., 0., 0.],
[0., 1., 0.],
[0., 0., 1.]])
Vision x Audio:
tensor([[0.8070, 0.1088, 0.0842],
[0.1036, 0.7884, 0.1079],
[0.0018, 0.0022, 0.9960]])
Model card
Please see the model card for detTop issues
#66IMU Input Dimensions are Unclear - Missing Information on Data Prep10 comments · 2023-06-01#42How to use ImageBind to generate image or audio? 10 comments · 2023-05-12#16the minimum requirement of gpu?8 comments · 2023-05-10#47Finetuning ImageBind with LoRA7 comments · 2023-05-13#37Thermal x Vision Support6 comments · 2023-05-11#34Varying the sound length6 comments · 2023-05-11#27import data error in Google Colab6 comments · 2023-05-10#14How to use Depth embedding.6 comments · 2023-05-10