Vision-CAIR /MiniGPT-4
Open-sourced codes for MiniGPT-4 and MiniGPT-v2 (https://minigpt-4.github.io, https://minigpt-v2.github.io/)
★ 25,558⑂ 2,854👁 209⑃ 57● 482▤ 0
-1
Daily
-17
Weekly
-30
Monthly
25,558
stars
2,854
forks
209
watchers
57
prs
482
issues
0
releases
README
MiniGPT-V
MiniGPT-v2: Large Language Model as a Unified Interface for Vision-Language Multi-task Learning
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong☨, Mohamed Elhoseiny☨
☨equal last author
MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny
equal contribution
King Abdullah University of Science and Technology
💡 Get help - Q&A or Discord 💬
Example Community Efforts Built on Top of MiniGPT-4
InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4 Lai Wei, Zihao Jiang, Weiran Huang, Lichao Sun, Arxiv, 2023
PatFig: Generating Short and Long Captions for Patent Figures.", Aubakirova, Dana, Kim Gerdes, and Lufei Liu, ICCVW, 2023
SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Model, Juexiao Zhou and Xiaonan He and Liyuan Sun and Jiannan Xu and Xiuying Chen and Yuetan Chu and Longxi Zhou and Xingyu Liao and Bin Zhang and Xin Gao, Arxiv, 2023
ArtGPT-4: Artistic Vision-Language Understanding with Adapter-enhanced MiniGPT-4.", Yuan, Zhengqing, Huiwen Xue, Xinyi Wang, Yongming Liu, Zhuanzhe Zhao, and Kun Wang, Arxiv, 2023
News
[Oct.31 2023] We release the evaluation code of our MiniGPT-v2.
[Oct.24 2023] We release the finetuning code of our MiniGPT-v2.
[Oct.13 2023] Breaking! We release the first major update with our MiniGPT-v2
[Aug.28 2023] We now provide a llama 2 version of MiniGPT-4
Online Demo
Click the image to chat with MiniGPT-v2 around your images
Click the image to chat with MiniGPT-4 around your images
MiniGPT-v2 Examples
!MiniGPT-v2 demos
MiniGPT-4 Examples
| | |
:-------------------------:|:-------------------------:
!find wild | !write story
!solve problem | !write Poem
More examples can be found in the project page.
Getting Started
Installation
1. Prepare the code and the environment
Git clone our repository, creating a python environment and activate it via the following command
bash
git clone https://github.com/Vision-CAIR/MiniGPT-4.git
cd MiniGPT-4
conda env create -f environment.yml
conda activate minigptv
2. Prepare the pretrained LLM weights
MiniGPT-v2 is based on Llama2 Chat 7B. For MiniGPT-4, we have both Vicuna V0 and Llama 2 version.
Download the corresponding LLM weights from the following huggingface space via clone the repository using git-lfs.
| Llama 2 Chat 7B | Vicuna V0 13B | Vicuna V0 7B |
:------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------Top issues
#237torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 31 comments · 2023-05-19#50有人成功了吗?28 comments · 2023-04-19#146部署成功了,但是识别图片结果为什么看起来很离谱26 comments · 2023-04-25#74Probability tensor contains either `inf`, `nan` or element < 025 comments · 2023-04-20#28Windows not supported19 comments · 2023-04-18#59无法结合原始重量和增重18 comments · 2023-04-19#46Loading the model on multiple GPUs18 comments · 2023-04-19#294Now you can run MiniGPT-4 locally!!!!17 comments · 2023-07-20
Discussions
#562MiniGPT-4 has a page on Shypd (AI Tool Directory)0 comments · 2026-03-30#559Dependency management: a working requirements.txt for demo.py with Cuda0 comments · 2025-08-07#556Request assistance for the situation where the dataset and checkpoint links cannot be opened during the fine-tuning of MiniGPT-40 comments · 2025-05-09#541Runtime Error: Online MiniGPT-4 Demo Not Working0 comments · 2024-12-23#529Strange modify date on pretrained data0 comments · 2024-09-19#515I want to just do the Inference on my custom dataset. I have Q/A file for my dataset.0 comments · 2024-05-26#498Can this model be used commercially0 comments · 2024-03-11#468Fine tuning after stage 1 ckpoint using Llama-21 comments · 2023-12-25