deepseek-ai /Janus
Janus-Series: Unified Multimodal Understanding and Generation Models
★ 17,769⑂ 2,229👁 143⑃ 45● 193▤ 0
+3
Daily
17,769
stars
2,229
forks
143
watchers
45
prs
193
issues
0
releases
README
🚀 Janus-Series: Unified Multimodal Understanding and Generation Models
--
--
--
📥 Model Download |
⚡ Quick Start |
📜 License |
📖 Citation
Janus, JanusFlow) | --
🤗 Online Demo (Janus-Pro-7B, Janus, JanusFlow)
News
2025.01.27: Janus-Pro is released, an advanced version of Janus, improving both multimodal understanding and visual generation significantly. See paper
2024.11.13: JanusFlow is released, a new unified model with rectified flow for image generation. See paper, demo and usage.
2024.10.23: Evaluation code for reproducing the multimodal understanding results from the paper has been added to VLMEvalKit. Please refer to this link.
2024.10.20: (1) Fix a bug in tokenizer_config.json. The previous version caused classifier-free guidance to not function properly, resulting in relatively poor visual generation quality. (2) Release Gradio demo (online demo and local).
1. Introduction
Janus-Pro: Unified Multimodal Understanding and
Generation with Data and Model Scaling
Janus-Pro is an advanced version of the previous work Janus. Specifically, Janus-Pro incorporates (1) an optimized training strategy, (2) expanded training data, and (3) scaling to larger model size. With these improvements, Janus-Pro achieves significant advancements in both multimodal understanding and text-to-image instruction-following capabilities, while also enhancing the stability of text-to-image generation.
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Janus is a novel autoregressive framework that unifies multimodal understanding and generation. It addresses the limitations of previous approaches by decoupling visual encoding into separate pathways, while still utilizing a single, unified transformer architecture for processing. The decoupling not only alleviates the conflict between the visual encoder’s roles in understanding and generation, but also enhances the framework’s flexibility. Janus surpasses previous unified model and matches or exceeds the performance of task-specific models. The simplicity, high flexibility, and effectiveness of Janus make it a strong candidate for next-generation unified multimodal models.
JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
JanusFlow introduces a minimalist architecture that integrates autoregressive language models with rectified flow, a state-of-the-art method in generative modeling. Our key finding demonstrates that rectified flow can be straightforwardly trained within the large language model framework, eliminating the need for complex architectural modifications. Extensive experiments show that JanusFlow achieves comparable or superior performance to specialized models in their respective domains, while significantly outperforming existing unified approaches across standard benchmarks. TTop issues
#62编译```sentencepiece```失败22 comments · 2025-01-28#55Jenus Pro for Mac17 comments · 2025-01-28#41cuda out of memory, but its not16 comments · 2025-01-27#40what ? ?11 comments · 2025-01-27#68model type `multi_modality` but Transformers does not recognize this architecture10 comments · 2025-01-28#212evaluation8 comments · 2025-04-15#147Fine tuning8 comments · 2025-02-06#73Code for finetuning Janus Pro and Janus Flow8 comments · 2025-01-28