microsoft /unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
★ 22,242⑂ 2,696👁 292⑃ 301● 1,454▤ 4
More from this author
+0
Daily
+5
Weekly
+3
Monthly
22,242
stars
2,696
forks
292
watchers
301
prs
1,454
issues
4
releases
README
aka.ms/GeneralAI Hiring We are hiring at all levels (including FTE researchers and interns)! If you are interested in working with us on Foundation Models (aka large-scale pre-trained models) and General AI, NLP, MT, Speech, Document AI and Multimodal AI, please send your resume to [email protected]. Foundation Architecture TorchScale - A Library of Foundation Architectures (repo) Fundamental research to develop new architectures for foundation models and AI, focusing on modeling generality and capability, as well as training stability and efficiency. Stability - DeepNet: scaling Transformers to 1,000 Layers and beyond Generality - Foundation Transformers (Magneto): towards true general-purpose modeling across tasks and modalities (including language, vision, speech, and multimodal) Capability - A Length-Extrapolatable Transformer Efficiency & Transferability - X-MoE: scalable & finetunable sparse Mixture-of-Experts (MoE) The Revolution of Model Architecture BitNet: 1-bit Transformers for Large Language Models RetNet: Retentive Network: A Successor to Transformer for Large Language Models LongNet: Scaling Transformers to 1,000,000,000 Tokens Foundation Models The Evolution of (M)LLM (Multimodal LLM) Kosmos-2.5: A Multimodal Literate Model Kosmos-2: Grounding Multimodal Large Language Models to the World Kosmos-1: A Multimodal Large Language Model (MLLM) MetaLM: Language Models are General-Purpose Interfaces The Big Convergence - Large-scale self-supervised pre-training across tasks (predictive and generative), languages (100+ languages), and modalities (language, image, audio, layout/format + language, vision + language, audio + language, etc.) Language & Multilingual UniLM: unified pre-training for language understanding and generation InfoXLM/XLM-E: multilingual/cross-lingual pre-trained models for 100+ languages DeltaLM/mT6: encoder-decoder pre-training for language generation and translation for 100+ languages MiniLM: small and fast pre-trained models for language understanding and generation AdaLM: domain, language, and task adaptation of pre-trained models EdgeLM(NEW): small pre-trained models on edge/client devices SimLM (NEW): large-scale pre-training for similarity matching E5 (NEW): text embeddings MiniLLM (NEW): Knowledge Distillation of Large Language Models Vision BEiT/BEiT-2: generative self-supervised pre-training for vision / BERT Pre-Training of Image Transformers DiT: self-supervised pre-training for Document Image Transformers TextDiffuser/TextDiffuser-2 (NEW): Diffusion Models as Text Painters Speech WavLM: speech pre-training for full stack tasks VALL-E: a neural codec language model for TTS Multimodal (X + Language) LayoutLM/LayoutLMv2/LayoutLMv3: multimodal (text + layout/format + image) Document Foundation Model for Document AI (e.g. scanned documents, PDF, etc.) LayoutXLM: multimodal (text + layout/format + image) Document Foundation Model for multi
Top issues
#627Finetune Trocr on my own dataset35 comments · 2022-02-18#1429[Kosmos-2] Fine-tune your checkpoint model on my downstream task22 comments · 2024-01-12#559LayoutReader: How can I inference if I only have layout information data?22 comments · 2021-12-05#1179TextDiffusers: question about Mario-LAION annotations21 comments · 2023-07-05#213How to use new image for prediction21 comments · 2020-08-05#464Unclear input data structure of layoutreader18 comments · 2021-10-07#170Question generation using multi-lingual minilm 18 comments · 2020-06-06#1336[kosmos-g] Problem about docker image setup15 comments · 2023-10-17
Latest release
YOCO yoco.v0 2024-05-09https://arxiv.org/abs/2405.05254