+0
Daily
9,001
stars
634
forks
1
watchers
43
prs
165
issues
0
releases
README
TinyLlama-1.1B English | 中文 Chat Demo | Discord The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. With some proper optimization, we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs 🚀🚀. The training has started on 2023-09-01. We adopted exactly the same architecture and tokenizer as Llama 2. This means TinyLlama can be plugged and played in many open-source projects built upon Llama. Besides, TinyLlama is compact with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint. News - 2023-12-18: Add two notes 1, 2 explaining the changes of training curves, project schedules, and bug fixes. - 2023-10-03: Add examples in speculative decoding with llama.cpp. Do check out the speculative_decoding/README.md. - 2023-10-02: 1. 1T-token checkpoint just dropped. 2. We document all intermediate checkpoints here. - 2023-09-28: Add a discord server. - 2023-09-18: 1. We added a chat demo so that you can play with TinyLlama-Chat-V0.1 right away. - 2023-09-16: 1. We released the intermediate checkpoint trained on 503B tokens. 2. We released a chat model finetuned on OpenAssisant and simple finetuning scripts is added. 3. More eval benchmarks are added and documented in EVAL.md. Evaluation You can find the evaluation results of TinyLlama in EVAL.md. Releases Schedule We will be rolling out intermediate checkpoints following the below schedule. Base models: | Date | HF Checkpoint | Tokens | Step | Commonsense Avg | |------------|-------------------------------------------------|--------|------| --------------- | | 2023-09-01 | Pythia-1.0B | 300B | 143k | 48.30 | | 2023-09-04 | TinyLlama-1.1B-intermediate-step-50k-105b | 105B | 50k | 46.11| | 2023-09-16 | TinyLlama-1.1B-intermediate-step-240k-503b | 503B | 240K | 48.28 | | 2023-10-01 | TinyLlama-1.1B-intermediate-step-480k-1T | 1T | 480k | 50.22 | | 2023-11-04 | TinyLlama-1.1B-intermediate-step-715k-1.5T | 1.5T |715k |51.28 | | 2023-11-20 | TinyLlama-1.1B-intermediate-step-955k-2T | 2T |955k |51.64 | | 2023-12-11 | TinyLlama-1.1B-intermediate-step-1195k-2.5T | 2.5T | 1195k |53.86 | | 2023-12-28 | TinyLlama-1.1B-intermediate-step-1431k-3T | 3T | 1431k | 52.99 | We are crafting a note offering possible explaination on why there is a significant improvement from 2T to 2.5T checkpoint (It is related to bos_id issue) Chat models: | Date | HF Checkpoint | Tokens | Step | Commonsense Avg | |------------|-------------------------------------------------|--------|------| --------------- | | 2023-09-16 | TinyLlama-1.1B-Chat-V0.1
Top issues
#10How do you plan on dealing with hallucinations due to knowledge compression?16 comments · 2023-09-07#36Resuming training8 comments · 2023-09-22#141Does it support spanish language?7 comments · 2024-01-17#77Why is there a significant drop in `val_ppl` after fixing data-loading bug?7 comments · 2023-10-25#74The queries generated a lot of repetitions. Possible to provide 1T again "fix"?6 comments · 2023-10-20#124Add support for sliding window context similar to Mistral?4 comments · 2024-01-03#168Help me pls3 comments · 2024-03-17#203is somebody knows how many validation file there are?2 comments · 2025-03-08