jzhang38 /TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

★ 9,001⑂ 634👁 1⑃ 43● 165▤ 0
PythonApache-2.0Created 2023-09-02
+0
Daily
9,001
stars
634
forks
1
watchers
43
prs
165
issues
0
releases

README

TinyLlama-1.1B
English | 中文

Chat Demo | Discord


The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. With some proper optimization, we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs 🚀🚀. The training has started on 2023-09-01. 


  


We adopted exactly the same architecture and tokenizer as Llama 2. This means TinyLlama can be plugged and played in many open-source projects built upon Llama. Besides, TinyLlama is compact with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint.

 News
- 2023-12-18: Add two notes 1, 2 explaining the changes of training curves, project schedules, and bug fixes.
- 2023-10-03: Add examples in speculative decoding with llama.cpp. Do check out the speculative_decoding/README.md.
- 2023-10-02: 1. 1T-token checkpoint just dropped. 2. We document all intermediate checkpoints here.
- 2023-09-28: Add a discord server.
- 2023-09-18: 1. We added a chat demo so that you can play with TinyLlama-Chat-V0.1 right away. 
- 2023-09-16: 1. We released the intermediate checkpoint trained on 503B tokens. 2. We released a chat model finetuned on OpenAssisant and simple finetuning scripts is added. 3. More eval benchmarks are added and documented in EVAL.md. 

 Evaluation
You can find the evaluation results of TinyLlama in EVAL.md.

 Releases Schedule
We will be rolling out intermediate checkpoints following the below schedule. 

Base models:

| Date       | HF Checkpoint                                   | Tokens | Step | Commonsense Avg |
|------------|-------------------------------------------------|--------|------| --------------- |
| 2023-09-01 | Pythia-1.0B                                     | 300B   | 143k   | 48.30 |
| 2023-09-04 | TinyLlama-1.1B-intermediate-step-50k-105b | 105B   | 50k   | 46.11|
| 2023-09-16 | TinyLlama-1.1B-intermediate-step-240k-503b                                            | 503B   | 240K    | 48.28 |
| 2023-10-01 | TinyLlama-1.1B-intermediate-step-480k-1T | 1T     | 480k | 50.22 |
| 2023-11-04 | TinyLlama-1.1B-intermediate-step-715k-1.5T                                            | 1.5T     |715k    |51.28 |
| 2023-11-20 | TinyLlama-1.1B-intermediate-step-955k-2T                                            | 2T     |955k    |51.64 |
| 2023-12-11 | TinyLlama-1.1B-intermediate-step-1195k-2.5T              | 2.5T     | 1195k    |53.86 |
| 2023-12-28 | TinyLlama-1.1B-intermediate-step-1431k-3T              | 3T   | 1431k  | 52.99 |

We are crafting a note offering possible explaination on why there is a significant improvement from 2T to 2.5T checkpoint (It is related to bos_id issue)

Chat models:

| Date       | HF Checkpoint                                   | Tokens | Step | Commonsense Avg |
|------------|-------------------------------------------------|--------|------| --------------- |
| 2023-09-16 | TinyLlama-1.1B-Chat-V0.1                              

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON