openai /tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
★ 19,668⑂ 1,680👁 208⑃ 235● 346▤ 15
+4
Daily
19,668
stars
1,680
forks
208
watchers
235
prs
346
issues
15
releases
README
⏳ tiktoken
tiktoken is a fast BPE tokeniser for use with
OpenAI's models.
python
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"
To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")
The open source version of tiktoken can be installed from PyPI:
pip install tiktoken
The tokeniser API is documented in tiktoken/core.py.
Example code using tiktoken can be found in the
OpenAI Cookbook.
Performance
tiktoken is between 3-6x faster than a comparable open source tokeniser:
!image
Performance measured on 1GB of text using the GPT-2 tokeniser, using GPT2TokenizerFast from
tokenizers==0.13.2, transformers==4.24.0 and tiktoken==0.2.0.
Getting help
Please post questions in the issue tracker.
If you work at OpenAI, make sure to check the internal documentation or feel free to contact
@shantanu.
What is BPE anyway?
Language models don't see text like you and I, instead they see a sequence of numbers (known as tokens).
Byte pair encoding (BPE) is a way of converting text into tokens. It has a couple desirable
properties:
1) It's reversible and lossless, so you can convert tokens back into the original text
2) It works on arbitrary text, even text that is not in the tokeniser's training data
3) It compresses the text: the token sequence is shorter than the bytes corresponding to the
original text. On average, in practice, each token corresponds to about 4 bytes.
4) It attempts to let the model see common subwords. For instance, "ing" is a common subword in
English, so BPE encodings will often split "encoding" into tokens like "encod" and "ing"
(instead of e.g. "enc" and "oding"). Because the model will then see the "ing" token again and
again in different contexts, it helps models generalise and better understand grammar.
tiktoken contains an educational submodule that is friendlier if you want to learn more about
the details of BPE, including code that helps visualise the BPE procedure:
python
from tiktoken._educational import
Train a BPE tokeniser on a small amount of text
enc = train_simple_encoding()
Visualise how the GPT-4 encoder encodes text
enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
enc.encode("hello world aaaaaaaaaaaa")
Extending tiktoken
You may wish to extend tiktoken to support new encodings. There are two ways to do this.
Create your Encoding object exactly the way you want and simply pass it around.
python
cl100k_base = tiktoken.get_encoding("cl100k_base")
In production, load the arguments directly instead of accessing private attributes
See openai_public.py for examples of arguments for specific encodings
enc = tiktoken.Encoding(
If you're changing the set of special tokens, make sure to use a different name
It should be clear from the name what behaviour to expect.
name="cl100k_im",
pat_str=cl100k_base._pat_str,
mergeable_raTop issues
#245Panic (stack overflow) when encoding a certain string13 comments · 2024-01-18#195Very slow for inputs like "a" * 10000012 comments · 2023-09-18#97Unofficial bindings / ports in other languages12 comments · 2023-04-05#24Tiktoken not published to cargo12 comments · 2023-01-24#281SSLError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url9 comments · 2024-04-10#22NPM package7 comments · 2023-01-22#511[RFC] Korean Tokenization: Jamo Decomposition as Pre-tokenizer (A Blueprint for Compositional Scripts)6 comments · 2026-03-22#366tiktoken under counting tokens for openai text-embedding-3-large6 comments · 2025-01-09