From ab3688a4016b4530231690f430c702727bb0da24 Mon Sep 17 00:00:00 2001 From: Shantanu Jain Date: Fri, 16 Dec 2022 03:26:13 -0600 Subject: [PATCH] README.md: minor improvements --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index cc1099c..8eaa823 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,12 @@ # ⏳ tiktoken -tiktoken is a fast tokeniser. +tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with +OpenAI's models. ```python import tiktoken enc = tiktoken.get_encoding("gpt2") -print(enc.encode("hello world")) +assert enc.decode(enc.encode("hello world")) == "hello world" ``` The open source version of `tiktoken` can be installed from PyPI: