GabrielTofvesson/tiktoken
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

README.md: minor improvements

Browse Source
This commit is contained in:
Shantanu Jain 2022-12-16 03:26:13 -06:00
parent 1f098ca4d7
commit ab3688a401
1 changed files with 3 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
README.md
View File

@ -1,11 +1,12 @@
# ⏳ tiktoken
tiktoken is a fast tokeniser.
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with
OpenAI's models.
```python
import tiktoken
enc = tiktoken.get_encoding("gpt2")
print(enc.encode("hello world"))
assert enc.decode(enc.encode("hello world")) == "hello world"
```
The open source version of `tiktoken` can be installed from PyPI: