GabrielTofvesson/tiktoken
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update the perf image path in readme (#68)

Browse Source 
Closes #67
This commit is contained in:
Mariatta Wijaya 2023-03-17 13:46:32 -07:00 committed by GitHub
parent 529de22652
commit 82facf911f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -27,7 +27,7 @@ Example code using `tiktoken` can be found in the
`tiktoken` is between 3-6x faster than a comparable open source tokeniser: `tiktoken` is between 3-6x faster than a comparable open source tokeniser:
![image](./perf.svg) ![image](https://raw.githubusercontent.com/openai/tiktoken/main/perf.svg)
Performance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from Performance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from
`tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`. `tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`.