GabrielTofvesson/tiktoken
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove redundant word in docstring (#7)

Browse Source 
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
This commit is contained in:
Nguyen-Khanh Vu 2022-12-17 03:41:08 +01:00 committed by GitHub
parent 4226a6c008
commit 0f8ec705e2
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
tiktoken/core.py
View File

@ -68,7 +68,7 @@ class Encoding:
and `disallowed_special` parameters. In particular: and `disallowed_special` parameters. In particular:
- Setting `disallowed_special` to () will prevent this function from raising errors and - Setting `disallowed_special` to () will prevent this function from raising errors and
cause all text corresponding to special tokens to be encoded as natural text. cause all text corresponding to special tokens to be encoded as natural text.
- Setting `allowed_special` to "all" will allow cause this function to treat all text - Setting `allowed_special` to "all" will cause this function to treat all text
corresponding to special tokens to be encoded as special tokens. corresponding to special tokens to be encoded as special tokens.
``` ```