GabrielTofvesson/tiktoken
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove duplicate words in docstring (#3)

Browse Source
This commit is contained in:
Henrik Torget 2022-12-16 22:55:30 +01:00 committed by GitHub
parent ab3688a401
commit 4226a6c008
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
tiktoken/core.py
View File

@ -59,7 +59,7 @@ class Encoding:
) -> list[int]: ) -> list[int]:
"""Encodes a string into tokens. """Encodes a string into tokens.
Special tokens are tokens are artificial tokens used to unlock capabilities from a model, Special tokens are artificial tokens used to unlock capabilities from a model,
such as fill-in-the-middle. So we want to be careful about accidentally encoding special such as fill-in-the-middle. So we want to be careful about accidentally encoding special
tokens, since they can be used to trick a model into doing something we don't want it to do. tokens, since they can be used to trick a model into doing something we don't want it to do.