starred/ollama-ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-04-18 00:03:27 +02:00
Code Issues Packages Projects Releases Wiki Activity
5,249 Commits 384 Branches 504 Tags
cb0033598ec36c9d10eceef8a0ba4e02329c5b35
Commit Graph

2 Commits

Author SHA1 Message Date
Daniel Hiltgen
cb0033598e tokenizer: add SentencePiece-style BPE support (#15162) 
* tokenizer: add SentencePiece-style BPE support

Add WithSentencePieceNormalizer option to BytePairEncoding for models
that use BPE with SentencePiece-style space markers (space to/from
U+2581).

NewBytePairEncoding is unchanged; the new NewBytePairEncodingWithOptions
constructor accepts BPEOption functions. Decoding handles the reverse
mapping of U+2581 back to spaces.

* review comments
 2026-03-31 17:00:36 -07:00
Michael Yang
f1373193dc move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00