ollama/tokenizer at 10e51c51771ea8536715876ee6707928712be41e - ollama - Brook's Gitea instance

bybrooklyn/ollama

mirror of https://github.com/ollama/ollama.git synced 2026-04-18 09:03:35 -04:00

Files

History

Michael Yang f1373193dc move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

..

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

bytepairencoding_test.go

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

bytepairencoding.go

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

sentencepiece_test.go

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

sentencepiece.go

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

tokenizer.go

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

vocabulary_test.go

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

vocabulary.go

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

wordpiece_test.go

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00

wordpiece.go

move tokenizers to separate package (#13825 )

2026-02-05 17:44:11 -08:00