|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
--- |
|
|
|
Original model from https://huggingface.co/openlm-research/open_llama_3b_600bt_preview. |
|
|
|
This repo includes: |
|
1) Ported `LlamaTokenizer` to `LlamaTokenizerFast` via a few lines of code. |
|
Loading via `AutoTokenizer` takes 3 to 4 minutes. Now, a few seconds! |
|
``` |
|
from transformers import LlamaTokenizerFast |
|
from tokenizers import AddedToken |
|
tokenizer = LlamaTokenizerFast.from_pretrained( |
|
"openlm-research/open_llama_3b_600bt_preview", |
|
add_bos_token = True, add_eos_token = True, |
|
bos_token = AddedToken("<s>", single_word = True), |
|
eos_token = AddedToken("</s>", single_word = True), |
|
unk_token = AddedToken("<unk>", single_word = True), |
|
pad_token = AddedToken("<unk>", single_word = True) |
|
) |
|
tokenizer.push_to_hub("open_llama_3b_600bt_preview") |
|
``` |
|
2) `AutoTokenizer` does not recognize the BOS, EOS and UNK tokens. All tokenizations weirdly prepend 0 and append 0 to the end, when actually, you're supposed to prepend 1 and append 2. |
|
3) Manually added BOS `<s>`, EOS `</s>`, UNK `<unk>` tokens, with PAD (padding) being also the `<unk>` token. |
