hotchpotch
/

xlm-roberta-japanese-tokenizer-24k

Inference Endpoints

Model card Files Files and versions Community

xlm-roberta-japanese-tokenizer-24k / README.md

hotchpotch's picture

Update README.md

fc6aaa9 verified 21 days ago

|

history blame contribute delete

212 Bytes

metadata

library_name: transformers
tags: []

vocab_size が 24576 サイズになるように作成した日本語 tokenizer
学習元は wikpedia (日本語のみ)
unidic + sentencepiece(unigram) で学習