한국어위키, 나무위키, aihub 구어 웹데이터 1%만 사용해서 8k 토크나이저 학습

Files changed (3) hide show

merges.txt CHANGED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json CHANGED Viewed

@@ -18,7 +18,7 @@
     "single_word": false
   },
   "errors": "replace",
-  "name_or_path": "tokenizer-8k-kowiki",
   "pad_token": null,
   "special_tokens_map_file": null,
   "tokenizer_class": "GPT2Tokenizer",

     "single_word": false
   },
   "errors": "replace",
+  "name_or_path": "tokenizer-8k",
   "pad_token": null,
   "special_tokens_map_file": null,
   "tokenizer_class": "GPT2Tokenizer",

vocab.json CHANGED Viewed

The diff for this file is too large to render. See raw diff