Hugging Face's logo

tokyotech-llm
/
Llama-3.1-Swallow-70B-Instruct-v0.1

Text Generation
Transformers
Safetensors
English
Japanese
llama
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
1
Taishi-N324 commited on
Commit
2c297e3
·
verified ·
1 Parent(s): a6c9e4a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -9
README.md CHANGED
@@ -66,21 +66,14 @@ The website [https://swallow-llm.github.io/](https://swallow-llm.github.io/) pro
66
  |---|---|---|---|---|---|---|---|---|---|---|
67
  ||4-shot|4-shot|4-shot|4-shot|4-shot|5-shot|4-shot|3-shot|0-shot||
68
  ||Acc|EMacc|Acc|EMacc|Acc|Acc|EMacc|CoTEMAcc|pass@1||
69
- | Gemma 2 27B IT | 0.4560 | 0.7660 | 0.6548 | 0.4012 | 0.9101 | 0.7624 | 0.8438 | 0.7876 | 0.6939 | 0.6973 |
70
- | Phi-3.5-MoE Instruct | 0.4960 | 0.6746 | 0.6901 | 0.3174 | 0.8903 | 0.7872 | 0.8317 | 0.7618 | 0.5561 | 0.6673 |
71
- | GRIN-MoE | 0.4660 | 0.7035 | 0.7046 | 0.3544 | 0.8976 | 0.7693 | 0.8287 | 0.7533 | 0.6841 | 0.6846 |
72
- | KARAKURI LM 70B Chat v0.1 | 0.4100 | 0.6873 | 0.6315 | 0.3677 | 0.9049 | 0.5941 | 0.3882 | 0.5724 | 0.2305 | 0.5319 |
73
- | Swallow-70b-instruct-v0.1 | 0.4440 | 0.7411 | 0.6567 | 0.3529 | 0.9166 | 0.6677 | 0.5095 | 0.6661 | 0.2835 | 0.5820 |
74
  | Llama 3 70B Instruct | 0.4400 | 0.7999 | 0.6552 | 0.4024 | 0.9127 | 0.7992 | 0.9052 | 0.8326 | 0.7555 | 0.7225 |
75
  | Llama 3.1 70B Instruct | 0.4300 | 0.8212 | 0.6621 | 0.3921 | 0.9157 | 0.8213 | 0.8764 | 0.8390 | 0.7915 | 0.7277 |
76
  | Llama 3 Youko 70B Instruct | 0.4500 | 0.7973 | 0.6863 | 0.3914 | 0.9153 | 0.8055 | 0.8923 | 0.7814 | 0.6598 | 0.7088 |
77
- | Llama-3.1-70B-Japanese-Instruct-2407 | 0.4220 | 0.8104 | 0.6481 | 0.3744 | 0.9170 | 0.8071 | 0.8893 | 0.8228 | 0.7463 | 0.7153 |
78
  | Llama 3 heron brain 70B v0.3 | 0.4460 | 0.8107 | 0.6682 | 0.4085 | 0.9174 | 0.7898 | 0.8772 | 0.7586 | 0.6713 | 0.7053 |
79
  | Llama 3 Swallow 70B Instruct | 0.4520 | 0.8174 | 0.6758 | 0.4050 | 0.9230 | 0.7883 | 0.8688 | 0.8152 | 0.6890 | 0.7150 |
80
  | Llama 3.1 Swallow 70B Instruct | 0.4520 | 0.8148 | 0.6834 | 0.4012 | 0.9157 | 0.7855 | 0.8886 | 0.8486 | 0.5823 | 0.7080 |
81
- | Qwen2-72B-Instruct | 0.4360 | 0.7588 | 0.6857 | 0.3913 | 0.9110 | 0.8391 | 0.8499 | 0.2436 | 0.6939 | 0.6455 |
82
- | Qwen2.5-72B-Instruct | 0.4540 | 0.6764 | 0.7064 | 0.3550 | 0.8895 | 0.8478 | 0.9113 | 0.4027 | 0.6165 | 0.6511 |
83
- | Mixtral-8x22B-Instruct-v0.1 | 0.4540 | 0.8265 | 0.7074 | 0.3927 | 0.9222 | 0.7733 | 0.8324 | 0.8306 | 0.7348 | 0.7193 |
84
 
85
  ## MT-Bench JA
86
 
 
66
  |---|---|---|---|---|---|---|---|---|---|---|
67
  ||4-shot|4-shot|4-shot|4-shot|4-shot|5-shot|4-shot|3-shot|0-shot||
68
  ||Acc|EMacc|Acc|EMacc|Acc|Acc|EMacc|CoTEMAcc|pass@1||
69
+ | Qwen2-72B-Instruct | 0.4360 | 0.7588 | 0.6857 | 0.3913 | 0.9110 | 0.8391 | 0.8499 | 0.2436 | 0.6939 | 0.6455 |
70
+ | Qwen2.5-72B-Instruct | 0.4540 | 0.6764 | 0.7064 | 0.3550 | 0.8895 | 0.8478 | 0.9113 | 0.4027 | 0.6165 | 0.6511 |
 
 
 
71
  | Llama 3 70B Instruct | 0.4400 | 0.7999 | 0.6552 | 0.4024 | 0.9127 | 0.7992 | 0.9052 | 0.8326 | 0.7555 | 0.7225 |
72
  | Llama 3.1 70B Instruct | 0.4300 | 0.8212 | 0.6621 | 0.3921 | 0.9157 | 0.8213 | 0.8764 | 0.8390 | 0.7915 | 0.7277 |
73
  | Llama 3 Youko 70B Instruct | 0.4500 | 0.7973 | 0.6863 | 0.3914 | 0.9153 | 0.8055 | 0.8923 | 0.7814 | 0.6598 | 0.7088 |
 
74
  | Llama 3 heron brain 70B v0.3 | 0.4460 | 0.8107 | 0.6682 | 0.4085 | 0.9174 | 0.7898 | 0.8772 | 0.7586 | 0.6713 | 0.7053 |
75
  | Llama 3 Swallow 70B Instruct | 0.4520 | 0.8174 | 0.6758 | 0.4050 | 0.9230 | 0.7883 | 0.8688 | 0.8152 | 0.6890 | 0.7150 |
76
  | Llama 3.1 Swallow 70B Instruct | 0.4520 | 0.8148 | 0.6834 | 0.4012 | 0.9157 | 0.7855 | 0.8886 | 0.8486 | 0.5823 | 0.7080 |
 
 
 
77
 
78
  ## MT-Bench JA
79