sthenno/tempesthenno-kto-0205-ckpt80
update: now checking for evaluations without chat templates
tempesthenno-icy-0130
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the SCE merge method using sthenno/tempesthenno-nuslerp-0124 as a base.
Models Merged
The following models were included in the merge:
- sthenno/tempesthenno-icy-0130-01
- sthenno/tempesthenno-icy-0130-02
- sthenno/tempesthenno-icy-0130-03
Configuration
The following YAML configuration was used to produce this model:
name: tempesthenno-icy-0130
merge_method: sce
parameters:
select_topk: 0.8
normalize: true
dtype: float32
out_dtype: bfloat16
base_model: sthenno/tempesthenno-nuslerp-0124
tokenizer:
source: base
chat_template: chatml
models:
- model: sthenno/tempesthenno-icy-0130-01
- model: sthenno/tempesthenno-icy-0130-02
- model: sthenno/tempesthenno-icy-0130-03
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 39.74 |
| IFEval (0-Shot) | 62.18 |
| BBH (3-Shot) | 50.10 |
| MATH Lvl 5 (4-Shot) | 37.99 |
| GPQA (0-shot) | 19.69 |
| MuSR (0-shot) | 19.84 |
| MMLU-PRO (5-shot) | 48.65 |
- Downloads last month
- 30
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
HF Inference API was unable to determine this model's library.
Model tree for sthenno/tempesthenno-kto-0205-ckpt80
Evaluation results
- strict accuracy on IFEval (0-Shot)Open LLM Leaderboard62.180
- normalized accuracy on BBH (3-Shot)Open LLM Leaderboard50.100
- exact match on MATH Lvl 5 (4-Shot)Open LLM Leaderboard37.990
- acc_norm on GPQA (0-shot)Open LLM Leaderboard19.690
- acc_norm on MuSR (0-shot)Open LLM Leaderboard19.840
- accuracy on MMLU-PRO (5-shot)test set Open LLM Leaderboard48.650