|
--- |
|
base_model: |
|
- Krystalan/DRT-o1-14B |
|
- sometimesanotion/Lamarck-14B-v0.3 |
|
- sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40 |
|
- CultriX/Qwen2.5-14B-Hyperionv4 |
|
- sometimesanotion/Qwenvergence-14B-v9 |
|
- sometimesanotion/Qwenvergence-14B-v9 |
|
- sometimesanotion/LoRA-32-tempesthenno-ppo-ckpt40 |
|
- sometimesanotion/Lamarck-14B-v0.6 |
|
- sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40 |
|
- sthenno/tempesthenno-ppo-ckpt40 |
|
- sometimesanotion/Qwenvergence-14B-v3-Prose |
|
- sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40 |
|
library_name: transformers |
|
tags: |
|
- mergekit |
|
- merge |
|
license: apache-2.0 |
|
language: |
|
- en |
|
--- |
|
# Notes |
|
|
|
For a model_stock merge, this has greatly exceeded my expectations. It beats Lamarck v0.7's average without introducing DeepSeek elements, mostly by scoring high on MATH without giving up much elsewhere. It also shows that the high-scoring Qwen2.5 14B merges are converging near the limits of the architecture. Here is how it benchmarks alongside the models it merges. |
|
|
|
 |
|
|
|
### Merge Method |
|
|
|
This model was merged using the [Model Stock](https://arxiv.org/abs/2403.19522) merge method using [sometimesanotion/Qwenvergence-14B-v9](https://huggingface.co/sometimesanotion/Qwenvergence-14B-v9) as a base. |
|
|
|
### Models Merged |
|
|
|
The following models were included in the merge: |
|
* [Krystalan/DRT-o1-14B](https://huggingface.co/Krystalan/DRT-o1-14B) |
|
* [sometimesanotion/Lamarck-14B-v0.3](https://huggingface.co/sometimesanotion/Lamarck-14B-v0.3) + [sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40](https://huggingface.co/sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40) |
|
* [CultriX/Qwen2.5-14B-Hyperionv4](https://huggingface.co/CultriX/Qwen2.5-14B-Hyperionv4) |
|
* [sometimesanotion/Qwenvergence-14B-v9](https://huggingface.co/sometimesanotion/Qwenvergence-14B-v9) + [sometimesanotion/LoRA-32-tempesthenno-ppo-ckpt40](https://huggingface.co/sometimesanotion/LoRA-32-tempesthenno-ppo-ckpt40) |
|
* [sometimesanotion/Lamarck-14B-v0.6](https://huggingface.co/sometimesanotion/Lamarck-14B-v0.6) + [sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40](https://huggingface.co/sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40) |
|
* [sthenno/tempesthenno-ppo-ckpt40](https://huggingface.co/sthenno/tempesthenno-ppo-ckpt40) |
|
* [sometimesanotion/Qwenvergence-14B-v3-Prose](https://huggingface.co/sometimesanotion/Qwenvergence-14B-v3-Prose) + [sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40](https://huggingface.co/sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40) |
|
|
|
### Configuration |
|
|
|
The following YAML configuration was used to produce this model: |
|
|
|
```yaml |
|
name: Qwenvergence-14B-v11 |
|
merge_method: model_stock |
|
base_model: sometimesanotion/Qwenvergence-14B-v9 |
|
tokenizer_source: base |
|
dtype: bfloat16 |
|
out_dtype: bfloat16 |
|
parameters: |
|
int8_mask: true |
|
normalize: true |
|
rescale: false |
|
models: |
|
- model: sometimesanotion/Lamarck-14B-v0.6+sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40 |
|
- model: sometimesanotion/Qwenvergence-14B-v3-Prose+sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40 |
|
- model: sometimesanotion/Qwenvergence-14B-v9+sometimesanotion/LoRA-32-tempesthenno-ppo-ckpt40 |
|
- model: sometimesanotion/Lamarck-14B-v0.3+sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40 |
|
- model: sometimesanotion/Lamarck-14B-v0.6+sometimesanotion/LoRA-64-tempesthenno-ppo-ckpt40 |
|
- model: CultriX/Qwen2.5-14B-Hyperionv4 |
|
- model: Krystalan/DRT-o1-14B |
|
- model: sthenno/tempesthenno-ppo-ckpt40 |
|
|
|
``` |
