---
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft
---

<style>
@import url('https://fonts.googleapis.com/css2?family=Vollkorn:ital,wght@0,400..900;1,400..900&display=swap');
</style>

<div style="background-color: #101010; border-radius: .5rem; padding: 2rem; font-family: monospace; font-size: .85rem; text-align: justify;">
  
![palmer-004](https://huggingface.co/appvoid/palmer-004-original/resolve/main/palmer-004.jpeg)

  
#### palmer turbo

This model has a slightly different architecture and training style:

1. The model was followed by a continual pretraining (lm_head + embedding layers were tuned).
2. Base model was trained on 15k instruction/response pairs.
3. Similar architecture than palmer series but smaller in context size (8192)

In short, palmer is now half the size, twice the speed and same overall performance with a dramatical boost on arc challenge instead of winogrande.

As all palmer models, the model is biased to respond to answers without using any specific prompt, feel free to further fine-tune it for your specific use case.

| Model                          | MMLU  | ARC-C | HellaSwag | PIQA   | Winogrande | Average |
|--------------------------------|-------|-------|-----------|--------|------------|---------|
| tinyllama                      | 0.2577| 0.3029| 0.5935    | 0.7329 | 0.5959     | 0.4966  |
| danube3-500m-chat (current sota)| 0.2554 | **0.3626** | 0.6072  | 0.7432 | 0.6140 | 0.5164  
| palmer-004-turbo               |**0.2736**|0.3558|**0.6179**|0.7367|0.6117|0.5191|
| palmer-004                     | 0.2661  |0.3490 | 0.6173  | **0.7481**  | **0.6417**  | 0.5244|


#### thanks to

- h2oai: performant base model provider
- teknium: openhermes dataset provider
- unsloth: tooling for training software

#### note

Next versions of this model will be available through my upcoming app. Keep tuned to not miss the release date on my X account.
</div>