File size: 1,977 Bytes
2863d67 3d04364 2863d67 ba8ef5f 2863d67 b4d546c 2863d67 ba8ef5f 2863d67 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
---
language:
- sv
license: llama3.1
library_name: transformers
tags:
- unsloth
datasets:
- neph1/bellman-7b-finetune
- neph1/codefeedback-swedish
---
# Model Card for Bellman
This version of bellman is finetuned from llama-3.1-instruct-8b.
It's finetuned for prompt question answering, based on a dataset created from
Swedish wikipedia, with a lot of Sweden-centric questions.
New from previous versions is questions from a translated code-feedback dataset, as well as a number of stories. It's not great at generating stories,
but better than previosly.
Try out the Q8 version here: https://huggingface.co/spaces/neph1/bellman (cpu)
## Model Details
Training run on 240724:
Step Training Loss Validation Loss<br>
25 1.352200 1.034565<br>
50 1.033600 1.009348<br>
75 1.022400 0.996665<br>
100 1.002900 0.988050<br>
125 1.014600 0.981633<br>
150 1.006300 0.975584<br>
175 0.988800 0.970966<br>
200 0.985300 0.967037<br>
225 0.992400 0.964120<br>
250 0.950000 0.962472<br>
275 0.931000 0.960848<br>
300 0.932000 0.958946 <-- picked checkpoint <br>
### Training Parameters
per_device_train_batch_size = 4,<br>
gradient_accumulation_steps = 16,<br>
num_train_epochs=3,<br>
warmup_steps = 5,<br>
learning_rate = 3e-5,<br>
logging_steps = 25,<br>
optim = "adamw_8bit",<br>
weight_decay = 0.01,<br>
lr_scheduler_type = "linear",<br>
seed = 3407,<br>
per_device_eval_batch_size = 2,<br>
eval_strategy="steps",<br>
eval_accumulation_steps = 32,<br>
eval_steps = 25,<br>
eval_delay = 0,<br>
save_strategy="steps",<br>
save_steps=50,<br>
### Model Description
- **Developed by:** Me
- **Funded by:** Me
- **Model type:** Instruct
- **Language(s) (NLP):** Swedish
- **License:** llama-3.1
- **Finetuned from model:** Llama3.1 Instruct 8b
## Model Card Contact
[email protected] |