|
--- |
|
base_model: [meta-llama/Meta-Llama-3.1-405B] |
|
library_name: transformers |
|
tags: |
|
- mergekit |
|
- merge |
|
|
|
--- |
|
|
|
|
|
|
|
# lobotollama-368b prune [Meta-Llama-3.1-405B-Base](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B). |
|
This is a negative-merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). |
|
|
|
# Just so you meow, this did not turn out all that great in the perplexity benchmarks. Needs healing, you'll probably need 32xh100 to do a full finetune. |
|
# Model was designed to fin in a M2 mac-studio 192gb in 4bit. |
|
|
|
```verilog |
|
perplexity: 167.37 seconds per pass - ETA 33.47 minutes - meta-405b-base - q8_0 - newest base was identical in bf16 and q8_0 |
|
[1]1.3927,[2]1.6952,[3]1.5905,[4]1.4674,[5]1.3652,[6]1.3054,[7]1.2885,[8]1.2673,[9]1.2397,[10]1.2179,[11]1.2149,[12]1.2162, |
|
Final estimate: PPL = 1.2162 +/- 0.02128 |
|
|
|
perplexity: 2197.87 seconds per pass - ETA 1 hours 49.88 minutes -- llama 405b - instruct - old BF16 -8head |
|
[1]2.1037,[2]2.4201,[3]2.0992,[4]1.8446,[5]1.6823,[6]1.5948,[7]1.5575,[8]1.5121,[9]1.4750,[10]1.4570,[11]1.4567,[12]1.4666, |
|
Final estimate: PPL = 1.4666 +/- 0.03184 |
|
|
|
./llama-perplexity -m /scratch-10/lobotollama-q8_0.gguf -f wiki.test.raw -t 96 --chunks 12 -b 1024 |
|
perplexity: 331.47 seconds per pass - ETA 33.13 minutes |
|
[1]2.6744,[2]3.4041,[3]2.9683,[4]2.8669,[5]2.7924,[6]2.7590,[7]2.8274,[8]2.8306,[9]2.7943,[10]2.7910,[11]2.8164,[12]2.9396, |
|
Final estimate: PPL = 2.9396 +/- 0.09497 |
|
``` |
|
|
|
|
|
|
|
## Merge Details |
|
### Merge Method |
|
|
|
This model was merged using the passthrough merge method. |
|
|
|
### Models Merged |
|
|
|
The following models were included in the merge: |
|
* /Meta-Llama-3.1-405B |
|
|
|
### Configuration |
|
|
|
The following YAML configuration was used to produce this model: |
|
|
|
```yaml |
|
dtype: bfloat16 |
|
merge_method: passthrough |
|
slices: |
|
- sources: |
|
- layer_range: [0, 29] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [30, 35] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [36, 40] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [41, 45] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [46, 49] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [50, 54] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [55, 59] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [60, 64] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [65, 69] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [70, 74] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [75, 79] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [80, 84] |
|
model: /Meta-Llama-3.1-405B |
|
- sources: |
|
- layer_range: [85, 126] |
|
model: /Meta-Llama-3.1-405B |
|
``` |
|
|
