End of training
Browse files
README.md
ADDED
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
base_model: roneneldan/TinyStories-33M
|
3 |
+
library_name: Distily
|
4 |
+
tags:
|
5 |
+
- generated_from_trainer
|
6 |
+
model-index:
|
7 |
+
- name: distily_bench_obj_cross_v2.6
|
8 |
+
results: []
|
9 |
+
---
|
10 |
+
|
11 |
+
# distily_bench_obj_cross_v2.6
|
12 |
+
|
13 |
+
This student model is distilled from the teacher model [roneneldan/TinyStories-33M](https://huggingface.co/roneneldan/TinyStories-33M) using the dataset (unspecified).
|
14 |
+
|
15 |
+
The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
|
16 |
+
|
17 |
+
It achieves the following results on the evaluation set:
|
18 |
+
- eval_enwikippl: 1915.6801
|
19 |
+
- eval_frwikippl: 17380.0898
|
20 |
+
- eval_zhwikippl: 80448.5391
|
21 |
+
- eval_tinystoriesppl: 689.9122
|
22 |
+
- eval_loss: 6.2077
|
23 |
+
- eval_runtime: 11.4682
|
24 |
+
- eval_samples_per_second: 87.198
|
25 |
+
- eval_steps_per_second: 10.9
|
26 |
+
|
27 |
+
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
28 |
+
should probably proofread and complete it, then remove this comment.
|
29 |
+
|
30 |
+
## Model description
|
31 |
+
|
32 |
+
More information needed
|
33 |
+
|
34 |
+
## Intended uses & limitations
|
35 |
+
|
36 |
+
More information needed
|
37 |
+
|
38 |
+
## Training and evaluation data
|
39 |
+
|
40 |
+
More information needed
|
41 |
+
-->
|
42 |
+
|
43 |
+
## Training procedure
|
44 |
+
|
45 |
+
### Training hyperparameters
|
46 |
+
|
47 |
+
The following hyperparameters were used during training:
|
48 |
+
- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=0, loss_fn=None, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=10, loss_fn=cos, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
|
49 |
+
- train_embeddings: True
|
50 |
+
- learning_rate: 0.01
|
51 |
+
- train_batch_size: 8
|
52 |
+
- eval_batch_size: 8
|
53 |
+
- seed: 42
|
54 |
+
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
|
55 |
+
- lr_scheduler_type: linear
|
56 |
+
- num_epochs: 1.0
|
57 |
+
|
58 |
+
### Resource Usage
|
59 |
+
Peak GPU Memory: 6.6276 GB
|
60 |
+
|
61 |
+
### Eval-Phase Metrics
|
62 |
+
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
|
63 |
+
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
64 |
+
| **teacher eval** | | 169.9865 | 47377.9414 | | | | | 3.9789 | 4998.1294 |
|
65 |
+
| 0 | 0 | 23978.2344 | 74374.2891 | 7.8203 | 11.5608 | 86.499 | 10.812 | 11110.3125 | 69209.9141 |
|
66 |
+
| 500 | 0.0404 | 3768063.25 | 1320083162923008.0 | 6.3913 | 11.5562 | 86.534 | 10.817 | 123.9350 | 35325253458591744.0000 |
|
67 |
+
| 1000 | 0.0808 | 40494.2930 | 34653921280.0 | 6.3220 | 11.5104 | 86.878 | 10.86 | 33.9454 | 1850179059712.0 |
|
68 |
+
| 1500 | 0.1212 | 1197.6969 | 168759.6875 | 6.2397 | 11.5411 | 86.647 | 10.831 | 117.6943 | 1388815.625 |
|
69 |
+
| 2000 | 0.1616 | 1228.2329 | 20482.3652 | 6.2140 | 11.4883 | 87.045 | 10.881 | 326.2055 | 114379.5781 |
|
70 |
+
| 2500 | 0.2020 | 1239.8959 | 19846.2070 | 6.2130 | 11.5386 | 86.665 | 10.833 | 336.5318 | 103600.4453 |
|
71 |
+
| 3000 | 0.2424 | 1914.1963 | 17243.5391 | 6.2077 | 11.5623 | 86.488 | 10.811 | 690.1402 | 78960.0391 |
|
72 |
+
| 3500 | 0.2828 | 1921.0294 | 17666.4023 | 6.2095 | 11.5099 | 86.882 | 10.86 | 689.5698 | 83088.0156 |
|
73 |
+
| 4000 | 0.3232 | 1873.1237 | 17426.6621 | 6.2077 | 11.4977 | 86.974 | 10.872 | 664.2795 | 81834.0312 |
|
74 |
+
| 4500 | 0.3636 | 1893.8420 | 17517.7246 | 6.2077 | 11.5637 | 86.477 | 10.81 | 677.2543 | 82140.2188 |
|
75 |
+
| 5000 | 0.4040 | 1924.9026 | 17336.0898 | 6.2075 | 11.5831 | 86.333 | 10.792 | 695.7541 | 80234.2344 |
|
76 |
+
| 5500 | 0.4444 | 1891.4965 | 17372.75 | 6.2075 | 11.496 | 86.987 | 10.873 | 675.6886 | 80878.9453 |
|
77 |
+
| 6000 | 0.4848 | 1897.3663 | 17453.6895 | 6.2073 | 11.5439 | 86.626 | 10.828 | 679.7222 | 81746.75 |
|
78 |
+
| 6500 | 0.5253 | 1904.1395 | 17316.5586 | 6.2075 | 11.5881 | 86.295 | 10.787 | 685.9882 | 80362.7344 |
|
79 |
+
| 7000 | 0.5657 | 1897.9545 | 17316.5586 | 6.2075 | 11.4888 | 87.042 | 10.88 | 679.2728 | 81485.4609 |
|
80 |
+
| 7500 | 0.6061 | 1886.2292 | 17407.0449 | 6.2077 | 11.5303 | 86.728 | 10.841 | 673.7922 | 81528.9062 |
|
81 |
+
| 8000 | 0.6465 | 1883.8922 | 17250.8242 | 6.2075 | 11.5743 | 86.399 | 10.8 | 673.7922 | 79892.4688 |
|
82 |
+
| 8500 | 0.6869 | 1894.4291 | 17448.7793 | 6.2070 | 11.4754 | 87.143 | 10.893 | 678.1508 | 82008.8828 |
|
83 |
+
| 9000 | 0.7273 | 1894.4291 | 17326.3223 | 6.2067 | 11.4686 | 87.195 | 10.899 | 677.5906 | 81746.75 |
|
84 |
+
| 9500 | 0.7677 | 1915.6801 | 17380.0898 | 6.2077 | 11.4682 | 87.198 | 10.9 | 689.9122 | 80448.5391 |
|
85 |
+
| 10000 | 0.8081 | 1907.6830 | 17326.3223 | 6.2075 | 11.5672 | 86.451 | 10.806 | 687.4073 | 81008.5469 |
|
86 |
+
| 10500 | 0.8485 | 1899.1313 | 17596.8730 | 6.2075 | 11.5368 | 86.679 | 10.835 | 677.2543 | 82955.0859 |
|
87 |
+
| 11000 | 0.8889 | 1887.6906 | 17461.0645 | 6.2073 | 11.5979 | 86.223 | 10.778 | 672.4567 | 82008.8828 |
|
88 |
+
| 11500 | 0.9293 | 1895.6039 | 17399.6934 | 6.2077 | 11.5089 | 86.889 | 10.861 | 678.9360 | 81790.3438 |
|
89 |
+
| 12000 | 0.9697 | 1898.5428 | 17429.1211 | 6.2077 | 11.5488 | 86.589 | 10.824 | 680.5093 | 81790.3438 |
|
90 |
+
| 12375 | 1.0 | 1898.5428 | 17429.1211 | 6.2077 | 11.5716 | 86.418 | 10.802 | 680.7906 | 81790.3438 |
|
91 |
+
|
92 |
+
### Framework versions
|
93 |
+
- Distily 0.2.0
|
94 |
+
- Transformers 4.44.0
|
95 |
+
- Pytorch 2.3.0
|
96 |
+
- Datasets 2.20.0
|
generation_config.json
ADDED
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"_from_model_config": true,
|
3 |
+
"bos_token_id": 50256,
|
4 |
+
"eos_token_id": 50256,
|
5 |
+
"transformers_version": "4.44.0"
|
6 |
+
}
|
logs/hs_loss_fn=cos, hs_weight=10, learning_rate=0.01/events.out.tfevents.1723850337.93d6cbb3ad53
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:a1779b913cb1d95bebfe2f9e2936883225318df17b4f94650a9a39e2d5ddeb76
|
3 |
+
size 307
|