---
base_model: roneneldan/TinyStories-33M
library_name: Distily
tags:
- generated_from_trainer
model-index:
- name: distily_bench_obj_cross_v2.6
  results: []
---

# distily_bench_obj_cross_v2.6

This student model is distilled from the teacher model [roneneldan/TinyStories-33M](https://huggingface.co/roneneldan/TinyStories-33M) using the dataset (unspecified).

The [Distily](https://github.com/lapp0/distily) library was used for this distillation.

It achieves the following results on the evaluation set:
- eval_enwikippl: 1915.6801
- eval_frwikippl: 17380.0898
- eval_zhwikippl: 80448.5391
- eval_tinystoriesppl: 689.9122
- eval_loss: 6.2077
- eval_runtime: 11.4682
- eval_samples_per_second: 87.198
- eval_steps_per_second: 10.9

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed
-->

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=0, loss_fn=None, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=10, loss_fn=cos, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
- train_embeddings: True
- learning_rate: 0.01
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1.0

### Resource Usage
Peak GPU Memory: 6.6276 GB

### Eval-Phase Metrics
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **teacher eval** |  | 169.9865 | 47377.9414 |  |  |  |  | 3.9789 | 4998.1294 |
| 0 | 0 | 23978.2344 | 74374.2891 | 7.8203 | 11.5608 | 86.499 | 10.812 | 11110.3125 | 69209.9141 |
| 500 | 0.0404 | 3768063.25 | 1320083162923008.0 | 6.3913 | 11.5562 | 86.534 | 10.817 | 123.9350 | 35325253458591744.0000 |
| 1000 | 0.0808 | 40494.2930 | 34653921280.0 | 6.3220 | 11.5104 | 86.878 | 10.86 | 33.9454 | 1850179059712.0 |
| 1500 | 0.1212 | 1197.6969 | 168759.6875 | 6.2397 | 11.5411 | 86.647 | 10.831 | 117.6943 | 1388815.625 |
| 2000 | 0.1616 | 1228.2329 | 20482.3652 | 6.2140 | 11.4883 | 87.045 | 10.881 | 326.2055 | 114379.5781 |
| 2500 | 0.2020 | 1239.8959 | 19846.2070 | 6.2130 | 11.5386 | 86.665 | 10.833 | 336.5318 | 103600.4453 |
| 3000 | 0.2424 | 1914.1963 | 17243.5391 | 6.2077 | 11.5623 | 86.488 | 10.811 | 690.1402 | 78960.0391 |
| 3500 | 0.2828 | 1921.0294 | 17666.4023 | 6.2095 | 11.5099 | 86.882 | 10.86 | 689.5698 | 83088.0156 |
| 4000 | 0.3232 | 1873.1237 | 17426.6621 | 6.2077 | 11.4977 | 86.974 | 10.872 | 664.2795 | 81834.0312 |
| 4500 | 0.3636 | 1893.8420 | 17517.7246 | 6.2077 | 11.5637 | 86.477 | 10.81 | 677.2543 | 82140.2188 |
| 5000 | 0.4040 | 1924.9026 | 17336.0898 | 6.2075 | 11.5831 | 86.333 | 10.792 | 695.7541 | 80234.2344 |
| 5500 | 0.4444 | 1891.4965 | 17372.75 | 6.2075 | 11.496 | 86.987 | 10.873 | 675.6886 | 80878.9453 |
| 6000 | 0.4848 | 1897.3663 | 17453.6895 | 6.2073 | 11.5439 | 86.626 | 10.828 | 679.7222 | 81746.75 |
| 6500 | 0.5253 | 1904.1395 | 17316.5586 | 6.2075 | 11.5881 | 86.295 | 10.787 | 685.9882 | 80362.7344 |
| 7000 | 0.5657 | 1897.9545 | 17316.5586 | 6.2075 | 11.4888 | 87.042 | 10.88 | 679.2728 | 81485.4609 |
| 7500 | 0.6061 | 1886.2292 | 17407.0449 | 6.2077 | 11.5303 | 86.728 | 10.841 | 673.7922 | 81528.9062 |
| 8000 | 0.6465 | 1883.8922 | 17250.8242 | 6.2075 | 11.5743 | 86.399 | 10.8 | 673.7922 | 79892.4688 |
| 8500 | 0.6869 | 1894.4291 | 17448.7793 | 6.2070 | 11.4754 | 87.143 | 10.893 | 678.1508 | 82008.8828 |
| 9000 | 0.7273 | 1894.4291 | 17326.3223 | 6.2067 | 11.4686 | 87.195 | 10.899 | 677.5906 | 81746.75 |
| 9500 | 0.7677 | 1915.6801 | 17380.0898 | 6.2077 | 11.4682 | 87.198 | 10.9 | 689.9122 | 80448.5391 |
| 10000 | 0.8081 | 1907.6830 | 17326.3223 | 6.2075 | 11.5672 | 86.451 | 10.806 | 687.4073 | 81008.5469 |
| 10500 | 0.8485 | 1899.1313 | 17596.8730 | 6.2075 | 11.5368 | 86.679 | 10.835 | 677.2543 | 82955.0859 |
| 11000 | 0.8889 | 1887.6906 | 17461.0645 | 6.2073 | 11.5979 | 86.223 | 10.778 | 672.4567 | 82008.8828 |
| 11500 | 0.9293 | 1895.6039 | 17399.6934 | 6.2077 | 11.5089 | 86.889 | 10.861 | 678.9360 | 81790.3438 |
| 12000 | 0.9697 | 1898.5428 | 17429.1211 | 6.2077 | 11.5488 | 86.589 | 10.824 | 680.5093 | 81790.3438 |
| 12375 | 1.0 | 1898.5428 | 17429.1211 | 6.2077 | 11.5716 | 86.418 | 10.802 | 680.7906 | 81790.3438 |

### Framework versions
- Distily 0.2.0
- Transformers 4.44.0
- Pytorch 2.3.0
- Datasets 2.20.0