|
--- |
|
datasets: |
|
- roneneldan/TinyStories |
|
--- |
|
Some very small and very simple models. |
|
|
|
29,960,200 parameters. |
|
|
|
"dim":256,"dim_head":32,"headcount":8,"ff_mult":4, |
|
"vocab_size":50304, "num_layers":4. |
|
|
|
this is nonstandard (for tinystories), |
|
reflecting a full gpt-2 vocabulary size (bloating the embedding layers), |
|
and the use of a swiglu activation function, |
|
(which doubles the width of one of the feedforward layers). |
|
|
|
|
|
training, inference, dataset preparation, and network definitions source available at |
|
https://github.com/SQCU/attn_demo |
|
|
|
|
|
training logs |
|
|
|
(unprocessed! unfiltered! it's a bunch of log prints of train and validation loss!) |
|
|
|
and training loader source for each run included with the demo models. |
