metadata
datasets:
- roneneldan/TinyStories
Some very small and very simple models.
29,960,200 parameters.
"dim":256,"dim_head":32,"headcount":8,"ff_mult":4, "vocab_size":50304, "num_layers":4.
this is nonstandard (for tinystories), reflecting a full gpt-2 vocabulary size (bloating the embedding layers), and the use of a swiglu activation function, (which doubles the width of one of the feedforward layers).
training, inference, dataset preparation, and network definitions source available at https://github.com/SQCU/attn_demo
training logs
(unprocessed! unfiltered! it's a bunch of log prints of train and validation loss!)
and training loader source for each run included with the demo models.