w2v-bert-bem-bembaspeech-model

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the BEMBASPEECH - BEM dataset. It achieves the following results on the evaluation set:

• Loss: 0.2620
• Wer: 0.5353

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 30.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
0.7066	0.2811	200	0.5066	0.7648
0.5538	0.5622	400	0.4313	0.7232
0.4574	0.8433	600	0.4102	0.6956
0.4084	1.1244	800	0.3529	0.6276
0.388	1.4055	1000	0.3004	0.5724
0.3803	1.6866	1200	0.3376	0.6477
0.367	1.9677	1400	0.2911	0.5802
0.3168	2.2488	1600	0.3106	0.5725
0.3227	2.5299	1800	0.2654	0.5348
0.3111	2.8110	2000	0.2621	0.5494
0.2823	3.0921	2200	0.2665	0.5422
0.2603	3.3732	2400	0.2623	0.5174
0.2735	3.6543	2600	0.2620	0.5353
0.2666	3.9353	2800	0.2753	0.5450
0.2248	4.2164	3000	0.2881	0.5818
0.2408	4.4975	3200	0.2748	0.5324

Framework versions

• Transformers 4.46.0.dev0
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0