--- library_name: peft base_model: NousResearch/CodeLlama-7b-hf tags: - axolotl - generated_from_trainer model-index: - name: ee29425a-d835-495b-8fb5-e11141a51ea1 results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.4.1` ```yaml adapter: lora auto_find_batch_size: true base_model: NousResearch/CodeLlama-7b-hf bf16: auto chat_template: llama3 dataloader_num_workers: 12 dataset_prepared_path: null datasets: - data_files: - 77f4d29f468bc2c0_train_data.json ds_type: json format: custom path: /workspace/input_data/77f4d29f468bc2c0_train_data.json type: field_input: input field_instruction: instruction field_output: response format: '{instruction} {input}' no_input_format: '{instruction}' system_format: '{system}' system_prompt: '' debug: null deepspeed: null early_stopping_patience: 3 early_stopping_threshold: 0.001 eval_max_new_tokens: 128 eval_steps: 40 flash_attention: false fp16: null fsdp: null fsdp_config: null gradient_accumulation_steps: 2 gradient_checkpointing: false group_by_length: false hub_model_id: mrferr3t/ee29425a-d835-495b-8fb5-e11141a51ea1 hub_repo: null hub_strategy: checkpoint hub_token: null learning_rate: 0.0003 load_in_4bit: false load_in_8bit: false local_rank: null logging_steps: 100 lora_alpha: 16 lora_dropout: 0.05 lora_fan_in_fan_out: null lora_model_dir: null lora_r: 8 lora_target_linear: true lr_scheduler: cosine micro_batch_size: 32 mlflow_experiment_name: /tmp/77f4d29f468bc2c0_train_data.json model_type: AutoModelForCausalLM num_epochs: 50 optimizer: adamw_bnb_8bit output_dir: miner_id_24 pad_to_sequence_len: true s2_attention: null sample_packing: false save_steps: 40 saves_per_epoch: 0 sequence_len: 512 special_tokens: pad_token: strict: false tf32: false tokenizer_type: AutoTokenizer train_on_inputs: false trust_remote_code: true val_set_size: 0.02 wandb_entity: null wandb_mode: online wandb_name: 48c3c8d8-6645-4a88-adc5-70cd656a2562 wandb_project: Gradients-On-Demand wandb_run: your_name wandb_runid: 48c3c8d8-6645-4a88-adc5-70cd656a2562 warmup_ratio: 0.05 weight_decay: 0.0 xformers_attention: null ```

# ee29425a-d835-495b-8fb5-e11141a51ea1 This model is a fine-tuned version of [NousResearch/CodeLlama-7b-hf](https://huggingface.co/NousResearch/CodeLlama-7b-hf) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.6540 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0003 - train_batch_size: 32 - eval_batch_size: 32 - seed: 42 - gradient_accumulation_steps: 2 - total_train_batch_size: 64 - optimizer: Use adamw_bnb_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 218 - num_epochs: 50 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | No log | 0.0014 | 1 | 2.6242 | | No log | 0.0571 | 40 | 2.2860 | | No log | 0.1142 | 80 | 1.9066 | | 4.9536 | 0.1713 | 120 | 1.7338 | | 4.9536 | 0.2284 | 160 | 1.6253 | | 3.6406 | 0.2855 | 200 | 1.5644 | | 3.6406 | 0.3426 | 240 | 1.4934 | | 3.6406 | 0.3997 | 280 | 1.4442 | | 3.1356 | 0.4568 | 320 | 1.3714 | | 3.1356 | 0.5139 | 360 | 1.3406 | | 2.8521 | 0.5710 | 400 | 1.3140 | | 2.8521 | 0.6281 | 440 | 1.2290 | | 2.8521 | 0.6852 | 480 | 1.1974 | | 2.7573 | 0.7423 | 520 | 1.1788 | | 2.7573 | 0.7994 | 560 | 1.1294 | | 2.3985 | 0.8565 | 600 | 1.1246 | | 2.3985 | 0.9136 | 640 | 1.0637 | | 2.3985 | 0.9707 | 680 | 1.0167 | | 2.2745 | 1.0278 | 720 | 1.0103 | | 2.2745 | 1.0849 | 760 | 0.9873 | | 1.8141 | 1.1420 | 800 | 0.9797 | | 1.8141 | 1.1991 | 840 | 0.9597 | | 1.8141 | 1.2562 | 880 | 0.9458 | | 1.5585 | 1.3133 | 920 | 0.9204 | | 1.5585 | 1.3704 | 960 | 0.9172 | | 1.5745 | 1.4276 | 1000 | 0.9020 | | 1.5745 | 1.4847 | 1040 | 0.8774 | | 1.5745 | 1.5418 | 1080 | 0.8412 | | 1.3623 | 1.5989 | 1120 | 0.8341 | | 1.3623 | 1.6560 | 1160 | 0.8323 | | 1.3262 | 1.7131 | 1200 | 0.7996 | | 1.3262 | 1.7702 | 1240 | 0.8067 | | 1.3262 | 1.8273 | 1280 | 0.7752 | | 1.2514 | 1.8844 | 1320 | 0.7176 | | 1.2514 | 1.9415 | 1360 | 0.7170 | | 1.2277 | 1.9986 | 1400 | 0.7039 | | 1.2277 | 2.0557 | 1440 | 0.7033 | | 1.2277 | 2.1128 | 1480 | 0.7025 | | 0.68 | 2.1699 | 1520 | 0.7079 | | 0.68 | 2.2270 | 1560 | 0.7214 | | 0.809 | 2.2841 | 1600 | 0.6897 | | 0.809 | 2.3412 | 1640 | 0.7047 | | 0.809 | 2.3983 | 1680 | 0.7042 | | 0.7931 | 2.4554 | 1720 | 0.6792 | | 0.7931 | 2.5125 | 1760 | 0.6777 | | 0.6957 | 2.5696 | 1800 | 0.6809 | | 0.6957 | 2.6267 | 1840 | 0.6466 | | 0.6957 | 2.6838 | 1880 | 0.6705 | | 0.7611 | 2.7409 | 1920 | 0.6549 | | 0.7611 | 2.7980 | 1960 | 0.6540 | ### Framework versions - PEFT 0.13.2 - Transformers 4.46.0 - Pytorch 2.3.1+cu121 - Datasets 3.0.1 - Tokenizers 0.20.1