---
license: apache-2.0
base_model: NousResearch/Meta-Llama-3.1-8B-Instruct
library_name: peft
tags:
- llama-factory
- lora
datasets:
  - Nekochu/novel17_train_alpaca_format
  - bofenghuang/vigogne
  - jpacifico/French-Alpaca-dataset-Instruct-110K
  - MaziyarPanahi/french_instruct_human_sharegpt
  - Snit/french-conversation
  
language:
  - fr
  - en
---

- Similar to the old [Nekochu/Llama-2-13B-fp16-french](https://huggingface.co/Nekochu/Llama-2-13B-fp16-french) with additional datasets.
- I've (alway) kept LoRA `QLoRA_french_sft` so it can be applied to any *LLaMA-3.1-8B* fine-tuned model but may affect performance.

<details>
  <summary>This training can be replicated using LLaMA-Factory. </summary>

Stage A: **P**re **T**raining, Raw text
```
set CUDA_VISIBLE_DEVICES=0 && llamafactory-cli train --stage pt --do_train True --model_name_or_path NousResearch/Meta-Llama-3.1-8B-Instruct --preprocessing_num_workers 16 --finetuning_type lora --template alpaca --rope_scaling linear --flash_attn fa2 --dataset_dir data --dataset french-raw-pt --cutoff_len 8192 --learning_rate 5e-05 --num_train_epochs 3.0 --max_samples 10000000 --per_device_train_batch_size 1 --gradient_accumulation_steps 1 --lr_scheduler_type cosine --max_grad_norm 1.0 --logging_steps 10 --save_steps 1000 --warmup_steps 0 --neftune_noise_alpha 5 --optim adamw_8bit --packing True --report_to none --output_dir saves\LLaMA3.1-8B-Chat\lora\QLoRA_french_pt --bf16 True --plot_loss True --ddp_timeout 180000000 --include_num_input_tokens_seen True --quantization_bit 4 --quantization_method bitsandbytes --lora_rank 32 --lora_alpha 64 --lora_dropout 0.15 --create_new_adapter True --lora_target all
```

Stage B: Continued **S**upervised **F**ine-**T**uning, QA
```
set CUDA_VISIBLE_DEVICES=0 && llamafactory-cli train --stage sft --do_train True --model_name_or_path NousResearch/Meta-Llama-3.1-8B-Instruct --preprocessing_num_workers 16 --finetuning_type lora --template alpaca --rope_scaling linear --flash_attn fa2 --dataset_dir data --dataset Acquiesce_french_vigogne,novel17_train --cutoff_len 8192 --learning_rate 5e-05 --num_train_epochs 3.0 --max_samples 10000000 --per_device_train_batch_size 1 --gradient_accumulation_steps 1 --lr_scheduler_type cosine --max_grad_norm 1.0 --logging_steps 10 --save_steps 1000 --warmup_steps 0 --neftune_noise_alpha 5 --optim adamw_8bit --packing True --report_to none --output_dir saves\LLaMA3.1-8B-Chat\lora\QLoRA_french_sft --bf16 True --plot_loss True --ddp_timeout 180000000 --adapter_name_or_path saves\LLaMA3.1-8B-Chat\lora\QLoRA_french_pt --quantization_bit 4 --quantization_method bitsandbytes --lora_rank 32 --lora_alpha 64 --lora_dropout 0.15 --lora_target all
```

Dataset convert to Alpaca: [Acquiesce_french_vigogne](https://huggingface.co/datasets/Nekochu/Luminia-mixture/tree/split-v2/General/French),french-raw-pt

<!-- DPO¿: https://huggingface.co/datasets/jpacifico/french-orca-dpo-pairs-revised -->

</details>